Like last Spring, I am looking for some college students to work with me on a project at IBM. This time though it’s not an Extreme Blue project, it’s a Summer/Fall co-op.
The problem area is this: How can we make development activities (e.g. design and coding), cloud-based infrastructures, and operational activities (e.g. deployment, monitoring) work together seamlessly to allow us to evolve IT and other types of living systems (e.g. buildings) faster and with higher quality?
The ideal candidate is a junior or rising senior undergrad or graduate student in computer science or similar major. Must be extremely passionate about technology, an proficient hacker, and able to make progress without micromanagement/babysitting. Interest in DevOps ideas and Cloud technologies a plus. Interest/knowledge in living ecosystems a plus.
The location is Research Triangle Park, North Carolina from May to December 2012.
If interested, send me an email (firstname.lastname@example.org) with the word “co-op” somewhere in the subject line. I will provide more details to candidates.
Like many, I was profoundly saddened by the death of a man I never met, but who has affected my life – Steve Jobs.
A hundred years from now, the world will be very different than if Steve Jobs had never lived. However, it won’t be (directly) because he helped create the Mac, the iPod, the iPhone, and the iPad.
It will be because he fundamentally altered the intellectual and creative DNA of people who create technology.
I don’t just mean Apple; there are a large number of technologists outside of Apple whose views on design and engineering have been shaped by Jobs and Apple circa 1997 to 2011. This influence will result in small and large changes to the technical landscape as these people deliver technology and teach the next generation of technologists.
How will it be different? It’s impossible to capture precisely, so let me instead sketch a few examples (admittedly a caricature):
Don’t worry about speeds and feeds; focus on addressing everyday human needs
Design not as “how it looks”; design as “how it works”
Money not as end unto itself; money as both “fuel” and also the result of helping to improve the human condition
Not settling; striving for excellence
It’s possible for large organizations to do amazing things 
It will be subtle, slow, and sometimes invisible, but this DNA will alter the evolution of technology at a pretty deep level. In fact, it has already begun.
In 2005 I joined the Rational Jazz project. I was relatively young at the time (28) and it was pretty cool when I saw a meeting invite to present to Erich Gamma on my technical area – “web UIs”. I worked hard to create a good presentation that described the basic vision, architectural approach, and issues we expected. Approximately five minutes into the presentation, Erich asked, “This is nice, but where’s the demo?”
I had no demo, so it was a bit awkward.
How do we solve problems? Well, it depends on the type of the problem. If the problem is “Dishes need to be put away”, it’s pretty easy because it’s a well-defined problem and there are not that many different ways you can choose to solve it – you will end up with the same result. But of course there are harder classes of problems such as “What business model and technical strategy should we adopt for the next five to ten years?” or “What do I want to accomplish in life?” Of course, these examples are on the other extremes – essentially wicked problems – but I find most of the problems I deal with these days are more wicked than they are trivial. And sometimes the hardest thing is figuring out what problem you should focus on trying to solve…
In the early days of Jazz, my manager was Scott Rich, and I remember when I first met him I was amazed by his technical knowledge and his ability to crank out hundreds of lines of good code. Over the course of several years, he got promoted to become an IBM Distinguished Engineer and I remember (half-) joking with him that he had changed his editor of choice from Eclipse to PowerPoint.
A Steve Jobs presentation is mesmerizing. Don’t believe me? Watch the original iPhone introduction then let me know if you still disagree. But what’s mesmerizing about it? To me the magic of the Steve Jobs presentation is that he shows us how to complete a puzzle that’s been unsolved for several years. After the presentation, the solution seems obvious. To quote Jony Ive, “How could it be any other way?” But of course, the presentation represents an end state. From a problem solving perspective, the interesting part is what happened that you didn’t see that led up to the presentation.
These days when I work on some new technical area, by default my inclination is to get a couple of smart developers together and start prototyping. My assumption is that not only is it impossible to solve a complex problem without getting our hands dirty in code, but we won’t even understand what problem we’re trying to solve until we’ve gotten our hands dirty in code.
But prototyping is not enough to solve the problem. Prototyping helps you understand the problem. Prototyping helps constrain the solution to the adjacent possible, as opposed to the fantastical. But prototyping alone doesn’t solve the problem. Prototyping doesn’t produce the narrative.
I really believe that for anything to succeed – a philosophy, a product, a movie, a technology – it has to tell a good story. I don’t know how to articulate this in conceptual terms – I just really have come to believe that if you can’t tell a compelling story, you’re doomed to failure or at best mediocrity. I think that’s the magic of Steve Jobs’ presentations – he’s a great storyteller and he describes the problems he’s trying to solve in very simple terms with which we can identify. And I think this is the value of presentations – IF – and it’s a big “IF” – you do them right.
What’s the difference between “getting it wrong” and “getting it right” with presentations? Unfortunately there’s many more examples of the former than the latter. But for canonical examples of each check out Edward Tufte’s “The Cognitive Style of PowerPoint” and Jobs’ iPhone introduction (above), respectively. I think in a nut, the difference is that when done right, a presentation should help visualize and complement a story that’s mainly told verbally, and when done wrong, well… there are many failure patterns for presentations.
In the past two weeks, I’ve spent approximately 60% of my time working on two variations of a presentation – one focused on customer value and adoption of of a new solution and one focused on internal execution of delivering that same solution. My colleague John Ryding chided me with the same “from IDE to PowerPoint” line as I used with Scott several years ago. But now that the shoe’s on the other foot, I believe that there’s potential real value in creating these presentations.
It forces me to take a step back from the code and try to clearly articulate what we’re building and why it’s valuable. This exercise has actually led to new insights on what we should build and how we should build it. From an even higher level, it forces me to think about what we’re building in terms of a story that’s simple, coherent and compelling.
Going back to Erich’s original question: ”This is nice, but where’s the demo?” In hindsight the problem was that I was trying to describe a set of concepts and a plan prior to having enough experience and running code to back up my assertions. But this isn’t to say that there’s no use to presentations. My view these days is that you have to work in a very iterative manner to learn at a concrete level, then take a step back and reflect on the problem you’re trying to solve and how, rinse and repeat. If you can’t articulate it in a simple and compelling manner, it’s a good sign you’re not done yet.
I’ll close with a Steve Jobs quote on the design of the iPod:
Look at the design of a lot of consumer products — they’re really complicated surfaces. We tried to make something much more holistic and simple. When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can often times arrive at some very elegant and simple solutions.
I remember when Mac OS X 10.6 Snow Leopard came out a few years ago, I updated the day it was released. A few days later, I asked my next door cube neighbor Pat Mueller what he thought of it and he made a face like “are you serious you dipshit?” and then said “I’ll try it in six months or so. I was sort of stunned.
Fast forward to this year. In July I got a new high-end iMac and upgraded to Mac OS X 10.7 Lion as soon as it was released.
Then the problems began.
Problem 1: The computer video would freeze often if I viewed videos either in Safari or iTunes. Given that one of the prime use cases for the Higgins family iMac is for Higgins children to watch cartoons, this was a big problem. The only “fix” was to power cycle the computer. Not good.
Problem 2: Wi-fi networking would just crap out after several hours. The workaround was to restart the computer whenever networking crapped out.
Problem 3: After a restart, user switch, or wake from sleep, the OS would report “Could not find any of your preferred wifi networks” and then proceed to show a list of available wifi networks… with my preferred network at the top of the list, which begs the question… if you can display the fracking network in the list, why can’t you connect to it?
Some strange data points:
- My two year-old MacBook Pro has exhibited exactly zero problems since upgrading to Lion.
- The new iMac had zero of these problems before I upgraded to Lion.
So the bad combination seems to be new Apple hardware + new Apple OS. My only speculation can be that the folks developing the new hardware and new software were testing with the previous generation of each others’ stuff.
Alas, as of a couple of hours ago, all of my problems are fixed. Problems 1 and 2 (video freeze + networking crapping out), were fixed via the first Lion fix pack (10.7.1) – in fact these two problems represent two of the four bullet points in the release notes. I became semi-obsessed with problem 3 and spent approximately twenty hours troubleshooting it by myself and with the help of Apple level 2 support. Finally tonight I decided to throw a hail mary and Google the symptom to see if anyone had discovered a fix since I first encountered the problem. Lo and behold, the first or second Google result had an Apple forum thread where someone explained that if you simply create a bogus new “Location” for your Networking preferences, it fixes the problem. I tried this and for reasons I won’t even attempt to comprehend, it worked.
The lesson I learned from this little fiasco is that Pat was right – best to wait six months or so and let other poor schmucks work out the kinks with Apple.
On the bright side, at least I don’t have to use Windows or Linux on my desktop every day
Now I can’t wait for iOS 5 and iCloud… I bet they work great…
I thought my family was on the do not call list for telemarketers but I guess not because recently we’ve been getting a lot of calls at the house. At first we got annoyed but one day by chance I found a way to have some fun. You can do this too. All it takes is a phone with a mute button and a young child.
If you receive a call from a number you don’t recognize, have your child answer it. If it’s a telemarketer, typically they talk for the first thirty seconds or so to give you their basic spiel. Turn the phone on mute and take this time to think of funny things to say. The basic game is to tell your child funny things to say while your phone is on mute. When the telemarketer asks you a question, turn the phone off mute and indicate to your child that they should say the funny thing. Repeat until the telemarketer finally gives up.
Here’s a sample conversation:
Telemarketer: “May I tell you about our new timeshares?”
Child: “What’s a timeshare?”
Telemarketer: <long-winded explanation of how a time share works, ending in a repeat of “May I tell you about our new timeshares?”>
Child: “Do you like flowers?”
Telemarketer: (pause, nervous laugh) “Well, yes.” (pause) ”May I tell you about our new timeshares?”
Child: “Is it fun to play with?”
Telemarketer: (pause) <somewhat tortured explanation about how it *is* fun to play with a timeshare, ending with “May I transfer you over to a sales specialist?”>
Child: “Did you see the new Winnie the Pooh?”
Telemarketer: (pause) ”May I transfer you over to a sales specialist?”
Child: “I saw the new Winnie Pooh with my friend Jill and Jill’s nanny Jaqlyn”
Telemarketer: (longer pause) ”May I transfer you over to a sales specialist?”
Child: “What’s your favorite flower?”
Telemarketer: (pause) ”Thank you ma’am. Have a good evening.”
Child: “I like purple flowers.”
A few interesting observations:
They go right ahead with their pitch, even when the person answering the phone obviously sounds like a young child.
Last year on the Jazz project, I helped design and implement a simple REST protocol to implement long-running operations, or long-ops. I’ve explained the idea enough times in random conversations that I thought it would make sense to write it down.
I’ll first write about the concrete problem we solved and then talk about the more abstract class of problems that the solution supports.
Example: Jazz Lifecycle Project Creation
Rational sells three particular team products that deal with requirements management, development, and test management, respectively. These products must work individually but also together if more than one is present in a customer environment. Each product has a notion of “project”. In the case where a customer has more than one product installed in their environment, we wanted to be able to let a customer press a button and create a “lifecycle project” that is basically a lightweight aggregation of the concrete projects (e.g. the requirements project, the development project, and the test project).
So we created a rather simple web application called “Lifecycle Project Administration” that logically and physically sits outside the products and gives a customer the ability to press a button and create a lifecycle project, create the underlying projects, and link everything together.
This presented a couple of problems, but I want to focus on the UI problem that pushed us towards the RESTy long-op protocol. Creating a project area can take between 30 seconds to a minute, depending on the complexity of the initialization routine. Since the lifecycle project creation operation aggregated several project creation operations plus some other stuff, it could take several minutes. A crude way to implement this UI would be to just show a “Creating lifecycle project area, please wait” message and perhaps a fakey progress monitor for several minutes until all of the tasks complete. In a desktop UI operating on local resources, you would use a rather fine-grained progress monitor that provides feedback on the set of tasks that need to run, the current running tasks, and the current percent complete of the total task.
We brainstormed on a way that we could come up with something like a progress monitor that could show fine-grained progress while running the set of remote operations required to create a lifecycle project and its subtasks. The solution was the RESTy long-op protocol. First I’ll talk about how one would typically do “normal, simple RESTful creation”.
Simple RESTy Creation
A common creation pattern in RESTful web services is to POST to a collection. It goes something like this:
HTTP/1.1 201 Created
The 201 status code of course indicates that the operation resulted in the creation of a resource and the Location header provides the URI for the new resource.
From a UI point of view, this works fine for a creation operation that takes a few seconds, but not so well for a creation operation that takes several minutes, like the lifecycle project administration case. So let’s look at the RESTy long-op protocol.
The RESTy Long-op Protocol
In this example, I’ll use a simplified form of lifecycle project creation:
Just to explain the request body, the name is simply the display name and the template is the ID of a template that defines the set of concrete projects that should be created and how they should be linked together.
Rather than responding with a URL for a resource that was created, the server responds with a 202 'Accepted' status, and the location of a “Job” resource, that basically reports on the status of the long-running task of creating (or updating) the resource.
Now the client polls the location of the “job”; the job is a hierarchal resource representing the state and resolution of the top level job and the sub-jobs (called ‘steps” below). It also includes a top-level property called resource that will eventually point to the URI of the resource that you are trying to create or update (in this case the lifecycle project).
At some point the top-level task has a non-null resolution and a non-null resource, at which point the client can GET the resource, which is the complete URI for the original thing you tried to create/update (in this case the lifecycle project).
GET /lifecycle-projects/bills-lifecycle-project HTTP/1.1
(I’ll omit the structure of the lifecycle project, as it’s not relevant to this discussion.)
Here’s a demo I recorded of an early version of Lifecycle Project Administration last year, that shows this protocol in action:
This protocol supports a set of related patterns:
You can use this protocol to support one or a combination of these patterns. E.g. you could have a single task (i.e. not a composite) that takes a long time and therefore you still want to use an asynchronous user experience.
Here are a few good things about this protocol:
Facilitates better feedback to people who invoke long-running, perhaps composite operations, through your UI.
Decouples the monitoring of a long-running composite operation from its execution and implementation; for all you know the composite task could be running in parallel across a server farm or it could be running on a single node.
Supports a flexible user experience; you could implement a number of different progress monitor UIs based on the information above.
Here are a few not-so-nice things about this protocol:
Not based on a standard.
Requires some expectation that the original create/update request might result in a long-running operation, and the only way you have to know that it’s a job resource (vs. the actual created or updated resource) is by the 202 Accepted response code (which could be ambiguous) and/or by content sniffing.
Doesn’t help much with recovering from complete or partial failure, retrying, cancelation, etc. though I’m sure you can see ways of achieving these things with a few additions to the protocol. We just didn’t need/want the additional complexity.
I would like to write a bit about some of the implementation patterns, but I think this entry is long enough, so I’ll just jot down some important points quickly.
Your primary client for polling the jobs should be a simple headless client library type thing that allows higher level code to register to be notified of updates. In most cases you’ll have more than one observer (e.g. the progress widget itself that redraws with any step update and the page that updates when the ultimate resource becomes available).
Your backend should persist the job entries as it creates and updates them. This allows you to decouple where the tasks in the composite execute from where the front-end can fetch the current status. This also allows you to run analytics over your job data over time to understand better what’s happening.
The persistent form of the job should store additional data (e.g. the durations for each task to complete) for additional analytics and perhaps better feedback to the user (e.g. time estimate for the overall job and steps based on historical data).
Of course you’ll want to cache all over the place on the job resources since you poll them and in most cases the status won’t have changed.
I don’t think this protocol is perfect, and I’m sure I’m not the first one to come up with such a protocol, but we’ve found it useful and you might too. I’d be interested if anyone has suggestions for improvement and/or pointers to similar protocols. I remember I first learned about some of these basic patterns from a JavaRanch article my IBM colleague Kyle Brown wrote way back in 2004.
Pretty much as soon as I published this, several folks on Twitter cited similar protocols:
I have never been a big fan of making public predictions about what might happen with the industry, a company, or a technology. I strongly agree with Alan Kay‘s famous quote: “The best way to predict the future is to invent it.” Of course, inventing the future is hard, especially if you’re spending precious time writing articles stating in unequivocal terms what will happen in the future (e.g. “Why and how the mobile web will win”).
Of course, none of us know precisely what will happen in the future (tired clichés aside), especially for things as complex and volatile as the economy or the technology industry. I frankly am baffled why people continue to make such confident and sometimes smug predictions on top of shaky or non-existent logical foundations. Luckily the web makes it easy to record these predictions and compare them to what really happened in the fullness of time, so there is some measure of accountability.
Of course, this doesn’t mean that it isn’t worth reasoning about potential futures – as long as you follow some simple guidelines:
State your evidence – What historical examples, current data, trends, and published plans lead you to your conclusions?
State your assumptions – What things have to happen for the potential future to become reality? What makes you think these things will happen?
State your conflicts of interest – Do you have something to gain if your predicted future becomes reality?
State your confidence level – Where are you on the continuum from wild-ass guess to high-probability outcome?
Another question to ask yourself is “Should you prognosticate publicly or privately?” I believe it’s very helpful to prognosticate privately (e.g. within a company) to help drive product strategy and semi-publicly to help customers chart their course (though in this case stating conflicts of interests is very important, for the obvious ethical reason and for the pragmatic goal of building customer trust). What I personally despise is predicting some future that aligns with your financial and/or philosophical interests and not stating the conflict of interest. It’s fine to advocate for some preferred future, but if you do so please be honest about your motivations – don’t dress up advocacy as prognostication.
Finally, if you have made prognostications, you should periodically perform an honest assessment of what you got right, what you got wrong, and why. Your retrospective should be at least as public as your prediction and you should be brutally honest – for one thing it’s unethical not to be brutally honest and for another thing people will quickly detect if you’re being honest or hedging which will obviously cause them to trust you more or less, respectively.
I originally planned to link to some of the prognosticating articles that put me in this obviously grumpy mood, but I’ve decided not to because A) Why promote trash? and B) I assume other people can think of plenty of examples of this sort of thing. Instead I will point to someone who I think does a great job of doing the reasoned, data-driven prognostication that I find incredibly valuable, Horace Dediu and his web site covering mobile information technology.
I don’t get angry very much. I’m usually pretty upbeat and when I hit something weird, scary, or uncool, I get stoic rather than upset. Yesterday a set of emails within IBM led me to get upset and I let the person know I was upset.
The unfortunate and somewhat funny thing (in hindsight) is that I had actually misinterpreted the person’s statements and intent so I actually got upset about something they didn’t really say.
The good news is that we worked it out, laughed about it, and I apologized for losing my cool. We reflected that the root cause was simply that email remains a crappy communications mechanism for anything but simple conversations, especially when you don’t know the people with whom you’re emailing very well.
This led me to make a personal resolution to myself: I won’t ever again react in anger to something I read in email (or a bug report or any other form of quickly composed written media). If I read something that makes me upset I will give the other person the benefit of the doubt and get on the phone or if possible walk over to their desk.
Then maybe I will find something to really get angry about
But then again, maybe I’ll find it was just a misunderstanding.
Hi. I normally don’t write directly about something I’m going to be working on because I hate vaporware, but in this case it’s necessary.
I am going to be running an IBM Extreme Blue project this summer where the intent is to build some technology to learn about how we can use Big Data techniques to analyze DevOps data. We will be using the Hadoop family of technologies for the data crunching and REST / Linked Data to help gather the data to be crunched.
It’s a twelve week project, I’ll be mentoring but the students really drive it, and we’ll also be getting a little help from my friends – my Tivoli colleagues Don Cronin and Andrew Trossman (who created some schweet new cloud technology we’ll be using), the Jazz team, Rod Smith and David Sink’s JStart team, and possibly some folks from the Yorktown Research lab.
Speaking of being crunched, we have a compressed schedule to find candidates and we already have some great candidates interviewing, so if you’re interested and you fit the profile below, please feel free to apply.
Here are the details:
When: May 23rd to Aug 12 2011
Where: Raleigh, NC
Requirements: Formal requirements here; my requirement: must be super passionate about technology and building great software
Pay etc.: Competitive pay, furnished apartment, travel/relocation, trip to IBM HQ in Armonk for a Big Demo, IBM swag (because you know you want it)
Basically I plan on making it a very intense and fun project. We will all learn a lot and have a lot of fun.
Again, if this sounds good to you and you fit the requirements, please feel free to apply … soon.
PS – I’ve asked the Extreme Blue folks to order Macs for the interns’ computers (if that’s important to you).
I’ve been working on a little Java web framework  for an exploratory work project. I am building the framework and a sample app as a set of OSGi bundles to drastically reduce API surface area between components . This also makes it easy to run my sample app directly within base Eclipse, using Eclipse’s built-in support for OSGi framework launches and a bundle-ized version of Jetty.
This configuration raises an interesting problem though, how do you inject the application code into the framework, since the framework obviously can’t statically depend on the application code, and OSGi will only let Java classes “see each other”  if one bundle makes a package (or packages) visible via an Export-Package manifest declaration (e.g. Export-Package: my.api.package) and another bundle declares an explicit dependency on that package via an Import-Package declaration (e.g. Import-Package: my.api.package)? In other words, how will you avoid hitting java.lang.NoClassDefFoundErrors when trying to load the application code via reflection?
I sure didn’t know. Luckily I have a good buddy here at IBM in Research Triangle Park named Simon Archer who is an OSGi and Equinox expert , so I ran the problem by him. He told me about an OSGi manifest declaration I had never heard of called DynamicImport-Package. My assumption that you can only get at code via explicit Import-Package declarations was actually wrong.
Simon explained that the way DynamicImport-Package works is that it basically allows a bundle to say “I want to be able to access any class that is part of an exported package in the runtime environment. So let’s say I have two bundles: bill.framework and bill.sampleapp. I want the code in bill.sampleapp to run inside the web framework implemented in bill.framework, but I obviously don’t want the bill.framework code to have a static (class-level) dependency on the bill.sampleapp code since the whole reason I’ve designed it as a framework is to allow build-time composition of arbitrary applications built on the framework . So I put the following in bill.framework‘s MANIFEST.MF file:
Then in my Sample App bundle’s MANIFEST.MF file, I put my application class in a package  that I export to the OSGi environment:
Now the framework is able to dynamically load the sample app via reflection:
 I know, because what the world needs now is another Java web framework. But as I observed in a journal entry, every framework is evil, except mine.
 Note that the framework itself doesn’t use or depend on OSGi. I build the bundles into a set of simple JARs that can run as part of a JEE web app or as standalone Java application again using an embedded Jetty web server.
 E.g. Simon co-wrote the book mentioned in . He is “Archer”
 Yes, I know. Most people call this pattern “dependency injection”. For the full treatise, see Fowler.
 The fact that you have to export the package for the code that you want to dynamically load wasn’t immediate obvious and Simon and I spent approximately twenty minutes staring at the screen wondering why we were getting java.lang.NoClassDefFoundError even though we were using DynamicImport-Package: *. After some unfruitful Googling, we decided to check out some bundle details using the OSGi console in Eclipse. As we were looking at the details for the sample app, I got the at the time unintuitive idea to try exporting the Sample App package. Sure enough this fixed it. Simon and I had a bit of a debate about whether or not it made sense to have to export application code since this effectively declares the application code to be API, which seems wrong – i.e. typically an application sits on top of a stack of code and depends on lots of stuff, but nothing depends on it.
But eventually we came to a reason that makes perfect sense for exporting the application code: If you didn’t have to explicitly export the code, theoretically any OSGi bundle’s code could get access to any other bundle’s code simply by declaring DynamicImport-Package: * and loading random classes via reflection, defeating the whole purpose of the OSGi modularity system. So by requiring that the to-be-dynamically-loaded class be available to the environment via an explicit Export-Package declaration you are still playing by the “normal rules” and just using reflection rather than static instantiation to poof up objects.
Of course this means that you should minimize your API surface area for the application class, so I put mine in its own package and its only public methods are those from the framework interface that it implements.