Like many, I was profoundly saddened by the death of a man I never met, but who has affected my life – Steve Jobs.

A hundred years from now, the world will be very different than if Steve Jobs had never lived. However, it won’t be (directly) because he helped create the Mac, the iPod, the iPhone, and the iPad.

It will be because he fundamentally altered the intellectual and creative DNA of people who create technology.

I don’t just mean Apple; there are a large number of technologists outside of Apple whose views on design and engineering have been shaped by Jobs and Apple circa 1997 to 2011. This influence will result in small and large changes to the technical landscape as these people deliver technology and teach the next generation of technologists.

How will it be different? It’s impossible to capture precisely, so let me instead sketch a few examples (admittedly a caricature):

• Don’t worry about speeds and feeds; focus on addressing everyday human needs
• Design not as “how it looks”; design as “how it works”
• Money not as end unto itself; money as both “fuel” and also the result of helping to improve the human condition
• Not settling; striving for excellence
• It’s possible for large organizations to do amazing things [1]

It will be subtle, slow, and sometimes invisible, but this DNA will alter the evolution of technology at a pretty deep level. In fact, it has already begun.

[1] From Josh Staiger’s comment below.

In 2005 I joined the Rational Jazz project. I was relatively young at the time (28) and it was pretty cool when I saw a meeting invite to present to Erich Gamma on my technical area – “web UIs”. I worked hard to create a good presentation that described the basic vision, architectural approach, and issues we expected. Approximately five minutes into the presentation, Erich asked, “This is nice, but where’s the demo?”

I had no demo, so it was a bit awkward.

—–

How do we solve problems? Well, it depends on the type of the problem. If the problem is “Dishes need to be put away”, it’s pretty easy because it’s a well-defined problem and there are not that many different ways you can choose to solve it – you will end up with the same result. But of course there are harder classes of problems such as “What business model and technical strategy should we adopt for the next five to ten years?” or “What do I want to accomplish in life?” Of course, these examples are on the other extremes – essentially wicked problems – but I find most of the problems I deal with these days are more wicked than they are trivial. And sometimes the hardest thing is figuring out what problem you should focus on trying to solve…

—–

In the early days of Jazz, my manager was Scott Rich, and I remember when I first met him I was amazed by his technical knowledge and his ability to crank out hundreds of lines of good code. Over the course of several years, he got promoted to become an IBM Distinguished Engineer and I remember (half-) joking with him that he had changed his editor of choice from Eclipse to PowerPoint.

—–

A Steve Jobs presentation is mesmerizing. Don’t believe me? Watch the original iPhone introduction then let me know if you still disagree. But what’s mesmerizing about it? To me the magic of the Steve Jobs presentation is that he shows us how to complete a puzzle that’s been unsolved for several years. After the presentation, the solution seems obvious. To quote Jony Ive, “How could it be any other way?” But of course, the presentation represents an end state. From a problem solving perspective, the interesting part is what happened that you didn’t see that led up to the presentation.

—–

These days when I work on some new technical area, by default my inclination is to get a couple of smart developers together and start prototyping. My assumption is that not only is it impossible to solve a complex problem without getting our hands dirty in code, but we won’t even understand what problem we’re trying to solve until we’ve gotten our hands dirty in code.

But prototyping is not enough to solve the problem. Prototyping helps you understand the problem. Prototyping helps constrain the solution to the adjacent possible, as opposed to the fantastical. But prototyping alone doesn’t solve the problem. Prototyping doesn’t produce the narrative.

—–

I really believe that for anything to succeed – a philosophy, a product, a movie, a technology – it has to tell a good story. I don’t know how to articulate this in conceptual terms – I just really have come to believe that if you can’t tell a compelling story, you’re doomed to failure or at best mediocrity. I think that’s the magic of Steve Jobs’ presentations – he’s a great storyteller and he describes the problems he’s trying to solve in very simple terms with which we can identify. And I think this is the value of presentations – IF – and it’s a big “IF” – you do them right.

—–

What’s the difference between “getting it wrong” and “getting it right” with presentations? Unfortunately there’s many more examples of the former than the latter. But for canonical examples of each check out Edward Tufte’s “The Cognitive Style of PowerPoint” and Jobs’ iPhone introduction (above), respectively. I think in a nut, the difference is that when done right, a presentation should help visualize and complement a story that’s mainly told verbally, and when done wrong, well… there are many failure patterns for presentations.

—–

In the past two weeks, I’ve spent approximately 60% of my time working on two variations of a presentation – one focused on customer value and adoption of of a new solution and one focused on internal execution of delivering that same solution. My colleague John Ryding chided me with the same “from IDE to PowerPoint” line as I used with Scott several years ago. But now that the shoe’s on the other foot, I believe that there’s potential real value in creating these presentations.

It forces me to take a step back from the code and try to clearly articulate what we’re building and why it’s valuable. This exercise has actually led to new insights on what we should build and how we should build it. From an even higher level, it forces me to think about what we’re building in terms of a story that’s simple, coherent and compelling.

—–

Going back to Erich’s original question: ”This is nice, but where’s the demo?” In hindsight the problem was that I was trying to describe a set of concepts and a plan prior to having enough experience and running code to back up my assertions. But this isn’t to say that there’s no use to presentations. My view these days is that you have to work in a very iterative manner to learn at a concrete level, then take a step back and reflect on the problem you’re trying to solve and how, rinse and repeat. If you can’t articulate it in a simple and compelling manner, it’s a good sign you’re not done yet.

—–

I’ll close with a Steve Jobs quote on the design of the iPod:

Look at the design of a lot of consumer products — they’re really complicated surfaces. We tried to make something much more holistic and simple. When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can often times arrive at some very elegant and simple solutions.

Last year on the Jazz project, I helped design and implement a simple REST protocol to implement long-running operations, or long-ops. I’ve explained the idea enough times in random conversations that I thought it would make sense to write it down.

I’ll first write about the concrete problem we solved and then talk about the more abstract class of problems that the solution supports.

Example: Jazz Lifecycle Project Creation

Rational sells three particular team products that deal with requirements management, development, and test management, respectively. These products must work individually but also together if more than one is present in a customer environment. Each product has a notion of “project”. In the case where a customer has more than one product installed in their environment, we wanted to be able to let a customer press a button and create a “lifecycle project” that is basically a lightweight aggregation of the concrete projects (e.g. the requirements project, the development project, and the test project).

So we created a rather simple web application called “Lifecycle Project Administration” that logically and physically sits outside the products and gives a customer the ability to press a button and create a lifecycle project, create the underlying projects, and link everything together.

This presented a couple of problems, but I want to focus on the UI problem that pushed us towards the RESTy long-op protocol. Creating a project area can take between 30 seconds to a minute, depending on the complexity of the initialization routine. Since the lifecycle project creation operation aggregated several project creation operations plus some other stuff, it could take several minutes. A crude way to implement this UI would be to just show a “Creating lifecycle project area, please wait” message and perhaps a fakey progress monitor for several minutes until all of the tasks complete. In a desktop UI operating on local resources, you would use a rather fine-grained progress monitor that provides feedback on the set of tasks that need to run, the current running tasks, and the current percent complete of the total task.

We brainstormed on a way that we could come up with something like a progress monitor that could show fine-grained progress while running the set of remote operations required to create a lifecycle project and its subtasks. The solution was the RESTy long-op protocol. First I’ll talk about how one would typically do “normal, simple RESTful creation”.

Simple RESTy Creation

A common creation pattern in RESTful web services is to POST to a collection. It goes something like this:

Request

POST /people HTTP/1.1
Host: example.com

{
    "name": "Bill Higgins",
    "userId": "billh"
}

Response

HTTP/1.1 201 Created
Location: http://example.com/people/billh

The 201 status code of course indicates that the operation resulted in the creation of a resource and the Location header provides the URI for the new resource.

From a UI point of view, this works fine for a creation operation that takes a few seconds, but not so well for a creation operation that takes several minutes, like the lifecycle project administration case. So let’s look at the RESTy long-op protocol.

The RESTy Long-op Protocol

In this example, I’ll use a simplified form of lifecycle project creation:

Creation Request

POST /lifecycle-projects HTTP/1.1
Host: example.com

{
    "name": "Bill's Lifecycle Project",
    "template": "com.ibm.team.alm.req-dev-test"
}

Just to explain the request body, the name is simply the display name and the template is the ID of a template that defines the set of concrete projects that should be created and how they should be linked together.

Here’s what the response looks like:

Response

HTTP/1.1 202 Accepted
Location: http://example.com/jobs/5933

Rather than responding with a URL for a resource that was created, the server responds with a 202 'Accepted' status, and the location of a “Job” resource, that basically reports on the status of the long-running task of creating (or updating) the resource.

Now the client polls the location of the “job”; the job is a hierarchal resource representing the state and resolution of the top level job and the sub-jobs (called ‘steps” below). It also includes a top-level property called resource that will eventually point to the URI of the resource that you are trying to create or update (in this case the lifecycle project).

Job Polling Request

GET /jobs/5933 HTTP/1.1
Host: example.com

Job Polling Response

HTTP/1.1 200 OK

{
    "title": "Creating lifecycle project 'Bill's Lifecycle Project'",
    "state": "IN_PROGRESS",
    "resolution": null,
    "resource": null,
    "steps": [
        {
            "title": "Creating requirements project",
            "state": "COMPLETE",
            "resolution": "SUCCESS"
        },
        {
            "title": "Creating development project",
            "state": "IN_PROGRESS",
            "resolution": null
        },
        {
            "title": "Creating project linkages",
            "state": "NOT_STARTED",
            "resolution": null
        },
        {
            "title": "Creating lifecycle project",
            "state": "NOT_STARTED",
            "resolution": null
        }
    ]
}

At some point the top-level task has a non-null resolution and a non-null resource, at which point the client can GET the resource, which is the complete URI for the original thing you tried to create/update (in this case the lifecycle project).

GET /lifecycle-projects/bills-lifecycle-project HTTP/1.1
Host: example.com

(I’ll omit the structure of the lifecycle project, as it’s not relevant to this discussion.)

Demo

Here’s a demo I recorded of an early version of Lifecycle Project Administration last year, that shows this protocol in action:

Uses

This protocol supports a set of related patterns:

• Long-running operations
• Asynchronous operations
• Composite tasks

You can use this protocol to support one or a combination of these patterns. E.g. you could have a single task (i.e. not a composite) that takes a long time and therefore you still want to use an asynchronous user experience.

Critique

Here are a few good things about this protocol:

• Facilitates better feedback to people who invoke long-running, perhaps composite operations, through your UI.
• Decouples the monitoring of a long-running composite operation from its execution and implementation; for all you know the composite task could be running in parallel across a server farm or it could be running on a single node.
• Supports a flexible user experience; you could implement a number of different progress monitor UIs based on the information above.

Here are a few not-so-nice things about this protocol:

• Not based on a standard.
• Requires some expectation that the original create/update request might result in a long-running operation, and the only way you have to know that it’s a job resource (vs. the actual created or updated resource) is by the 202 Accepted response code (which could be ambiguous) and/or by content sniffing.
• Doesn’t help much with recovering from complete or partial failure, retrying, cancelation, etc. though I’m sure you can see ways of achieving these things with a few additions to the protocol. We just didn’t need/want the additional complexity.

Implementation Notes

I would like to write a bit about some of the implementation patterns, but I think this entry is long enough, so I’ll just jot down some important points quickly.

• Your primary client for polling the jobs should be a simple headless client library type thing that allows higher level code to register to be notified of updates. In most cases you’ll have more than one observer (e.g. the progress widget itself that redraws with any step update and the page that updates when the ultimate resource becomes available).
• Your backend should persist the job entries as it creates and updates them. This allows you to decouple where the tasks in the composite execute from where the front-end can fetch the current status. This also allows you to run analytics over your job data over time to understand better what’s happening.
• The persistent form of the job should store additional data (e.g. the durations for each task to complete) for additional analytics and perhaps better feedback to the user (e.g. time estimate for the overall job and steps based on historical data).
• Of course you’ll want to cache all over the place on the job resources since you poll them and in most cases the status won’t have changed.

In Closing

I don’t think this protocol is perfect, and I’m sure I’m not the first one to come up with such a protocol, but we’ve found it useful and you might too. I’d be interested if anyone has suggestions for improvement and/or pointers to similar protocols. I remember I first learned about some of these basic patterns from a JavaRanch article my IBM colleague Kyle Brown wrote way back in 2004.

Updates

Pretty much as soon as I published this, several folks on Twitter cited similar protocols:

• William Vambenepe referenced this Tim Bray blog entry from 2009 on “Slow REST”
• Sam Johnston referenced the O’Reilly RESTful Web Services Cookbook book’s chapter on “How to Use POST for Asynchronous Tasks” Here’s a peak.
• Dims Srinivas referenced Leonard Richardson and Sam Ruby’s RESTful Web Services book’s chapter on REST and ROA Best Practices. Here’s a peak.

Thanks very much William, Sam, and Dims.

I have never been a big fan of making public predictions about what might happen with the industry, a company, or a technology. I strongly agree with Alan Kay‘s famous quote: “The best way to predict the future is to invent it.” Of course, inventing the future is hard, especially if you’re spending precious time writing articles stating in unequivocal terms what will happen in the future (e.g. “Why and how the mobile web will win”).

Of course, none of us know precisely what will happen in the future (tired clichés aside), especially for things as complex and volatile as the economy or the technology industry. I frankly am baffled why people continue to make such confident and sometimes smug predictions on top of shaky or non-existent logical foundations. Luckily the web makes it easy to record these predictions and compare them to what really happened in the fullness of time, so there is some measure of accountability.

Of course, this doesn’t mean that it isn’t worth reasoning about potential futures – as long as you follow some simple guidelines:

• State your evidence – What historical examples, current data, trends, and published plans lead you to your conclusions?
• State your assumptions – What things have to happen for the potential future to become reality? What makes you think these things will happen?
• State your conflicts of interest – Do you have something to gain if your predicted future becomes reality?
• State your confidence level – Where are you on the continuum from wild-ass guess to high-probability outcome?

Another question to ask yourself is “Should you prognosticate publicly or privately?” I believe it’s very helpful to prognosticate privately (e.g. within a company) to help drive product strategy and semi-publicly to help customers chart their course (though in this case stating conflicts of interests is very important, for the obvious ethical reason and for the pragmatic goal of building customer trust). What I personally despise is predicting some future that aligns with your financial and/or philosophical interests and not stating the conflict of interest. It’s fine to advocate for some preferred future, but if you do so please be honest about your motivations – don’t dress up advocacy as prognostication.

Finally, if you have made prognostications, you should periodically perform an honest assessment of what you got right, what you got wrong, and why. Your retrospective should be at least as public as your prediction and you should be brutally honest – for one thing it’s unethical not to be brutally honest and for another thing people will quickly detect if you’re being honest or hedging which will obviously cause them to trust you more or less, respectively.

I originally planned to link to some of the prognosticating articles that put me in this obviously grumpy mood, but I’ve decided not to because A) Why promote trash? and B) I assume other people can think of plenty of examples of this sort of thing. Instead I will point to someone who I think does a great job of doing the reasoned, data-driven prognostication that I find incredibly valuable, Horace Dediu and his web site covering mobile information technology.