On the Jazz project, one of my jobs is leading the software development for our platform-level web UI stuff [1]. Erich Gamma is the overall Jazz technology lead. Twice a month we have a sync up meeting where we discuss platform-level web UI progress.

In our last chat in mid-December, one of the things we chatted about was the status of the Jazz Server’s Admin Web UI. We began bootstrapping the Admin Web UI in September and I thought we’d made good progress in such a short amount of time. Erich didn’t seem so impressed with what he saw. I was a bit frustrated by this because I thought Erich was being unreasonable – we’d made good progress for the short amount of time we’d been working on it. But after a few more minutes of chatting, I realized that there was simply a mismatch in what we were evaluating. I was evaluating the state of the Admin Web UI vs. the resources (time and people) we’d had to work on it; Erich was evaluating the state of the Admin Web UI vs. his vision of what an awesome Admin Web UI should look like. Put more tersely, I was measuring “current vs. previous” while Erich was measuring “current vs. ideal”.

I found this difference in approach fascinating and insightful. I’m a very methodical person; when someone gives me a job I very methodically list its goals and measures of success, then I work backwards given the time and people I have to reach the goals and then over the course of weeks or month work towards the goals. But during my conversation with Erich I realized that a risk of my methodical approach is that I can become too bogged-down on the blocking and tackling of day-to-day software development and become overly focused on incremental progress and lose sight of the big vision of what you’re trying to build. I found that when I adopted Erich’s point of view and thought about the current state of the Admin Web UI vs. what it could become in the very long-term, I became very dissatisfied with its current state and highly motivated to be more ambitious with the next iteration plan.

It’s ironic because most of the iterative software development literature talks about the converse problem – you’re too focused on the big picture that you don’t make day-to-day progress. Indeed it’s become a cliché over the past several years to make fun of architects who are too far removed from day-to-day coding, testing, and building.

So the insight I gained was that I need to think in terms of both “current vs. previous” and “current vs. ideal”. Fortunately our iterative development process supports finding this balance. At the beginning of iterations it’s fine to dream big dreams about what you’re trying to achieve but then to become very pragmatic about how to make incremental progress towards realizing those dreams in the course of a four week iteration. Then over the four weeks you can be extremely methodical about making progress. Then at the end of the iteration you can reflect on what you’ve accomplished and feel some short satisfaction that the software is significantly better now than it was four weeks ago. Then you can reflect that the software’s not nearly as good as you ideally want it to be which provides the motivation to start the cycle again.

[1] By “platform-level” I simply mean more of the infrastructure. That is, frameworks and APIs rather than tangible applications.

Jen pointed out the most pressing problem (font style in form elements, if memory serves) and I said “Oh, I can change that” and one line of CSS later, the major problem was fixed. Hey, this was pretty easy! Jen mentioned a few other things and I realized that they were easy as well. Erich and the other folks dropped off the call and we decided to start a NetMeeting and try to knock out as many UI polish items we could. Two hours later we were completely done – not only had we fulfilled the spirit of the spec, we’d rethought a few spec’d decisions and tweaked a few additional items in the process of our extremely rapid iterations.

A few days later, I reflected on this and wondered why the heck we had the UI designers sending us super-detailed specs in the first place. If someone were to send me a highly-detailed technical spec on how I should code a particular feature, I’d ask them if I were being Punk’d. Yet with regards to UI design, this is standard practice. Not anymore, at least for me. The UI designers are welcome to draw up whatever specs they like, but I’m only going to do major UI work via pair programming with a UI designer.

Pair programming with a UI designer has the same benefits as pair programming something in [fill in your programming language of choice]. You can experiment, discuss, learn, and refine very quickly, and the real-time verbal and visual communication is much higher bandwidth than email and documents. This is especially true in a web environment, where you can hack CSS literally in real-time via EditCss and a screen sharing program.

This experience and another recent email conversation with James Governor on my reticence to publish a new Ajax article “before it is completely ready” has made me decide to reflect on how I can apply the principles of experiment/discuss/learn/refine to everything I do.

But I haven’t done this.

Fortunately for users but unfortunately for my idea, the function in question already runs as fast (1-4 seconds) as users expect it to run, given the nature of the function.  So although the 20-40% improvement is certainly relatively significant, it isn’t of any practical significance to users.  After all, who cares if something that currently runs in a ‘speedy’ 2 seconds were reduced to 1.5 seconds?

So it’s been with great discipline (and a bit of angst) that I’ve abstained from writing this clever code, basically because of the YAGNI principle, which the team as a whole rigorously observes.  The benefits (0.2 – 1.6 second reduced response time) simply don’t justify these costs:

• time to implement
• time to test
• potential defects (the current implementation has been stable for six months)
• increased complexity (it’s clever, you know)
• opportunity cost of not doing other work which is of practical use to users (backlogged enhancement requests, UI/UX improvements, bug fixes)

The reason I bring this up is because it demonstrates the powerful temptation to implement a clever or elegant solution despite the fact that it lacks a compelling problem.  I consider YAGNI to be a fundamental principle of my personal development philosophy, yet I still found myself tempted to implement the solution once I had fallen in love with its cleverness and efficiency.

So … I’ll be patient for now and my solution will remain unimplemented, but documented in a bug report.  If, over time, other system factors cause the response time to degrade, perhaps the solution’s value may increase enough to justify its costs.

Or not.

I felt fortunate for this interaction because I was impressed both with the Jazz strategy and the Jazz team – composed primarily of the folks who created the Eclipse platform. However, I felt like I wasn’t adding as much value as I could have since I didn’t interact much with the development team, who were driving much of the direction.

I spoke with Kurt Bittner of the product management team about my concerns and he told me “You’ve got development experience, why don’t you talk to Scott Rich about joining the development team?”. To me this was a pretty wild idea, since in Global Services I’d worked hard to ‘escape’ the Developer caste in favor of architecture. But Kurt explained that in the Eclipse/Jazz team, everyone, up to the executive level, participated in technical design and even wrote code.

After thinking about this a while, I decided it was a good idea, so I spoke with Scott about working on his Jazz server team. Our talk went well and we both had a good vibe but the outcome was a bit muddy. Because no one on the Jazz development team had seen my technical skills first-hand, and because I hadn’t done active development for more than a year, they wanted to bring me on in a sort of trial period. If I did well, there was an opportunity to join full-time, and if things didn’t go well, no harm done.

At first this was quite surprising, because since I joined IBM, I’d always been a top technical performer and had a very good reputation around Global Services. But the more I thought about it, the more I realized that this was a very positive sign about the Jazz development culture. My glowing performance reviews weren’t enough – I had to prove and show my capabilities – this team would not risk bringing on dead weight. Now I wanted to join this team even more and worked very hard. So on September 14th 2005, after two months of working on a trial basis, Scott asked me to join the team full-time. This was probably the happiest moment of my IBM career. I even saved the chat:

Scott Rich – hey, still there?

Bill Higgins – yes

Scott Rich – what would you think about joining the Jazz team?

Bill Higgins – !!!!!!!!!

Bill Higgins – are you serious?!?!?!?!

Scott Rich – yep, you’re in. your final test is to figure out how to apply for JobReq #006282 in the jobs system…

We observed that in American culture, you’re expected to take what food you want from the center of the table, and finish what you take.  In Chinese culture, you can take food from the table, but other people at the table are just as likely to eagerly give you food in a spirit of generosity, especially if they think you like something.  What makes your Chinese friends and family think that you like something?  If you finish it!

This can easily lead to some crossed cultural wires.  Americans feel culturally-obligated to finish what they take, and Chinese people feel culturally-obligated to give more food to someone who seems to enjoy it.  The trick is that when you want to be done with a particular dish, you simply leave a small but visible portion of the food on your plate, so none of your Chinese friends and family assume that you want more.  This way you don’t overeat, and you don’t get into an awkward “No, I really am full!” conversation with your Chinese friends and family as they try to put more food on your plate.

One example of this happened at the end of 2006. The day before we were supposed to freeze code for the year, I assisted another developer on a late-breaking severe defect. Unfortunately we didn’t test the fix adequately and it led to another defect which I discovered two hours after the final scheduled build completed. Sleep-deprived and stressed, I forgot about the code freeze and delivered a fix to the second defect. Fifteen minutes after delivering the patch, one of our senior technical people sent out an email reminding everyone that the codebase was frozen. I felt horrible because I’d committed a major faux pas at the most critical period in the development cycle. With great embarrassment, I sent a follow-up email to the project, notifying them of my mistake.

The next morning I came into the RTP lab for the end-game planning call with the PMC and component leads. I went to a meeting room and found John Wiegand and Scott Rich, who at this point were fully aware of the mistake I made. With a sheepish smile, I asked “Can I buy anyone a coffee?” Scott replied “You’re not forgiven that easily. What you did last night requires coffee and donuts.” John said, “Well, we all screw up from time to time, and the important thing is that you recognized the mistake and, to look at the positive aspect of it, the feature now works.” And that was that. In the end it turned out that there were a few other lingering bugs in the final scheduled build so we did one more build and all was well.

Another example was my mid-year checkpoint with Erich Gamma, reviewing the progress on the subsystem that I lead. My team was struggling at the time. In some very new technical territory, we were progressing more slowly than anyone would have liked. I was dreading the call, because Erich’s sort of a professional hero of mine, so I really wasn’t looking forward to hearing him tell me that things weren’t going well. But the call wasn’t like that at all. He began by reflecting on the things we’d accomplished and what had gone well. Only after a few minutes of discussing accomplishments did he gently segue into the discussion of areas that needed improvement. We prioritized a list of architectural features and user scenarios, and then worked through the details of what my team would need from other teams to succeed. I left the call feeling energized about what I was confident we could achieve, and over the past six months, I’m confident that we’ve met or exceeded those expectations.