Do I Need User Stories?

How do you create a worthwhile specification of the requirements for a system?

As an Agile Scrum practitioner I’m supposed to answer this by pointing to user stories, but as others have found before me, these are a slippery beastie to define. Cat Schwamm takes a stab at a definition, but finds Scott Bellware telling her she is following Scrum too blindly and should go change what she’s doing.

Scott’s point of view is interesting, but like a lot of his comments he is coming at the problem from the frame of his lean methodology, which is not fully explained and therefore extremely hard to argue with/understand. This is not Scott’s fault, he is after all busy consulting not writing Agile methodologies, but it does make his commentary a tad alarming when one first reads it.

The classic format of a user story is:

As a [user] I need/want [feature] so that [benefit].

Or as Cat puts it:

As a [role] I need [feature] so that [business value].

Cat points out that this is fairly barebones, and leaves lots of scope for you to choose the wrong level of granularity for your user story. She points out that her team also follows the INVEST acronym and use the concept of minimum marketable feature (MMF) as defined by James Shore.

INVEST = Independent, Negotiable, Valuable, Estimable, Small, Testable

MMF = “Smallest possible set of functionality that, by itself, has value in the marketplace”

This is obviously working for her team, but I suspect they benefit from having a single Business Analyst who works everything through for them and ensures some consistency across user stories. This is actually very Agile, as the manifesto points out we value “individuals and interactions over processes and tools”. However, we still need some processes/tools for those individuals.

Scott Bellware’s suggestion was to change the format to:

As a [role] I want to [goal] so that [motivation].

He wants us to avoid mentioning the software altogether and have the user story simply capture what the users actually do (or will do once they have a system to help them). He then divorces the user story from work items altogether, and simply relegates them to the role of being information about the goals of the system. This may work for him, but it then begs the question of how he defines what the work item is all about, which is where we started talking about the need for user stories …

I have been reading a great book called Designing for Interaction by Dan Saffer that gave me another viewpoint on this issue. One of the things I learnt from this book was that user-centred design (UCD) is not the best way to design interactions, it is simply one of the possible ways you could design them and its relevance depends upon the goals of the thing you are designing.

Dan Saffer lists 4 different approaches to interaction design:

1. User-Centred Design
2. Activity-Centred Design (aka Use-Centred Design ???)
3. Systems Design (aka Service Design???)
4. Genius Design (aka Rapid Expert Design???)

Cat’s user stories actually sounded like activity-centred design, and Scott sounds like he is more interested in user-centred design. Quite possibly they are both right in the contexts they are working in. Neither one may be better than the other (Scott at this point would probably call BS on me, he has previously talked about the need for user stories to communicate expectation).

Coming back to my own issues with user stories, I need to ask what sort of design approach am I taking, otherwise I will find my user stories being all over the place. Is the user story meant to encapsulate the MMF in a single story, or is it supposed to be used to specify all ways of looking at the functionality.

As an example, when looking at the search function for administrators of an organisation I have specified:

• As an admin I want to be able to search for a member by their Member Id so that I can find a specific member quickly.
• As an admin I want to be able to search for one or more members by their Member Type so that I can target an action against all members of a specific type.
• As an admin I want to be able to search for one or more members by a wildcard search of first and last names so that I can find a member just by knowing their name.
• As an admin I want to be able to search for one or more members by region or industry so that I can find members from specific industries or regions.
• As an admin I want to be able to search for one or more members by a wildcard search of job title so that I can find members with specific job titles.
• As an admin I want to be able to search for one or more members by member status for my organisation so that I can target an action against all members of a specific member status.
• As an admin I want to be able to search for one or more members by activation status so that I can target an action against all members of a specific activation status.
• As an admin I want to be able to conduct a search against multiple facets of a member at once (e.g. by member status, industry, member type and job title) so that I can get very specific search results.
• As the system I want to give admins a common page which allows them to search for members for all sorts of reasons and then target actions against the members found.
• As the system I want to ensure that admins can only ever see their organisation’s own members in search results.

You can see the format slipping towards the end of that list, with the “so that [benefit]” element disappearing. There is also no regard to the INVEST principles, particularly the independent one. If I adopted another (MMF?) approach I might reduce all of that to:

• As an admin I need to be able to find the members I’m interested in from my organisation so that I can work with those members’ details.

Scott’s format might leave me with:

• As an admin I want to work with my members so that I can help them use our organisation’s online tools.

I think this demonstrates that there is a lot of variance possible within this simple definition of user story. In the past I think I have over-specified functionality with user stories, leaving my developers with a weird tension between overly specific (and hence out of date) user stories and little or no supporting documentation around other design issues.

This actually speaks to the N in INVEST, as Cat explains it:

“Negotiable: User stories are not a contract; they are reminders for the team to discuss and collaborate to clarify the details near time of development

What a user story really is is a placeholder for a conversation.  We’ve gotten away from writing strict, rigid use cases and into these user stories, which are much better for the way the business world really works.  By writing a user story this way, I am opening up a higher bandwidth conversation.  Instead of a developer just reading what I have and doing it word for word, now we can talk about things and make sure everyone is on the same page and fully understands the business value and the needs of the user.  I like that Kelly specified “near time of development” though; that’s one thing you’ll always find: CHANGE IS A CONSTANT.  Friggin annoying, but it is inevitable.  The closer to development time you talk about something the more likely it is that you’ll have the best information and the most clarity on the item.”

I especially like the idea of a user story as a placeholder for a conversation. Of course this depends on your developers actually being willing to talk to anyone about the requirements (sometimes they still seem to think that it is faster to code away on assumptions and correct later than to have a 30 min conversation up front and ensure you know what you are doing!).

Scott Reynolds works with Cat, and he points out that not everything can be captured in user stories:

“Here’s the thing, stories and tasks are only part of the equation. Stories alone aren’t enough to define a system, and trying to define everything in text is a fool’s task. (I’ve been that fool). You need a full arsenal of specification tools to do the best job possible.”

I think I’ve been the fool Scott mentioned myself (once or twice). Scott (and Cat) mentions 3 other tools that they use as well as user stories:

1. Mockups
   Basically a low-fidelity prototype. They are great for eliciting feedback about visual design and interaction issues without leaving anyone feeling like these ideas are final. They are poor for actually designing off of as they frequently ignore or skimp on the design of areas that were not pertinent to the original design issues being explored in them.
2. High Fidelity Design
   High-fidelity screenshots showing exactly how the screen should look. These are great when you need signoff on something from outside the development team as they show everything that needs to be taken into account. They are poor when the design needs to change and you don’t have time to re-do those rich initial screenshots for the change request. Actually they are poor for Agile, full-stop, as they reek of BUFD.
3. Conversation and Whiteboarding
   This is the higher bandwidth conversation that Cat referred to. This is a chance to meet, collaboratively engage each other and work out what the business need for this feature is. The caveat is that it is only as good as the conversation and whiteboarding ability of your developers, and it requires that the customer be available immediately when the developers need them (this is why Agile doesn’t work for many projects).

There are other good ideas for artifacts to capture interaction design ideas, such as:

• Acceptance Tests
  Here I mean a light version of the full test scripts that are possible. Ones that match user stories come in the form:
  Given that [context], when [event/action], then [expected outcome].
  Like user stories they have the same problems with being at the right scale.
• Key Scenarios
  Penny Hagen describes these as combinations of user stories that help define the strategic direction of the website. Whilst she uses them as a tool to help define design strategy early on, they have a useful purpose in helping developers understand the context of the user stories they are working on.
• User Pathways
  Again Penny Hagen describes these as a detailed view of the pathway a user needs to follow for a key scenario. They don’t explicitly restrict themselves to pages, or data, but they do help identify the user’s view of what is happening when they interact with the system – and give insight into the user’s experience over time.

Personally I’ve used mockups, high fidelity design, conversation/whiteboarding and acceptance tests to varying levels of success. One of the problems you face in Agile development is that the more artifacts you ask for, the longer it takes to go from designing to building – and the greater the chance that your artifacts are out of date and, frankly, wrong.

I’m still not sure about how to set my user stories, I sure appreciate the discipline they grant even quick feature specifications, but I’m loathe now to simply bang them out when the reality is that I might need to use either less, or more, detail for a given feature. I’m going to keep experimenting because that’s what Agile is all about. In the meantime let me know if you find something better to use!

When Agile Doesn’t Work

I am a huge fan of Agile software development and since becoming Elcom’s Technical Director I have made it my business to push us to become more Agile at every step. But a few months ago I realised I was pushing a square block into a round hole.

It started with a realisation that most proponents of Agile software development had a common single clause that modified their promises of great results.

Customers must get involved for Agile to work.

In Scrum it’s the Product Owner, and it must be the single wringable neck – the person accountable to the business for the project’s delivery of real value. It is the person who not only understands, but can make decisions about functionality, business needs and the user interface.

Oops.

While this sounds great, and you would think that people paying you tens (if not hundreds) of thousands of dollars would want to get involved, we had found all too often that big promises of ongoing involvement disappeared as soon as the project kicked off. Clients wanted to tell us what they wanted, approve our quote, pay us the money, and then pretend we didn’t exist until we had finished their project. Or they had a committee of thousands that wanted to give us their pent up IT requirements, regardless of whether it was in scope or not, and often in conflict with each other.

To be fair to our clients, we often are simply implementing our software and the amount of customer involvement is quite low – however that didn’t fit the Agile methodology we wanted to work within.

Things became clearer for me when I attended a ThoughtWorks Quarterly Technology Briefing in Sydney, and got the chance to hear Martin Fowler speak about how to fail in Agile adoption. During the Q&A I asked him how he suggested we handle customers who were reluctant to get involved.

Martin pointed out that there are in general two sorts of projects:

1. Strategic: Key to business success, and as such important.
2. Infrastructure: Necessary to do, but not seen as particularly vital for business success.

Immediately I saw that for many of our clients their projects were internally seen as necessary (new intranet, re-skinned website) but were not particularly risky or impactful on the organisation’s success. I mulled that answer over for a while and wondered what it might mean for our use of Agile.

A few weekends later I was re-reading David Maister’s classic book Managing the Professional Service Firm and came across his definition of the three basic types of professional service delivery. He writes for any type of professional service and uses examples from disciplines as far apart as architecture, accounting, engineering and management consulting to make his point.

Brains/Expertise

• Client problem is at forefront of professional or technical knowledge
• “Hire us because we’re smart”
• One-off projects
• Highly skilled and highly paid professionals

Grey Hair/Experience

• Clients require highly customised output, more familiar problems
• “Hire us because we’ve done it before”
• Specialised knowledge in project area
• We sell our knowledge, experience and judgement

Procedure/Efficiency

• Client problems are well-known and familiar, solutions also well-known
• “Hire us because we can do it for you”
• Repeat projects, well understood, standard solutions
• Well trained and supervised junior staff

Maister points out that you can see a continuum from Brains to Grey Hair to Procedure for a range of different service features.

Using this idea, I could see that most of the “infrastructure” projects we had involved procedure or efficiency services. A simple deployment of CommunityManager.NET with implementation of a specified basic website design and some custom styling of forms and other modules. Agile simply slowed these projects down and was generally ignored by the team doing most of this work.

On the other hand some of our custom development work fell into the realm of grey hair or experience services – enhancing or extending CommunityManager.NET for a client. These usually did not need to be Agile projects, but were being forced into our product sprint cycle, even when they were a day or two of custom work not dependent upon enhancements to our core product.

Our product work deserves a mention here. Whilst not for any specific client, this is in fact the work that provides the greatest return on investment for the business. However a need for cash flow meant there was a limit to how much could be done at any given time. We had a settled sprint duration of 6 weeks (5 weeks dev + 1 week regression testing and release preparation), wit 1 week in between sprints, that the team had a good rhythm with. Agile was a great fit for this work.

After this analysis there was a small, but important part of our business. Small in terms of number of projects, but important in terms of revenue. This was the projects that involved significant custom development work and had little dependency upon CommunityManager.NET. Sometimes these jobs could have been delivered without our product at all, but using it gave a convenient head start to the project. These projects were strategic to the client, but as a result often had loose, or non-existent requirements because it was acknowledged that the requirements would change during the course of the project. This type of work suits Agile methodologies, but the business needed to recognise that this work shouldn’t be sold the same way our other services were.

Clearly Agile suited some parts of our business, but was a poor fit for others. Our sales material made a big deal of our Agile methodology, but it was not always relevant or appropriate for our clients. Worse, working in an Agile manner was causing many projects to have a greater chance of failure, and to delay client solutions.

Agile Wasn’t Working

I now knew why Agile wasn’t a great fit for much of our business, but it still had value in some areas. Maister also made the point that when for a professional service firms to mix Brains, Grey Hair and Procedure work successfully it is necessary to create divisions within the firm to allow each to have a management style and culture that fit their style of work.

I really like the way that Jurgen Appelo organises his development teams, with a project manager assigned 3-5 other staff as their permanent team, with them needing to complete each of their projects using that team. By pushing resource constraints to the lowest level (that of the project team) they ensure that other projects’ resource variances do not adversely affect them. However I know that Elcom is much smaller than Jurgen’s company, and such a profound shift would create more friction than it would generate in gains.

I discussed these ideas with my boss and some of my colleagues, but felt that we quickly needed to make a change as a large custom development project was about to kickoff its second stage.

The team structure we are now heading towards is this:

Delivery Team

This is a team made up of web designers, front-end web developers, and .NET analyst/programmers. Their role will be to implement new customers’ versions of CommunityManager.NET as painlessly and efficiently as possible. They will handle custom development work that is tightly integrated to, or built around, CommunityManager.NET. They work in traditional BUFD (big up-front design) projects with well-scoped requirements. Project duration would usually be anything from 1-8 weeks.

Solutions Team

This is a team made up of user experience (UX) consultants, designers, and web developers. It is the renaissance team that is responsible for finding new ways to deliver innovative solutions for strategic client problems using design thinking and Agile software development methodologies. They work in a more flexible manner, preferably with a pseudo time and materials approach, with the client purchasing entire sprint’s worth of work. Project duration would usually be more than 12 weeks, with usually two week sprints.

Product Team

This is a team made with a web designer, front-end web developer and .NET developers. It is responsible for enhancements to, and strategic development of our products, particularly CommunityManager.NET. They follow an Agile software development methodology and are very much the traditional product-centred Agile team. Sprint duration would depend upon the ability to regression test in a timely manner – but for the moment this would be around the same 6 week duration we had used previously.

Right Now

When we first made this change, there wasn’t enough strategic product work for us to justify having a dedicated Product Team. Instead members of the Delivery Team are made available as and when required to work on the product. We still have 6 week product sprints.

The Delivery Team has started transitioning away from Agile, and their time is now more easily shifted from one piece of client work to another (they previously were committed  up to 6 weeks in advance for a particular sprint). Our clients have started to notice that we have become more flexible and responsive to sudden requests. Our project managers have been learning a new resource planning tool which is slightly clunky (built in Excel, requiring double data entry) but that we soon hope to replace with a more effective system.

The Solutions Team has had some success working in a more Agile manner, but they need to learn to do it without the moral, emotional or practical support of the rest of the technical teams. By removing many of the restrictions that product work placed on them, they are now free to explore new technologies and test them in practice before we roll them into our products. For instance they have used Domain-Driven Design (DDD) and LINQ2SQL.

The teams are differentiated by more than just name. They have different KPIs, different management styles and different things to get good at.

The Delivery Team needs to become more process-driven. Too many things are left to individual choice when the best solution is already known and has been tested in practice. Also documenting requirements up front is a skill set that needs to be re-discovered and applied. After all if you are going to do BUFD projects then you need to be good at it.

The Solutions Team needs to become more Agile, and look at ideas like TDD/BDD and more comprehensive automated testing so that short sprints do not lead to a reduction in quality. Selling a job as an Agile development project is proving harder than we would have thought. It seems lip service to Agile is easy to sell, it sounds good and requires few changes in the way clients work. Real Agile development requires an altogether different mindset, and this is much harder to sell up front. We may well need to lead these projects with an initial fixed price project that uses BDUF, and then transition to a more Agile approach once we have some credits in the trust account with our client.

The Future

The Solutions Team will be undertaking a major strategic product initiative for Elcom over the Christmas period and will bring their Agile approach and technical expertise to bear in order to give us a major new element to our product for CommunityManager.NET version 7.0. After this we hope to have another major project lined up, but if not then they may become our Product Team.

If another Solutions Team project does come up, then the next step will be to create a full-time Product Team. We believe that version 7.0 will create enough demand that a full-time team will be justified. An important part of this strategy is to find partners that are comfortable enough implementing CommunityManager.NET that our Delivery Team is not a bottleneck on our sales. One of the keys to version 7’s success will be the way that it makes it far easier for digital agencies and systems integrators to use our product for their own web projects.

After that fame, fortune and fun might consume our days – or more likely we will find ourselves looking beyond enterprise content management to create a new market niche in the manner of good Blue Ocean strategists. Whatever we do, Agile will remain a part of it, but only when it deserves to be.

Keep Blogging, Don’t Stop!

This is a reminder for myself as much as anything else. With my meeting schedule and responsibilities it can be easy to avoid blogging for weeks at a time. However, I think that is a mistake – and I want to record why. In fact, blogging about not blogging is a meme all bloggers seem to fall into at one stage or another, so this might seem a little trite. My excuse is that I’m encouraging my team to consider blogging more themselves (and I need to kick my own backside from time to time too!).

In summary I think the Reasons to Keep Blogging are:

• Career
• Mental health/sharpness
• Community
• Memory
• Discipline

Career

I have had the unfortunate experience of working somewhere for a relatively long time (5 years) and then being retrenched through no fault of my own. What I found was that I had been ignoring my career in general in favour of my career in the company that employed me.

Let me unwrap that a little. It is easy when working for someone to find yourself learning just what you need to for the tasks at hand. To restrict your professional relationships to colleagues and clients. To ignore new ideas that are not acceptable where you work. To stunt your growth in other words.

Just as it’s important to have a life outside work, it is important to have a professional life outside of work. If you get to know people outside your normal range of professional contacts, explore ideas outside what is needed for work and generally get interested in your industry then it is surprising how much more effective it makes your job search chances later.

For information works, especially creative types like developers, blogging can be a very effective way of building a portfolio of work and ideas that future employers can be impressed by. As an employer, I know that it also helps potential employers validate their face to face opinion of you.

Mental Health

As a creative type you need to keep your mental faculties well exercised and challenged. Work is not always the best place where you can get a chance to push yourself. Sometimes it helps to try something in public, but not to have too much hanging on the result.

Use your blog as a place to explore new ideas, techniques and ways of doing things. It is not just the content that needs to be fresh, so does the design and layout of the site. (reminder to self: take your own advice!)

Community

Don’t just blog on your own. Leave comments on other people’s site and link your name back to your blog so people can come see more of what you think if they liked it. Get into conversations, write articles criticising other’s ideas, commenting on them, stretching them, validating them, taking them further, or writing them off.

In other words, get involved with people.

Memory

Sometimes I wish I’d record how I fixed something, or where I found an idea out, because often I need it again later and can’t find it (easily). I discussed this with Neil Houghton the other day, sometimes you use a nifty technique in apiece of code, but then requirements change and it gets deleted, and you;re left hunting through sourced code control (Subversion in our case) to find it. Far better to make the effort to blog about it, and then some other poor schmuck might get help too!

Discipline

Aaaah the “D” word. Need anything else be said?

Self-discipline is something to be much admired, the enemy within is always vastly more difficult to conquer than the one without. But it seems to have a bad name.

Discipline is not about some rigid, heavily controlled lifestyle that simply denies and restricts our actions to the ones most acceptable, or least harmful to others.

Discipline is useful because it builds up the strength of will, and willpower is often the only difference between success and failure. Can you keep going even when you fail? Does one loss make you a loser? I think discipline is the habit of trying to succeed just one more time. It is denial, and restriction, but the purpose behind it is to do what we really want to do and not get distracted by the unnecessary, the mundane or the superficial.

If you really want to blog, if it fulfils one of the purposes I’ve outlined above, then you will find that continuing to do it helps build in you the discipline needed to do other things you really want to do. And that is the best reason I can think of to keep blogging.

Software projects are like sailing voyages

I have been reading Greg Smith's new Becoming Agile book and was struck by the false analogy that is often applied to software projects, which compares them to construction projects, where Smith points out you typically "gather requirements, create a design, excavate, lay the foundation, and then build the walls."

Smith points out two problems with applying this analogy:

"1. Additional requirements will be discovered during design, coding, demonstration, and testing.
2. Software development does not require completion of each step before the following step can be initiated. You can start design and coding based on a few initial requirements. You can iteratively build out the system, revisiting it and going deeper on requirements, design, and code as you make discoveries and refine the vision for the project."

Analogies are dangerous because they build in these sorts of inaccuracies, and that can lead to dangerous assumptions and attitudes later on. However they are excellent for helping non-experts in a field understand complex topics. I dearly want a better analogy for software projects and I have never heard a really satisfactory one, so I got the mental juices going and looked for one that makes sense to me. This is what I found ...

Software Projects are Sailing Voyages

Software projects, at least the ones I have been involved in, feel more like ocean voyages in small yachts than anything else. Like any sailing voyage you will have an idea of where you want to get to, and if you are smart you will have checked the weather forecast and planned your route - even calculating what you average speed should be and when you will reach your destination.

from: http://www.flickr.com/photos/14132971@N05/3100906233/ 

This is reasonably similar to the planning and preparation for a software project where a goal will be set, the estimated time to reach it mapped out and the current environment and conditions researched.

However, in sailing you can expect to find a number of things changing your plans. Perhaps the wind changes in unexpected ways, or your plan failed to take into account the wind shadow of a particularly large island, or the strength of local currents or tides. The helmsman may have neglected to keep his eye on where you were headed, being distracted by a pretty sight, or other crew may not have kept the sails properly trimmed or closed the bilge valve.

from: http://www.flickr.com/photos/flasporty/2791384256/ 

Mishaps can occur, such as equipment failure, collisions with other vessels or even crew sickness or injury. Your destination port might be closed, and you find that you need to make landfall either sooner or later than planned. Even the captain's intent may change as it becomes apparent that the passengers would prefer a leisurely cruise around the coast to a deep ocean adventure.

from: http://www.flickr.com/photos/wili/2691809503/ 

Very similar changes can be expected in software projects. Your tools vendor may bring out a new release that opens up new development options, business goals can change or be re-assessed, leading to changes in requirements. Staff changes can cause a change in vision, or highlight mistakes or negligence in the work previously performed. Individuals' performance may fall short or exceed expectations for their role, and unexpected leave or changes in priorities may derail timelines.

These problems occur on construction projects too, but they have a real, physical object they have been developing with building approvals already acquired so it is far more likely that they will simply continue to the end (even then, you might end up with a project like Sydney's World Square, where it was just a huge hole in the ground with some cement in it for years).

from: City of Sydney 

With software it always seems easier to throw it all away and start again (and indeed this is sometimes the best course to take). Simply having a shared analogy with the customer that makes more sense than a building will help them understand that while we can give up the voyage now, we can't avoid the fact that we have sailed this far, and perhaps we should take that learning and make use of it.

Agile software development in this case can be seen as a series of small hops around the coast, rather than the usual grand exploratory voyage undertaken by many traditional software projects. Each hop allows us to review where we have gotten to, and lets passengers get off if they so desire.

Of course analogies have their limitations, and using them to model a complex system by comparing it to another simpler, or more familiar, system does not guarantee success. It just makes client meetings shorter (which by some definitions is success!).

Half of TEDx Sydney 2009

Thanks to Josh Anstey's generosity I was able to attend TEDx Sydney last night. Whilst it was enjoyable, I ended up leaving at the half-time break and spending the evening with my family instead. Nevertheless if you're interested in TED then I would recommend giving an event like this a go, if nothing else there were some great people there like Mark Cohen, Jodie Miner and Brian Maguire.

Some background: TED is an event that focuses on thought leadership in the Technology, Entertainment and Design fields every year and has as its catch-cry "Ideas worth spreading". The TED conference sells out quickly every year (despite some unusual conditions) and attracts all sorts of luminaries and leaders. A TED event is the place to meet movers and shakers and has been the birthplace of some wonderful ideas over the years. They make the videos of each event available online during the year, and part of the focus of a TEDx event is to create that same buzz in a local venue by mixing TED videos with local speakers.

So without further ado here is my thoughts on TEDx Sydney 2009:

Venue

The Australian Technology Park hosted the event, and as that is where Elcom is based that proved convenient for me. The theatre used was a great venue for watching TED videos and live speakers alike and I think it did a good job of providing some atmosphere - although I wonder whether other venues could have made a bigger impression? (e.g. Darling Harbour Convention Centre)

MC/Host (Nils Vesk)

I don't think a Gen-Y host was the right choice for an event like this - I think less entertainment and more credibility was needed (although it was Mark Cohen who blew Nils' credibility for me). However, Nils got the job done, and managed to do the more cheesy sales aspects of his role without making it seem too cheesy (more about that later).

Craig Rispin (Futurist)

I have heard of Craig Rispin, and indeed knew someone who he'd invited to the event. Unfortunately his recursive telling of the history of futurists who tell the history of the future left me bored and dual-tweeting with Mark Cohen.

I'm not sure Craig had an idea worth spreading, perhaps he intended it to be that we are all futurists, but it felt more like he was telling us that we were smart and forward-thinking people - which was nice of him, but didn't give me a reason to care about his talk. He also pointed out the prophetic nature of speculative/science-fiction in a way that might have impacted those who didn't yet care about it - but I've known about that for most of my life so it was wasted on me.

Patti Maes (Video, MIT Media Lab)

Patti Maes is from the MIT Media Lab and demonstrated a form of augmented reality where her students had cobbled together a wearable apparatus that linked a camera, phone and some software so that you could make any surface around you into a computer work area, and interact with your environment in ways that could both instruct the computer/camera to do something (e.g. take a photo by holding your hands up in a rectangle) or could bring digital information to you based on your context (e.g. talking to a person, reading a book, looking at a product label).

She pointed out that the technology could be cobbled together for a few hundred dollars easily enough, but their point was that they were investigating what were reasonable ways to augment reality. Some of this has been done before with for example the head-up displays (HUD) that Boeing engineers use when wiring plane bodies.

This was the most exciting talk, but hardly surprising to anyone who has been following technology for the last 15 years or so.

Peter Baines (Hands Across the Water)

Peter Baines was one of the forensic specialists sent to Thailand in the aftermath of the recent Boxing Day Tsunami. He gave us a fairly somber account of that, especially dealing with the human side when giving bodies back to relatives. He went on to show how this created a desire in him and his colleagues to do more than they were for the Thais - out of that desire was birthed Hands Across the Water.

I think that Peter's idea worth spreading was that if you just focus on the results, and not the obstacles, then you can achieve your dreams - whatever they are, whoever you are. I think this was a great message, and Peter is obviously a good example of this in action. Focus, and the single-minded pursuit of an idea, is too often given lip-service as a reason for success, when I believe it might just be the major factor in it.

A large part of Peter's story was a dig at the international aid agencies that quickly went into Thailand and then just as quickly pulled out around a year later. One of his ideas is that charities should be forced to put some stats on every piece of marketing literature telling the reader how much money they spent on admin versus aid in the last financial year. The idea (much like the MIT Media guys) being to give people information to make informed decisions about where to give their money. I think this ignores part of the bigger picture, which is a question of how much admin is required to run an international charity that coordinates volunteers from many nations into situations that are often not as attractively packaged as the Thai one was. However that discussion can't happen without making the information available, and so I would welcome it form that point of view.

Bill Gates (Video, Curing Malaria and Making Teachers Better)

I had seen this one before, but hung around to watch it on the big screen. Bill and Melinda Gates' foundation is really trying to focus money and attention onto big issues that the market and governments are ignoring.

The first issue he covered was curing malaria, which seems both an achievable aim and one that has been largely ignored now that wealthy nations are free of it. His idea here was that if we focus on eradicating it then a relatively modest effort will see it gone, whilst if we tackle it half-heartedly we only make things worse.

The second issue was about how do you make teachers great, and he gave a long analysis of why this mattered, which seemed obvious, but I guess the hard numbers he pulled out showed people that it was an issue worth focusing on. I think his idea here was that we should care about the quality of teaching - mainly because in the US at least, the quality of a teacher is not easily discovered or rewarded.

Event

Overall the event had a good vibe, seemed interesting, and was a good way to share the watching of a TED video with people you knew. I think I missed too much of the schmoozing part of it to know whether that worked - but I was happy catching up with who I did see there.

There was a cheesy side to the event which was the blatant pushing of wares for the main sponsors, especially the silent auction of the Thought Leaders' services (with some fairly suspicious dollar figures on their market value). I understand that some of this is necessary to get the event paid for - but I do wonder if this could have been handled in a more subtle and positive way.

Top 5 classic programmer folly projects

Back in the day when the British Empire was still new, wealthy landowners who had heard about great places elsewhere (usually ones where it didn't rain all the time) started a trend of re-creating these great places on their own land - often right down to the current decrepit state of said places.

Sometimes they didn't have a specific place in mind, but wanted a "ruined Greek temple" or similar stereotypical "interesting" structure (sometimes they were just plain strange). The point seemed to be to prove that a) you were interesting, b) you could pay for a spurious building for the sake of it, and c) foreign travel was all so unnecessary. These were known, rather aptly, as follies (presumably because they represented the folly of spending your children's inheritance on an expensive monument when investing it might have proven more prudent).

From: Xerones 

Programmers it turns out are no less susceptible to follies, although they might not realise it, and they are not deliberate fakeries. There are several classic programmer follies which tend to get repeated by each generation, with ever-increasing zeal and lack of awareness of their futility.

Now, don't get me wrong, these are all fine examples of throwaway projects that you might play around with and practice on, but for some reason programmers can't seem to leave them stay as just throwaway projects - they always want to upgrade them to the status of "next big thing".

Without any further ado, here they are.

Top 5 classic programmer folly projects

1. Bug tracking

Every software program has bugs, and most programmers find they need a good system to track said bugs. Some companies, Atlassian and Fog Creek Software have made their living providing great, simple yet powerful solutions for exactly this problem.

However, there is always a temptation to take shortcuts, customise the bug tracking, or get a cheaper solution, and so inevitably every programmer gets tempted to build their own bug tracking application.

It's a folly because ...
This is actually a fairly complicated domain, especially once you add in the bug resolution workflow, web interface and email integration that even free, open-source applications have. Of course there is also the problem of what to do when your bugs are in your bug tracking software ... evil recursion in the real world!

2. To do list

I just want to track what I need to do ... and what I want is to improve this to do list. Remember the Milk has the best free implementation I've seen of this type of application, but BaseCamp, your mobile phone and even GMail are worth mentioning (not to mention trusty old pen and paper).

It's a folly because ...
You really don't need to waste brain cycles re-inventing something this simple, and when you look at the sophistication of Remember the Milk (web, offline Google Gears, Twitter, email and mobile integration) anything you do yourself looks kinda ... sad.

3. Website/blog engine

I want to use something I built myself, and it's a good chance to try my hand at content management ... except it isn't. If you are blogging then do your readers and yourself a favour and use an existing blogging application. You will have a more consistent user interface and will never find yourself not posting because the build is broken after that last refactor and you can't be bothered fixing it yet.

It's a folly because ...
The free alternatives are great, the cheap ones are even better and frankly it's not that great a domain for learning how to code, especially web stuff.

4. Hobby management

My hobby (roleplaying games, model robots, high performance cars, salsa dancing, etc) is out of control and I need a way of managing it. Actually no you don't, you probably just need to get out and live more.

It's a folly because ...

You don't really want to mix up your hobby and your work, it just makes one less fun and the other less professional. it is also an area where you will never really be satisfied with the result because it's your hobby. Finally, when you show it to other people they are likely to go "Ummm ... very nice. What is it?" Hugh MacLeod defines a geek as "Somebody who socializes via objects" and congratulations, that's exactly what you've just become!

5. Application Building Applications

This is the one where you build an application that uses meta-data to build an application. It seems this is the purvey of the very smart and the very dumb. The really smart guys believe they can be the ones to break through and nail this problem, the really dumb ones just don't realise how hard it is.

It's a folly because ...

It's possible to do this in theory, but in practice you only ever get around 80% of the way there, and it turns out the other 20% is the bit that an application really needs to live and breathe. Really, really smart guys manage to push that 80% up to the giddy heights of 81.63%, but hey, it still doesn't do what you really wanted it to.

Bonus! Installer software

OK, here is number six. What do you do when you want to distribute your application to clients? Use the MSI installer for Visual Studio? Heck, no! You go ahead and create your own installer software, complete with license number check, built-in unzipper and some cross-platform nastiness.

It's a folly because ...

You already know you shouldn't be doing this. Go use something someone else created and love the fact that you don't need to do it by hand!

 

OK, time for confession time, I've been guilty of each of these (at least through to the design stage). What about you, what were your follies?

Loading objects' fields/properties automagically

There is a very nice function in the new ASP.NET MVC code that allows a Controller to load an object from values passed in from a HttpRequest (UPDATE: called UpdateFrom, see ScottGu's post). It did a best guess on matching values to object properties, but most (99%?) of the time that is all you need. Well, when I started using Domain-Driven Design (DDD) and Test-Driven Development (TDD) on the latest additions to our Community Manager.NET product I wanted to find something similar - but I couldn't find anything as terse and useful.

So I did what any good dev would, I read the sourcecode and derived my own functions to do something similar, but with the SqlDataReaders that my repositories would be getting out of SQL Server (it almost feels like the ActiveRecord pattern ...). Below are the result, two LoadToObject() functions (and a PopulateTypeException class) in my BaseRepository class that give me the functionality I need. I'm putting these out there in case someone else needs this, and in the hope that any glaringly obvious errors might get picked up by others!

One issue I had with the Microsoft version of this function was they only load public properties of an object, not public fields. Now I know that in C# you can setup properties as easily as fields, but in VB it's still a hassle. I personally like using fields for simple value storing because it is clearer to see what they do in the code and leave more space for real method calls.

The final problem I ran into was that my custom Html data type needed to implement it's own TypeConverter so that the HTML in the SQL Server ntext field could be converted to a strongly typed Html type. But that can go into another blog post if people are interested.

    ''' <summary>
    ''' Populates an object's public fields and properties with values from a SqlDataReader.
    ''' </summary>
    ''' <param name="readerToLoad">The SqlDataReader that needs to be loaded into the object.</param>
    ''' <param name="obj">The object we want to populate.</param>
    ''' <param name="objectPrefix">A prefix to the object field/property names. Tries to match using "." and "_".</param>
    ''' <returns>The object passed in with fields/properties loaded.</returns>
    ''' <remarks>
    ''' Does a best guess as to how to match names from the SqlDataReader to the object.
    ''' </remarks>
    Protected Function LoadToObject(ByVal readerToLoad As SqlDataReader, ByVal obj As Object, ByVal objectPrefix As String) As Object
        Dim newColl As New System.Collections.Specialized.NameValueCollection()
        Dim x As Integer

        If obj Is Nothing Then
            Throw New ArgumentNullException("obj", "Object must have a value in order to be loaded to.")
        End If

        For x = 0 To (readerToLoad.FieldCount - 1)
            newColl.Add(readerToLoad.GetName(x), readerToLoad.GetValue(x))
        Next

        Return LoadToObject(newColl, obj, objectPrefix)
    End Function

    ''' <summary>
    ''' Populates an object's public fields and properties with values from a NameValueCollection.
    ''' </summary>
    ''' <param name="valueCollectionToLoad">The NameValueCollection that needs to be loaded into the object.</param>
    ''' <param name="obj">The object we want to populate.</param>
    ''' <param name="objectPrefix">A prefix to the object field/property names. Tries to match using "." and "_".</param>
    ''' <returns>The object passed in with fields/properties loaded.</returns>
    ''' <remarks>
    ''' Does a best guess as to how to match names from the NameValueCollection to the object.
    ''' </remarks>
    Protected Function LoadToObject(ByVal valueCollectionToLoad As System.Collections.Specialized.NameValueCollection, ByVal obj As Object, ByVal objectPrefix As String) As Object
        Dim objType As Type = obj.GetType()
        Dim objName As String = objType.Name
        Dim exceptionList As New StringBuilder()
        Dim props As PropertyInfo() = objType.GetProperties()
        Dim fields As FieldInfo() = objType.GetFields()
        Dim ex As PopulateTypeException = Nothing
        Dim prop As PropertyInfo
        Dim field As FieldInfo

        ' try writing to the object's properties
        For Each prop In props
            'check the key, going to be forgiving here, allowing for full declaration or just propname
            Dim key As String = prop.Name

            If objectPrefix <> String.Empty Then
                key = objectPrefix + key
            End If

            If valueCollectionToLoad(key) = Nothing Then
                key = objName + "." + prop.Name
            End If

            If valueCollectionToLoad(key) = Nothing Then
                key = objName + "_" + prop.Name
            End If

            If valueCollectionToLoad(key) <> Nothing Then
                Dim conv As TypeConverter = TypeDescriptor.GetConverter(prop.PropertyType)
                Dim value As Object = valueCollectionToLoad(key)

                If conv.CanConvertFrom(System.Type.GetType("System.String", True, True)) Then
                    Try
                        value = conv.ConvertFrom(valueCollectionToLoad(key))
                        prop.SetValue(obj, value, Nothing)
                    Catch e As Exception
                        Dim message As String = prop.Name + " is not a valid " + prop.PropertyType.Name + "; " + e.Message

                        If ex Is Nothing Then
                            ex = New PopulateTypeException("Errors occurred during object binding - review the LoadExceptions property of this exception for more details")
                        End If

                        Dim info As New PopulateTypeException.ExceptionInfo()

                        info.AttemptedValue = value
                        info.PropertyName = prop.Name
                        info.ErrorMessage = message

                        ex.LoadExceptions.Add(info)
                    End Try
                Else
                    Throw New Exception(String.Format("No type converter available for type: {0}", prop.PropertyType))
                End If
            End If
        Next

        ' now try writing to the object's public fields
        For Each field In fields
            'check the key, going to be forgiving here, allowing for full declaration or just fieldname
            Dim key As String = field.Name

            If objectPrefix <> String.Empty Then
                key = objectPrefix + key
            End If

            If valueCollectionToLoad(key) = Nothing Then
                key = objName + "." + field.Name
            End If

            If valueCollectionToLoad(key) = Nothing Then
                key = objName + "_" + field.Name
            End If

            If valueCollectionToLoad(key) <> Nothing Then
                Dim conv As TypeConverter = TypeDescriptor.GetConverter(field.FieldType)
                Dim value As Object = valueCollectionToLoad(key)

                If conv.CanConvertFrom(System.Type.GetType("System.String", True, True)) Then
                    Try
                        value = conv.ConvertFrom(valueCollectionToLoad(key))
                        field.SetValue(obj, value)
                    Catch e As Exception
                        Dim message As String = field.Name + " is not a valid " + field.FieldType.Name + "; " + e.Message

                        If ex Is Nothing Then
                            ex = New PopulateTypeException("Errors occurred during object binding - review the LoadExceptions property of this exception for more details")
                        End If

                        Dim info As New PopulateTypeException.ExceptionInfo()

                        info.AttemptedValue = value
                        info.PropertyName = field.Name
                        info.ErrorMessage = message

                        ex.LoadExceptions.Add(info)
                    End Try
                Else
                    Throw New Exception(String.Format("No type converter available for type: {0}", field.FieldType))
                End If
            End If
        Next

        If Not (ex Is Nothing) Then
            Throw ex
        Else
            Return obj
        End If
    End Function

    ''' <summary>
    ''' Contains exceptions that arise during the populating of an object by the LoadToObject() method.
    ''' </summary>
    <Global.System.Serializable()> Public Class PopulateTypeException
        Inherits Exception

        Public Class ExceptionInfo
            Public PropertyName As String
            Public AttemptedValue As Object
            Public ErrorMessage As String
        End Class

        Public LoadExceptions As New List(Of ExceptionInfo)
        Private PopulateTypeException()

        Public Sub New(ByVal message As String)
            MyBase.new(message)
        End Sub

        Public Sub New(ByVal message As String, ByVal inner As Exception)
            MyBase.new(message, inner)
        End Sub

        Protected Sub New(ByVal info As System.Runtime.Serialization.SerializationInfo, ByVal context As System.Runtime.Serialization.StreamingContext)
            MyBase.New(info, context)
        End Sub

    End Class