Thursday, November 19, 2009

When Agile Doesn’t Work

imageI am a huge fan of Agile software development and since becoming Elcom’s Technical Director I have made it my business to push us to become more Agile at every step. But a few months ago I realised I was pushing a square block into a round hole.

It started with a realisation that most proponents of Agile software development had a common single clause that modified their promises of great results.

Customers must get involved for Agile to work.

In Scrum it’s the Product Owner, and it must be the single wringable neck – the person accountable to the business for the project’s delivery of real value. It is the person who not only understands, but can make decisions about functionality, business needs and the user interface.

Oops.

While this sounds great, and you would think that people paying you tens (if not hundreds) of thousands of dollars would want to get involved, we had found all too often that big promises of ongoing involvement disappeared as soon as the project kicked off. Clients wanted to tell us what they wanted, approve our quote, pay us the money, and then pretend we didn’t exist until we had finished their project. Or they had a committee of thousands that wanted to give us their pent up IT requirements, regardless of whether it was in scope or not, and often in conflict with each other.

To be fair to our clients, we often are simply implementing our software and the amount of customer involvement is quite low – however that didn’t fit the Agile methodology we wanted to work within.

Things became clearer for me when I attended a ThoughtWorks Quarterly Technology Briefing in Sydney, and got the chance to hear Martin Fowler speak about how to fail in Agile adoption. During the Q&A I asked him how he suggested we handle customers who were reluctant to get involved.

Martin pointed out that there are in general two sorts of projects:

  1. Strategic: Key to business success, and as such important.
  2. Infrastructure: Necessary to do, but not seen as particularly vital for business success.

Immediately I saw that for many of our clients their projects were internally seen as necessary (new intranet, re-skinned website) but were not particularly risky or impactful on the organisation’s success. I mulled that answer over for a while and wondered what it might mean for our use of Agile.

MaisterProfessionalServicesFirm A few weekends later I was re-reading David Maister’s classic book Managing the Professional Service Firm and came across his definition of the three basic types of professional service delivery. He writes for any type of professional service and uses examples from disciplines as far apart as architecture, accounting, engineering and management consulting to make his point.

Brains/Expertise

  • Client problem is at forefront of professional or technical knowledge
  • “Hire us because we’re smart”
  • One-off projects
  • Highly skilled and highly paid professionals

Grey Hair/Experience

  • Clients require highly customised output, more familiar problems
  • “Hire us because we’ve done it before”
  • Specialised knowledge in project area
  • We sell our knowledge, experience and judgement

Procedure/Efficiency

  • Client problems are well-known and familiar, solutions also well-known
  • “Hire us because we can do it for you”
  • Repeat projects, well understood, standard solutions
  • Well trained and supervised junior staff

image

Maister points out that you can see a continuum from Brains to Grey Hair to Procedure for a range of different service features.

image

Using this idea, I could see that most of the “infrastructure” projects we had involved procedure or efficiency services. A simple deployment of CommunityManager.NET with implementation of a specified basic website design and some custom styling of forms and other modules. Agile simply slowed these projects down and was generally ignored by the team doing most of this work.

On the other hand some of our custom development work fell into the realm of grey hair or experience services – enhancing or extending CommunityManager.NET for a client. These usually did not need to be Agile projects, but were being forced into our product sprint cycle, even when they were a day or two of custom work not dependent upon enhancements to our core product.

Our product work deserves a mention here. Whilst not for any specific client, this is in fact the work that provides the greatest return on investment for the business. However a need for cash flow meant there was a limit to how much could be done at any given time. We had a settled sprint duration of 6 weeks (5 weeks dev + 1 week regression testing and release preparation), wit 1 week in between sprints, that the team had a good rhythm with. Agile was a great fit for this work.

After this analysis there was a small, but important part of our business. Small in terms of number of projects, but important in terms of revenue. This was the projects that involved significant custom development work and had little dependency upon CommunityManager.NET. Sometimes these jobs could have been delivered without our product at all, but using it gave a convenient head start to the project. These projects were strategic to the client, but as a result often had loose, or non-existent requirements because it was acknowledged that the requirements would change during the course of the project. This type of work suits Agile methodologies, but the business needed to recognise that this work shouldn’t be sold the same way our other services were.

Clearly Agile suited some parts of our business, but was a poor fit for others. Our sales material made a big deal of our Agile methodology, but it was not always relevant or appropriate for our clients. Worse, working in an Agile manner was causing many projects to have a greater chance of failure, and to delay client solutions.

Agile Wasn’t Working

I now knew why Agile wasn’t a great fit for much of our business, but it still had value in some areas. Maister also made the point that when for a professional service firms to mix Brains, Grey Hair and Procedure work successfully it is necessary to create divisions within the firm to allow each to have a management style and culture that fit their style of work.

I really like the way that Jurgen Appelo organises his development teams, with a project manager assigned 3-5 other staff as their permanent team, with them needing to complete each of their projects using that team. By pushing resource constraints to the lowest level (that of the project team) they ensure that other projects’ resource variances do not adversely affect them. However I know that Elcom is much smaller than Jurgen’s company, and such a profound shift would create more friction than it would generate in gains.

I discussed these ideas with my boss and some of my colleagues, but felt that we quickly needed to make a change as a large custom development project was about to kickoff its second stage.

The team structure we are now heading towards is this:

Delivery Team

This is a team made up of web designers, front-end web developers, and .NET analyst/programmers. Their role will be to implement new customers’ versions of CommunityManager.NET as painlessly and efficiently as possible. They will handle custom development work that is tightly integrated to, or built around, CommunityManager.NET. They work in traditional BUFD (big up-front design) projects with well-scoped requirements. Project duration would usually be anything from 1-8 weeks.

Solutions Team

This is a team made up of user experience (UX) consultants, designers, and web developers. It is the renaissance team that is responsible for finding new ways to deliver innovative solutions for strategic client problems using design thinking and Agile software development methodologies. They work in a more flexible manner, preferably with a pseudo time and materials approach, with the client purchasing entire sprint’s worth of work. Project duration would usually be more than 12 weeks, with usually two week sprints.

Product Team

This is a team made with a web designer, front-end web developer and .NET developers. It is responsible for enhancements to, and strategic development of our products, particularly CommunityManager.NET. They follow an Agile software development methodology and are very much the traditional product-centred Agile team. Sprint duration would depend upon the ability to regression test in a timely manner – but for the moment this would be around the same 6 week duration we had used previously.

Right Now

When we first made this change, there wasn’t enough strategic product work for us to justify having a dedicated Product Team. Instead members of the Delivery Team are made available as and when required to work on the product. We still have 6 week product sprints.

The Delivery Team has started transitioning away from Agile, and their time is now more easily shifted from one piece of client work to another (they previously were committed  up to 6 weeks in advance for a particular sprint). Our clients have started to notice that we have become more flexible and responsive to sudden requests. Our project managers have been learning a new resource planning tool which is slightly clunky (built in Excel, requiring double data entry) but that we soon hope to replace with a more effective system.

The Solutions Team has had some success working in a more Agile manner, but they need to learn to do it without the moral, emotional or practical support of the rest of the technical teams. By removing many of the restrictions that product work placed on them, they are now free to explore new technologies and test them in practice before we roll them into our products. For instance they have used Domain-Driven Design (DDD) and LINQ2SQL.

The teams are differentiated by more than just name. They have different KPIs, different management styles and different things to get good at.

The Delivery Team needs to become more process-driven. Too many things are left to individual choice when the best solution is already known and has been tested in practice. Also documenting requirements up front is a skill set that needs to be re-discovered and applied. After all if you are going to do BUFD projects then you need to be good at it.

The Solutions Team needs to become more Agile, and look at ideas like TDD/BDD and more comprehensive automated testing so that short sprints do not lead to a reduction in quality. Selling a job as an Agile development project is proving harder than we would have thought. It seems lip service to Agile is easy to sell, it sounds good and requires few changes in the way clients work. Real Agile development requires an altogether different mindset, and this is much harder to sell up front. We may well need to lead these projects with an initial fixed price project that uses BDUF, and then transition to a more Agile approach once we have some credits in the trust account with our client.

The Future

The Solutions Team will be undertaking a major strategic product initiative for Elcom over the Christmas period and will bring their Agile approach and technical expertise to bear in order to give us a major new element to our product for CommunityManager.NET version 7.0. After this we hope to have another major project lined up, but if not then they may become our Product Team.

If another Solutions Team project does come up, then the next step will be to create a full-time Product Team. We believe that version 7.0 will create enough demand that a full-time team will be justified. An important part of this strategy is to find partners that are comfortable enough implementing CommunityManager.NET that our Delivery Team is not a bottleneck on our sales. One of the keys to version 7’s success will be the way that it makes it far easier for digital agencies and systems integrators to use our product for their own web projects.

After that fame, fortune and fun might consume our days – or more likely we will find ourselves looking beyond enterprise content management to create a new market niche in the manner of good Blue Ocean strategists. Whatever we do, Agile will remain a part of it, but only when it deserves to be.

Friday, November 13, 2009

Keep Blogging, Don’t Stop!

This is a reminder for myself as much as anything else. With my meeting schedule and responsibilities it can be easy to avoid blogging for weeks at a time. However, I think that is a mistake – and I want to record why. In fact, blogging about not blogging is a meme all bloggers seem to fall into at one stage or another, so this might seem a little trite. My excuse is that I’m encouraging my team to consider blogging more themselves (and I need to kick my own backside from time to time too!).

In summary I think the Reasons to Keep Blogging are:

  • Career
  • Mental health/sharpness
  • Community
  • Memory
  • Discipline

Career

I have had the unfortunate experience of working somewhere for a relatively long time (5 years) and then being retrenched through no fault of my own. What I found was that I had been ignoring my career in general in favour of my career in the company that employed me.

Let me unwrap that a little. It is easy when working for someone to find yourself learning just what you need to for the tasks at hand. To restrict your professional relationships to colleagues and clients. To ignore new ideas that are not acceptable where you work. To stunt your growth in other words.

Just as it’s important to have a life outside work, it is important to have a professional life outside of work. If you get to know people outside your normal range of professional contacts, explore ideas outside what is needed for work and generally get interested in your industry then it is surprising how much more effective it makes your job search chances later.

For information works, especially creative types like developers, blogging can be a very effective way of building a portfolio of work and ideas that future employers can be impressed by. As an employer, I know that it also helps potential employers validate their face to face opinion of you.

Mental Health

As a creative type you need to keep your mental faculties well exercised and challenged. Work is not always the best place where you can get a chance to push yourself. Sometimes it helps to try something in public, but not to have too much hanging on the result.

Use your blog as a place to explore new ideas, techniques and ways of doing things. It is not just the content that needs to be fresh, so does the design and layout of the site. (reminder to self: take your own advice!)

Community

Don’t just blog on your own. Leave comments on other people’s site and link your name back to your blog so people can come see more of what you think if they liked it. Get into conversations, write articles criticising other’s ideas, commenting on them, stretching them, validating them, taking them further, or writing them off.

In other words, get involved with people.

Memory

Sometimes I wish I’d record how I fixed something, or where I found an idea out, because often I need it again later and can’t find it (easily). I discussed this with Neil Houghton the other day, sometimes you use a nifty technique in apiece of code, but then requirements change and it gets deleted, and you;re left hunting through sourced code control (Subversion in our case) to find it. Far better to make the effort to blog about it, and then some other poor schmuck might get help too!

Discipline

Aaaah the “D” word. Need anything else be said?

Self-discipline is something to be much admired, the enemy within is always vastly more difficult to conquer than the one without. But it seems to have a bad name.

Discipline is not about some rigid, heavily controlled lifestyle that simply denies and restricts our actions to the ones most acceptable, or least harmful to others.

Discipline is useful because it builds up the strength of will, and willpower is often the only difference between success and failure. Can you keep going even when you fail? Does one loss make you a loser? I think discipline is the habit of trying to succeed just one more time. It is denial, and restriction, but the purpose behind it is to do what we really want to do and not get distracted by the unnecessary, the mundane or the superficial.

If you really want to blog, if it fulfils one of the purposes I’ve outlined above, then you will find that continuing to do it helps build in you the discipline needed to do other things you really want to do. And that is the best reason I can think of to keep blogging.

Tuesday, August 11, 2009

Software projects are like sailing voyages

I have been reading Greg Smith's new Becoming Agile book and was struck by the false analogy that is often applied to software projects, which compares them to construction projects, where Smith points out you typically "gather requirements, create a design, excavate, lay the foundation, and then build the walls."

Smith points out two problems with applying this analogy:

"1. Additional requirements will be discovered during design, coding, demonstration, and testing.
2. Software development does not require completion of each step before the following step can be initiated. You can start design and coding based on a few initial requirements. You can iteratively build out the system, revisiting it and going deeper on requirements, design, and code as you make discoveries and refine the vision for the project."

Analogies are dangerous because they build in these sorts of inaccuracies, and that can lead to dangerous assumptions and attitudes later on. However they are excellent for helping non-experts in a field understand complex topics. I dearly want a better analogy for software projects and I have never heard a really satisfactory one, so I got the mental juices going and looked for one that makes sense to me. This is what I found ...

Software Projects are Sailing Voyages

Software projects, at least the ones I have been involved in, feel more like ocean voyages in small yachts than anything else. Like any sailing voyage you will have an idea of where you want to get to, and if you are smart you will have checked the weather forecast and planned your route - even calculating what you average speed should be and when you will reach your destination.

3100906233_d9fa131d23_b

from: http://www.flickr.com/photos/14132971@N05/3100906233/ 

This is reasonably similar to the planning and preparation for a software project where a goal will be set, the estimated time to reach it mapped out and the current environment and conditions researched.

However, in sailing you can expect to find a number of things changing your plans. Perhaps the wind changes in unexpected ways, or your plan failed to take into account the wind shadow of a particularly large island, or the strength of local currents or tides. The helmsman may have neglected to keep his eye on where you were headed, being distracted by a pretty sight, or other crew may not have kept the sails properly trimmed or closed the bilge valve.

2791384256_cb84d28653_b

from: http://www.flickr.com/photos/flasporty/2791384256/ 

Mishaps can occur, such as equipment failure, collisions with other vessels or even crew sickness or injury. Your destination port might be closed, and you find that you need to make landfall either sooner or later than planned. Even the captain's intent may change as it becomes apparent that the passengers would prefer a leisurely cruise around the coast to a deep ocean adventure.

2691809503_bd8ec4d118_b

from: http://www.flickr.com/photos/wili/2691809503/ 

Very similar changes can be expected in software projects. Your tools vendor may bring out a new release that opens up new development options, business goals can change or be re-assessed, leading to changes in requirements. Staff changes can cause a change in vision, or highlight mistakes or negligence in the work previously performed. Individuals' performance may fall short or exceed expectations for their role, and unexpected leave or changes in priorities may derail timelines.

These problems occur on construction projects too, but they have a real, physical object they have been developing with building approvals already acquired so it is far more likely that they will simply continue to the end (even then, you might end up with a project like Sydney's World Square, where it was just a huge hole in the ground with some cement in it for years).

Hole-in-the-ground

from: City of Sydney 

With software it always seems easier to throw it all away and start again (and indeed this is sometimes the best course to take). Simply having a shared analogy with the customer that makes more sense than a building will help them understand that while we can give up the voyage now, we can't avoid the fact that we have sailed this far, and perhaps we should take that learning and make use of it.

Agile software development in this case can be seen as a series of small hops around the coast, rather than the usual grand exploratory voyage undertaken by many traditional software projects. Each hop allows us to review where we have gotten to, and lets passengers get off if they so desire.

Of course analogies have their limitations, and using them to model a complex system by comparing it to another simpler, or more familiar, system does not guarantee success. It just makes client meetings shorter (which by some definitions is success!).

Friday, May 29, 2009

Half of TEDx Sydney 2009

Thanks to Josh Anstey's generosity I was able to attend TEDx Sydney last night. Whilst it was enjoyable, I ended up leaving at the half-time break and spending the evening with my family instead. Nevertheless if you're interested in TED then I would recommend giving an event like this a go, if nothing else there were some great people there like Mark Cohen, Jodie Miner and Brian Maguire.

Some background: TED is an event that focuses on thought leadership in the Technology, Entertainment and Design fields every year and has as its catch-cry "Ideas worth spreading". The TED conference sells out quickly every year (despite some unusual conditions) and attracts all sorts of luminaries and leaders. A TED event is the place to meet movers and shakers and has been the birthplace of some wonderful ideas over the years. They make the videos of each event available online during the year, and part of the focus of a TEDx event is to create that same buzz in a local venue by mixing TED videos with local speakers.

So without further ado here is my thoughts on TEDx Sydney 2009:

Venue

The Australian Technology Park hosted the event, and as that is where Elcom is based that proved convenient for me. The theatre used was a great venue for watching TED videos and live speakers alike and I think it did a good job of providing some atmosphere - although I wonder whether other venues could have made a bigger impression? (e.g. Darling Harbour Convention Centre)

MC/Host (Nils Vesk)

I don't think a Gen-Y host was the right choice for an event like this - I think less entertainment and more credibility was needed (although it was Mark Cohen who blew Nils' credibility for me). However, Nils got the job done, and managed to do the more cheesy sales aspects of his role without making it seem too cheesy (more about that later).

Craig Rispin (Futurist)

I have heard of Craig Rispin, and indeed knew someone who he'd invited to the event. Unfortunately his recursive telling of the history of futurists who tell the history of the future left me bored and dual-tweeting with Mark Cohen.

I'm not sure Craig had an idea worth spreading, perhaps he intended it to be that we are all futurists, but it felt more like he was telling us that we were smart and forward-thinking people - which was nice of him, but didn't give me a reason to care about his talk. He also pointed out the prophetic nature of speculative/science-fiction in a way that might have impacted those who didn't yet care about it - but I've known about that for most of my life so it was wasted on me.

Patti Maes (Video, MIT Media Lab)

Patti Maes is from the MIT Media Lab and demonstrated a form of augmented reality where her students had cobbled together a wearable apparatus that linked a camera, phone and some software so that you could make any surface around you into a computer work area, and interact with your environment in ways that could both instruct the computer/camera to do something (e.g. take a photo by holding your hands up in a rectangle) or could bring digital information to you based on your context (e.g. talking to a person, reading a book, looking at a product label).

She pointed out that the technology could be cobbled together for a few hundred dollars easily enough, but their point was that they were investigating what were reasonable ways to augment reality. Some of this has been done before with for example the head-up displays (HUD) that Boeing engineers use when wiring plane bodies.

This was the most exciting talk, but hardly surprising to anyone who has been following technology for the last 15 years or so.

Peter Baines (Hands Across the Water)

Peter Baines was one of the forensic specialists sent to Thailand in the aftermath of the recent Boxing Day Tsunami. He gave us a fairly somber account of that, especially dealing with the human side when giving bodies back to relatives. He went on to show how this created a desire in him and his colleagues to do more than they were for the Thais - out of that desire was birthed Hands Across the Water.

I think that Peter's idea worth spreading was that if you just focus on the results, and not the obstacles, then you can achieve your dreams - whatever they are, whoever you are. I think this was a great message, and Peter is obviously a good example of this in action. Focus, and the single-minded pursuit of an idea, is too often given lip-service as a reason for success, when I believe it might just be the major factor in it.

A large part of Peter's story was a dig at the international aid agencies that quickly went into Thailand and then just as quickly pulled out around a year later. One of his ideas is that charities should be forced to put some stats on every piece of marketing literature telling the reader how much money they spent on admin versus aid in the last financial year. The idea (much like the MIT Media guys) being to give people information to make informed decisions about where to give their money. I think this ignores part of the bigger picture, which is a question of how much admin is required to run an international charity that coordinates volunteers from many nations into situations that are often not as attractively packaged as the Thai one was. However that discussion can't happen without making the information available, and so I would welcome it form that point of view.

Bill Gates (Video, Curing Malaria and Making Teachers Better)

I had seen this one before, but hung around to watch it on the big screen. Bill and Melinda Gates' foundation is really trying to focus money and attention onto big issues that the market and governments are ignoring.

The first issue he covered was curing malaria, which seems both an achievable aim and one that has been largely ignored now that wealthy nations are free of it. His idea here was that if we focus on eradicating it then a relatively modest effort will see it gone, whilst if we tackle it half-heartedly we only make things worse.

The second issue was about how do you make teachers great, and he gave a long analysis of why this mattered, which seemed obvious, but I guess the hard numbers he pulled out showed people that it was an issue worth focusing on. I think his idea here was that we should care about the quality of teaching - mainly because in the US at least, the quality of a teacher is not easily discovered or rewarded.

Event

Overall the event had a good vibe, seemed interesting, and was a good way to share the watching of a TED video with people you knew. I think I missed too much of the schmoozing part of it to know whether that worked - but I was happy catching up with who I did see there.

There was a cheesy side to the event which was the blatant pushing of wares for the main sponsors, especially the silent auction of the Thought Leaders' services (with some fairly suspicious dollar figures on their market value). I understand that some of this is necessary to get the event paid for - but I do wonder if this could have been handled in a more subtle and positive way.

Thursday, May 14, 2009

Top 5 classic programmer folly projects

Back in the day when the British Empire was still new, wealthy landowners who had heard about great places elsewhere (usually ones where it didn't rain all the time) started a trend of re-creating these great places on their own land - often right down to the current decrepit state of said places.

Sometimes they didn't have a specific place in mind, but wanted a "ruined Greek temple" or similar stereotypical "interesting" structure (sometimes they were just plain strange). The point seemed to be to prove that a) you were interesting, b) you could pay for a spurious building for the sake of it, and c) foreign travel was all so unnecessary. These were known, rather aptly, as follies (presumably because they represented the folly of spending your children's inheritance on an expensive monument when investing it might have proven more prudent).

142697575_46dc01af64_b
From: Xerones 

Programmers it turns out are no less susceptible to follies, although they might not realise it, and they are not deliberate fakeries. There are several classic programmer follies which tend to get repeated by each generation, with ever-increasing zeal and lack of awareness of their futility.

Now, don't get me wrong, these are all fine examples of throwaway projects that you might play around with and practice on, but for some reason programmers can't seem to leave them stay as just throwaway projects - they always want to upgrade them to the status of "next big thing".

Without any further ado, here they are.

Top 5 classic programmer folly projects

1. Bug tracking

Every software program has bugs, and most programmers find they need a good system to track said bugs. Some companies, Atlassian and Fog Creek Software have made their living providing great, simple yet powerful solutions for exactly this problem.

However, there is always a temptation to take shortcuts, customise the bug tracking, or get a cheaper solution, and so inevitably every programmer gets tempted to build their own bug tracking application.

It's a folly because ...
This is actually a fairly complicated domain, especially once you add in the bug resolution workflow, web interface and email integration that even free, open-source applications have. Of course there is also the problem of what to do when your bugs are in your bug tracking software ... evil recursion in the real world!

2. To do list

I just want to track what I need to do ... and what I want is to improve this to do list. Remember the Milk has the best free implementation I've seen of this type of application, but BaseCamp, your mobile phone and even GMail are worth mentioning (not to mention trusty old pen and paper).

It's a folly because ...
You really don't need to waste brain cycles re-inventing something this simple, and when you look at the sophistication of Remember the Milk (web, offline Google Gears, Twitter, email and mobile integration) anything you do yourself looks kinda ... sad.

3. Website/blog engine

I want to use something I built myself, and it's a good chance to try my hand at content management ... except it isn't. If you are blogging then do your readers and yourself a favour and use an existing blogging application. You will have a more consistent user interface and will never find yourself not posting because the build is broken after that last refactor and you can't be bothered fixing it yet.

It's a folly because ...
The free alternatives are great, the cheap ones are even better and frankly it's not that great a domain for learning how to code, especially web stuff.

4. Hobby management

My hobby (roleplaying games, model robots, high performance cars, salsa dancing, etc) is out of control and I need a way of managing it. Actually no you don't, you probably just need to get out and live more.

It's a folly because ...

You don't really want to mix up your hobby and your work, it just makes one less fun and the other less professional. it is also an area where you will never really be satisfied with the result because it's your hobby. Finally, when you show it to other people they are likely to go "Ummm ... very nice. What is it?" Hugh MacLeod defines a geek as "Somebody who socializes via objects" and congratulations, that's exactly what you've just become!

5. Application Building Applications

This is the one where you build an application that uses meta-data to build an application. It seems this is the purvey of the very smart and the very dumb. The really smart guys believe they can be the ones to break through and nail this problem, the really dumb ones just don't realise how hard it is.

It's a folly because ...

It's possible to do this in theory, but in practice you only ever get around 80% of the way there, and it turns out the other 20% is the bit that an application really needs to live and breathe. Really, really smart guys manage to push that 80% up to the giddy heights of 81.63%, but hey, it still doesn't do what you really wanted it to.

Bonus! Installer software

OK, here is number six. What do you do when you want to distribute your application to clients? Use the MSI installer for Visual Studio? Heck, no! You go ahead and create your own installer software, complete with license number check, built-in unzipper and some cross-platform nastiness.

It's a folly because ...

You already know you shouldn't be doing this. Go use something someone else created and love the fact that you don't need to do it by hand!

 

OK, time for confession time, I've been guilty of each of these (at least through to the design stage). What about you, what were your follies?

Friday, February 20, 2009

Loading objects' fields/properties automagically

There is a very nice function in the new ASP.NET MVC code that allows a Controller to load an object from values passed in from a HttpRequest (UPDATE: called UpdateFrom, see ScottGu's post). It did a best guess on matching values to object properties, but most (99%?) of the time that is all you need. Well, when I started using Domain-Driven Design (DDD) and Test-Driven Development (TDD) on the latest additions to our Community Manager.NET product I wanted to find something similar - but I couldn't find anything as terse and useful.

So I did what any good dev would, I read the sourcecode and derived my own functions to do something similar, but with the SqlDataReaders that my repositories would be getting out of SQL Server (it almost feels like the ActiveRecord pattern ...). Below are the result, two LoadToObject() functions (and a PopulateTypeException class) in my BaseRepository class that give me the functionality I need. I'm putting these out there in case someone else needs this, and in the hope that any glaringly obvious errors might get picked up by others!

One issue I had with the Microsoft version of this function was they only load public properties of an object, not public fields. Now I know that in C# you can setup properties as easily as fields, but in VB it's still a hassle. I personally like using fields for simple value storing because it is clearer to see what they do in the code and leave more space for real method calls.

The final problem I ran into was that my custom Html data type needed to implement it's own TypeConverter so that the HTML in the SQL Server ntext field could be converted to a strongly typed Html type. But that can go into another blog post if people are interested.

    ''' <summary>
''' Populates an object's public fields and properties with values from a SqlDataReader.
'''
</summary>
''' <param name="readerToLoad">The SqlDataReader that needs to be loaded into the object.</param>
''' <param name="obj">The object we want to populate.</param>
''' <param name="objectPrefix">A prefix to the object field/property names. Tries to match using "." and "_".</param>
''' <returns>The object passed in with fields/properties loaded.</returns>
''' <remarks>
''' Does a best guess as to how to match names from the SqlDataReader to the object.
'''
</remarks>
Protected Function LoadToObject(ByVal readerToLoad As SqlDataReader, ByVal obj As Object, ByVal objectPrefix As String) As Object
Dim
newColl As New System.Collections.Specialized.NameValueCollection()
Dim x As Integer

If
obj Is Nothing Then
Throw New
ArgumentNullException("obj", "Object must have a value in order to be loaded to.")
End If

For
x = 0 To (readerToLoad.FieldCount - 1)
newColl.Add(readerToLoad.GetName(x), readerToLoad.GetValue(x))
Next

Return
LoadToObject(newColl, obj, objectPrefix)
End Function

''' <summary>
''' Populates an object's public fields and properties with values from a NameValueCollection.
'''
</summary>
''' <param name="valueCollectionToLoad">The NameValueCollection that needs to be loaded into the object.</param>
''' <param name="obj">The object we want to populate.</param>
''' <param name="objectPrefix">A prefix to the object field/property names. Tries to match using "." and "_".</param>
''' <returns>The object passed in with fields/properties loaded.</returns>
''' <remarks>
''' Does a best guess as to how to match names from the NameValueCollection to the object.
'''
</remarks>
Protected Function LoadToObject(ByVal valueCollectionToLoad As System.Collections.Specialized.NameValueCollection, ByVal obj As Object, ByVal objectPrefix As String) As Object
Dim
objType As Type = obj.GetType()
Dim objName As String = objType.Name
Dim exceptionList As New StringBuilder()
Dim props As PropertyInfo() = objType.GetProperties()
Dim fields As FieldInfo() = objType.GetFields()
Dim ex As PopulateTypeException = Nothing
Dim
prop As PropertyInfo
Dim field As FieldInfo

' try writing to the object's properties
For Each prop In props
'check the key, going to be forgiving here, allowing for full declaration or just propname
Dim key As String = prop.Name

If objectPrefix <> String.Empty Then
key = objectPrefix + key
End If

If
valueCollectionToLoad(key) = Nothing Then
key = objName + "." + prop.Name
End If

If
valueCollectionToLoad(key) = Nothing Then
key = objName + "_" + prop.Name
End If

If
valueCollectionToLoad(key) <> Nothing Then
Dim
conv As TypeConverter = TypeDescriptor.GetConverter(prop.PropertyType)
Dim value As Object = valueCollectionToLoad(key)

If conv.CanConvertFrom(System.Type.GetType("System.String", True, True)) Then
Try
value = conv.ConvertFrom(valueCollectionToLoad(key))
prop.SetValue(obj, value, Nothing)
Catch e As Exception
Dim message As String = prop.Name + " is not a valid " + prop.PropertyType.Name + "; " + e.Message

If ex Is Nothing Then
ex = New PopulateTypeException("Errors occurred during object binding - review the LoadExceptions property of this exception for more details")
End If

Dim
info As New PopulateTypeException.ExceptionInfo()

info.AttemptedValue = value
info
.PropertyName = prop.Name
info
.ErrorMessage = message

ex
.LoadExceptions.Add(info)
End Try
Else
Throw New
Exception(String.Format("No type converter available for type: {0}", prop.PropertyType))
End If
End If
Next

' now try writing to the object's public fields
For Each field In fields
'check the key, going to be forgiving here, allowing for full declaration or just fieldname
Dim key As String = field.Name

If objectPrefix <> String.Empty Then
key = objectPrefix + key
End If

If
valueCollectionToLoad(key) = Nothing Then
key = objName + "." + field.Name
End If

If
valueCollectionToLoad(key) = Nothing Then
key = objName + "_" + field.Name
End If

If
valueCollectionToLoad(key) <> Nothing Then
Dim
conv As TypeConverter = TypeDescriptor.GetConverter(field.FieldType)
Dim value As Object = valueCollectionToLoad(key)

If conv.CanConvertFrom(System.Type.GetType("System.String", True, True)) Then
Try
value = conv.ConvertFrom(valueCollectionToLoad(key))
field.SetValue(obj, value)
Catch e As Exception
Dim message As String = field.Name + " is not a valid " + field.FieldType.Name + "; " + e.Message

If ex Is Nothing Then
ex = New PopulateTypeException("Errors occurred during object binding - review the LoadExceptions property of this exception for more details")
End If

Dim
info As New PopulateTypeException.ExceptionInfo()

info.AttemptedValue = value
info
.PropertyName = field.Name
info
.ErrorMessage = message

ex
.LoadExceptions.Add(info)
End Try
Else
Throw New
Exception(String.Format("No type converter available for type: {0}", field.FieldType))
End If
End If
Next

If Not
(ex Is Nothing) Then
Throw
ex
Else
Return
obj
End If
End Function

''' <summary>
''' Contains exceptions that arise during the populating of an object by the LoadToObject() method.
'''
</summary>
<Global.System.Serializable()> Public Class PopulateTypeException
Inherits Exception

Public Class ExceptionInfo
Public PropertyName As String
Public
AttemptedValue As Object
Public
ErrorMessage As String
End Class

Public
LoadExceptions As New List(Of ExceptionInfo)
Private PopulateTypeException()

Public Sub New(ByVal message As String)
MyBase.new(message)
End Sub

Public Sub New
(ByVal message As String, ByVal inner As Exception)
MyBase.new(message, inner)
End Sub

Protected Sub New
(ByVal info As System.Runtime.Serialization.SerializationInfo, ByVal context As System.Runtime.Serialization.StreamingContext)
MyBase.New(info, context)
End Sub

End Class

Tuesday, February 10, 2009

Challenging software development blogs

I am not hugely keen on top 100 lists or similar trite listings of people, blogs or posts, however sometimes I notice that I have too many tabs open in Chrome (hello, 58?!) and need to itemise some of them. Many of the ones I keep open are blogs that challenge me enough that I want to read them frequently (I know ... I should be using an RSS reader, so call me old-school! ;).

I think these are some of the more challenging blogs* out there with regards to software development - read these and you will find your brain, assumptions and dealer-held beliefs under high-voltage attack.

* OK, so I am going to miss a couple of fantastic ones - what is important is that you check out the ones I do mention!

59459065_0ed25c902e_o

Joel on Software (visit)

Joel Spolsky of Fog Creek Software and Stack Overflow fame has had one of the most well-read and discussed software development blogs on the planet for years now. He is an original thinker in a market filled with duplicates, he also has a deep commitment to giving his developers great office space.

The Software Simplist (visit)

Udi Dahan's blog specialises in Service Oriented Architecture (SOA), but I read him for his refreshing view of domain-driven design (DDD). I got a new insight into DDD from his DDD & Many to Many Object Relational Mapping post.

Scott Bellware's Blog (visit)

Scott Bellware is probably the most hated/respected .NET rebel on the planet. His abrasive manner and obvious chip on his shoulder with Microsoft and the ALT.NET mainstream put some people off, but he has a deep insight into what Agile development is meant to be all about and a straightforward manner in sharing what he knows. He also seems pretty sure that most developers are learning impaired. A good example of his (more positive) thinking is Productivity: It Comes from Software Design Rather than Software Tools.

Noop.nl (visit)

Jurgen Appelo's software development management blog has some of the most pragmatic and interesting advice available for functional managers of developers. His company's situation is similar to Elcom's, so I appreciate his honest and smart insight into how to make Scrum work with large teams working on many disparate projects. He has useful posts like How to Handle Many Simultaneous Projects which apply Lean/Scrum principles to the real world of custom software development.

You'd think with all my video game experience that I'd be more prepared for this (visit)

Jason Yip from ThoughtWorks focuses his blog on lean manufacturing principles and how they apply to the reality of software development. A challenging thinker and opinionated professional, he manages to stay as radical as someone like Scott Bellware with a more pleasant demeanour. A typical (longer) post from him would be No matter how many times you say it, we still don't need a QA on the team.

Greg Young's Blog (visit)

Greg Young is a Microsoft MVP who has some very interesting opinions on domain-driven design, namely that the domain model is only there for write operations. Check out his DDDD: Master-Detail Question post for more details on this and how he uses a thin DTO layer to power his read-only UI pages.

Chad Myers' Blog (visit)

I first found out about Chad from a talk he and Jeremy Miller gave on Opinionated MVC (see videos here and here). Jeremy Miller developed StructureMap, which is a Dependency Injection/Inversion of Control framework. Chad is also the driving force behind the Virtual ALT.NET (VAN) group which has helped developers connect from places as far afield as Texas and Australia!

James Bach's Blog (visit)

Interested in software testing? Then you need to quickly head over to James Bach's blog and check out his controversial, but smart and honest posts on the topic. To understand where he is coming from, check out his Schools of Testing ... Here to Stay blog post.

So ...

Who did I miss? Leave a comment and let me know!

Thursday, February 05, 2009

The Fat Controller must die!

As most parents with little boys know the Fat Controller is a key figure in the Thomas the Tank Engine TV series (and books, toys, clothing, linen, bags, crockery, snacks, etc.). he is a cheery fellow, unfortunately prone to angrily shouting at train engines, but equally kindly and caring about his beloved vehicles (especially steam powered ones like Elizabeth the lorry). For the sake of my children I must point out that I have no problem with this jolly fellow.

fat controller

There is another domain where fat controllers exist, and that is the world of the MVC (Model-View-Controller) pattern. The pattern has experienced a renaissance recently in web applications, particularly because of Ruby on Rails, and now Microsoft's ASP.NET MVC offering.

Now these Ruby on Rails guys have been doing this for a little while longer than the ASP.NET guys, and one key paradigm became clear fairly early on:

"Try to keep your controller actions and views as slim as possible."

This was most clearly explained by Jamis Buck in his excellent Skinny Controller, Fat Model post back in 2006, which is still worth reading even if you don't use Ruby on Rails. More recently Ian Cooper has pointed out this is the same problem webforms had:

"This is the good old problem of domain-logic in code-behind that we had in ASP.NET webforms. Indeed Webforms are just another type of controller, a page controller, and switching to an application controller model does not remove the need for us to watch for domain logic creeping into the controller. There can be a temptation to believe that just because the controller is easier to test it is now safe to put domain logic in there. Do not fall into that trap."

In his book, Domain-Driven Design, Eric Evans identifies this as the Smart UI Anti-Pattern.

“Put all the business logic into the user interface. Chop the application into small functions and implement them as separate user interfaces, embedding the business rules into them. Use a relational database as a shared repository of the data. Use the most automated UI building and visual programming tools available.”

SmartUI
Image from David Hayden's blog

Eric is kind (and pragmatic) enough to point out when it is useful.

“A project needs to deliver simple functionality, dominated by data entry and display, with few business rules. Staff is not composed of advanced object modelers.”

Clearly ASP.NET's webforms model encouraged this sort of practice (it is just far too easy to do), but surely a controller is a safe place for this code? After all, a controller can use many views, so it is more clearly separated than code behind. The problem is that whilst application logic ("Which view do I show next?", "What domain object is handling this request?") makes perfect sense in a controller, there are too many examples of them being used as stores for business/domain logic - which is the province of the domain model.

Hang on, we probably have a (ubiquitous) language problem here. You see, the "Model" in MVC != the "domain model" in Domain-Driven Design (DDD). In fact we need to map the DDD concept of layered architecture to MVC in order to see what we truly have.

MvcMapToDdd

Instead of giving the Model layer a fully fleshed out domain and supporting infrastructure (repository and services), many ASP.NET developers seem to want to treat the Model as pure database infrastructure and DAL (like the one in the Smart UI graphic above). This leads to the problem of where to put the business logic, and the natural assumption is that it belongs in the controller ... which is like making the navigator the captain of the ship.

Ian and Jamis have much more (technical) stuff to say on this subject so I'd advise you to check their posts out if you're into .NET or Rails, respectively, but do take this message with you, the Fat Controller must die!