Wednesday, October 22, 2014

Evaporating Web Records

You’re hip, you’re cool – ‘business enablement’ is your middle name, and you’ve got social media accounts, blogs, forums, Atom/RSS feeds and wikis rocking and rolling. As Chief Records Officer (CRO) you help your agency move into the 21st Century full speed ahead.

Except. Things are never that easy, and what we find is web records are sadly missing.

We know that a short URL was used in this tweet, but 2 years later the URL has been re-used and no longer points where it did when we created the tweet. Worse yet, we relied on identifying records at creation and we missed one, it never got recorded and Facebook’s API is refusing to give it to us.

The wiki system was migrated to a new platform and the old edit history has been lost – worse the new system tracks comments in a different form and they have been lost too.

An ill-considered rollout of a new website neglected to ensure that all of our old URLs were migrated, and apart from losing Google ranking, we also now can’t identify what content a user might have seen on a given date for a given URL.

In other words, our web records are evaporating. It’s not your EDRMS that’s failing, it’s the fact that all of these web systems exist outside the EDRMS and compliance needs are seen as a secondary (unimportant?) requirement for replacement systems. Practical needs for delivering services now are overwhelming the old centralised compliance needs.

The “Review of Social Media and Defence” report in 2011 by George Patterson Y&R is a good example of the sorts of problems agencies face:

“Given the dynamic nature of social media communications and the collaborative approach to the creation of user generated content, Defence will need to take particular care to ensure that such content is properly identified as a Commonwealth record as and when it is created. An accurate and authentic copy of such content will need to be captured and saved as a record so as to ensure that obligations under the relevant auditing, recordkeeping and disclosure legislation can be met. This is likely to require the development of a specific Defence social media records policy that provides guidance for each particular social media channel to be used by Defence during Professional Use.”
Review of Social Media and Defence, p.102

“The simplest interpretation of international record-keeping policy is that all outgoing communication should be housed on an official website that provides both a credible source for the community and a method of archiving content. The content can then be shared easily into social media, and important or significant conversations can be selected for archiving.”
Review of Social Media and Defence, p.124

“Because the National Archives of Australia (NAA) considers social media to simply be channels in which Commonwealth records can be shared, existing record management and archiving protocols need to be followed. The challenge lies in identifying commonwealth records worthy of archiving but also in the resourcing and processes required to ensure compliance. The government’s response to the Government 2.0 Taskforce (p. 15) states explicitly that the Archives will produce guidance on what constitutes a Commonwealth record in the context of social media. The NAA should be consulted to provide greater clarification for DEOC.
Review of Social Media and Defence, p.157

Rebecca Stoks produced an academic paper in October 2012 that summarised a survey of actual recordkeeping practices for social media records amongst Australian government agencies (mostly state (33), but some local (20) and federal(9) agencies). Her summary was damning:

“The transient nature of social media opposes traditional recordkeeping methods; consequently, most government agencies are not meeting their legal obligation to keep records.”
Taming the Wild West: Capturing Public Records Created on Social Media Websites, p.8

“In this study, only a minority of government agencies were found to be capturing social media records. Most of those capturing records were not very confident that they are meeting their legal obligations or that their methods are sustainable. Within the sample, the level of internal support, be it strong or lacking, was found to affect the degree to which social media records were being captured. Although well regarded as a resource, the guidance provided by PROs did not seem to have an impact on whether or how agencies were capturing records, with several respondents expressing a desire for more practical advice.”
Taming the Wild West: Capturing Public Records Created on Social Media Websites, p.48

What do we want to know about web records when we capture them?

  • URL
  • AGLS meta-data (author, publish date/time, country, copyright, etc)
  • Re-use (trackbacks, retweets, inbound links, ratings, likes, votes)
  • Outbound links, and their status (if they redirect, then to what URL? do they have meta-tags set like NoFollow?)
  • Linked resources (images, JavaScript, iframes, Flash files, video/audio) – not always useful, but worth bringing images into content as an embedded image at very least
  • Conversations started by the record (comments, replies, threads in general)
  • Relative site-map location compared to other web records (requires the concept of a site, perhaps leverage Google site maps?)

Much of this comes from Atom/RSS feeds, but some of it requires post-capture processing.

How do we want to see web records that we capture?

  • As an HTML page, even though stored as XML.
  • As a PDF, even though originally seen as an HTML page and stored as XML.

Of course this only gets us 80% of what we need, there will always be the missing context of what the page design looked like when that content was displayed (and what other content was dynamically displayed alongside it). With social media there is also the context of an responses, retweets, likes, shares or trackbacks to consider.

Do we organise web records by the site they belong to, the Atom/RSS feed they come from, or by some other more definite measure?

I don’t know anyone that has all the answers to those questions, I’m not even sure I know that many people that care about all those questions! However, I do know that without those answers there are essential government records that are literally evaporating every minute of the day, never to be seen again, or known about. They may not be important now, they may not ever be important, but our lack of care with them is likely to be lamented by future generations seeking to understand what motivated, inspired and drove us into action (or not).

Tuesday, October 21, 2014

Collaborate 2014 Roundup

We just finished another Collaborate conference for Objective Corp's international customers and it was a great effort by all; staff, partners and customers included.

The amount of effort that goes into the event is extraordinary, but the payoff is an event that informs, excites and inspires our customers to get more value from their Objective products (ECM, ECC and Connect) and change the way their organisations impact the world.

A big part of that is helping them drive business process innovation so they can deliver better services and products to the public.

Business Process Innovation

Something we are doing across our products is looking to deliver great user experiences, especially ones that are customised for our customers' organisations so that they are received by their end users with credibility.

One of the challenges that raises is what an old colleague of mine used to call the "pink flamingo effect":

Pink Flamingos

This is the tendency of some people to go wild with any sort of HTML customisation and add what they assume is an attractive element (e.g. pink flamingos) to an otherwise functional page.

We can't prevent that altogether, but a big part of the move we're making is to educate customers about the sorts of things they should consider customising (logos, fonts, colours) and how to consider the user experience when customising things that we have carefully designed, such as our responsively designed sample email templates.


To that end the slide above was used to help explain the difference between user interfaces (UI) and user experiences (UX).

Edit: See more about UI vs UX at my colleague David Eade's blog.

Updated blog template

It's been a few years since I updated the template my blog uses, so I've taken the chance now to simplify things and apply a template that seems less busy and more modern. I've also updated the fonts to make them more readable on modern screens.

Friday, January 18, 2013

You need consumer-grade usability

It’s interesting watching the English language evolve in response to technology and changing cultural ideas. Take the term “consumer-grade”, this used to (and sometimes still does) mean, lower quality, less robust construction.

Compare it to “weapons-grade”, a term commonly applied to uranium and plutonium, but simply meaning it is a substance pure enough to to be used to make a weapon or has properties that make it suitable for weapons use. “Export-grade” is similarly used to define food or beverage that is better than average, and worthy of consumption away from its place of production. Some people seem to confuse the two.

sign-24092_640 (1)

So you might be confused by someone espousing “consumer-grade usability” as a good thing … confused that is, if you lived in 1999 and knew nothing about the history of consumer devices in the 21st century … you know, the whole iPod, iPhone, iPad, Alienware, LCD televisions, Android and “just google it” thing?

Now that we are living in an increasingly BYOD world it is obvious what the term means. It is a purity of user interface design that brings the essential features of the device or application to the fore, and frees anyone to use it with confidence and passion.

It’s easy to see how this came about, “consumers” includes everyone, “enterprise users” are a small subset of the working population, itself a subset. When creating for the consumer it has become obvious that success increasingly comes from great usability – even when competing in the budget end of the market. This means everyone gives usability lip-service, hence the rise in usability consultancies.

Consumers have access to free tools that are developed with such care that they make the average enterprise application look clunky in comparison – and they are beginning to get sick of it!!

The challenge as I see it is how do I as an enterprise product manager bring consumer-grade usability to my products, without making people see it as just a gimmick, or a band-aid fix?

Update: The Consumerization Revisited – Why Aesthetics Matter article from QlikTech points out that aesthetics matter when it comes to usability and user adoption.

Monday, November 26, 2012

Windows Live Writer ate my blogger login

Well thank you Microsoft.

The latest release of Windows Live Writer demands that I configure a blog when I start it up - fair enough, it does want to know my blog styles in order to show me formatted posts, so I go to login and ...BAM! Login fails.

Oops! Retry. Login fails.

Expand to full email rather than just username. Retry. Login fails.

Try every combo possible. Login fails, fails, fails ...

I finally worked out that because I use 2-step verification with Google authentication, and this app doesn't have an app-specific password, it will always fail. Now this is a failing of the Windows Live Writer devs, but I wonder how many other dev teams would think of this?

My solution? Fallback to online editing. Nowhere nears as nice as Windows Live Writer, nor as available, but it works*.

Although the damn thing did put in DIVS in place of P tags and I instead of EM tags ... sigh.

EDIT: Matt was right, using an app-specific password solved my problem – apologies Microsoft devs, not your problem (well, except for perhaps not providing nice help around the problem).

Friday, September 14, 2012

Why is social media so ... anti-social?

Perhaps this is a stupidly obvious question, but why is so much social media so anti-social? Let me expand that a bit before you deride my lack of Facebook ‘friends’, Twitter followers or Klout score (currently a healthy 46 thank you very much).

Sure, you can be social with your friends on these sites, and off-them thanks to mobile apps, but there are some key issues that I at least run into with them:

  1. If we don’t share a network then my friends don’t see my updates, share my life or catch my drift.
  2. Twitter and Facebook (at least), seem hell-bound on making third-party developers lives hard, they depend on you ending up on their apps, on their site to make advertising money.
  3. If I like the features of a new network better, I need to bring across my friendships or contacts, and re-apply my privacy settings and do a bunch of other work before I can use it fully.

Recently people have been comparing social platforms to early email clients and networks. Albert Wenger (a VC, ex-President of del.icio.us) wrote:

It would (be) a huge benefit to society if we can get with social networking to where we are with email today: it is fundamentally decentralized with nobody controlling who can email whom about what, anyone can use email essentially for free, there are opensource and commercial implementations available and third parties are offering value added services.

Thomas Baekdal (social media commentator) writes about the future of social media and makes the same comparison to email:

In the early 1980s, email worked pretty much as social services do today. Each email provider used their own proprietary protocols and systems, and each system was unable to communicate with any other.

So email didn't take off because the process was simply too complicated. It wasn't until every email provider finally decided on an open, non-proprietary format, that email started to work and became the massively popular communication mechanism that we know today.

The question is then, what is the next step? What is the future of social media? And the answer is painfully obvious because we have already seen it happen with email. The future is when social becomes a protocol.

More than just anti-social, the current networks lock up our content as Scott Hanselman (Microsoft employee and prolific blogger) recently pointed out:

You are not blogging enough. You are pouring your words into increasingly closed and often walled gardens. You are giving control - and sometimes ownership - of your content to social media companies that will SURELY fail. These companies are profoundly overvalued, don't care about permalinks, don't make your content portable, and have terms of service that are so complex and obtuse that there are entire websites dedicate to explaining them.

Scott is more railing against social media than suggesting an alternative (FWIW, I think RSS is great, but it provably does not solve all the issues here). I do think he’s right about not stopping blogging, and I said it myself back in 2009 too.

So if the future of social is to be a protocol, how will this happen? Google tried something like this with Google Wave, which morphed into Google+, but nobody trusts them anymore it seems, so it kinda died before it could be really interesting.

How could this happen with social media?

  • Facebook could die, and then people pick up the pieces with smaller shared networks. But Zuckerberg has nailed that one with the successful IPO.
  • Twitter could die, and then on the way out offer its data and API to the world for free. More likely than Facebook going, but again not hugely likely.
  • A dominant OS, browser or device vendor could introduce the ability to use such a protocol without caring about controlling it. Microsoft could give this a shot with WIndows 8, but that will have its own issues to face before then. Google+ as a service is trying to be this, and might just be there if they bake it into Android.
  • Amazon could surprise everyone by creating a platform for it and being open about it. It would probably generate huge storage and bandwidth revenue for them.
  • Yahoo! could be a surprise player in this area, Marissa Mayer needs to give them something to aim for, and they have form in creating useful technical solutions for others (Yahoo! Pipes, Yahoo! Developer Network, YUI Library).

Baekdal is the most bullish that social must become a protocol, but he doesn’t offer us a way forward, other than pointing to the need for social to be an activity we do, not a place we go.

I think there might be a surprising way forward, but that can wait for another post.

Friday, May 18, 2012

AusCloud Forum May 2012

I had the pleasure this week of being OrionVM’s guest on the panel for this month’s AusCloud Forum (see tweets).

The panel I was on followed the three keynote speakers:
  1. David Yuile, CEO of AAPT on Powering the Cloud
  2. David Werdiger, CEO of Billing Brureau on SmartBilling
  3. Sheng Yeo, CEO of OrionVM on Carrier Clouds
The speakers were interesting in the most part, I think David had us the most entertained, but unfortunately he was in the dark for most of it (and really needs to read Beyond Bulletpoints as his first slide had tiny icons I couldn’t even read from the 5th row.
The panel was about “Enterprise in the Cloud” and put me in some very flattering company:
We had several great questions to answer, and some interesting variances in opinion. I wasn’t tracking well enough to keep everything in order, but some of the high points from my point of view were:

Cloud Has Arrived

It is clear that enterprises are switching to use the cloud, at least on new/internal projects. This was backed up at yesterday’s AWS Summit where News Ltd and Realestate.com.au both shared they have moved their development/test environments over to Amazon’s cloud. This is what we’re doing at Elcom. Our release testing now occurs on servers at OrionVM that get spinned up when we need them and archived  in between.

Hardware Must Be Sweated

Both Roger Lawrence and David Yuile were emphatic that enterprises have significant hardware (and skill) investments that need to be sweated before they will feel free to move to the cloud. Sometimes these can be re-tasked for other duties, but all too often they can’t be (few banks are moving their mainframe code just yet (although MicroFocus and Heirloom Computing both offer COBOL in the cloud).

Give Me Standards

David Yuile was heavy on the need for cloud standards, and he thinks that larger players should put some money up to ensure standards get worked on properly. He likened the current trepidation about cloud to the same feelings enterprises had about VLAN – and in a similar fashion he thinks some standards will help it go mainstream.
Roger pointed out that there’s a difference between standards and certifications, with the later being what most enterprises care about. I agree to a point, as a small ISV we can’t always afford to get all the certifications we’d like to, and being able to assure customers we can adhere to a standard is often all we need to do.

CIOs See New Savings, CEOs See New Possibilities

Roger had a very interesting point to make about how different occupants of the C-suite tend to approach cloud. CIOs tend to see the cost savings and efficiencies to be gained, whilst CEOs come from the point of view of what new possibilities the cloud offers their business. A case in point was that of a large construction company where IT costs were around 1% of revenue – any saving being negligible – but being able to reduce the wait between delivery of goods to clients and payment of invoices from 27 to 26 days was worth $34M in extra cashflow.

We Ain’t Seen Nothing Yet

I think we have barely seen what the real effect of the cloud will be in the enterprise arena. Most technology shifts are not really embraced until the generation that grows up with it gets the chance to flex their muscles in the workforce and for the cloud that is still 5 years or so away. When you have grown up with streaming media as the norm, mobile access the primary means of internet access, true ubiquitous computing and commoditised cloud services how differently will you regard the problems enterprises (and countries) face today?

Cloud Helps Startups, But Not That Much

There is some lurking optimism that if we just commoditise the cloud and computing enough then startups will suddenly multiply and change the world. Unfortunately for that rather utopian idea the lowering of the barriers to entry helps everyone, big enterprise innovator and  garage startup alike – it also means more competition and noise to rise above.
An example comes from my wife’s cousin who has worked in the Australia games industry for over 15 years. The industry is currently deep in recession, but it looks like it’s healthy because so many developers are trying their hands at mobile games. The problem is that few (if any) of them are making any money (Halfbrick being a notable exception).

Microsoft’s Future (bonus)

Not particularly cloud related, but a subject dear to Roger and my heart and something that came out of one of the questions asked. We both have strong opinions on this, but what it boiled down to was the success of Windows 8 will define how much affect Microsoft has on the future.
Roger pointed out that Microsoft might lose it’s way, but like IBM, still remain a giant in the IT space. He also noted that Microsoft’s partners, as heavily invested as they are in its success, are a mighty (not so) secret weapon when it comes to their success in the business world.
Personally I think their Windows 8 Metro UI is inspired, and will probably be a great hit on tablets and mobile devices … it’s penetration into the business world will be limited by old non-touch LCD monitors (see “Hardware Must Be Sweated” above). However, Microsoft will both find ways to maximise the switch, and will care less than it used to if it can wrest majority control of the mobile/tablet space from Apple/Android.