Evaporating Web Records

You’re hip, you’re cool – ‘business enablement’ is your middle name, and you’ve got social media accounts, blogs, forums, Atom/RSS feeds and wikis rocking and rolling. As Chief Records Officer (CRO) you help your agency move into the 21st Century full speed ahead.

Except. Things are never that easy, and what we find is web records are sadly missing.

We know that a short URL was used in this tweet, but 2 years later the URL has been re-used and no longer points where it did when we created the tweet. Worse yet, we relied on identifying records at creation and we missed one, it never got recorded and Facebook’s API is refusing to give it to us.

The wiki system was migrated to a new platform and the old edit history has been lost – worse the new system tracks comments in a different form and they have been lost too.

An ill-considered rollout of a new website neglected to ensure that all of our old URLs were migrated, and apart from losing Google ranking, we also now can’t identify what content a user might have seen on a given date for a given URL.

In other words, our web records are evaporating. It’s not your EDRMS that’s failing, it’s the fact that all of these web systems exist outside the EDRMS and compliance needs are seen as a secondary (unimportant?) requirement for replacement systems. Practical needs for delivering services now are overwhelming the old centralised compliance needs.

The “Review of Social Media and Defence” report in 2011 by George Patterson Y&R is a good example of the sorts of problems agencies face:

“Given the dynamic nature of social media communications and the collaborative approach to the creation of user generated content, Defence will need to take particular care to ensure that such content is properly identified as a Commonwealth record as and when it is created. An accurate and authentic copy of such content will need to be captured and saved as a record so as to ensure that obligations under the relevant auditing, recordkeeping and disclosure legislation can be met. This is likely to require the development of a specific Defence social media records policy that provides guidance for each particular social media channel to be used by Defence during Professional Use.”
Review of Social Media and Defence, p.102

“The simplest interpretation of international record-keeping policy is that all outgoing communication should be housed on an official website that provides both a credible source for the community and a method of archiving content. The content can then be shared easily into social media, and important or significant conversations can be selected for archiving.”
Review of Social Media and Defence, p.124

“Because the National Archives of Australia (NAA) considers social media to simply be channels in which Commonwealth records can be shared, existing record management and archiving protocols need to be followed. The challenge lies in identifying commonwealth records worthy of archiving but also in the resourcing and processes required to ensure compliance. The government’s response to the Government 2.0 Taskforce (p. 15) states explicitly that the Archives will produce guidance on what constitutes a Commonwealth record in the context of social media. The NAA should be consulted to provide greater clarification for DEOC.”
Review of Social Media and Defence, p.157

Rebecca Stoks produced an academic paper in October 2012 that summarised a survey of actual recordkeeping practices for social media records amongst Australian government agencies (mostly state (33), but some local (20) and federal(9) agencies). Her summary was damning:

“The transient nature of social media opposes traditional recordkeeping methods; consequently, most government agencies are not meeting their legal obligation to keep records.”
Taming the Wild West: Capturing Public Records Created on Social Media Websites, p.8

“In this study, only a minority of government agencies were found to be capturing social media records. Most of those capturing records were not very confident that they are meeting their legal obligations or that their methods are sustainable. Within the sample, the level of internal support, be it strong or lacking, was found to affect the degree to which social media records were being captured. Although well regarded as a resource, the guidance provided by PROs did not seem to have an impact on whether or how agencies were capturing records, with several respondents expressing a desire for more practical advice.”
Taming the Wild West: Capturing Public Records Created on Social Media Websites, p.48

What do we want to know about web records when we capture them?

• URL
• AGLS meta-data (author, publish date/time, country, copyright, etc)
• Re-use (trackbacks, retweets, inbound links, ratings, likes, votes)
• Outbound links, and their status (if they redirect, then to what URL? do they have meta-tags set like NoFollow?)
• Linked resources (images, JavaScript, iframes, Flash files, video/audio) – not always useful, but worth bringing images into content as an embedded image at very least
• Conversations started by the record (comments, replies, threads in general)
• Relative site-map location compared to other web records (requires the concept of a site, perhaps leverage Google site maps?)

Much of this comes from Atom/RSS feeds, but some of it requires post-capture processing.

How do we want to see web records that we capture?

• As an HTML page, even though stored as XML.
• As a PDF, even though originally seen as an HTML page and stored as XML.

Of course this only gets us 80% of what we need, there will always be the missing context of what the page design looked like when that content was displayed (and what other content was dynamically displayed alongside it). With social media there is also the context of an responses, retweets, likes, shares or trackbacks to consider.

Do we organise web records by the site they belong to, the Atom/RSS feed they come from, or by some other more definite measure?

I don’t know anyone that has all the answers to those questions, I’m not even sure I know that many people that care about all those questions! However, I do know that without those answers there are essential government records that are literally evaporating every minute of the day, never to be seen again, or known about. They may not be important now, they may not ever be important, but our lack of care with them is likely to be lamented by future generations seeking to understand what motivated, inspired and drove us into action (or not).

KEVIN07 - update

Thanks again to everyone who commented on my original KEVIN07 post. The newsfeeds are full of interesting points of view about the site:

For a start I think it's un-Australian to shorten Kevin Rudd's name to just Kevin. No, it needs to become the truly iconic Kev for him to have a real chance to win the election in 2007.

Anyway, back to the commentary:

Craig Sherborne at the Herald Sun points out the American elements in the site:

“Lashings of American-style red, white and blue, which are also Labor Party colours, create a bold backdrop for video links and a quick, simple manifesto for hot topics from job security to climate change.”

The Herald Sun also wonders where the KEVIN07 t-shirts are made:

“Kevin Rudd's "prototype" campaign T-shirts may have been made in the sweatshops of Asia.

A "Kevin07" T-shirt obtained by the Herald Sun had its label cut out.

A Labor spokesman could not rule out that it had been made overseas. ”

NineMSN have attracted hundreds of user comments about the KEVIN07 website. Most pointed was this one:

“Hypocrisy best describes this lastest stunt. Kevin Rudd is my local member and before he became opposition leader I sent an email for him to address some issues on my behalf...........I am still waiting for a reply!!!!!”

Finally, Lachlan Connor (independent for Victoria) gave us the following hacked image of Kev and his crew:


I'm looking forward to more good parody images coming out over the next week or so ... mind you being parodied and made fun of never did John Howard any harm, so it may all come good for Kev in the end.

KEVIN07 - are Labor out of their minds?

We've hit a new low in Australian political life with the launch of KEVIN07, the campaign website for Kevin Rudd - oh, and the Labor party. It is full of the mock enthusiasm so prevalent in American politics, and the similarities are made more obvious by the use of red, white and blue, the presence of bumper stickers, and the ability to buy KEVIN07 gear ...


In a blatant acknowledgement that they must fight the Liberals on personality alone, the site launch represents the rock-bottom for Australian politics - that is, until we have the Rudd girl battling the Howard girl online (like the yanks do with Obama girl vs Hilary girl). Mind you Kev looks like he's just swallowed a prune in this photo, so he probably needs some cheering up.


The buzz around the office is that this is likely to backfire. It is all too easy to parody and rip something like this off, not to mention the average Aussie's instinctive dislike of anyone so blatantly espousing American qualities and values.


We might like working with and cooperating with yanks, but that doesn't mean we want our politicians to become them!

The buzz in the blogosphere is that Labor has made a bold play, but might just have shot themselves in the foot with this one. I think the idea, using the web effectively, is a smart one - but the out and out cheesiness of the site backfires badly.

Nuclear power, economically and enviromentally good

It seems my remarks about local councils seeing the light re nuclear power have been noticed on the NEI Nuclear Notes blog. Their post pointed out a long list of papers looking at the economic benefits of nuclear power stations to the local communities.


I also recently came across a documentary following on from An Inconvenient Truth, called The Nuclear Option.

“The film looks at the potential of nuclear energy to provide sustainable, non-greenhouse gas emitting energy for our society in the future. In-depth look at Fission and How Nuclear Power Works, the Three Mile Island and Davis-Besse Accidents, Nuclear Safety, Waste Transportation, Yucca Mountain, Recycling/Reprocessing Nuclear Fuel, Breeder Reactors, Sustainability, and the Potential of Fusion Energy.”

I truly believe that the time is right for the world to take a closer look at nuclear power generation. It is not an option without issues (but no mass power generating solution is) but it has a low environmental impact and in several countries, especially France, it has proven a boon to their economy.

PBS' Frontline looks at the French experience of nuclear power:

“Civaux in southwestern France is a stereotypical rural French village with a square, a church and a small school. On a typical day, Monsieur Rambault, the baker, is up before dawn turning out baguettes and croissants. Shortly after, teacher Rene Barc opens the small school. There is a blacksmith, a hairdresser, a post office, a general store and a couple of bars. But overlooking the picturesque hamlet are two giant cooling towers from a nuclear plant, still under construction, a half-mile away. When the Civaux nuclear power plant comes on line sometime in the next 12 months, France will have 56 working nuclear plants, generating 76% of her electricity.

In France, unlike in America, nuclear energy is accepted, even popular. Everybody I spoke to in Civaux loves the fact their region was chosen. The nuclear plant has brought jobs and prosperity to the area. Nobody I spoke to, nobody, expressed any fear. From the village school teacher, Rene Barc, to the patron of the Cafe de Sport bar, Valerie Turbeau, any traces of doubt they might have had have faded as they have come to know plant workers, visited the reactor site and thought about the benefits of being part of France's nuclear energy effort.”

What about nuclear waste? Let's see what the IAEA says:

“The spent fuel that comes out of a nuclear power plant is highly radioactive. Although its volume is small - all the spent fuel produced annually by the world´s 442 nuclear power plants would cover a space the size of a soccer field to a depth of 1.5 meters - it must be securely contained for tens of thousands of years. Today´s spent fuel is stored mainly on-site at the power plant where it was produced.

For the long term, the scientific and technical communities generally agree that high-level waste and spent fuel can be disposed of safely by deep geological burial in suitable hard rock, salt or clay formations, using both natural and engineered barriers to isolate the waste. Finland, Sweden and the US have made the most progress.
...
There is also renewed interest in the possibility of international repositories, both because of limited domestic options for waste disposal and because of new proposals to strengthen the global non-proliferation regime through international control of important parts of the nuclear fuel cycle, like uranium enrichment and spent fuel management. The IAEA is actively pursuing this issue in connection with a study of possible multilateral oversight of the nuclear fuel cycle.”

Back with the naysayers, The Age claims that John Howard's pro-nuclear push in simply 'playing politics' and Rudd's geosequestration option is similarly bogus as neither would generate results in the 10-15 year timeframe that economist Nicholas Stern warned was critical to “avoid catastrophic climate change”. They do admit that nuclear is the better option:

“Compared with clean coal, nuclear power could be a better bet. The cost and risks of storing carbon are likely to be far higher than storing nuclear waste. Nuclear power plant designs could be bought off the shelf, provided a suitable regulatory regime was in place.”

Perhaps our State and Local governments could see their way clear to encouraging nuclear power for a change?

The Honest Broker

Roger Pielke Jr runs the Prometheus science policy blog and is releasing an interesting sounding new book called The Honest Broker: Making Sense of Science in Policy and Politics.

“For scientists seeking to play a positive role in policy and politics and contribute to the sustainability of the scientific enterprise, scientists have choices in what role they play. This book is about understanding this choice. Rather than prescribing what course of action each scientist ought to take, the book aims to identify a range of options. Using examples from a range of scientific controversies, The Honest Broker challenges us all - scientists, politicians and citizens - to think carefully about how best science can contribute to policy-making and a healthy democracy.”

Regardless of where you might sit in the climate change debate, I think it is worthwhile taking the time to think carefully about the role that scientists should play in the formation of public policy.

This will definitely be going on my wishlist at Amazon!

Climate change update

The classy science policy guys at the Prometheus blog ask what does "policy neutral" mean to the IPCC? They've got an interesting point of view on the slant that the IPCC's latest report will take, especially as the IPCC claims to be policy neutral:

“The past reaction to my comments on political advocacy by IPCC leadership has been mixed. Some who share the IPCC's advocated agenda see no problem in the IPCC leadership engaging in such advocacy. Who wouldn’t want such a group perceived as authoritative and legitimate on their side? (Similarly, I am sure neo-cons would welcome a CIA Director advocating action on Iran!) By contrast some opposed to the advocated agenda have seized upon the obvious inconsistency in the IPCC’s views on "neutrality" to try to impinge the credibility of the organization. From my perspective, while both of these perspectives are to be expected (and I am sure will make their views known in response), there is a third view that matters most -- and that is the question of the appropriate role of organized expertise in decision making, whether it is the CIA or IPCC. This last view is quite independent of (or it should be) what one thinks about the issues of climate policy.”

Margaret Wente at the Globe and Mail has written a great article that condenses much of the current climate change debate into a manageable summary; Climate change a 'questionable truth' is fantastically quotable due to the number of respected scientists she talks to, and the depth of their responses about climate change issues. Here are a few of the more interesting ones (emphasis mine):

“The very great uncertainty of long-term climate impacts is a point that often gets lost in the debate. The scenarios range from mild to severe, but it's the extreme ones that get the ink.”

and

“The climate debate focuses almost entirely on mitigation (how we can slow down global warming). But climate scientists and policy experts say that in the short term — our lifetimes — our most important insurance policy is adaptation. Nothing we do to cut emissions will reduce the risk from hurricanes or rising seas in the short term. But there are other ways to reduce the risk. We can build storm-surge defences, stop building in coastal areas and make sure we protect our fresh-water supplies from salination. We also can develop crops that will do well in hotter climates.

“Adaptation” is not a word that figures much in climate-change debates. Activists (and much of the general public) think it sounds lazy and defeatist. But the experts talk about adaptation all the time.

“Climate-change policy requires that both of these issues — adaptation and carbon reduction — be addressed simultaneously,” Yale's Prof. Mendelsohn says.”

Environmental campaigner Tim Flannery (and 2007 Australian of the Year) laments that scientists need more freedom to research Australian-specific climate change factors and to spend less time helping make industries such as coal 'clean'. The reality is that both are required, but he does bring up some interesting research:

“Over the past 50 years, forest clearing and industrialisation in Asia have produced a haze of tiny particles called aerosols that hang perpetually over the south Asian landmass. This haze cools the region, and in doing so alters the temperature and air pressure gradients across the Indian Ocean.

It is most likely these changes that have increased the tendency of monsoonal winds, and the rain they bring, to flow across Australia. So important is their impact that without these aerosols, some models show that global warming would cause northern Australia to become even drier and sunnier than it was in 1950.

Studies such as these underline the need for government policy to be underpinned by fundamental science. Unfortunately, this is not what we have seen in Australia over the past decade. Instead, scientists have been told to become more relevant to industry, and increasingly organisations such as the CSIRO have been reorganised to work with partners such as the coal industry. This has left us in a dangerous position, flying partially blind into a rapidly changing future.

Australians need to be sure we are making the right decisions on climate change. Aerosols last only weeks or days in the atmosphere, and if Asia cleans up its industry and air (there are signs that this is already happening), northern Australia's rainfall may decline precipitously.”

Great. So now we don't want our neighbours to stop polluting their air ... clearly whether the world addresses climate change (or not) adaptation will be required.

Neville Nichols from Monash University, an IPCC contributor, is brought in on Australia's current dry spell (emphasis added):

“Nicholls says while the consensus of most climate scientists is that it is possible that man-made climate change may be at least contributing to the present dry period, it is certainly not its root cause. He sees a real danger in the public being swept up in the belief that the drought has been caused by climate change.

"I think there is this feeling in the community linking this drought to climate change, and the great worry with that is that when it does rain they will think all that stuff about climate change is rubbish," he says. "We are experiencing this really intense one-year drought which probably has almost nothing to do with climate change, it just seems like it's a natural thing, and there is the possibility that the longer-term drought that we've had might just be natural variability, but the problem is (that) as far as anything in science is certain, we're definitely getting warmer temperatures because of the enhanced greenhouse effect."

Nicholls's dollar-each-way observations are typical of those closest to the debate and most qualified to comment.

They have left a vacuum in the public debate that has been filled by commentators less constrained by the science.

...

High-profile environmental commentators such as Tim Flannery have done plenty to fuel this populist thinking. His engaging blend of genuine science with hand-picked factoids and a dose of fear has long since blamed the drought on climate change, warning of rising food prices and referring to the erroneous claim at the water summit last year by South Australian Premier Mike Rann that this is a one-in-1000-year drought.”

The Australia Institute have presented a report about possible nuclear reactor sites, which they point out is “one of the most politically contentious aspects of the nuclear debate.” Unsurprisingly, given their left-wing position, they chose to focus on this aspect in their report, emphasising people's fears:

“Report author Andrew Macintosh said the fact that nuclear energy attracted moderate levels of support at a general level but fierce opposition from local communities when concrete proposals were put forward suggested the presence of the not-in-my-backyard (NIMBY) phenomenon.

...

Greens senator Kerry Nettle said the report unsurprisingly showed that populations close to the suggested sites did not want nuclear power plants.

"Instead of talking about 25 possible nuclear power plants, the prime minister should be looking for another 25 sites for major wind power stations and another 25 solar power stations," she said.”

Unfortunately we are unlikely to get anyone to want wind stations or solar power sites next door to them either ... so why does this study invalidate the pursuit of nuclear energy as a viable alternate energy source? I wonder how long it takes before some enterprising local council works out that the impact on jobs, industry and local living conditions of a nuclear power plant far outweighs the potential and theoretical downsides? All I know is that if I move up to the Central Coast I would far rather have the local coal power plant shutdown and a smaller footprint (physical and environmental) nuclear plant take its place.

The NSW State Government has asked the CSIRO to reflect on the potential impacts to Sydney of global warming, and it appears that devastation is expected unless we adapt:

“The CSIRO report warned that the city must work out how to adapt quickly, with the impacts of human-caused global warming now apparently inevitable.”

It's a shame that Narrabeen is one of the suburbs they say will be most damaged - but in reality much of it is built on sandbar anyway, and in 'normal' years you can expect big storms to wash over much of it.

In summary, climate change alarmists are getting shriller, Australian State and Federal governments are still using 'independent' reports to fight their political battles, and real policy approaches to adapt to and minimise the effects of climate change are still nowhere in sight. The next Federal and NSW State elections will be very interesting - if things continue as they are we can expect more gradual policy changes, if either government changes then we might see a much faster move in either direction.

Words fail me ... but the VIC govt failed this little girl

Just venting a little over over at Cracked Jug.

Citizen vs Tribesman

America's ABC has an interesting article on Why We Will Never See Democracy in the Middle East. Steven Pressfield says the key is the longstanding conflict between the mindset of 'citizen' and 'tribesman'.

“What exactly is the tribal mind-set? It derives from that most ancient of social organizations, whose virtues are obedience, fidelity, warrior pride, respect for ancestors, hostility to outsiders and willingness to lay down one's life for the cause/faith/group. The tribe's ideal leader is closer to Tony Soprano than to FDR and its social mores are more like those of Geronimo's Apaches than the city council of Scarsdale or Shepherd's Bush.”

He makes a particular point about how the citizen takes part in war, but the tribesman takes revenge:

“The tribesman does not operate by a body of civil law but by a code of honor. If he receives a wrong, he does not seek redress. He wants revenge. The taking of revenge is a virtue in tribal eyes, called badal in the Pathan code of nangwali. A man who does not take revenge is not a man. Al Qaeda, Hezbollah, and the sectarian militias of Iraq are not in the war business, they are in the revenge business. The revenge-seeker cannot be negotiated with because his intent is bound up with honor. It is an absolute.”

If true, then that places the War on Terrorism not within the religious grounds of Christianity vs Islam, nor within the political arena of Democracy vs Theocracy, but on the more philosophical grounds of State vs Tribe - or from the terrorists' view, of personal revenge.

I think that there is clear example of citizens winning this war once before ... Australia's ABC aired an interesting documentary last night called Carthage: The Roman Holocaust that looked at how Carthage was copied, and eventually surpassed, by Rome. The point made was that Rome took the advanced culture, agriculture and trading routes of the Carthaginians and added an extra element - the cult of the Republic. Suddenly people who were once Roman enemies, came to see themselves as Roman citizens, and part of what their ancestors once fought. Historian Richard Miles points to the baths, the theatre and the temple as the tools Rome used to bind newly conquered peoples to the idea of citizenship.

Carthaginian Veterans
Is there a lesson in this for our conflict with the tribesmen of the Middle East, or closer to home, the Solomon Islands?

via Chaos Manor's RSS feed

The Iron Law of Bureaucracy

From sci-fi writer Jerry Pournelle's Chaos Manor blog:

“Pournelle's Iron Law of Bureaucracy is that in any bureaucracy, the people devoted to the benefit of the bureaucracy itself always get in control, so that those dedicated to the goals the bureaucracy is supposed to accomplish have less and less influence, and sometimes are eliminated entirely.”