Future Proof – Protecting our digital future

ICA Congress photo

At the end of August Janet and I, as well as several other colleagues, attended the excellent International Council on Archives (ICA) 2012 Congress in Brisbane. We thoroughly enjoyed ourselves, chatting with and learning from 1000 other archivists from 95 countries around the world. (Drink. *)

The ICA Congress only runs once every four years and so it is a very significant event in the international recordkeeping calendar. As a result it is chock-a-block with content. Seven parallel conference sessions ran every half an hour for nearly the full three days of the conference, as speakers reported on recordkeeping developments from across the globe.

Full papers of virtually all presentations are available via the Congress website at but here I am going to report on some of the conference sessions I attended that focussed on metadata in one tangential way or another.

As one conference delegate put it, it all comes down to metadata and, to me at least, the significance of metadata to all aspects of the recordkeeping endeavour was a really interesting theme that emerged through the conference.

Considerations for enabling metadata reuse

In one of the first presentations at the Congress, Helen Morgan, Dr Joanne Evans and Ailie Smith spoke about the need to build resilient metadata. They discussed their research which examined online descriptive metadata. They particularly focussed on metadata that had originally been created as a paper-based index to archival collections but which had then been harvested as online description for archival resources.

The key lessons to be learned from this research are as applicable to current recordkeeping environments as they are to archival ones. Their research shows that data reuse exposes you to the weaknesses in underlying data models and to the challenges of harmonising different data models. Morgan, Evans and Smith recommend that when developing metadata, we need to build resilient metadata models so that data can be easily harvested for multiple purposes and does not have to be individually recrafted each time to support reuse.

They provided some useful examples of how underlying data models can affect the reuseability of metadata. For example, in one systems they were assessing, the researchers noted that only 21% of database entries for record collections had titles in the title field. This is because when the system was designed, the Unique ID field was sufficient to identify collections. The database was also constructed to automatically present connections between related entities and collections and this all gave sufficient context so that a stand-alone title for each database entry was not seen as necessary.

When you remove the metadata from this data model, however, and try to use it in another context, suddenly the lack of a title means that all the records cannot actually be searched for. Their unique identifiers still give some means of distinguishing them but they hold absolutely no meaning for researchers trying to search for and identify relevant records.

So the key lesson to take from this experience is, when you are developing systems and the metadata within them, think long term. Think strategic, think reuse, think, ‘What else do I need to do to sustain and use this data through time? and ‘How can I best enable this data for reuse in the future?’ These conclusions are really important for anyone developing metadata structures and schema to consider. The full text for this presentation is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00185.pdf

How to identify high value public sector information

The presentation by Trish O’Kane, Selena Smeaton and Lisa Austin was not specifically about metadata but the ideas presented could be seen as providing a model for helping to prioritise metadata and other recordkeeping efforts, and ensure they are being deployed where they are most needed.

Developed in conjunction with Archives New Zealand, the research presented was based on an investigation of the business functions performed by NZ government agencies. The objective of the research was to develop a methodology for identifying the government functions with the highest ‘value’, and through this identify the digital information that supports these functions.  The researchers argued that it is this information that should be the priority for active management, protection and preservation as archives, and they anticipate that the results of the research will be used by Archives New Zealand to prioritise their transfer and disposal activities.

While functional assessments have long been the basis for recordkeeping best practice and have been the core of State Records’s appraisal policies for many years, this research was really interesting as it provided a model to perform functional assessments at a whole-of-government level. It also allows functions to be mapped against many different community, business, risk and research-based value characteristics and for these values to be updated as community or government requirements change. The abstract for this paper is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Abstract00275.pdf

We really need to build strategic alliances

The presentation by Ineke Deserno, Recordkeeping: The evidence base for corporate social responsibility again did not refer directly to metadata, but it too has relevance for how recordkeeping requirements, including metadata requirements, are perceived and how they are able to be implemented across organisations.

Ineke’s research has involved assessing the practices supporting a specific form of corporate reporting known as sustainability reporting in major multinational corporations. Ineke has interviewed regulators, consultants, auditors, records managers and others to try to understand how records and records management contributes to this important reporting process.

Unfortunately, so far Ineke’s research is showing that corporate recordkeeping frameworks are not recognised as relevant business systems that should support this whole-of-organisation reporting requirement. Indeed, her research is showing that the recordkeeping frameworks that operate in each of these companies are generally unknown to many other professional groups in the organisation. One auditor she spoke to said, ‘Records management – what is that?’. Another when asked about their corporate records manager had no idea of who this person was and said, ‘A lot of companies would not have such a person, would they?’.

While it is not new to most of us, Ineke’s presentation was a sobering reminder of how much the recordkeeping profession still needs to do in order to connect with allied business and information professionals. Ineke has reviewed the requirements for sustainability reporting and they contain many recordkeeping requirements, but they make no reference to recordkeeping specifically, nor to recordkeeping standards.

Ineke’s recommendations are to link recordkeeping more specifically to reporting and audit requirements. She also recommends integrating recordkeeping requirements directly into relevant business standards. She thinks recordkeeping professionals need to proactively reach out to business units and help them identify recordkeeping requirements in their processes and determine how existing corporate recordkeeping frameworks can help to support their operations. She recommends performing a risk analysis of corporate business functions in order to focus on those areas that present the highest levels of corporate risk.

So, while not related to metadata, Ineke’s presentation helps to demonstrate why recordkeeping metadata requirements are often left out of business system redevelopment plans, or left out of system requirement statements. As recordkeepers we need to get more directly involved with business, with the auditors who measure internal business processes and systems, and demonstrate the value of recordkeeping strategies to key corporate operations. A few more key things for the to-do list everyone! Ineke’s really interesting presentation is available at http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12final00197.pdf

Building sustainable metadata

Quite a few great presentations focussed on the challenges of metadata reuse. Kuldar Aas of the National Archives of Estonia spoke about the challenges of ingesting metadata from EDRMS applications into archival systems and using this contemporary metadata as the basis for archival description. Kuldar has been working on automating the reuse of metadata generated in EDRMS environments for archival description purposes and has practical experiences of the challenges involved. He reports that a lack of metadata standardisation is a challenge to metadata reuse, as is the diversity of EDRMS used across the Estonian government, each of which require an independent technical approach to the export and ingest of metadata.

Kuldar’s experience in arranging for the transfer of digital records into archival systems has lead him to conclude believe that we need to move away from prescriptive metadata schemas and away from manual mapping processes a towards more flexible metadata approaches that be added on top of existing business generated metadata, rather than replacing existing metadata. Kuldar’s full paper is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00221.pdf

Managing records through administrative change

Fiona Sim gave an excellent presentation on the challenges of being a records manager charged with the responsibility of amalgamating and de-amalgamating corporate information systems in the wake of administrative change.

Fiona’s fascinating talk had a lot to say on the subject of metadata. In the wake of their corporate restructuring, Fiona and her staff were responsible for moving more than 4 million records into corporate EDRMS and this process was completely reliant on good metadata. Fiona said that the key challenges when managing records through administrative change are to maintain accessibility and business continuity.

To determine how corporate business information should be split or amalgamated and managed through the administrative change, Fiona and her staff firstly took a long term view of each of the business areas affected. Using a lot of work they had already done, they compiled administrative histories of the business areas, some of which went back over 150 years. These were quite critical in identifying which areas of business were connected with which business area, and were particularly useful in ensuring that the appropriate management and responsibilities were allocated for the many legacy records identified.

Fiona said that to manage information effectively though this level of change, records staff need detailed skills in metadata mapping and business analysis. There are significant risks associated with poor metadata management in administrative change as metadata that supports record use and understanding can easily be lost. Fiona’s team also identified that metadata that had already undergone several migrations before their migration suffered some degree of degradation. The team also identified that metadata aggregation levels tended to be designed and deployed differently across all business areas and this made migration very challenging. Also challenging was the poor metadata that was often applied by business areas to their records. This delayed and complicated the work that Fiona’s team needed to perform.

Fiona also said in these situations, where there is little time and a massive amount of work to be done, it is critical to be strategic in your approach. Her team used their administrative understandings of the affected business areas to focus in on key business functions and then prioritise their work on the key records series and systems that support these business areas. They also used their knowledge of retention requirements to target records that were key long term underpinnings for business. With so much administrative change to deal with, there was a limit as to what could actually be achieved within the timeframes, but Fiona’s strategic approach ensured that business critical and long term accountability information was sustained and managed throughout the process.

Fiona recommended that all records staff dealing with administrative change have skills in rapid appraisal, data mapping, data migration and internal staff training development and delivery.

Public Record Office of Victoria’s digital archive strategy

Andrew Waugh gave a very good overview of PROV’s experience to date with its VERS project, its digital archiving initiatives and the lessons they have learned from all their experiences. While emphasising its focus on accountability and context, Andrew emphasised that the user experience, and access and use requirements were key drivers in PROV’s implementation of their digital archive.

Andrew also made the important point that, as fabulous as metadata is, it is not all that we want the digital archives of the future to be populated with. Both records and their metadata need to be connected and sustained in order to have a viable digital archival collection. And with the profusion of metadata that can be created and kept today, we may need to be selective in determining what metadata is needed to maintain meaning and accountability in the long term.

Digital recordkeeping at The National Archives, UK

Oliver Morley, Chief Executive and Keeper at The National Archives (TNA) gave an excellent presentation where, in a model that I should definitely learn to follow, he focussed on the good news and not just the bad news associated with digital recordkeeping.

Morley said that we should be pleased that a number of the key challenges that initially emerged in the digital recordkeeping world have successfully been addressed:

• there is now generally good legal clarity around digital recordkeeping
• physical record formats are not a significant preservation challenge any more
• there is a lot of standardisation now in digital record formats which makes their management easier and
• successful web archiving strategies at TNA mean that, they estimate, 30-40% of government records will be captured through web archiving strategies.

In terms of challenges remaining, Morley said that data volumes are an issue, but in shrinking public sector environments he feels that these challenges are not insurmountable. He said that email management continues to be an ongoing challenge because capturing email remains a clumsy process that is not aligned to business process. To Morley the management of sensitivities in records also remains a challenge and he believes that TNA needs to work closely with government departments to resolve sensitivity issues.

Morley also sees social media as a potential problem. He reported that Twitter and Facebook regularly change their APIs and often minimise the amount of data that can be extracted from them which raises problems for recordkeeping. Morley suggested it might be better to copy and paste content from social media channels into recordkeeping systems, rather than rely on other extraction methods.

He did caution however that we must not get carried away with social media recordkeeping strategies. Morley’s own research into UK government’s use of social media earlier this year showed that 93% of surveyed government tweets referred to stories on government websites. This information would already be captured under web recordkeeping rules and so he suggests that in many scenarios it may not actually be necessary to capture information in its social media form. In terms of the cloud, Morley said that the most important recordkeeping consideration is to ensure you can actually get your records out of any cloud storage arrangement they are in. He said it is important to check whether you can move your records to other providers if you want and if can you import them successfully back into your home systems at the conclusion of any cloud arrangements.

Morley outlined TNA’s ‘parsimonious preservation’ strategy where he flagged the importance of metadata and the need to deploy and use it as a powerful, multipurpose tool. He then introduced TNA’s new online search tool, Discovery. This tool provides fast search and filtering capacities based on time, geography and topic filters. Discovery is also a thoroughly integrated search tool, facilitating searches through both TNA’s digital and paper collections. The abstract to Oliver’s paper is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Abstract00342.pdf

Rethinking the archival function in the digital era

Hans Hofman from the National Archives of the Netherlands spoke about the really interested work going on in the government sector in the Netherlands to standardise recordkeeping. A government directive has stated that all government ministries must work entirely digitally by 2015. In preparation for this the National Archives has been involved in the development of an e-government infrastructure and have been actively integrating recordkeeping requirements and capacities into this.

The focus of their work has been on the business processes that government needs to perform, and then integrating information management and recordkeeping as a part of the information architectures and systems that will support these processes.

A key recommendation from Hans is to get involved in the planning and design stages of the system development life cycle to firstly make sure all necessary records are created, and then to ensure they can be sustained. A systematic approach to recordkeeping needs also to be based on a risk management approach, to ensure high risk business is supported by comprehensive recordkeeping strategies. The full text from Han’s paper is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12final00187.pdf

Using Linked Open Data

Jane Stevenson of the University of Manchester then spoke on the potential offered by machine readable metadata. Metadata that can be understood by machines creates opportunities for making connections between a whole host of online resources. In the archival environment, this enables linkages to be made between related collections in other domains, and allows you to build on other people’s linked data and to create entire networks of rich, meaningful, interconnected metadata.

Jane discussed some of the many fascinating projects that are currently exploring the potential of linked open data in the archival domain but there are also many possible uses in the contemporary recordkeeping space as well, particularly in administrative change scenarios or cross-jurisdictional collaborations. It would be great if someone could start exploring the potential of these scenarios! The full text from Jane’s presentation is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00029.pdf

Digital archives experience in Queensland State Archives

Adrian Cunningham from the Queensland State Archives spoke about QSA’s digital continuity strategy. Adrian said that the objective of QSA’s strategy is to ensure ongoing access to essential evidence of government business.

The digital continuity strategy is designed to be applied to records that need to be kept for five or more years, essentially those digital records that will outlive the business systems and environments in which they are created. Adrian reported that QSA’s surveys of government show that government agencies are concerned about the responsibility for the ongoing preservation of their long term temporary records. Adrian said that QSA’s digital archive strategy is likely to investigate how to tackle this problem first, and will then use this experience to inform its digital archiving processes.

Another key consideration in Queensland is that a recent audit report found that 2000 major business systems across government are currently due for decommissioning. This is a very large number of key systems all terminating at once and QSA is working to provide advice on how to integrate recordkeeping into the replacement systems and also on the management of the records contained in the decommissioned systems.

Towards generous interfaces for archival collections

Mitchell Whitelaw then spoke on different ways to access archival collections. His suggestions however have good validity for information searching in any context.

Mitchell argued that traditional search structures are generally the only way to access information but these structures are problematic. They often rely on users being able to use the same words that are contained in the record metadata and are often limited by the content, structure and quality of the existing metadata.

In contrast a generous search interface offers up information, rather than relying on a user to find it. Mitchell recommends that to enable generous interfaces, you should show as much information as possible to users and not demand a query before information will be provided. He recommends that you use metadata in innovative ways to characterise your information, offer samples of the information available and orient the user so that they are always able to understand the data you are presenting to them. He also recommends that you use metadata to provide a lot of context and a lot of relationships between your information and to illuminate as many macro and micro relationships as possible. Experiment with your information and with different ways of presenting it to users, to provide many possible pathways for exploring and using information. Mitchell’s full paper is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00423.pdf

Public rights to information and privacy

On the final morning of the conference, John McMillan, the Australian Information Commissioner, Jennifer Stoddart, the Privacy Commissioner of Canada and Miriam Nisbet, Director of the Office of Government Information Services in the United States formed a panel to discuss public rights to information and to privacy.

Miriam Nisbet said that it is important not to treat people who are trying to access government information in an adversarial fashion as it is, essentially, their information. She also made the point that records management (and good metadata) is critical if there is to be widespread public access to government information. Miriam’s paper is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/nisbet_m.pdf

Jennifer Stoddart said that we need to ensure that technology is used as an enabler of key privacy and recordkeeping principles and that it is not allowed to drive or distort these principles. She also made the point that it is very important to think up front at system design and information collection about privacy principles and recordkeeping, and to build in mechanisms for these to be achieved simply and effectively.

The metadata game

Jorien Weterings of the National Archives of the Netherlands spoke about a board game she and a colleague co-created called The Metadata Game.

In the Netherlands, a Ministerial regulation states that all government bodies should develop a metadata schema in accordance with the requirements of the international metadata standard, ISO 23081. The Metadata Game has been designed to help colleagues at the National Archives really understand the complexities of metadata and to help them explain these to Ministerial staff across government.

Using cards and a game board, the game explains challenging concepts such as metadata mapping, metadata quality, metadata schema, metadata encoding schemes, metadata aggregations and a range of other concepts in a fun and interactive way.

Jorien’s presentation was a really engaging lesson in how complex topics can be taught in innovative and clever ways. Jorien’s full presentation is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00184.pdf

State Records’s presentations

A number of State Records staff also gave presentations at the conference. Cassie Findlay gave one of the most acclaimed presentations of the entire conference, People, Records and Power: What can archives learn from WikiLeaks. Cassie’s full and inspiring presentation is available via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00220.pdf

To bring us back to metadata again, the very busy Richard Lehane gave two presentations. The first was a co-presentation with Damien Hassan from the State Records Office of Western Australia on their work to create software tools to manage the appraisal process. This paper can be accessed via http://ica2012.ica.org/files/pdf/Full%20papers%20upload/ica12Final00063.pdf

Richard’s second paper was on State Records’ award-winning API which provides online access to State Records’ extensive metadata about its archival collections.

Janet and I also gave a presentation where we reported on the digital disposal and social media surveys that we have conducted and where we discussed the challenges of sustainable recordkeeping in digital business systems.

Director Alan Ventress also spoke about third party digitisation projects and online accessibility of archival materials.

Professional reinvigoration and inspiration

In addition to all the incredible knowledge that was shared through the presentations I have described above and the 100+ other sessions that were on offer, one of the most fantastic things at a conference like this is the opportunity it provides for professional reinvigoration and inspiration. It was wonderful to mix with recordkeepers from across the world, hear about the diverse range of projects and challenges that people are engaging with and learn from the incredible array of professional expertise that was gathered in the once place at the once time.

The photo right up the top of this post shows Janet and me meeting with some fantastic archival colleagues. From left to right are Linqing Ma, Sherry Xie, Kate Cumming, Janet Knight, Jian Wang, and Yuenan Liu. Sherry is an adjunct professor at the School of Library, Archives and Information Studies at the University of British Columbia, Vancouver, Canada and her colleagues are archival scholars from Renmin University, Beijing. We had a great meeting for over an hour one evening where we discussed digital recordkeeping practices and the challenges of maintaining digital archives into the future. It was lovely to meet you, ladies!

To conclude, my summary above has barely touched the sides of all the issues that were raised at the conference. I can recommend the series of blog posts from the  Netherlands Coalition for Digital Preservation which are fantastic and which cover some of the other sessions I didn’t get to.  With very nice pictures too! The Twitter hash tag #ICA_2012 was also used extensively during the conference and the Twitter record provides a useful overview of many of the sessions (but get in fast while you can still access it!). The good news is that the National Archives has also used Storify to capture most #ICA_2012 tweets.

Finally, I would like to say a big thank you to State Records for supporting its staff’s involvement at the ICA Congress and a big congratulations to the National Archives of Australia for organising this fantastic event. Thanks!

* A number of excellent phrases were repeated throughout the conference and ‘1000 archivists from 95 countries around the world’ was one of them. A bingo/drinking game was quickly established on Twitter where people would tweet ‘Drink!’ as they reported hearing one of the key words or phrases. Possibly you had to be there, but it was an excellent diversion!