Future Proof – Protecting our digital future

photo credit: justindoub

In October 2011, State Records distributed a survey on public sector attitudes and practices in relation to digital disposal to all NSW public offices. During the two week period it was available, 83 individual survey responses were received.

The key issues that emerge from this survey are:

• limited record destruction is taking place in the EDRMS environment and in the business system environment
• the disposal of records in business systems is regarded by many public offices as costly, time consuming and difficult
• many organisations are not aware of the retention requirements that apply to their records held in business systems
• there is a widespread belief that digital disposal can ‘wait till later’ and for many paper record disposal is seen as a much more pressing issue, and
• most records managers are not confident that their significant records can be maintained for more than 10 years.

Each of these issues is of concern to State Records as they pose risks to the useability, accessibility and maintenance of government business information. The responses to the Digital disposal survey will inform State Records’ future work program for the development of tools and advice in relation to digital disposal.

In addition, the issue of the importance of digital disposal will be a focal point of our communications to chief executives, ICT staff, business managers and records managers. Through our communications channels, and at other available opportunities, we will highlight the importance of digital disposal as a tool for enabling digital continuity and digital longevity.

The survey responses quite clearly illustrate that digital disposal is seen as a process that can be deferred for some time into the future. However, the realities of digital data mean that unless long-term records are identified at or close to their creation, and unless active intervention and management of key records is performed, it is possible that these records will not survive for as long as they are needed to fulfil their legal and business obligations.

Digital disposal is therefore a key tactic to help ensure digital longevity, as well as a tool for managing increasing data storage and management costs, and therefore should be more actively encouraged across the NSW public sector.

Full details about the specific survey responses are available below. At the bottom of the post is a list of all the survey questions and also the responses provided by respondents in the free text field in the survey where they were asked for any additional comments about digital disposal.

If you have any issues, comments, concerns or advice about digital disposal issues, please contact us here at Future Proof. And please note that for a few of the questions, percentages add up to more than 100% because respondents could check all responses that applied to them.

Destruction in electronic document and records management system (EDRMS) environments is occurring

The survey shows that records managers generally have good knowledge about the disposal practices in their EDRMS (electronic document and records management system) environments.

32% of respondents reported that they had destroyed digital records from an EDRMS using disposal authorities. This is an encouraging statistic, showing that some digital disposal is being performed.

47% of respondents said that they had not destroyed digital records in EDRMS environments.

Another 17% of respondents reported that they do not have an EDRMS and 4% said that they do not know whether records have been destroyed in their EDRMS.

Destruction in business system environments is more problematic

Respondents appear to be less confident about implementing disposal in the business system environment.

Only 9% reported that records had been destroyed from their business systems using authorised retention and disposal authorities.

66% of respondents said that disposal had not been performed in their business systems.

25% reported that they did not know whether authorised disposal had taken place in their business systems.

There are a range of issues preventing organisations from destroying digital records

The survey provided a list of 9 issues that might affect their ability to destroy digital records. Respondents were asked to select any or all that might apply.

67 respondents answered this question and most nominated more than one issue that affected their ability to destroy digital records.

There are problems with the implementation and scope of disposal tools

The top two responses to this question were that ‘It is too hard to map retention and disposal authorities to the records in systems’ and ‘ Triggers in disposal authorities are too hard to apply to systems’. These options received 27 responses each, meaning 40% of respondents believe the disposal tools themselves are affecting their ability to destroy records.

21% of respondents said that ‘Retention and disposal authorities do not cover all records in systems’.

There are problems with systems

Many respondents believe that systems inhibit good disposal practices.

For example:
• 23 respondents (34%) believe ‘Systems are not able to be configured to manage disposal’
• 22 respondents (33%) believe ‘Too much manual intervention is required to destroy records in our systems’
• 21 respondents (31%) believe ‘It is too expensive, difficult or time consuming to configure systems for disposal or to integrate them with our EDRMS’.

The cloud is an issue

Only 2 respondents nominated that they do not have control over disposal as their records are in the cloud. However, this is worth noting as a potential risk on the horizon.

But most organisations do see disposal as important

Only 4 (6%) respondents answered ‘My organisation does not see the need for digital disposal’. This is a heartening response. In the current environment where storage is cheap, there is the risk that organisations could fail to understand the risks and liabilities associated with a lack of regular disposal. It is good to see that most organisations still see the value in good disposal processes.

The full extent of business systems may not be covered by existing retention and disposal authorities

When asked specifically what proportion of their organisation’s digital records in business systems were covered by retention and disposal authorities, 45% of respondents reported that their current retention and disposal authorities provide coverage for 51 – 100% of their business system records.

However, 37% of respondents reported that less than half of their business systems were covered by existing retention and disposal authorities.

It is of some concern that 18% of respondents do not know if their business systems have disposal coverage.

This appears to confirm what is understood anecdotally: that records managers are frequently isolated from business system development and implementation in their organisations. It is a concern because business systems need to be designed and implemented with recordkeeping requirements in mind, particularly retention requirements which identify how long the data within business systems needs to be maintained for.

Retention and disposal requirements and generally not being managed in business systems

The survey provided a list of 9 ways in which retention and disposal authorities could be applied in business systems. Respondents were asked to select any or all that might apply. 55 respondents answered this question and most nominated more than one way in which disposal authorities were or were not being applied.
Mostly disposal authorities are not being applied

49% of respondents said that ‘My organisation has not decided how to manage retention and disposal requirements in business systems’, indicating that no disposal action or planning is occurring in the majority of business systems.

Disposal is not occurring in business systems because it is hard

27% of respondents reported that their organisations would like to manage their retention requirements in business systems but do not because it is ‘too expensive, difficult or time consuming’.

Some organisations do not know what records they have in business systems and therefore can’t apply retention and disposal actions to them

22% of respondents stated that ‘My organisation has not yet identified what records are in the systems’.

Some plan to manage disposal at migration

16% of respondents said that ‘My organisation will manage requirements at system migration’.

Some are baking disposal requirements in at system design

There was some good news in amongst the responses to this question. 11% of respondents reported that ‘My organisation builds requirements into systems at design/customisation’.

Others are linking business systems closely to EDRMS environments

Further positive news was provided by the 18% of respondents that said ‘My organisation links requirements by integrating our EDRMS with the systems’ and ‘My organisation exports relevant records and captures them into an EDRMS’.

A small but significant percentage are not sure how it is done

7% stated that they did not know how disposal was being managed in their business systems.

And a small but significant percentage does not care

6% said that ‘My organisation sees no need to manage requirements as it intends to keep everything’.

Genuine risks are threatening long term retention

Respondents were asked ‘For your digital records that need to be retained 10+ years, are you confident that they can be maintained for the length of their retention period?’

Only 19% of respondents indicated that they were confident that their long term value records could be retained. 17% were not confident, 41% were confident for some records but not for all and 23% did not know.

Unfortunately, if 80% of respondents indicate some lack of confidence in maintaining significant records for more than 10 years, then this is a critical risk for the long term accountability and business continuity of many government organisations.

Applying disposal decisions to legacy systems is problematic

Respondents were next asked to nominate how much of a problem it would be to apply disposal decisions to legacy systems.

44% said it would be a major problem, 36% a minor problem and 11% didn’t know. 9% said it would not present a problem. The vast majority of respondents therefore believe legacy systems pose disposal problems for them.

Informal advice suggests there are growing numbers of legacy systems across public offices and that these systems are often very large and uncontrolled. This advice also suggests that many of these systems are under immediate threat as a result of administrative change, with no one effectively being responsible for their ongoing support and maintenance.

Metadata is at risk

Respondents were then asked to nominate the degree to which maintaining a metadata record after the destruction of digital records would be a problem.

42% said it would be a major problem, 31% a minor problem and 14% didn’t know. 13% said it would not present a problem.

Ongoing accountability requirements have always stated that an adequate record should be kept of disposal processes. These responses indicate that this is possibly becoming less achievable in the digital environment.

What will convince management to support digital disposal?

The survey presented 5 options and respondents could tick all that they believed would inspire their management to support digital disposal, and also nominate those that would not appeal to their management.

Interestingly, the biggest driver identified was the option ‘improving risk management through better identification and management of key business records’ (86%). This is a very heartening response that shows people believe their managers are very concerned with risk and good business operations. In future communications with managers and chief executives, the clear relationship between good recordkeeping and good business and good risk management should be emphasised.

The next biggest driver was ‘speeding up system response times’ (79%). The other more IT focussed drivers, ‘freeing up storage space’ and ‘making systems migration easier’ received responses from 70% and 68% of respondents respectively. Therefore there is still great relevance in linking recordkeeping requirements to good data management practices, and promoting the ability of recordkeeping tools to assist with IT management issues.

68% of respondents flagged that ‘helping to manage the cost of digital preservation’ was a driver for management.

The responses which identified the options that would not work as drivers for management were also revealing:

• 22% of respondents said that ‘freeing up storage space’ was not a driver
• 22% reported that ‘helping to manage the cost of digital preservation’ was not a driver
• 21% said ‘making systems migration easier’ was not a driver.

This tends to confirm that:

• there is still a core belief among some managers that storage is cheap and unlimited
  there is still a lack of understanding among management of the need for the ongoing retention and management of digital records
• some managers fail to understand the costs and risks of not considering disposal requirements early in the life of the records.

Other comments

The final question gave respondents the opportunity to make any additional comments about the disposal of digital records or how digital disposal is performed in their organisation. 23 respondents made additional comments. A full list of comments received is provided below.

Key themes that emerge through these comments are:

• paper records are often still seen as the priority
• there is a lack of knowledge, resources and skills to deal with digital disposal
• State Records should do more to promote good disposal practices to management rather than convince records managers
• State Records should do more to help IT understand disposal requirements
• disposal in business systems is a significant challenge
• some managers believe deleting ‘as much as possible as soon as possible’ is a risk reduction strategy.

Conclusion

This survey provided a range of very useful data on digital disposal practices and State Records is very grateful to all those who took the time to provide this valuable information.

State Records will use the data provided to assist in the future development of digital recordkeeping tools and advice. As an immediate priority, however, State Records will use the survey data to promote the importance of digital disposal and its links to information longevity. Using various communications mechanisms State Records will continue to promote the importance of proactive disposal and the consideration of disposal requirements in business system design to:

• Chief executives
• Managers
• Business managers
• ICT managers
• Records managers
• EDRMS vendors.

It is important for all organisations to be aware that with digital records, there is not necessarily the luxury of time. In the digital world, disposal needs should not be seen as a post-hoc, antiquated process that occurs at the end of a record’s lifespan. It should instead be an active process that should be considered at record creation. Disposal authorities should be used as system design and implementation tools and applied at or before record creation.

If disposal information is not used from system design and record creation to plan for the retention and management of digital records, it is increasingly unlikely that records of long term organisational value will survive and be accessible for as long as they are legally required to be kept.

Organisations place themselves at significant risk if they defer disposal considerations for digital records and State Records will continue to communicate this point.

Break down of survey responses by sector

Of the survey responses received:

• 42% were government agencies
• 34% were local government
• 10% were State owned corporations
• 7% were health sector
• 6% were universities
• 1% were courts

Survey questions from State Records NSW survey on digital disposal, October 2011

1. Are you from the NSW public sector?

2. Is your organisation:

• part of the NSW health sector
• part of the NSW university sector?
• part of the NSW local government sector?
• a NSW government agency?
• a NSW State owned corporation?
• part of the NSW Parliament or the court system?

3. Has your organisation destroyed records from an electronic document and records management system (EDRMS) using retention and disposal authorities?

• Yes
• No, my organisation has an EDRMS but we have not destroyed records
• No, my organisation does not have an EDRMS
• Don’t know

4. Has your organisation destroyed records from business systems using retention and disposal authorities? (Business systems are purpose built or customised systems such as e-commerce systems, client relationship management systems, finance or HR systems etc, not EDRMS).

• Yes
• No
• Don’t know

5. What might prevent your organisation from destroying digital records? (Tick any that apply)

• My organisation does not see the need for digital disposal
• Retention and disposal authorities do not cover all records in systems
• It is too hard to map retention and disposal authorities to records in systems
• Triggers in retention and disposal authorities are too hard to apply to systems
• Systems are not able to be configured to manage disposal
• It is too expensive, difficult or time consuming to configure systems for disposal or integrate them with our EDRMS
• Too much manual intervention is required to destroy records in our systems
• Our records are in the cloud and we do not have control over disposal
• None of the above

6. What proportion of the records in your organisation’s digital business systems (core or administrative systems, not EDRMS) are covered by authorised retention and disposal authorities?

• 0%
• 1-25%
• 26-50%
• 51-75%
• 76-100%
• Don’t know

7. How does your organisation apply retention and disposal requirements to business systems? (Tick any that apply)

• My organisation builds requirements into systems at design/customisation
• My organisation links requirements by integrating our EDRMS with the systems
• My organisation exports relevant records and captures them into an EDRMS
• My organisation will manage requirements at systems migration
• My organisation has not decided how to manage requirements
• My organisation would like to manage requirements but it is too expensive, difficult or time consuming
• My organisation has not yet identified what records are in the systems
• My organisation sees no need to manage requirements as it intends to keep everything
• Don’t know

8. For your digital records that need to be maintained for 10+ years, are you confident that these can be fully exported out of their current business and/or EDRMS environment, imported fully into a new system and maintained for the length of their retention period?

• Yes
• No
• For some systems but not all
• Don’t know

9. Please rate how much of a problem you think each of the following disposal scenarios may be.

• Applying disposal decisions to legacy systems (eg old databases, superseded business systems) [Options: A major problem, a minor problem, not a problem, don’t know]
• Destroying digital records while maintaining an appropriate metadata record [Options: A major problem, a minor problem, not a problem, don’t know]

10. Do you think these drivers for digital records disposal would appeal to your management?

• Freeing up storage space
• Speeding up system response times
• Making systems migration easier
• Helping to manage the cost of digital preservation
• Improving risk management through better identification and management of key business records

11. Are there any other comments you would like to make about the disposal of digital records or how digital disposal is performed in your organisation?

Individual comments received about digital disposal issues

The specific comments received in response to Question 11 in the survey, ‘Are there any other comments you would like to make about the disposal of digital records or how digital disposal is performed in your organisation?’ are as follows:

We are still in the planning phase of how to dispose of digital records from the myriad of different business systems generating digital records in our organisation.

Disposal of digital records will continue to be a major problem for business systems unless steps are taken to mandate built in disposal functionality when these systems are built. Eg there is a push to ensure electronic information security is mandated for systems covered by the NSW panel contracts – the same should be considered for digital disposal.

In the absence of the ability to correctly sentence digital records held in business systems (outside of the EDRMS) they are currently being maintained “forever” with the knowledge that they must be migrated if/when there is a change of business system.

It would be great if GA30 and Keywords for Councils had some built in mapping, thus standardising and simplifying the application of disposal and retention criteria.

The strategy to keep digital records is about identifying non current records and moving them to relatively cheap storage solutions. In some ways this is like an internal disposal repository but accessible via line of business applications.

More information needs to go to Senior Management in regards to the importance of preserving digital records especially Record Metadata. Perhaps some information sessions specifically for Senior Managers or CEO’s. This is more relevant now with the merging of Agencies into Super Departments. The migration of data into relevant the EDRMS seems to be low on the list of priorities

We have designed our EDRMS so all staff in the orgasation can capture records easily into a classification they can understand. This has been succesfull and we are capturing an increased number of records. This new classification will make disposal of records difficult as we are reliant on the user to select appropriate classification – this not always correct.

Triggers for the destruction of electronic records are proving to be a major stumbling block

Our emphasis has been on improving capture through the implementation of an EDRMs We are still trying to better manage that, it is a long term process. In the meantime, any disposal done is performed on paper records stored offsite so we can reduce storage costs. The digital records can wait, they are not a priority

I think disposal of digital records in our EDRMS system will be done in the most efficient manners the issue here is the cloud computing and those records resides off site, there is little direction as to how the agencies approach those records as more and more of our business will find its way in the cloud, so clear guidelines on cloud computing will be well received. (USA published its Terms of Services as a guidance to their agencies in dealings with cloud computing and we would like to see something similar in NSW)

If IT and Records dept work together towards a common goal regarding Digital disposal, Council should become compliant with Digital Records requirements. If they cannot work together there will be problems

The capture of electronic records needs to be managed and highlighted as integral to business functioning. The importance needs to be embedded across the public sector from top down!

A more respectful regard for the role of recordkeeping and disposal of digital records is a necessary cultural change for our organisation. A coordinated approach to digital records disposal would help avoid duplication of records into the EDRMS. A requirement to have a sentencing program in local government and a requirement to employ at least one sentencing officer for digital (and paper) records disposal would greatly assist.

Only minimal records have been destroyed in our EDRMS

Not easy to respond as not all systems managed the same way, with same level of compliance implementation…difficult integrating requirements or systems on limited budgets

The disposal of digital records held on network at not in the EDRMS system is being managed by the IT Manager, not the Records Manager. The IT Manager has a destruction process that does not refer to any disposal authorities and simple destroys back up tapes every few years. Perhaps a reminder that a penalty can be levied for this sort of disregard of compliance requirements?

Current disposal authorities designed for paper – need to be linked to process not activity to facilitate implementation.

Resources prevent us from looking at the disposal of digital records at the moment. This will change in the future though.

In raising the requirements of the standard on digital recordkeeping with IT professionals the response has been that State Records do not understand the difficulties and complexities and the issues facing Information Technology professionals in what is being requested of them. Business systems are never going to meet the requirements to be stand alone recordkeeping systems, therefore the only other action is to integrate the system with TRIM or export the records into TRIM. Council now ensures that TRIM integration is raised with software vendors for any new business system but often the response is noncommittal. Vendors seek to sell their product and are very wary of integration points that see them have to work with other vendors of SDK’s. This makes life very difficult for Records Managers in this electronic age who rely on the involvement of IT professionals to a large degree with the complexities of digital records.

I am in the fortunate situation of working for an organisation that takes these matters seriously. Even so, the challenges are not small ones.

This issue has absolutely no visibility in the first two executive management levels of this ( & every) agency. Ditto for Ministers. You need to stop trying to win over records people (you’ve won that one) and start at the top.

My management are very keen to reduce legal risk by deleting as much as possible as soon as possible. This is a big factor you have not included. Not everyone is an archivist!

My organisation uses TRIM as its ERDMS and use a Classification Scheme KFC with retention metadata applied. Updating from GDA10 to GA39 is seen by TRIM as an action and thus changes last action date for all records effecting retention periods.