Future Proof – Protecting our digital future

We are keeping too much information.

The growing costs of information storage and management are mounting to the extent that very soon they will become unsustainable.  Unless proactive action is taken, the data bubble is going to burst and key information will be lost when it does.

To date, however, the belief that storage is cheap and that we can afford to keep all our data forever continues to dominate.

An October 2012 blog post by David Rosenthal of Stanford University, however, provides much evidence to challenge this belief.

Its conclusions about storage costs reinforce the view that, in order to prevent the data bubble from bursting, we need to:

• be much more strategic with our information management practices
• proactively use recordkeeping strategies to manage burgeoning ICT costs
• use existing record retention and disposal strategies to mitigate information risks.

 In a paper presented in September 2012 to UNESCO’s Memory of the World in the Digital Age conference, David Rosenthal and his co-researchers Daniel Rosenthal, Ethan Miller, Ian Adams, Mark Storer and Erez Zadok provide evidence that in the very near future ‘storage will be a lot less free that it used to be’.

Citing IDC figures for annual world-wide data growth (an extraordinary 60% per year), the decreasing bit density on the platters of disk drives and the 0-2% annual growth rates of ICT budgets, Rosenthal observes that “10 years from now, storing all the accumulated data would cost over 20 times as much as it does this year. If storage is 5% of your IT budget this year, in 10 years it will be more than 100% of your budget. If you’re in the digital preservation business, storage is already way more than 5% of your IT budget. Its going to consume 100% of the budget in much less than 10 years.”

Analysis by Rosenthal and his research colleagues shows that Kryder’s Law is no longer holding true. For the last thirty years, data storage developments followed Kryder’s Law and ‘customers got roughly double the capacity for the same price every two years. Thus the cost of storing a given digital object rapidly becomes negligible. The perception was that the delta between storing an object for a few years and storing it forever was too small to worry about.’ 

Rosenthal says current industry projections are for no more than a 20% per year improvement in bit density and, for a variety of reasons, his research suggests that this much more moderate improvement will not follow the usual pattern under Kryder’s Law and will not result in an annual 20% reduction in storage costs.

Thus each year we are going to create 60% more data, storage costs are going to increase rather than decrease and IT budgets are not going to keep pace. 

Combined with this, State Records’ research shows that most NSW government organisations have no regular programs for the strategic retention of their core, long term business information and for the regular, timely destruction of their time-expired digital business information. Thus no risk management and strategic assessment is generally being applied to digital business information, and blanket decisions mean all information, irrespective of its value, is generally being kept. The risks of this inattention are drastically starting to rise.

In relation to the question of whether cloud storage provides a possible solution, Rosenthal concludes his post with: ‘I will leave you with this thought experiment. Suppose we wanted to keep all the world’s data in S3. Each year, we would need to endow that year’s data. 2011’s endowment would be about $11.4T, or 14% of the gross world product. According to IDC, the data to be stored grows by 60%/yr. Gross world product is growing about 5%/yr. The endowment needed for 2018’s data would exceed the gross world product.’

We cannot therefore afford to keep everything, no matter where it is stored. We therefore need to be much more strategic with our information retention and we need to start regularly throwing appropriately time-expired digital information away. From an economic point of view these strategic information management frameworks have never had to be built into digital business decisions but it is critical now that they are.

In an article in Forbes about Rosenthal’s research, David Feinleib comments that ‘You can’t save everything forever, because storage technology has not, in recent years, delivered the increased capacity and associated reduction in cost historically promised by Kryder’s Law. Meanwhile, “the rate of finding things to measure and ‘sense’ is going up exponentially,” according to Jonathan Feiber, venture capitalist and Adjunct Professor at Stanford. ”These two things are in conflict and at some point will collide for enterprises.” ‘

No matter how you look at it, we are maintaining more information than we can afford to keep and we urgently need to be so much more strategic in our decision-making. Once perhaps we could be complacent but we can’t anymore. We cannot save everything forever so we need to be incredibly proactive and strategic and design ways to keep what we need to keep, and to appropriately destroy the information we no longer require.

Every organisation needs to proactively do this now and not allow depleted ICT budgets with no capacity for rising data storage costs to make general, sweeping data storage cuts.

Information is a core business asset. Risk management strategies need to be put in place and existing record retention and disposal requirements leveraged to ensure core, long term value business records are kept and short term value information is routinely destroyed.

Establishing this strategic, managed approach to information retention in your organisation is necessary to help you to manage your ICT costs, avoid the impacts of the data storage bubble bursting and to ensure your organisation has the core information it needs now and into the future.