Are we coming to an information event horizon at which point the speed we collect new information will move faster than our ability to delete it; let alone manage it.

Up to this point the computing power and cost of storage have been a self limiting factor in our ability to hold on to this much data.  Previously only large institutions and kings had vast libraries of information. Now this limitation has gone. When taking pictures at your next family event, you will notice that the memory card in the camera allows you to take hundreds of pictures. Even though many pictures are not worth keeping you will never delete all the bad photographs. Instead you copy the full contents of the memory card to your home computer.

This is not to say that you will use all of these photo’s. It is very unlikely that you will upload all 300 pictures of Aunty Mildred’s birthday to your Facebook account. However, at the same time you will not remove most of the unwanted photographs as there might be something in the future that you decide you will want. This shows that the cost of storage is no longer a limiting factor in technology.

This is only the start of the level of information we will collect over our life time as the cost of storage further reduces.

The Internet and cloud computing is one of the major factors for moving this event horizon closer. The formation of huge, seeming endless storage, hosted over multiple locations means we are not limited by any physical constraints anymore. In fact we are not bound by any single location.  The comical idea of a bookshelf with a collection of books is further removed through latest eReaders that have arrived. These devices in combination with the Internet remove the need for us to physically go to a store to buy books and allow us to get new books without moving form our house. This would have been seen as magic only a few decades ago.

This is not to say that any of these inventions are bad or wasteful, far from it. Technology has enabled many people to study and express their views of an event in ways that have never been possible before. With no censorship people can capture their own version of history. In fact the filming of events and social uprising via video has been one of the strongest benefactor of this new low cost storage.

The real issue is our ability to recall information and manage it. As humans we are have not evolved at the same rate as our technology. Our brain was built with a specific ability to recall facts and automatically filter and remove memory that have no obvious purpose to us anymore. For example not many of us will remember where we parked the car four weeks ago at the supermarket, even though we visit it every week. This can be very useful as we can focus on the information that is most important to us, and remove all the clutter of everyday life.

However electronic storage provides no way of automatically filtering the information that we find useful from the information that we are not bothered about. This does not mean we want a automated delete system, just a better way to find things that are important to us. We need technology to work in a way that more closely models the way our brain works.

So what is an event horizon? The term is used to describe the distance from the centre of a black hole when crossed you can never escape. When inside this horizon the gravitational pull will never let you go. Information technology is starting to behave in a way that mimics this same point of no return.

With more complex smart phones and cheaper recording devices like digital cameras we will soon be able to capture everything that is happening around us and store it at a speed we can never expect to be able to use.

So what is the answer? There are many possible outcomes, smarter ways in which we pick what we want to keep, and smart automation that helps highlight information. However the level of information kept will still be greater than our own brain is able to remember. And searching all of it every time we need something will become slower and slower the more we have.

Therefore we need to change the way in which we retrieve information, we need to find methods that allow us to forget all the details but focus on only a subset of topics that will enable us to obtain the detail that we want. In this way we will not need to hold the full index of the information in our heads just the subset of information that enables us to navigate it.

Search engines have tried to do this, however they are unable to allow the person to truly control the way in which the index is created. Search engines today only allow you to manipulate a set of standard rules that try to provide information in a common way. We all know that the way in which you remember an event is not the same way that another person will remember the event.  Therefore  it is incredibly complex to provide a single tool and method of information retrieval that will work for everybody. It is also why these search engines can only provide meaningful results when the output has be further refined.  The act of entering a search term and then select the third item in the list is a method of refinement. It changes the importance of that entry to the terms over time. This method is used by many search engines to improve the way to find things in the Internet.

This works well when many millions of people look for similar information over a common source (the Internet). However it has been shown that the similar methods do not result in the same level of productivity when used in isolation. For example how many of us have tried desktop search tools to find a file, only to get a long list of meaningless results.  Unless we can narrow the search by dates or file type we do not get a meaningful result.

Therefore a better more natural method of building a reference is required one which is directly tailored to an individual and allows the person to use the way that they think to arrange and mange information.

New taxonomies and linguistic technologies are some of the possible ways forward if they can be made to adapt at the same speed at which we can store the information. Many methods today still require post analysis of the information and manual configuration that make the speed of retrieval not real time. Personal tagging that automatically adapts to your experience and knowledge is a better way in which we will be able to speed up this process. Combined these ideas with better retrieval based on personal key terms will be the next major step forward.

Lets hope the technology  we use to collect information will also find ways to stop us from falling completely into the information black hole that we are creating.