Is the big data farce or force with your company?
So what is dark data?
IDC has estimated that as much as up to 90 per cent of the world’s big data is dark data. Gartner describes dark data as “Information assets that organizations collect, process and store in the course of their regular business activity, but generally fail to use for other purposes.”
With the increasing adoption of big data, dark data has come to mean the growing amount of both structured and unstructured – but also untagged and untapped – data that’s accumulating in organisations. The data that a business creates and saves but hasn’t yet noticed or used for business intelligence. It also includes data that a business creates but doesn’t currently save.
An example of the dark data that can lurk in organisations, such as Telcos, is the reams of information on ‘customer experience’. There are now so many sources feeding into this, such as social media, that dark data is in danger of consuming us all.
The dangers of the dark side.
Firstly, dark data can be complex to analyse. It’s generally stored in numerous, and difficult to access, locations, where analysis is difficult – meaning it can be quite costly.
Secondly, due to its unruly nature dark data can pose many security risks. It’s often unknown, unmanaged, and may be out of compliance with internal or external requirements. Security is the third major big data adoption challenge – a potential show-stopper for nearly one in five.
And finally, much of it isn’t actually very useful – much of dark data is 99.999% noise and of no value. For example, the transition from voice only to data networks has created copious dark data for Telco operators; the information gathered used to be 20 voice transactions a day, now it’s more like 200 web transactions, which are of much less value than a voice call.
What should be done with dark data – can it be brought over to the light side?
It can be very hard to find the brightest star in all that galaxy but you don’t have to surrender to the dark side. Dark data can present opportunities if you have the tools to take advantage of it in the right way – and the .001% data of value can be identified.
Organisations should never start by analysing the data they have, they should begin by understanding what type of information could be in the data that could serve a strategic purpose. Items that used to be dark, mobile network outage history for example, may have significant value when brought to light. It just needs a light saber-type tool to banish the darkness and find the force.
There are now technology solutions available to help businesses access, understand, control and take action on dark data. For example, the Zen big data applications suite can predict and automatically action events, such as network outages, before they occur. So, when thinking about what potential dark data may hold, it pays to think as broadly as possible.
So remember: with the decision in mind and the right analysis, dark data can be brought over to the light side, and with it, its potential ROI.