The average cost of a gigabyte of storage has gone from a whopping $10,000 to $0.10 in just 20 years! With data storage costs continually reducing, the cost of big data storage is an issue of the past; disks really are very cheap. But within the telecoms industry, CSPs are faced with a new challenge – finding the right data architecture which can meet all of their big data needs, from real-time analytics to long term data retention.
In today’s market, someone says big data and the marketing dollars spent lead you to automatically think Hadoop. And this is with good reason; in the 10 years since its launch, Hadoop has built up an amazing brand and has been associated with the explosive growth of the many Web 2.0 giants. However it clearly isn’t a nirvana for all. Many organisations have followed the Hadoop pathfinders with very limited, if any, success. It just isn’t the right platform for every application.
In fact, within the telecommunications industry we are beginning to see the very many Hadoop data lakes stagnate, filled with untapped, undervalued and unusable data. Not surprisingly, CSPs are now looking for solutions to get ‘life’ out of their Hadoop data lakes so that they can utilise the undoubted value of their broad network and customer data sets. So what is the solution for telecoms operators?
What do CSPs actually need when it comes to big data management?
When it comes to big data, CSP’s create a lot! They are generating millions of data records every second, all of which need to be processed, analysed and stored. For the most part, there are four fundamental requirements driving big data management in telecoms:
- Real-time data and complex analytics: Real-time (or near real-time) data loading and processing is required for applications such as network monitoring and service operations; late or slow data is of no use to these teams. To successfully gain insights from big data, complex data analytics is needed to understand network performance and usage, and predict future trends.
- Transactional Activity: As data volume increases, so does the requirement for automated alarming and alerting. By its very nature, event based data changes state, i.e. individual records constantly need to be updated. The complete corollary of the traditional network data that is entirely unchanging.
- Relational Management: For mapping and understanding network topology, CSPs need to map network elements, both physical and virtual, and constantly refresh their relationships and attributes. Many customer relationships also follow similar relationship based data models.
- Long Term data retention: CSPs are required to store some data for periods of two years and more. But much of this data is of indeterminate value and may be useful or essential in future – but what data is potentially useful and what is not? The challenge is to effectively store what may have value, i.e. nearly everything!
With Big Data, one datastore cannot fit all
When talking big data, hadoop is the de facto solution for data storage, but let’s be clear, it can only address item 4. Similarly, the real-time analytics databases are not well suited to updates and long time archive, although they are the perfect model for real-time, high volume data capture. And graph databases are perfect for relationship management, but not for transactional data.
In a “one use case” world, one data store can do the job, and with “small data,” a general purpose database will meet any use cases. But big data is big and challenging as its name implies, and each use case demands a different data management model – one data management tool will always fall short.
Therefore CSPs need to utilise a combination of data stores to meet the four fundamental requirements of big data management in the telecoms space. This approach will give CSPs a futureproof foundation for their big data management, meeting all of their big data analytics needs. One thing we must remember, is that to achieve this within telecoms, we need not only the architecture, but also the understanding of each big data use case to be a truly functional and impactful success.