Big Data is a buzzword in every innovative business right now! Why? Companies are vacuuming up data from an expansive network of online and offline sources (from phones, credit cards to the infrastructure of cities) – but it is not the quantity that is remarkable, its what we can do with it. It is now possible to process this data cost effectively and timely to provide more valuable insights and become more empowered through business intelligence to make more informed business decisions.
Ultimately, Big Data provides companies with a competitive advantage through better understanding of the 5c’s (consumer, competitor, company, competitor, climate) and foresight about what actions can help drive the business into the future. For example, insights can help create cost efficiency, improve processes to reduce time and generate economies of scale, understand customer needs better to assist with new product development and; improve customer experience or to detect risks/fraud. This is just the tip of the iceberg.. I think now you can see why its such a hot topic! Show me the MONEY!
So…What is Big Data?
According to Chen et Al , Big Data and Big Data analytics can be defined as:
“the data sets and analytical techniques in applications that are so large (from terabytes to exabytes) and complex (from sensor to social media data) that they require advanced and unique data storage, management, analysis, and visualization technologies.”
Simply put, Big Data is a term used to describe extremely large volumes of data that can be analysed via computer programming to see patterns, trends and associations. It can be of both a quantitative and qualitative nature, stemming from a multitude of sources – such as streaming data (a web of connected devices- IOT), social data and publicly available data sourced through secondary research.
2.5 Quintillion bytes of data is created every day and by 2020 it will equal 40 Zettabytes. While 20% of data is structured (governed by a relational table), a staggering 80% is either semi-structured or unstructured data stemming from videos, images, social networks etc, so it is essential that data storage, data analysis, data visualisation and data reporting is at the core of how businesses operate.
What is important is not the volumes of raw data, but what companies do with the data to analyze and generate insights for better decision making and enhanced strategic planning through Business Intelligence.
This is reinforced through Gartners definition of Big Data:
“Big data” is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
As above, Gartner Analyst, Doug Laney, defined Big Data as 3 Vs (volume, velocity and variety) in 2001 and since then it can be simplified with the following formula:
Big Data = Transactions + Interactions + Observations
Data is being generated by people, machines, applications and a combination of all of these. Classical data is Transactional data, which is highly structured stemming from an event, with a time, numerical value and objective- e.g. purchase, payments, inventory shipping- and is usually accessed through SQL (Structured Query Language). Whereas, Interactional data, which stems from relationships and interaction – e.g. web logs, social interactions and UGC-and observational data which stems from Internet of Things – e.g. RFID, NFC, sensors for lights/pressure/alarms- are both from multi-structured sources and require NOSQL.
From a marketing perspective, data has evolved over time from being one of transactional data (where the customer was unknown), to demographic data (what the customer looks like) to psychographic data (defining people by their interests) to an age now where we can evaluate attitudinal data (understanding sociographics i.e. how people think or feel). Big Data tells us what has happened and Business Intelligence helps us to understand behaviours and what can influence them. These steps sum up the use of Big Data:
Aggregate > Analyze > Articulate > Act
Big Data Challenges
Due to the large volume, velocity, variety and requirement for validity of Big Data, there are huge challenges for IT, from use of infrastructure (data capacity, data speed), to platforms (end-to-end, easy to use and fully integrated platforms) and databases (scalability and ability to manage semi/non-structured data). This is being addressed by HPC (High Performance Computing) through Parallelism, Clusters and cloud computing.
- Business Intelligence and Analytics: from Big Data to Big Impact, Chen et al (2012).