10 Things to Know About Big Data Analytics

1. The 5 Vs of big data

There are a few characteristics that stand out when we talk about big data. One is the velocity or the rate at which it gets generated. The second is the volume, which is a vast amount of data. There are also many varieties of data sources that usually characterize big data. We are also talking more about this thing called veracity when it comes to big data. Since the type of data is different and can sometimes be incomplete or inconsistent, deriving direct insights with available technology depends on the quality or veracity of data. You also need to be able to use innovative technologies and to derive value or make sense of such large volumes of data. That is what differentiates big data from a large volume of data.

2. Types of data

Maybe you have some business data or transaction data which is coming from the products, i.e., the revenue, the customer data, the sales data. Those are the structured data that you have, so the sources can be many. It could be coming from the operational data source, or from the transactional data source. Many businesses are creating lots of data daily. Then there is this social media data that is about customers and their interactions with different brands, which is now called experience data. What we are doing at SAP is trying to look at how this operational data and experience data can be brought together to be able to derive insights so that things can be done better.

3. The first uses

If you look at the history of data and what it was being used for, for example, to make decisions, it probably dates back thousands of years. Let’s take accounting, for instance. From the time accounting started, data was already playing a role. The first major data project was created in 1937 by the Franklin D. Roosevelt administration in the US for the Social Security Act. IBM oversaw the project, and it was one of the first times American’s data was tracked and managed at such a large scale. But then, if you look back just 50 years ago, the amount of data that was being generated in the entire world could have fit in the average laptop that you use now. According to the IDC, data that is generated today is in the range of 59 zettabytes — that's a lot of zeroes. While data has been playing a role in our lives for many years, in the last 10 to 20 years, the amount of data, as well as the kind of data that is being generated has massively increased, and hence, the overall complexity when it comes to dealing with big data has amplified as well.

4. How big is the data

To be able to get a sense of how big, big data is and to put some practical examples to quantify the amount of data, basically a zettabyte is 1000 exabytes and one exabyte is 1000 petabytes. Each petabyte is 1000 terabytes. According to a quote from storage company Seagate, a zettabyte is enough storage for over 30 billion movies, 60 billion video games, or 75 trillion MP3 songs.

5. Its impact

The potential that big data has is something that really stands out, because it really has the capability to make us more effective and efficient in using data. It is impacting our lives in several ways, the very latest being the speed at which the COVID-19 vaccine was developed. The thing that intrigues me, is the application of big data in our lives, the impact that it has in both our personal and working lives. If we can make the right decision at the right time, most things would fall into place. And big data has the potential to be able to enhance the complete decision-making cycle. So that's why it really plays an important role in enhancing our jobs and our lives.

6. The most promising fields

A main advancement of big data is healthcare, where there are more use cases on how we can provide better healthcare to a higher number of people and how the data can be used to do more things automatically. The thing that I like about artificial intelligence and machine learning is that it is making all of us more intelligent with its application. This technology will play a very important role in the coming years with the large amount of data at our disposal. According to Andrew NG, a leader in the big data space, it is not so much about having the best model but about having the quality data.

7. An interesting use of big data analytics

Recently, I read an interesting news case on the creation of an early-warning system to determine which animals are endangered species. With the help of sensors and cameras, we can now collect data on these animals and their locations. This application of big data is helping to save these endangered species. Another area big data is impacting is the creation of autonomous cars. We are now able to collect data on different drivers and traffic patterns to be able to generate an algorithm for the self-driving car.

8. Misconceptions of big data

There are quite a few misconceptions when it comes to big data. A big one is that many times we feel that big data will solve all our business problems and our problems in general. However, it's not always so easy to derive insights from data. It requires strategy and the right skill sets and understanding. It's basically about having the business and technical domain experience to be able to make use of all this data. It's not as simple as just pushing a button and immediately getting insight. You must become very efficient in applying big data.

9. Obstacles

From capturing such a large amount of data to storing it, there are challenges in every step. The other issue is getting real-time analysis out of this data. Real-time analysis is required to be able to understand the data and then to be able to apply it. If you find some insight which is already out-of-date that's not really going to help you, or your company improve. So being able to capture the data, being able to store that data, and analyzing it in real-time is crucial.

Then, of course, privacy and security are other critical factors to consider. Businesses also need to take precautions to make sure that the data is securely moving, whether it is being captured or stored – what encryption mechanism is being used, whether we can satisfy the auditing requirements while the data is changing, and who is accessing it when and from where. So, I would say, even though there are many challenges in big data, there are ways to tackle some of these challenges.

10. The future of big data analytics

We are a better version of ourselves with the use of big data, and that applies to everybody and that applies to every job. If something is more appropriate to be automated and more value can be derived by doing something else, then we should be focusing on more value creating jobs. That's where big data plays an important role. It is making us more effective in the jobs that we do so that we can focus on things which are more value adding than what, let's say, we were doing ten years back.