Big data provides a way of predicting behaviour. Does it provide a means of understanding behaviour pattern? You have large amount of data, technology and algorithm know-how to apply on the data and predicted some insights. when the prediction goes out of way from actuals, the business owner is interested to know and where is the error and how and what needs to done to mitigate the same in future. How to identify origin of the error? data or technology or algorithm? On success of the prediction, the business owner wants to know what is customer behaviour leading to the success and also would want to validate success with real data points from real customers.
A billion things can correlate with each other, depending on how the data is structured and what are getting compared as part of the analysis. While correlation helps to derive meaningful interpretation from meaningless ones, it needs a analyst with the understanding of the domain to come with causal hypothesis about what is leading to what. Humans has their own limits and correlation needs human inputs and there is limit of correlation.
“Data is Bad” means that the data has not been collected accurately or consistently or the data has been defined differently. The huge amount of “bad” data that arrives as input for analysis makes it difficult for system to independently observe linkages. There comes efforts of the analyst to focus in on what is important to the business and identify relevance of data islands and dismiss spurious relationships in the data. Humans come with their bias and preferences.
In retail space, the analysis of Big Data helps to maximize profits by predicting what people will buy and retailer wants to focus on profits (and therefore shareholder profits and owner profits). They might also want analyst to identify customer segments to enable cross selling of new products to specific customer segments. Will they also focus on customer service? When you look in pharma space and the company want to establish causes for flu outbreaks, diagnosis. We need to examine whether the question is ” How to profit from misery of a disease?” or “How to eradicate the disease? How to reduce the cost of drugs to cure the disease?” The problem does not lie in Big Data or act of analysis of big data. It lies in the question to frame problem and find the solution.
When you look at the fastest route to go from home to office, Big Data can help a straightforward decision. For subjective ones like who to get married and what career to purse and what college to join involves to find things that are in harmony with person. If you are relying just on data, there is a tendency to trust preferences and anticipate a continuation of what is happening right now. if the reality is misinterpreted wrongly, there might be a vicious cycle of self-enforcing feedback. Are you looking for patterns of error in addition to patterns of preferences?When
when faculty is provided with report of which students are likely to fall behind. To actually help the student, you might need to personally intervene to help that student from the world of responsibility. Let us acknowledge that in regular world, when humans are asked to perform x, they perform y.While big data can help the players to see patterns that they miss otherwise, the intuition of the players and the coach continue to play a larger role. Big Data might be good at telling what to pay attention to.
While Big Data focus on numbers, it is important to focus on the needs of the business and to convey story that conveys benefits to the business owner. Where is the talent to come with the story and share the same is a entirely different topic.