How does a successful big data project look like?

A successful big data project is  more about articulating a testable hypothesis, designing that test, and evaluating the results. Big Data is science and Big Data project is more of a scientific method. Without the business context, the big data query is simply an academic exercise.

Most of the big data project in the enterprise start with marketers and business men asking the question  “How do I capitalize on the currently available internal data ?“.  Their objective generally is to get better understanding the usage and preference trends of its current customers.  Big Data teams needs to have both business men and the engineers.

The first question for stakeholders to tackle is  “Understand the available data and Where You Want To Go”. The engineer needs to understand what is data currently available and how it is structured  and they do not know the insights that are needed by business.  Business does know what data is available and what is possible and did not know to phrase the question in meaningful way.

The engineer needs to share the currently available data with business men. The business men need to collaborate with the engineers to arrive at different scenarios to tie the available data together and transform the same to meaningful data. The engineer might have to handle huge amount of the data to get meaningful results and need to be open to connect and correlate data  from various data sources(different formats and schema) to find answers to meaningful big data question.

While the business men look for faster answers, they need to be aware of the fact that it takes a longer time to analyze data from various data sources. The greater the time period the data is available, the greater the complexity of structures and formats you might encounter. The business men need to articulate the business requirement in place, the data you want to get back, and the hypothesis that is tested with this exercise. The business mean needs to patient to answer long questions asked by engineers and be part of lengthy discussions, different  from the quick questions and answers in the sales and marketing functions.

Irrespective of the side you are in the big data question,  the success of the big data solution depends on all the stakeholders. The engineers develop tools to handle the business requirements and defining process to deal with the data you collect and arrive at analysis.  The business men try the tool, examine the result, and most of the time, the engineer needs to iterate again to make tool better or redo the analysis again.The business men keeps asking relevant questions and  the engineer keeps displaying different results and  this feedback loop enables to obtain better results.