Success factors for Big Data Projects 

In this age of digital transformation and fast access to information, companies are constantly striving to find that competitive edge in order to attain more market share and achieve their business goals. The recent rise of Big Data and the field of business analytics has been a major transformative force into providing more valuable insights to a company’s performance. When done right, business analytics can allow discovery of new market opportunities or underline some unknown negative outcomes. Randy Bean of The Harvard Business Review surveyed top industry executives and found that 80.7% percent of the responses confirmed the successful investment that was made in Big Data.

One has to wonder however, whether such initiatives did not succeed in achieving the required goals. After all, while many technologies are “open source” and available for free to software developers, the expertise and creativity are still needed in order to undertake a Big Data project. Companies like MongoDB, a leading NoSQL data storage outfit, offer their solution for free but charge a sizable rate for hardware support and maintenance costs [1].

At times, when companies hear of a buzzword like Big Data, they jump on it without prior planning, which is the main reason that Bernard Marr cites for project failures [2]. Lack of clear objectives coupled with cost overruns may derail analytics projects and render the outcome ineffective. Google had an analytics tool dubbed Google Flu Trends (GFT), which was supposed to predict outbreaks of flu and fever based on users’ search patterns. But, due to inaccurate aggregation and data cleansing, the GFT tool missed the mark in 2013 in a high flu season cases, and now the tool has been retired [3].

While data scientists enjoy lucrative careers, they are still a scarce skillset to be found in the technology talent pool. Various technologies ranging from predictive analytics, data visualization and forecasting have become essential tasks in business analysis and company strategy across multiple industries. The expansion of Internet of Things and ubiquity of Cloud services will bode well for data enthusiasts in the job market. Major employment pockets in California and the East Coast will continue attract analysts for a hefty annual salary exceeding $100,000 [4]

In the end, I think the field of Big Data needs more time to mature. The merging of talented business folk with technologists will enable creative ideas and spurn new product development. I think it starts with a company’s culture and desire for introspection, as well as tapping into their customer base to get more meaningful insights on their sales performance.

[1] “A Total Cost of Ownership Comparison of MongoDB & Oracle, ” A MongoDB White Paper, June, 2016.

[2] Marr, Bernard.  “Why Big Data Projects Fail.” Forbes. March 17, 2015. Accessed June 2, 2017. https://www.forbes.com/sites/bernardmarr/2015/03/17/where-big-data-projects-fail/2/#3cee04c768b5

[3] Lazer, David. Kennedy, Ryan. “What We Can Learn From the Epic Failure of Google Flu Trends.” WIRED.  October 1, 2015. Accessed June 2, 2017. https://www.wired.com/2015/10/can-learn-epic-failure-google-flu-trends/

[4] Columbus, Louis. “Where Big Data Jobs Will Be In 2016.” Forbes. November 16, 2015. Accessed June 2, 2017. https://www.forbes.com/sites/louiscolumbus/2015/11/16/where-big-data-jobs-will-be-in-2016/#46fb538c608c