Executive Summary
We live in a Big Data environment today. The term “Big Data” refers to massive amounts of data in multiple formats generated inside and outside organizations on day-to-day basis. The three main characteristics of Big Data are commonly referred to as 3V's: (1) volume, i.e. massive amounts of data; (2) variety, i.e. structured, semi-structured and unstructured data generated from multiple sources; and (3) velocity, i.e. data generated at fast speeds. The explosive growth of semi-structured and unstructured data — just think of social media postings — is a key factor that lies behind the Big Data phenomenon. It is estimated that semi-structured and unstructured data account for 80% to 90% of all data. Traditional data technology cannot handle these data efficiently and effectively, and this implies that organizations were mainly relying on just 10% to 20% of all data to guide their decision making in the past. The emergence of Big Data technologies has enabled organizations to leverage Big Data and discover previously hidden insights.
Big Data can provide richer, more granular and faster insights to power smarter and faster decision making in a wide range of business problems. Procter and Gamble (P&G) searches for new product ideas from customers' comments on social media; Merck, a US-based pharmaceutical firm, leverages Big Data and predictive analytics for deeper root-cause analysis to improve production yield; United Parcel Service (UPS) analyzes real time road traffic data for route optimization; and Celcom delivers targeted promotions to customers based on their actual behaviors. Several reputed research centers found that firms with higher business analytics competency outperform those with less. International Data Corporation (IDC) found that analytically oriented firms are 20% more likely to be among the most competitive within their respective industries. According to Gartner, firms that use predictive analytics may increase profitability by 20% by 2017. A joint research by MIT Center for Digital Business and McKinsey's Business Technology found that data-driven firms perform on average 5% more productive and 6% more profitable than their competitors. These findings show that there is a strong case for Big Data adoption.
Big firms are generally good at leveraging data in their decision making. In contrast, smalland medium-sized enterprises (SMEs) are generally less data-driven and rely more on gut feeling and past experience in their decision making, hindered by smaller data sets and budget for technology investments. Nevertheless, in today's Big Data environment with increasing availability of quick-to-deploy, affordable and flexible technology products, SMEs have better and cheaper access to data, and this represents a big opportunity for them to develop Big Data capabilities to win in the ever more competitive marketplace. SMEs can leverage cloud-based products which require no hardware, no setup time and can be 6 deployed with just a few mouse clicks. These products also have affordable and flexible pricing plans that allow users to make incremental increase as their needs grow. Some cloud-based analytics products also provide access to external data, e.g. Google Analytics provides access to web traffic data and IBM Watson Analytics provides access to Twitter.
Big Data adoption is picking up across the world with more and more firms jumping on the bandwagon. IDC forecasts the global Big Data technology and services market to grow robustly at a compound annual growth rate (CAGR) of 23.1% from 2014 to reach USD48.6 billion in 2019. However, there is a huge Big Data talent shortage. The US alone is expected to face a shortage of 140 000 to 190 000 workers with deep analytical skills as well as 1.5 million managers and analysts with know-how of Big Data analysis to make effective decisions by 2018. As a late comer in the industry, Malaysia too lacks Big Data talent. According to Malaysia's Minister of Higher Education, there were about 4 000 Big Data scientists in 2014 and an additional 12 000 will be needed in the next five years. SMEs are generally seen as less attractive employers compared to big firms. It will therefore be less attractive for SMEs to embark on Big Data projects since it is useless to invest in the necessary infrastructure if no one is there to make sense of data and make insights actionable. SMEs which lack internal Big Data talent can consider engaging external consultants instead of developing an in-house team, or turn to Kaggle for help — an online platform that matches data-related requests with data scientists from around the world at a far more affordable cost.
It is a tough task to deliver value from Big Data, even for big, data-driven firms. Many big firms which are committed in Big Data and advanced analytics failed to achieve the big impact they expected. Nevertheless, there are also plenty of successful examples where firms have been able to improve their top-line and bottom-line performances. To increase chances of success, firms should develop a data-driven culture throughout their organizations; carefully evaluate the business case of their proposed Big Data projects; and clearly define the specific outcomes, i.e. think of what to ask of the data, how the firms will react to the answers, and what are the actionable operational measures.
Data is the new oil. Firms that are not data-driven risk of being outcompeted by those that are. The complexity of today's business environment warrants them not to only rely on gut feeling and past experience but to adopt a more data-driven approach in their decision making. Big Data can be the new strategy to help firms stay competitive and so they should invest in Big Data technology, talent and culture to prepare themselves for the Big Data era.
