EMC: Big Data Crippled By Huge Skills Shortage

The opportunity to profit from the intersection of big data and data analytics is huge, but a new study from EMC reports a "rampant scarcity" for the prerequisite skills necessary for a company to capitalize on these opportunities. Only one-third of the companies participating in the EMC Data Science Study--covering the United States, the United Kingdom, France, Germany, India and China--are able to effectively use new data to assist their business decision making, gain competitive advantage, drive productivity growth, yield innovation and reveal customer insights, says the storage giant.

The big data era has arrived in full force, bringing with it an unprecedented opportunity to transform business and the way we work and live, says EMC. Through the convergence of massive scale-out storage, next-generation analytics and visualization capabilities, the technology is in place. What’s needed to fully realize its value is a vibrant, interconnected, highly skilled and empowered data science community to reveal relevant trend patterns and uncover new insights hidden within.

According to Gartner, big data is a term used to acknowledge the exponential growth, availability and use of information in the data-rich landscape of tomorrow. Worldwide information volume is growing annually at a minimum rate of 59% annually.

A year ago, IDC predicted that the IT industry's next dominant platform--built on mobile computing, cloud services, social networking and big data analytics technologies--would begin its transition into the mainstream. This week it reported that spending on these technologies is growing at about 18% per year and is expected to account for at least 80% of IT spending growth between now and 2020 (IDC Predictions 2012: Competing for 2020).

IDC expects big data to earn its place as the next "must-have competency" in 2012 as the volume of digital content grows to 2.7 zettabytes, up 48% from 2011. More than 90% of this information will be unstructured (for example, images, videos, MP3 files, and files based on social media and Web-enabled workloads) and full of rich information, but challenging to understand and analyze.

As businesses seek to squeeze high-value insights from this data, IDC expects to see offerings that more closely integrate data and analytics technologies--such as in-memory databases and business intelligence tools--move into the mainstream. And, like the cloud services market, 2012 is likely to be a busy year for big data-driven mergers and acquisitions as large IT vendors seek to acquire additional functionality.

The EMC study revealed that the explosion of digital data created by the likes of mobile sensors, social media, surveillance, medical imaging and smart grids combined with new tools for analyzing it all has created a corresponding business demand for data scientists that has outpaced the supply of talent. The survey included nearly 500 respondents from the global data science community, including data scientists and professionals from related disciplines such as data analysts, data specialists, business intelligence analysts, information analysts and data engineers. All have IT decision-making authority.

The major barriers to data science adoption include lack of skills or training (32%), budget/resources (32%), the wrong organizational structure (14%) and lack of tools/technology (10%). Only 12% of respondents saw today’s business intelligence professionals as the most likely source to meet that demand, although the survey found an increasing need for data scientists in their firms.Scripting languages, including Python, Perl, BASH and AWK, are more likely to be used by data scientists rather than BI pros, states the study. However, Excel, followed closely by SQL, remain the tools of choice for both scientists and executives.

See more on this topic by subscribing to Network Computing Pro Reports The Data Mastery Imperative (free, registration required).