The disruptive forces converging Enterprise Computing, HPC & Big Data
Enterprise Computing is the driving force behind many of the major Cloud Provider’s strategies today. Sure, it was small businesses and startups that drove the early innovation on Amazon Web Services but the Enterprise category is responsible for audacious predictions by the likes of Gartner, forecasting $107billion in Public Cloud spending by 2016. Enterprise Computing is roughly an $800billion business so it’s not surprising that we’re seeing this sort of momentum given the industry is migrating to the cloud on-masse. HPC on the other hand is a relatively niche market today. The overall spend is somewhere around $34billion. Cloud adoption has been modest to-date but is rapidly accelerating. Big Data, a term almost as overloaded and ambiguous as Cloud, is the fastest growing IT sector right now. At least in hyperbole if nothing else. Analyst CAGR numbers are typically around 40% which is pretty remarkable.
There are two disruptive forces at play that will ultimately see the convergence of HPC, Big Data and Enterprise computing.
The hype surrounding Big Data has a lot to do with the value this offers to the Enterprise. As the quantity and complexity of unstructured data increases, so does the need to process it. Businesses are finding clever and innovative ways of turning this data into a source of revenue, not directly, but through a deeper understanding of their business, their customers and the habits exhibited by them.
Typically large silos of structured and unstructured data are spread across various divisions and applications within the enterprise. Big Data technology allows questions of this data to be more easily asked to gain insight that allows companies to serve their customers better and compete more effectively.
So Big Data is about performing analysis and computations against large amounts of data. What about HPC? Yep, same thing. Sure, there are some important distinctions and subtleties that I won’t delve into here but the point is we’re already seeing a strong convergence of Big Data and HPC technology.
Putting Big Data hype aside for a moment, the second, and most important, disruptive force at play is Cloud Computing. The cloud offers many advantages over traditional on premise computing that make HPC-like capabilities far more accessible and affordable. Here is a summary of some of the key benefits as they relate to HPC-like workloads;
Cost: For spikey workloads, the cost savings in the cloud can be significant. What’s more, the cost is an operational rather than capital expense so is often more palatable for many businesses as costs can be attributed to a particular project.
Ease of use: The cloud can facilitate easy dynamic provisioning for specific workloads. The ability to have OS/Software configurations particular to a workload is a key benefit.
Speed of deployment: The ability to rapidly provision new environments/clusters in minutes is incredibly empowering to many businesses.
Scalability: Elastically scaling out to meet increased capacity demands is a powerful concept. The public cloud promises “infinite” scale. The reality is somewhat different: there are some real limits even in the cloud. But the computing capacity that can be obtained from large providers such as Microsoft Azure is far greater than what most users can muster with internal hardware.
Resiliency: The ability to snapshot workloads as they are running can allow for check-pointing of MPI workloads. Combine this with active monitoring and the ability to dynamically move a guest VM from one physical host to another, and your workloads can keep running even in the face of hardware failure.
Portability: The ability to move a workload from one cloud platform to another on the fly without any application changes allows for powerful options such as for bursting from a private cloud out to a public cloud, High Availability where a workload is run on multiple clouds simultaneously, scaling across multiple clouds to meet extremely high resource requirements or, to take advantage of shifts in the spot pricing market.
In summary, enterprises across many industries have much to gain in adopting Big Compute and Big Data technology and taking advantage of the many benefits cloud computing offers for these applications. This is why I predict wider adoption and convergence of these technologies is inevitable. Software Vendors and Cloud Service Providers that are building solutions for this market and addressing the challenges around data management and governance will be positioned very well.
Follow me on twitter:@dave_fellows
