Mesosphere's new big data solution: Add Spark, hold the Hadoop
Mesosphere, corporate backers of the Apache Mesos cluster-management project, are ginning up a big-data stack that eschews Hadoop, but embraces Spark (and Kafka, and Cassandra, and the Akka event framework) for real-time processing.
Mesosphere Infinity, as the new product is called, is "a turnkey, full-stack offering optimized for big data and IoT," and its main aim is to provide an easily erected stack for businesses for real-time data work. But it also stands as a recent example of how many of the technologies reflexively associated with the Hadoop stack don't require Hadoop to be useful.
Matt Trifiro, chief marketing officer for Mesosphere, explained in a phone conversation how Infinity is managed by wanother of Mesosphere's creations -- Mesosphere DCOS, which allows entire data centers full of applications to be stood up easily. Infinity, in turn, is for managing a relatively narrow range of applications: Spark for data processing; Kafka for real-time data ingestion; and another Apache Foundation project, Cassandra, for data storage.
While Infinity "doesn't exclude Hadoop," said Trifiro, "it doesn't require it, either. You can use [Hadoop's] HDFS as a persistent data store, and you may have Hadoop processing over data pushed into Cassandra, but in terms of real-time acquisition, you need a specialized stack."
Spark has drawn attention as of late from a roster of A-list technology firms, all interested in both investing in the project and leveraging it for heavy-duty business analytics work. Still, like many other open source data tools, Spark is by itself far more "project" than "product," meaning it isn't a trivial effort to use in an enterprise environment.
Trifiro claims Spark and the rest of the Infinity stack "was built from observation of what people were putting into production." Businesses were attempting to put together Spark and Kafka stacks for real-time analysis, said Trifiro, because "the demand for processing real-time data by non-Web companies is relatively new, and there's immense pressure on IT teams to do this." Standing up an entire such stack has "historically required a lot of expertise," and Infinity is meant to require minimal work to get up and running.
Another way Mesosphere plans to make Infinity's stack even easier to consume is to offer it by way of existing cloud services. Right now, though, the only named partner for cloud-based enterprise distribution is Cisco, the same company that worked hand-in-hand with Mesosphere to build Infinity.
One possible analogy is with running applications in containers, versus using virtualization and OpenStack. Containers offer a potentially more precise solution to the problems of running applications at scale than VMs did. Likewise, Spark alone, as opposed to Spark plus Hadoop, might present a better fit for the data-processing problems faced by enterprises -- as long as deployment and management of a Spark stack doesn't just put them back to square one.