Resource Management, Programming Tools, Task Monitoring, Data Analytics, & Web Security
Apache Mesos is an open source project that abstracts the CPU, RAM, storage, and other computing resources available across multiple hardware instances in a data center in a unified manner so that systems administrators can operate the entire network as a single computer. Apache Mesos provides a set of unique APIs for resource management in cloud computing that can scale to over 10,000 nodes, integrating with other frameworks such as Hadoop, Chronos, Spark, Cassandra, & Kafka for complex web/mobile app deployments. Apache Mesos is used by Apple, Microsoft, PayPal, Twitter, Verizon, Samsung, Netflix, eBay, Bloomberg, AirBNB, Yelp, Uber, China Mobile, & many other companies for their data center management requirements. Benjamin Hindman, one of the principle founders of Mesos/Nexus at UC Berkeley, launched the start-up company Mesosphere in 2013 which produces DC/OS (Datacenter Operating System) as a production ready distribution of the code designed for enterprise use. Mesosphere DC/OS includes elastic cluster resource management tools that enable cloud orchestration at scale using Docker Swarm, Kubernetes, or Marathon. The most recent release of Apache Mesos (version 1.5) includes support for the Container Storage Interface (CSI) that allows the system to automatically create storage partitions based on preset application parameters with multi-cloud platform compatibility. The new release also includes better configuration support for Windows environments, a standalone container format, and improved container image garbage collection.
Apache Mesos & Mesosphere DC/OS – The Datacenter Operating System
Apache Mesos began in 2009 at the University of California – Berkeley with the research project named “Nexus” developed by Ben Hindman, Andy Konwinski, & Matel Zaharia. At this time, Google had not yet released the code for Kubernetes to the public and many corporations were searching for a way to replicate their “Borg” functionality for large scale data center management in IT corporations. In 2010, the trio gave a presentation at Twitter about the project and released an academic paper entitled “Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center” on the subject. The same year, Mesos entered the Apache incubator on open source licensing foundations for further development by programmers internationally. The use of Mesos in large scale data center management was adopted by Twitter, eBay, PayPal, and many other start-up companies for cluster server deployment, meeting CI/CD requirements in Agile project management, automating system tasks, & scheduling aggregated resources for production applications. Many of the largest telecom and IT companies now use Apache Mesos & DC/OS as a leading industry standard. For example, Microsoft has deployed it as part of their Azure Cloud product suite, Apple uses it to run Siri with J.A.R.V.I.S, Netflix implements it for their Titus cloud container orchestration requirements, and Uber has developed their “big data” analytics platform around it. Apache Mesos & DC/OS are recommended by devops teams in these companies for the extensive Jenkins, Chef, Puppet, & Ansible integration facilities provided, with the ability to run in both private cloud and public cloud environments agnostically.
Mesosphere DC/OS: ‘Concepts like microservices, containerization, “fast data” analysis and response, distributed computing, and edge data collection and response were groundbreaking concepts when the company began in 2013. Building on top of Apache Mesos, Mesosphere sought to bring together all of the tools needed to operate data-intensive modern applications such as container orchestration, distributed databases, message queues, data streaming and processing, machine learning, monitoring and management capabilities, security tools, deployment automation, and more. Since its launch in 2015, the Mesosphere DC/OS “operating system” has made it easy to deploy, connect, and elastically scale over 100 open source and commercial services with a single click, and underpins everything from web-scale applications, to IoT and autonomous cars, to banking and trading systems.’ Learn More About Kubernetes on Mesosphere DC/OS.
Container Orchestration: Docker Swarm, Kubernetes, & Marathon
Mesosphere DC/OS is extremely flexible under open source licensing guidelines and modular under object-oriented principles so that third-party programmers, development teams, & corporations can build extensions for the platform that allow it to be customized in new ways. Similar to Linux, Apache Mesos functions as the “kernel” unifying all of the data center hardware resources with the required utility software, where Mesosphere DC/OS is the operating system that can be installed with different applications on the top layers. Mesosphere DC/OS stacks integrate with Apache Hadoop, Spark, Cassandra, Kafka, Infinity, Velocity, Zeppelin, etc. so that new web/mobile apps can be developed including functionality that would not be possible using other software tools or database platforms. Mesosphere DC/OS saves money for companies by not requiring them to rebuild all of the fundamental elements for data center management repeatedly on top of the Apache Mesos codebase, and the platform is relatively simple to use “out of the box” for a wide range of different scenarios. For container orchestration, developers have the choice of Docker or Mesos’ own native container standard that is part of the core distribution. Similarly, organizations can choose to operate with Docker Swarm, Kubernetes, or Marathon for cluster server management, resource allocation, scheduling, synchronization, and artifact staging in web/mobile app production. Marathon guarantees high availability with 100% uptime for multiple container runtimes, stateful apps, service discovery, load balancing, health checks, event subscription, virtual IP routing, and metrics. Mesosphere DC/OS can be installed in private cloud data centers running together with VMware vSphere or OpenStack integration, as well as on public cloud platforms like AWS, Microsoft Azure, and Google Cloud Engine. With this system, companies can configure their resources to support multiple development teams and software product deployments with identity management, access control, cluster-wide connected device encryption, and reduced privilege escalation.
Microservices: “Data pipelines rely on multiple tightly-coupled technologies in order to be successful. Building a data pipeline involves integrating varied technologies such as distributed analytics engines (Apache Spark), distributed message queues (Apache Kafka), and distributed storage systems (Apache Cassandra). However, many organizations struggle to build and maintain data pipelines because they are exceedingly complex they are comprised of multiple components that require careful operation and administration to obtain the maximum value and avoid data loss… Large-scale web organizations pioneered the use of a data center cluster managers such as Apache’s Mesos to efficiently build and maintain complex distributed systems. Mesosphere DC/OS, powered by Apache Mesos, helps data engineers and DevOps teams simplify the deployment and operations of data pipelines on any infrastructure.” Learn More About Using Apache Mesos with Cassandra, Kafka, Zeppelin, & Spark.
Apache Mesos & Mesosphere DC/OS – Resources
As Apache Mesos & Mesosphere DC/OS have become widely adopted in enterprise data center management and cloud orchestration for web/mobile apps, there has grown to be an increasingly large number of learning resources for systems administrators, programmers, and developers who are seeking to work with the platform. Foremost among these are the academic papers published by the founders which articulate the basic principles and philosophy behind the code:
- Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center (2010)
- The Datacenter Needs an Operating System (2011)
- Omega: flexible, scalable schedulers for large compute clusters (2013)
- An Architecture for Fast and General Data Processing on Large Clusters (2014)
Additionally, there are a number of excellent books, slideshows, videos, and use case scenarios on the topic, with GitHub the recommended resource site to access code and information on third-party projects that plug into Mesosphere DC/OS to extend its functionality.
- Apache Mesos Essentials – Dharmesh Kakadia (2015)
- Big Data SMACK: A Guide to Apache Spark, Mesos, Akka, Cassandra, and Kafka – Raul Estrada & Isaac Ruiz (2016)
- Mesos in Action – Roger Ignazio (2016)
- Building Applications on Mesos: Leveraging Resilient, Scalable, and Distributed Systems – David Greenberg (2016)
- Cloud Native Infrastructure: Patterns for Scalable Infrastructure and Applications in a Dynamic Environment – Justin Garrison & Kris Nova (2017)
- Introduction to Apache Mesos (Thomas Barton)
- Introduction to Apache Mesos (Joe Stein)
- Scaling Like Twitter with Apache Mesos
- John Wilkes – Google Faculty Summit (2011)
- John Wilkes – Omega: flexible, scalable schedulers for large compute clusters (2013)
- Container Pods with Docker Compose in Apache Mesos (PayPal)
- MesosCon Europe 2017
Use Case Scenarios:
Apache Mesos: “Mesos consists of a master daemon that manages agent daemons running on each cluster node, and Mesos frameworks that run tasks on these agents. The master enables fine-grained sharing of resources (CPU, RAM, etc.) across frameworks by making them resource offers… The master decides how many resources to offer to each framework according to a given organizational policy, such as fair sharing or strict priority. To support a diverse set of policies, the master employs a modular architecture that makes it easy to add new allocation modules via a plugin mechanism. A framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework’s tasks… While the master determines how many resources are offered to each framework, the frameworks’ schedulers select which of the offered resources to use. When a framework accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding agents.” Learn More About Apache Mesos Architecture.
Mesosphere has received $122 million USD in venture start-up funding since 2013 from Microsoft, Hewlett Packard, Khosla Ventures, Andreessen Horowitz, Fuel Capital, and other firms to continue the development of the DC/OS platform. The company has hosted four MesosCon conventions in North America, three in Europe, and two in Asia allowing companies to present their applications and working methods to the public. Follow Mesosphere on Twitter.