On the eve of Twillio IPO, here’s an Oldie but a goodie - Build your own Twillio Platform leveraging #Mesos

Andulka
Cosimo Galluzzi
Alisa U Zemlji Chuda

roma★

tannertan36
cherry valley forever
TVSTRANGERTHINGS

Origami Around

izzy's playlists!

★
NASA
YOU ARE THE REASON

shark vs the universe

Discoholic 🪩
h
tumblr dot com
Today's Document
🪼
he wasn't even looking at me and he found me
Monterey Bay Aquarium

seen from United Kingdom
seen from United States
seen from United States

seen from Japan
seen from Russia
seen from Brazil

seen from United States
seen from United States

seen from United States
seen from Germany
seen from Puerto Rico

seen from United States

seen from United States
seen from Italy
seen from United States

seen from Malaysia

seen from United Kingdom
seen from Malaysia
seen from United States

seen from Canada
@strikesme
On the eve of Twillio IPO, here’s an Oldie but a goodie - Build your own Twillio Platform leveraging #Mesos
Why I Joined Mesosphere
You better lose yourself in the music, the moment
You own it, you better never let it go
You only get one shot, do not miss your chance to blow
This opportunity comes once in a lifetime
– Eminem
After many conversations with the founders of Mesosphere, I had the aha’ moment one late saturday evening when I realized the Mesosphere was the ‘opportunity’ that Eminem talks about above and I knew that I had pass on a carefully planned career break faster than a Japanese Shinkansen bullet train and get on the Mesosphere Rocket ship. In other words:
When you are given a seat on a Rocket ship, you don’t ask what seat? you get on it!
– Sheryl Sandberg
Operating Systems 101
It was a chilly winter morning at The University of Texas at Austin when I showed up for my Operating Systems class -- CS 372. The class was taught by Professor Mike Dahlin ( Current Principal Engineer of Google’s Cloud Platform ) and there I attended this particular lecture in CS 372 to realize the foundational elements and functions that are required of an Operating System are:
Co-ordination
Security or Isolation -- This is the notion of processes that are isolation boundaries.
Communication -- Inter-Process Communication (IPC) or sockets for network programming.
Resource Management -- The heart and soul of the Co-ordination service is the resource manager that carves out CPU, Memory and other scarce resources.
Abstraction & Standard Services
Abstract the physical reality (hardware) with convenient interfaces for Developers
Provide “Standard Services” to enable developer to focus on specialized programs.
As you can see, when dealing complex and novel concepts and ideas such as the idea of Datacenter Operating System, I tend to go back to first principles and piece together the new idea to make sure that the new idea doesn’t break first principles.
Why the Datacenter needs an Operating System?
Now, before getting into “What is a Datacenter Operating System”, I would like to go over why does the world need a Datacenter Operating System?
First, our world is going through a massive transformation powered by the ubiquitous connectivity that modern Mobile phones have enabled. A world where there are now Billions of “computing endpoints” or Smart Phones which is orders of magnitude more connections into datacenter when compared to the previous Personal Computer (PC) generation. A world where the amount of data being generated is growing exponentially. A world where customer expectations are moving to real-time request fulfillment -- be it in Media ( Netflix), Transportation (Uber) or Real Estate ( AirBnB)
Secondly, our Datacenters today run at under 10% utilization, most Datacenters operate leveraging Virtualization and Systems Management tooling at best and manual provisioning at worst.
So, the challenges of Mobile-Cloud era seem enormous but the way our current Datacenters operate, without a Datacenter Operating System, seems antiquated.
Whats the solution?
The Mesosphere RocketShip & Datacenter Operating System (DCOS)
If you understood the first principles of Operating System design above then you already are further along than most folks in understanding Mesosphere’s Datacenter Operating System -- DCOS.
Mesosphere’s team has taken the Open Source Apache Mesos kernel and created an Operating System just as I described in OS 101 section above but at the Scale of an entire modern datacenter consisting of thousands and thousands of physical or virtual machines.
DCOS is the Operating System which abstracts away the underlying “physical reality” and provides the convenience of an API, CLI & UI to the Datacenter developer who will have the entire datacenter at her disposal to run massive scalable new generation applications such as Spark, Hadoop, Cassandra, MemSQL, Kubernetes or migrate some of the existing application onto the same Datacenter OS that runs your new generation Cloud Native Applications.
Mesosphere DCOS runs on top the “physical reality” which can be physical machines, virtual machines or IaaS Clouds, then it layers DCOS software on top which makes the entire capacity behave as a single datacenter with isolation/security, resource management & communication services built into the OS and “standard services” such as datacenter scale package management, init system etc pre-installed. The killer distributed apps of the Datacenter such as hadoop can be installed with a simple CLI command.
All in all, DCOS makes Jill the developer a Super datacenter developer and Jack the operator a Super datacenter operator.
But, don’t take my word for it, give it a try over here.
eBay, Paypal, NTT, DreamHost, Aptira providing a retrospective on bringing OpenStack to Enterprise environments.
It's now 2014, 3 years since I first authored the blog posts below introducing Neutron (then known as Quantum) for the world. In this year's Summit, I recap our progress and differences between Nova-Network & Neutron. Learn and decide for yourself if Neutron is ready for you.
OpenStack, Quantum and Open vSwitch – Part III
[ Originally authored in July 2011 on Openvswitch.com . This is the final post in a three part series to describe OpenStack, its new networking stack called Quantum, and the first Quantum plug-in which is based on Open vSwitch.]
Open vSwitch is a production quality, multilayer virtual switch licensed under the Open Source Apache 2.0 license. Open vSwitch can operate both as a soft switch running within the hypervisor, and as the control stack for switching silicon. It is the default switch in XenServer “Project Boston”, theXen Cloud Platform and also supportsXen,KVM,Proxmox VEandVirtualBox.
Open vSwitch is also the first Open Source and built-in Quantum plugin that is available out of the box. The way it operates is to manage all Open vSwitch instances within the compute hypervisor (whether Xen or KVM) to create a multi-tenant L2 overlay over the physical network. The plugin itself has no requirements of the underlying physical network beyond basic IP reachability, therefore it should work equally well on an L2, L2/L3 or pure L3 network while still providing the guests with an L2 service model.
The operation is equally straighforward. The plugin contains a “controller” which exposes and API that contains the plugin methods for creating/deleting/managing logical networks. On tenant creation, it will create the necessary tunnels to provide that tenant with an L2 service model. If VMs move or are taken down, the controller will similarly update the state.
Below, we will dig into the details a little.
Key components of the Quantum Open vSwitch Quantum Plugin:
Quantum Controller Plug-in: The Controller is the brains of the Open vSwitch Quantum plugin. It is responsible for exposing all “tenant” and “administrator” APIs. This module essentially fulfills requests by “tenants” and “administrators” to create/delete/update isolated “virtual” L2 networks, ports on virtual networks and manage connectivity of “Network Interfaces” to these virtualized Quantum networks.
The Controller achieves the Quantum Networking model by essentially maintaining a centralized data model repository that contains the “virtual” network view of the OpenStack Cloud. This repository contains the the entire “virtual” network topology consisting of various isolated L2 Networks, an inventory of ports on these L2 Networks, and an account of all network “interfaces” that are plugged into these Quantum networks as well as the admin status of each “virtual” network link i.e. If the link is “Up” or “Down”
As API requests come in, Quantum updates its centralized data model and upon completion, the API request succeeds. At the same time, all Open vSwitch agents, “phone home” to check on the state of Quantum’s centralized network model. If there are any changes to the centralized network model then those changes are pushed down to affected Open vSwitch agents.
Network Data Model: The data model is a data store that is responsible for modelling and maintaining the representation of various network elements contained in the cloud network fabric, such as:
* Virtual Quantum Networks
* Ports on Quantum Networks
* Network Interfaces attached to Quantum Networks
* Link status of each Network Interface that is attached to a Quantum Network
Open vSwitch Quantum Agent: The agent is responsible for establishing and maintaining a communication channel with the centralized Quantum controller. This agent is also responsible for retrieving the changes made to Quantum’s data model and then subsequently communicating those changes to Open vSwitch to establish network connectivity and network isolation based on the network topology that has been specified in Quantum controller’s centralized data store.
Current Status
Core Quantum API - The Core Quantum API consists of “tenant” and “administrator” facing APIs and relevant infrastructure to map those APIs to backend plugins. Quantum Core API is currently in beta and is available for early adopter to test and create interesting Cloud Networking topologies. Launchpad blueprint tracking this effort is available here.
Core Infrastructure - The Quantum Core Infrastructure consists of infrastructure “glue” that ties components together as well as daemonizes the Quantum service and keeps the service running reliably. Those who are interested in contributing towards Quantum’s reliability, availability and serviceability pieces, should contact the Quantum team here. Launchpad blueprint tracking this effort is available here.
Quantum Authentication and Authorization - The Quantum Authorization and Authentication blueprint is currently in development and this functionality is scheduled to leverage OpenStack’s Keystone project. Launchpad blueprint tracking this effort is available here.
Built-in Open vSwitch plugin - The Open vSwitch plugin for Quantum is the first supported plugin that is available for beta testers today. Launchpad blueprint tracking this effort is available here.
Plugin Qualification test harness - The Quantum test suite not only helps Quantum developers in helping maintain our code quality but also doubles as a plugin “quanlificaiton” test suite for folks who are interested in writing plugins for Quantum. Launchpad blueprint tracking this effort is available here.
Quantum API Extensibility framework - A key feature of Quantum API is that its extensible by the community, Quantum Extensibility framework is responsible for enabling the Quantum eco-system to innovate beyond the core API. This effort is under active development and code should be available for beta testers in the near future. Launchpad blueprint tracking this effort is available here.
Quantum CLI Tools - Quantum provides a Web Services based CLI that allows administrators to run Quantum commands and automate against Quantum using our CLI tools. Quantum CLI is currently in beta and available from the Quantum trunk. Launchpad blueprint tracking this effort is available here.
Quantum GUI Tools - Quantum Graphical User Interface efforts are in their early stages and are being integrated into OpenStacks “Dashboard” project. Launchpad blueprint tracking this effort is available here.
Quantum Client Library - Quantum client library is another tool for the eco-system to build additional tools, orchestration mechanisms and interesting automation techniques leveraging the Quantum’s Core API. Quantum client library is under active development and the launchpad blueprint tracking this effort is available here.
OpenStack Compute refactoring to support Quantum “out of the box” - Quantum team has been actively involved with OpenStack Compute aka “Nova” project to refactor the the networking subsystem in OpenStack Compute to seamlessly work with an external Cloud Networking fabric such as Quantum. The launchpad blueprint tracking this effort is available here.
Summary
Open vSwitch plugin for Quantum demonstrates a practical example of how one creates Quantum plugins that enable creation of sophisticated virtualized networks, that are decoupled from the physical networking infrastructure. Cloud infrastructure stacks such as OpenStack can leverage OpenStack Quantum’s high-level networking abstractions to create highly flexible and dynamic cloud networking topologies that can be programmatically manipulated without any interactions with or complex configuration of physical switching infrastructure.
Other OpenStack Quantum plugins include Cisco’s plugin that leverages Unified Computing Systems(UCS) and Nexus based switching infrastructure that leverages 802.1Qbh Network Interface Cards(NICs)
OpenStack, Quantum and Open vSwitch – Part II
[Originally published on Openvswitch.com/openstack/blog by Somik Behera & others. This is the second post in a three part series to describe OpenStack, a new OpenStack-compatible networking service called Quantum, and the first Quantum plug-in which is based on Open vSwitch.]
In the previous post, we provided some background for OpenStack and described how it has major sub components for Compute (Nova), Object Storage (Swift), and Image Management (Glance), yet networking is missing as a first class component (currently it is a part of Nova). While this made sense in the initial context in which OpenStack was designed, there has been a lot of effort within the OpenStack community to define a network service allowing networking to evolve easily and independently.
In this post, we describe Quantum, a recently created networking service compatible with OpenStack.
The Quantum Project – A Virtual Networking Service for OpenStack
The goal of Quantum is simple, to provide a clean network service abstraction within OpenStack. While we’ll get into the particulars below, here is a cheat sheet of some of the benefits its design offers:
Provides a flexible API for service providers or their tenants to manage OpenStack network topologies (e.g., create multi-tier web applications)
Presents a logical API and a corresponding plug-in architecture that separates the description of network connectivity from its implementation. This allows virtual networking to be implemented in virtual switches, physical switches, or both, and supports incorporating technologies from multiple vendors / open source projects.
Offers an API that is extensible and evolves independently of the compute API, allowing plugins to easily introduce more advanced network capabilities (e.g., QoS, ACLs, etc.)
Provides a platform for integrating advanced networking solutions such as:
Existing firewall services
Load balancer services
MPLS infrastructure
The Big Picture – Quantum Cloud Networking Fabric
We’ll start by describing how Quantum fits within the existing OpenStack component architecture, shown in the diagram below.
Quantum’s primary interface is a programmatic RESTful API. The abstractions over which it operates are, by design, extremely simple.
The Quantum API allows for creation and management of “virtual networks” each of which can have one or more “ports”. A port on a virtual network can be attached to a “network interface”, where a “network interface” is anything which can source traffic, such as a vNIC exposed by a virtual machine, an interface on a load balancer, and so on. These abstractions offered by Quantum (virtual networks, virtual ports,and network interfaces) are the building blocks for building and managing logical network topologies.Of course, the technology that implements Quantum is fully decoupled from the API (that is, the backend is “pluggable”).
So, for example, the logical network abstraction could be implemented using simple VLANs, L2-in-L3 tunneling, or any other mechanism one can imagine and build. The only requirement is that the actual implementation provide the L2 connectivity described by the logical model.While the native Quantum API does not support more sophisticated network services such as, say, QoS or ACLs, it does provide an API extensibility mechanism that plugins can use to expose them. This is the conduit by which developers and vendors in the OpenStack ecosystem can innovate within Quantum. If an extension proves useful and generally applicable, it may become a part of the core Quantum API in a future version.
Quantum Internals
There are 3 key functional layers of abstraction that make up the Quantum service:
1) REST API layer: This layer is responsible for implementing the Quantum API and routing API requests to the correct end-point within Quantum’s pluggable infrastructure. The REST API layer also contains various infrastructure glue around launching the Quantum service, marshalling & unmarshalling requests and responses, and validating data format & data correctness. This layer can also contain security and stability infrastructure such as rate-limiting logic on inbound API calls to protect against Denial of Service attacks and make sure that the Service remains responsive under load.
REST API Extensions: Quantum provides an “extensibility” mechanism that enables anybody to extend the Core API and add additional features and functionality that are not currently part of the Core API. Taking today’s Core API as an example, one could use the extensibility mechanism to create a QoS extension that enables setting up Quality of Service parameters associated with Quantum networks. Similarly, you can imagine multiple parties can easily integrate advanced networking functionality using Quantum’s extensibility mechanism. Quantum community is actively working on implementing the extensibility framework (to follow the progress, check out the blueprint here).
Key Quantum API methods:
Method: Create Network
REST URL: POST /tenants/{tenant-id}/networks HTTP Request Body: Specified the symbolic name for the network being created. E.g. { "network": { "name": "symbolic name for network1" } }
Description: This operation creates a Layer-2 network in Quantum based on the information provided in the request body.
Method: List all networks for a particular tenant
REST URL: GET /tenants/{tenant-id}/networks HTTP Request Body: Not Applicable
Description: This operation returns the list of all networks currently defined in Quantum
Method: Update Network
REST URL: PUT /tenants/{tenant-id}/networks/{network-id} HTTP Request Body: Specify a new symbolic name for a particular Quantum network E.g. { "network": { "name": "new symbolic name" } }
Description: This operation updates a network’s attributes in Quantum. The only attribute that can be updated, as it stands now, is the network’s symbolic name. Method: Delete Network
REST URL: DELETE /tenants/{tenant-id}/networks/{network-id} HTTP Request Body: Not Applicable
Description: This operation deletes the network specified in the Request URI.
Keeping in line with principles of RESTful interfaces, we have shown examples of Create, Read, Update, Delete(CRUD) methods for the “network” resource. Similar CRUD API methods are available for other resources managed by Quantum, such as “Ports” and “Port Attachments.”
Detailed API Specification is available as a living document at the following Wiki.
2) Authentication & Authorization: This layer is not yet complete, but will be reponsible for validating incoming API Requests and ensuring that an authenticated user is making the request before letting the API call pass through the rest of the stack. Quantum authentication will use the OpenStack Keystone Identity Service.
In addition to basic authentication, the API will enforce client access rights to REST resources. This is an important component for enforcing access rights in multi-tenant environments in which network configuration is exposed to the tenants.
Key Authorization roles that are currently under consideration are:
Super User/System – Super User can operate on any network within the Quantum namespace, this account will be typically reserved for Service Providers.
Tenant – A Tenant can create networks from the “pool” of service provider networks. The Tenant can create ports on these networks and attach interfaces to the ports, as long as the Tenant owns those interfaces (e.g., it is an interface of one of the Tenant’s nova VMs).
Quantum community is still actively defining the Authentication and Authorization capabilities of the API. The blueprint tracking this effort is available here.
3) Pluggable Backend: As we’ve mentioned, Quantum decouples the abstractions from the implementations. The primary mechanism for doing so is through a pluggable architecture in which the mechanism that implements the virtual networks is implemented as a plugin.
So, how do I implement a Quantum Plugin?
This is far from comprehensive, but to give you a general idea of how plugins work, we have put together an architectural sketch of a sample Quantum plugin (that is loosely based on the Open vSwitch plugin we will describe in the next post). Our Reference Architecture for Quantum plugin contains:
a) A centralized Quantum controller: This is the box titled “Quantum Cloud Network Fabric Plugin”, and this controller is responsible for servicing users of Cloud environments such as OpenStack. The controller helps tie the end-user facing cloud network model, with the physical networks that are being imlemented by connecting vSwitches and pSwitches.
b) A centralized data model: This is the data model represents the network connectivity model as desired by the cloud users, where they care about creating virtual networks that provide connectivity between a set of virtual machines. A common mechanism to store this cloud network model would be for the plugin to contain an internal database. This database can be used to store the network model being desired by the cloud users. This database will also serve as the single source of “truth” for responding to Quantum API calls as well as for implementing the cloud network model by communicating with various physical and virtual switches that a particular plugin supports.
c) Virtual and Physical Switch communication channel: The final component of our reference architecture is a mechanism for the centralized Quantum controller to communicate with various virtual and physical switches to implement the physical connectivity. One possible mechanism for establishing such a communication channel would be to install a Quantum plugin agent on every Hypervisor that is running a vSwitch as well as on every physical switch that is supported. The agent can then “phone home” to get updates from the centralized Quantum controller. The agent here is responsible for communicating with various switches using the appropriate APIs and establishing L2 connectivity. The agent can choose a mechanism (e.g., VLANs) to create the isolation between packets from different Quantum networks.
For a more detailed view on how to create a Quantum plugin, take a look at the Open Source Quantum plug-in that uses Open vSwitch. Or, just wait for the next post … :)
OpenStack , Quantum and Open vSwitch – Part I
[Originally published on Open vSwitch blog on July 25, 2011 by Somik Behera & others, This is the beginning of a three part post to describe OpenStack, a new OpenStack-compatible networking service called Quantum, and the first Quantum plug-in which is based on Open vSwitch.]
We will start by providing some background on OpenStack, focusing on the original network model and the rational for creating Quantum. This is not meant to be an exhaustive (nor authoratative!) overview. If you are interested in learning more, or keeping track of the progress, check out the project homepage, openstack.com.
OpenStack Overview
OpenStack is a cloud management system (CMS) that orchestrates compute, storage, and networking to provide a platform for building on demand services such as IaaS. Traditionally, OpenStack has consisted of the following major subcomponents:
OpenStack Compute (Code named Nova) – Nova documentation
OpenStack Object Storage (Code named Swift) – Swift documentation
OpenStack Image Services (Code named Glance) – Glance documentation
The following diagram provides a illustration of how the components are organized to provide the OpenStack platform.
The system operates roughly as follows. The “Cloud Controller” cluster is responsible for managing the global state of the system. It interacts with LDAP (not shown), OpenStack Object Storage, and OpenStack Compute. OpenStack Object Storage and OpenStack Compute packages contain a centralized piece that usually runs on the Cloud Controller as well as a “worker” piece that usually runs on a particular compute or storage node. The “workers” can be thought of as agents that manage storage or compute resources locally on a single system. To communicate with compute and storage “workers”, the cloud controller uses Rabbit MQ.
The responsibilities of the primary components are:
OpenStack Compute (Nova), is responsible for managing the Hypervisor hosts, overseeing virtual machine configuration, creation, deployment and destruction on a “pool” of compute resources. Nova currently supports KVM, XEN,VMware vSphere and Hyper-V. Nova also orchestrates deployment across Availability Zones, which are often called Nova Zones{link to Nova Zones}. An availability zone represents a single unit of failure. An example of availability zone could be a Cloud provider’s San Francisco Datacenter, while another availability zone would be New York Datacenter. The rational of availability zone is that when one of the availability zone experiences failure, none of the other zones or impacted. This enables a Cloud user to mitigate risk by distributing and replicating workloads across availability zones. However availability zones need not necessarily span an entire datacenter, zones can very well consist of a single rack of servers that is situated at the north end of datacenter while another zone can be a rack of servers situated in the south end of the datacenter.
OpenStack Object Storage (Swift), allows the creation of clusters that can store, retrieve and update multi-petabyte objects, all within a single storage system. Swift handles data replication and integrity across the cluster to maintain availability during failure. The primary use cases tend to be around static, long-term data storage needs. For example, swift is used for storing virtual machine images and long term backups.
OpenStack Image Service (Glance), is a multi-format Virtual Machine Image repository that provides discovery, registration and delivery services for Virtual Machine Images and disks. Glance doesn’t actually provide the storage of the Virtual Machine Images. Rather, in order to store Virtual Machine images, Glance uses local disk, iSCSI/NAS/SAN ,and Swift as storage backends.
What about networking?
Noticeably absent from the list of major subcomponents within OpenStack is networking. The historical reason for this is that networking was originally designed as a part of Nova which supported two networking models:
Flat Networking – A single global network for workloads hosted in an OpenStack Cloud.
VLAN based Networking – A network segmentation mechanism that leverages existing VLAN technology to provide each OpenStack tenant, its own private network.
While these models have worked well thus far, and are very reasonable approaches to networking in the cloud, not treating networking as a first class citizen (like compute and storage) reduces the modularity of the architecture. Next we’ll briefly describe some of these shortcomings which precipitated the work towards Quantum, which is the proposed to sit alongside Nova, Swift, and Glance as a standalone service.
Limitations of the Current OpenStack Network Architecture
Limited network options: As we’ve mentioned, OpenStack only supports two network models. While suitable for some environments, they don’t cover all use cases.
For example, the flat model exposes one large “flat” network with no real isolation primitives, making it difficult to build support for multi-tenancy.
The VLAN model, on the other hand, provides basic isolation via 802.1Q tagging. However, it is burdened with the classic set of problems associated with VLANs such as, difficulty in supporting overlapping IP and MAC addresses, difficulty of extending L2 across subnets, VLAN scaling limits, MAC table scaling issues, dependency on proprietary interfaces for configuration of networking gear, etc. Additionally, the VLAN model only supports having a single interface per VM.
No well-defined network interface: In addition to having limited network options, there is no well defined interface by which a new one can be created and slotted in non-disruptively. The original developers did a good job designing flexibility into how networks are orchestrated, but the concepts of VLANs are bridges are embedded deeply in the nova codebase.
Simplistic network model: The current networking model is built around L2 and L3. However, interposition of higher-level services is a growing need and not well accommodated in the current model. For example, how would a new “Firewall as a service” offering plug into a private nova network? Or how would one model connecting a tenant’s network to a remote data center using an MPLS circuit?
It is for these reasons that the Quantum network service was proposed. Quantum is designed to be a first class subcomponent which defines the interface for a network service. It can accommodate the existing FLAT and VLAN models, as well as many other service implementations including virtualized L2 using soft switches, service interposition, hardware integration etc.
In the next post, we will describe Quantum in detail, and describe an implementation of it usingOpen vSwitch.
All three major open source IaaS platforms -- OpenStack, CloudStack and Eucalyptus -- have made much progress this year in testing Hadoop deployments on their stacks.
Come learn about VMware and OpenStack in Action and Visit my Session on using Network Virtualization to create Nicira's OpenStack Cloud!
Raised $52K+ while volunteering for KQED, Public Broadcasting Company for San Francisco area.
I am on the first row, third from left, working the frontline, taking calls. Fun experience!
Do Intellectual Property laws serve their intended purpose of encouraging creation and innovation? You decide.
Ethics for the Real World
In companies, money is a terrible motivator for employees, culture change starts at upper echelons and spreads
http://bit.ly/5pHF3H