On the multi-dimensional evolution of platforms and applications

I’d like to touch on a topic that I am seeing in several of my areas of interest right now. In general, it’s related a lot to the overall topic of “First, Second and Third Platform” but I’d like to focus more on the individual implications for multiple domains. Over the course of the last few months, I have been involved in several discussions around different platforms and applications as well as their individual evolution and maturity. My personal observation is that both don’t necessarily evolve synchronously. Therefore, it is important to not only identify the phase that you are currently in but also to understand the operational implications of the “generation disconnect” between app and platform.

 

Evolution of Platforms

As mentioned above, I’d like to relate my observations to the three platform generations below. I’d like to point out that these three generations are subdivided in several different technologies and can look and feel different in specific use-cases or fields of application. The common sense around the generations is:

platform_evolution

 

In addition to these phases, I see different “implementations” of the respective platform generation. Take Client-Server as one example – this can be a physical-server-only model, this also stretches to server virtualization and potentially even to “VM-oriented” hosting or even cloud services. My friend Massimo also wrote a nice piece on this.

 

Evolution of Applications

One of my key observations is that there is no simple 1:1 connection between applications and platforms. With the rise of 2nd generation platforms, not all applications from the 1st platform have been dropped and immediately been available for the next-generation platform. It’s actually an evolution for applications that are still business-relevant and therefore make sense to be optimized for the next-generation platform. And here comes the important observation: I believe there are (at least) three phases in an application evolution cycle that is happening for each platform generation – or potentially even in each concrete implementation of the platform generation. I’ll call theses phases “Unchanged”, “Optimized” and “Purpose-built” for now:

application_evolution

But how does that fit in the overall platform picture? I’ll try to merge the previous two pictures into one. It also shows a potential application evolution path across platform generations. As you can see, there can be a slight overlap between “purpose-built” of the previous and the “unchanged” phase of the next-generation platform.

evolution

But let’s move on to two concrete examples that I see applicable.

 

Example 1: Network Functions Virtualization

I’ll start with Network Functions Virtualization (NFV). NFV is a Telco industry movement that is supposed to provide hardware independence, new ways of agility, reduced time to market for carrier applications & services, cost reduction and much more – basically, it’s about the delivery of promises of Cloud Computing (and Third Platform) for the Telco industry (read more about it here). The famous architectural overview as described by ETSI can be seen below:

ETSI NFV

NFV differentiates between several functional components such as the actual platform (NFVI = Network Functions Virtualization Infrastructure), the application (VNF = Virtual Network Function), the application manager (VNF manager) and e.g. the orchestration engine.

So how could this look in reality? Let’s assume the VNF manager detects a certain usage pattern of its VNF and that VNF is reaching it’s potential maximum scale for the currently deployed amount of VNF instances. The VNF manager then talks to the Orchestrator that could then trigger e.g. the scale-out of the VNF by deploying additional worker instances on the underlying infrastructure/platform resources. The worker instances of the VNF could then automatically be included in the load distribution and have instant integration into necessary backend services where applicable. All of that happens via open APIs and standardized interfaces – it looks and feels a lot like a typical implementation example for a “third platform” including the “purpose-built” app.

Now into a quick reality check. ETSI’s initial NFV whitepaper is from October 2012. It basically describes the destination that the industry is aiming for. And while there might be some examples where VNFs, NFVI and Orchestration are already working hand in hand, there is still a lot of work to do. Some of the “NFV applications” (or VNFs) might have been just „P2V“’ed (1:1 physical to virtual conversion) onto a virtualization platform and basically have the same configuration, same identity and are kept as close to its physical origins as possible. This allows a VNF provider/vendor to keep existing support procedures and organizations while offering their customers a “NFV 1.0 product” that is providing some early benefits of NFV (hardware independence, faster time to market, …). But this also implies that you transfer some of the configurations that made perfect sense in the physical world over to the virtual world – where it only makes questionable sense. In this case, I’d actually talk about a move from a “purpose-built” app from the first platform to an “unchanged” app on the second platform. 

One example: one physical server in a telco application had 30*300GB harddisks, had 2*4Core CPUs and 128GB RAM. It never used more than 1TB of storage and average utilization has been below 4 CPUs and 32GB RAM. The “unchanged” version of this app would be a 1:1 conversion with all (unnecessary) resource overhead provided in a virtual machine. The “optimized” version of this app is a right-sized application (so only 1TB storage, 4 CPUs and 32GB RAM) that is also leveraging easy configuration files for installation as well as crash-consistent and persistent data management to allow backup & restore as VM. But a “purpose-built” version of that app would leverage the underlying NFVI APIs, would allow scale-out deployment options based on actual demand as well as optimizations that are e.g. encryption at every layer of the application to ensure global deployment models even in the face of lawful interception relevance, etc.

 

Example 2: Microservices, Containers & Docker

My next example are microservices and their close friends containers. They are promising a new generation of application architecture and are drivers for the “3rd platform” architecture. One of this movements famous poster-childs is Docker. Docker is a great (new) way to package and distribute applications in “containers” that contain applications or just pieces of a larger application architecture. Newly developed applications usually follow a scale-out design, some might be written with something like the “12 factor app” manifesto in mind (or the 15 factors according to Pivotal). Coming back to the pictures above: a 12 factor app could be considered “purpose-built” for the “third platform”.

But how many applications have been built for this? There are many great examples for microservices-oriented applications by the “cloud-native” companies such as Google, Amazon, Facebook and the likes. Adrian Cockcroft also gives inspirational talks about these topics around the globe. But I actually expect many applications to stay mainly unchanged as they are optimized for their current platform. At the same time, some of them might become available as (Docker) containers as part their next release. But again – if you look into the details, you’ll find the same application in a different wrapper. RAR is now ZIP (for my German readers: “Aus Raider wird nun Twix…”). But will these potentially “single-container-applications” run well on a Cloud-Native/third platform architecture? They might not! To put it in a picture:

So in this case, it is actually important to understand these application limitations and expectations towards the platform (what about data persistence, security, platform resilience, networking, …) to make sure it runs smoothly in production. Coming back to Massimo’s blogpost – you can run your old Windows NT4 on a Public Cloud, but does it make sense?

Summary

Just like the continuous evolution of platforms that expose new characteristics and capabilities, there is also an ongoing evolution of applications. It is important to understand the key aspects of the application architecture and it’s deployment model before making a platform decision. The word “VNF” does not necessary imply the alignment with NFV and the word “Docker” does not automatically describe a Cloud-Native or microservices-oriented application.

 

Edits:

18.05.2016: added picture (containerizing legacy applications)

Install vSphere Integrated Containers v0.1 via VMware Photon OS TP2

I just wanted to share a few simple steps on how to set up vSphere Integrated Containers (VIC) v0.1 (released on April 4, 2016) on VMware Photon OS. You can follow most steps easily via copy and paste but please be aware that you should not run this in your production environment! I worked with a local installation running in VMware Fusion. The vSphere Integrated Containers Code on GitHub is in a preview state but please read the project status directly on GitHub. It also requires a few environmental variables to be available (e.g. DHCP). You can find the binaries for the install on Bintray as they are available as a pre-packaged .tar.gz file (direct download link).

First we need to install Photon OS TP 2 (fresh install from the ISO) and give it 1 CPU and 2GB of memory. I used the “full” install type while going through the installer. You can find the Photon OS TP2 ISO files on Bintray – I used the full ISO (direct download link). Cormac just told me that you can also use the “smaller” Photon OS or the OVA and add git, wget, tar and go to the OS as prerequisites for the install of VIC.

Why 2GB RAM? I ran into some issues (like “go build github.com/vmware/govmomi/vim25/types: /usr/lib/golang/pkg/tool/linux_amd64/6g: signal: killed”) while installing govmomi with less than 2GB. It was fixed by adding more memory – you should be able to go back to less than 2GB after the install.

After the install, we login via DCUI (e.g. via Fusion) and enable remote SSH access. You can either edit the sshd_config file with the editor of your choice of follow the commands below.

# Enable root login in /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication/PasswordAuthentication/g' /etc/ssh/sshd_config
sed -i 's/PermitRootLogin no/PermitRootLogin yes/g' /etc/ssh/sshd_config
systemctl restart sshd

Once you noted the IP address of your system, you can connect to it via ssh. I updated tdnf (Photon OS package manager) and the Photon OS system as a first action.

# Upgrade tdnf and Photon packages
tdnf upgrade tdnf -y
tdnf upgrade -y

Then, we set up a working directory for VIC, I called it “VCHmaster”.

# Create working directory /home/VCHmaster
mkdir /home/VCHmaster
cd /home/VCHmaster

Next up is the download and extraction of the VIC artifacts from Bintray (placed in /home/VCHmaster)

# Download files from https://bintray.com/vmware/vic/Download
wget https://bintray.com/artifact/download/vmware/vic/vic_0.1.0.tar.gz
tar -xzf vic_0.1.0.tar.gz
chmod +x vic/install.sh

Before installing govmomi, you need to set some environmental variables (GOPATH and adding the bin directory to your PATH):

# Set Go variables
mkdir /home/VCHmaster/govmw
export GOPATH=/home/VCHmaster/govmw
PATH=$PATH:/home/VCHmaster/govmw/bin

We can then install the vSphere API Go library:

# install Go library for the VMware vSphere API
go get github.com/vmware/govmomi/govc

At this point, we are all set to install the first Virtual Container Host (VCH). The command is pretty straight forward. All you need is a deployment endpoint (in my case my homelab ESXi host) with a datastore.

# install your first VCH
cd vic
./install.sh -g -t 'user:password@IP_ADDRESS' -i ESXi_datastore VCH-name

When we run the install.sh script, this is the output that we are seeing:

root [ /home/VCHmaster ]# ./install.sh -g -t 'root:PASSWORD@ESXi_IP_ADDRESS' -i ESXi_datastore VCH-name
# Generating certificate/key pair - private key in VCH-name-key.pem
# Logging into the target
# Uploading ISOs
[05-04-16 12:00:38] Uploading... OK
[05-04-16 12:00:41] Uploading... OK
# Creating vSwitch
# Creating Portgroup
# Creating the Virtual Container Host appliance
# Adding network interfaces
# Setting component configuration
# Configuring TLS server
# Powering on the Virtual Container Host
# Setting network identities
# Waiting for IP information
#
# SSH to appliance (default=root:password)
# root@VCH_APPLIANCE_IP_ADDRESS
#
# Log server:
# https://VCH_APPLIANCE_IP_ADDRESS:2378
#
# Connect to docker:
docker -H VCH_APPLIANCE_IP_ADDRESS:2376 --tls --tlscert='VCH-name-cert.pem' --tlskey='VCH-name-key.pem'


DOCKER_OPTS="--tls --tlscert='VCH-name-cert.pem' --tlskey='VCH-name-key.pem'"
DOCKER_HOST=APPLIANCE_IP_ADDRESS:2376

To make the variables persistent across multiple sessions, you’ll have to add these e.g. to your bash profile. I’ll keep it simple (edit in /root/.bash_profile):

# Begin ~/.bash_profile
# Written for Beyond Linux From Scratch
# by James Robertson <jameswrobertson@earthlink.net>
# updated by Bruce Dubbs <bdubbs@linuxfromscratch.org>


# Personal environment variables and startup programs.
GOPATH=/home/VCHmaster/govmw
PATH=$PATH:/home/VCHmaster/govmw/bin

At this point in time, there is not much more to explore. You run a few docker commands already but again – this a v0.1 with limited functionality.

(Edits: added the Bintray download links for VIC and Photon OS. Thanks Manuel & Cormac for your feedback!)

Cloud-Native Applications – Link Collection

vmware_cna

I started to collect the most comprehensive and important links around (VMware) Cloud-Native Applications on a dedicated page on this blog. I’ll keep it updated over time. If you feel something is missing, just ping me on Twitter and I’ll add the link/material.

You can find the page at:

Cloud-Native Applications

VMworld Tipp: Cloud-Native Applications & vSphere Integrated Containers

Die VMworld 2015 in Barcelona steht unmittelbar bevor. Eine Teilnahme an der VMworld hat für mich schon immer bedeutet über den eigenen Tellerrand zu schauen und ein Gefühl für die aktuellen Trends und Themen der nächsten Jahre zu bekommen. Dazu gehört natürlich auch der persönliche Austausch mit Kollegen aus anderen Firmen, Sessions, Labs, Diskussionsrunden und vieles mehr.

Auf die Frage nach den interessantesten Sessions zum Thema “Zukunft” verweise ich in den letzten Wochen gern auf alles, was sich im Cloud-Native Applications (CNA) Bereich abspielt: die Sessions mit dem Kürzel “CNA”, die Hands-On Labs zum Thema, etc. Warum ich das Thema besonders für vSphere Administratoren und Manager von Virtualsierungsteams als wichtig ansehe werde ich in einem separaten Blogpost erläutern.

Als Teaser und 3:30min Zusammenfassung eines der wichtigsten Announcements rund um Cloud-Native Applications von der VMworld 2015 in San Francisco möchte ich an dieser Stelle auf ein Video hinweisen – viel Spaß mit vSphere Integrated Containers: