#vK8s 2021 edition – friends don’t let friends run Kubernetes on bare-metal

Three years ago, I wrote a blogpost on why you wouldn’t want to run Kubernetes on bare-metal. VMware released a number of platform enhancements over these years and there is a lot of updated material and feedback – also coming from customers. So what are (my personal) reasons to run containers and Kubernetes (short “K8s”) on a virtual infrastructure & vSphere in particular?

Operations: Running multiple clusters on bare-metal is hard

  • Multiple clusters in a virtual environment are a lot easier and each cluster can leverage e.g. it‘s own lifecycle policies (e.g. for K8s version upgrades) instead of forcing one bare-metal cluster to upgrade. Running multiple Kubernetes versions side-by-side might be already or become a requirement in the near future.
  • It also makes lots of sense to run Kubernetes side-by-side with your existing VMs instead of building a new hardware silo and operational complexity
  • VMware’s compute platform vSphere is the de-facto standard for datacenter workloads in companies across industries and operational experience and resources are available across the globe. Bare-metal operations typically introduces new risks and operational complexity.

Availability/Resilience and Quality of service: you can plan for failures without compromising density

  • Virtual K8s clusters could benefit even in „two physical datacenter” scenarios where the underlying infrastructure is spread across both sites. A “stretched” platform (e.g. vSphere with vSAN Stretched Cluster) allows you to run logical three-node Kubernetes control planes in VMs and protect the control plane and workload nodes using vSphere HA.
  • vSphere also allows you to prioritize workloads by configuring policies (networking, storage, compute, memory) that will also be enforced during outages (Network I/O Control, Storage I/O Control, Resource Pools, Reservations, Limits, HA Restart Priorities, …)
    • Restart a failed or problematic Kubernetes node VM before Kubernetes itself even detects a problem.
    • Provide the Kubernetes control plane availability by utilizing mature heartbeat and partition detection mechanisms in vSphere to monitor servers, Kubernetes VMs, and network connectivity to enable quick recovery.
    • Prevent service disruption and performance impacts through proactive failure detection, live migration (vMotion) of VMs, automatic load balancing, restart-due-to-infrastructure failures, and highly available storage

Resource fragmentation, overhead & capacity management: single-purpose usage of hardware resources vs. multi-purpose platform

  • Running Kubernetes clusters virtually and using VMware DRS to balance these clusters across vSphere hosts allows the deployment of multiple K8s cluster on the same hardware setup and increasing utilization of hardware resources
  • When running multiple K8s clusters on dedicated bare-metal hosts, you lose the overall capability to utilize hardware resources across the infrastructure pool
    • Many environments won‘t be able to (quickly) repurpose existing capacity from one bare-metal host in one cluster to another cluster in a short timeframe
  • From a vSphere perspective, Kubernetes is yet another set of VMs and capacity management can be done across multiple Kubernetes clusters; it gets more efficient the more clusters you run
    • Deep integrations with existing operational tools like vRealize Operations allow operational teams to deliver Kubernetes with confidence
  • K8s is only a Day-1 scheduler and does not perform resource balancing based on running pods
    • In case of imbalance on the vSphere layer, vSphere DRS rebalances K8s node VMs across the physical estate to better utilize the underlying cluster and delivers best-of-both-worlds from a scheduling perspective
  • High availability and „stand-by“ systems are cost intensive in bare-metal deployments, especially in edge scenarios: in order to provide some level of redundancy, some spare physical hardware capacity (servers) need to be available. In worst case you need to reserve capacity per cluster which increases physical overhead (CAPEX and OPEX) per cluster.
    • vSphere allows you to share failover capacity incl. incl strict admission control to protect important workloads across Kubernetes clusters because the VMs can be restarted and reprioritized e.g. based on the scope of a failure

Single point of integration with the underlying infrastructure

  • A programmable, Software-Defined Datacenter: Infrastructure as Code allows to automate all the things on an API-driven datacenter stack
  • Persistent storage integration would need to be done for each underlying storage architecture individually, running K8s on vSphere allows to leverage already abstracted and virtualized storage devices
  • Monitoring of hardware components is specific to individual hardware choices, vSphere offers an abstracted way of monitoring across different hardware generations and vendors

Security & Isolation

  • vSphere delivers hardware-level isolation at the Kubernetes cluster, namespace, and even pod level
  • VMware infrastructure also enables the pattern of many smaller Kubernetes clusters, providing true multi-tenant isolation with a reduced fault domain. Smaller clusters reduce the blast radius, i.e. any problem with one cluster only affects the pods in that small cluster and won’t impact the broader environment.
  • In addition, smaller clusters mean each developer or environment (test, staging, production) can have their own cluster, allowing them to install their own CRDs or operators without risk of adversely affecting other teams.

Credits and further reading

#vK8s – friends don’t let friends run Kubernetes on bare-metal

So, no matter what your favorite Kubernetes framework is these days – I am convinced it runs best on a virtual infrastructure and of course even better on vSphere. Friends don’t let friends run Kubernetes on bare-metal. And what hashtag could summarize this better than something short and crips like #vK8s ? I liked this idea so much that I created some “RUN vK8s” images (inspired by my colleagues Frank Denneman and Duncan Epping – guys, it’s been NINE years since RUN DRS!) that I want to share with all of you. You can find the repository on GitHub – feel free to use them whereever you like. 

Work from home: productivity & tools

In my previous post, I wrote about my homeoffice setup and hardware. Today, I’d like to add a few tools that helped me over the past few years and specifically over the last couple of months…

Whiteboard: sometimes, standing in front of a physical whiteboard is the beginning of some amazing brainstorming. While I incorporated lots of online tools and virtual whiteboards into my daily workflows, I don’t want to miss my “real” whiteboard anymore. Sometimes, the whiteboard is a quick way to dump ideas, tasks or other “loose ends” from my brain before heading to bed. It’s sometimes the easiest way to get rid of some open thoughts, materialize them somewhere and then categorize and work on them the next day. Especially when working on things in parallel, the amount of ideas and things to consider can be overwhelming – getting them out of my head has become an important strategy in general. For collaboration, an online whiteboard is super helpful. Miro has also done a great job for me and has even replaced my physical whiteboard for some occasions.

Calendar: as mentioned above, I try to dump thoughts, tasks and plans from my memory and persist them in the appropriate format/tool. Events, special dates, deadlines, birthdays, trips, … it all has to be in the work or personal calendar to be helpful for me. Remembering where I have to be next week or next month is not a helpful brain cycle for me – I try to outsource that to a tool. And when a trip or activity requires some preparation in advance, the related efforts have to be planned and documented as To-Do items with a due date on my list (see below) as well. An example from not too long ago: is the passport still valid for the trip to the US? That needs to be checked at least three months in advance. Even better: directly putting a reminder for six months prior to the passport expiry date directly on the To-Do list…

To-Do-App: I tried many ways to keep track of my to-do items – from minimalist (txt file) to note-taking apps to notes on the physical whiteboard to post-it notes… they all had their shortcomings and issues. Universal access and ease of use are key features for me as I believe in dumping stuff from my mind into a tool to not spend time on remembering it. Over the past year or so I have been using Todoist very successfully. Todoist is not only available on all my devices, it also has a very intuitive way to get stuff onto your lists. You basically type in the title of the task, naturally write a date (“tomorrow, next tuesday, every sunday, …”) and add a #-add the project name and todoist makes it so. You can even mention someone if you work in a team (or e.g. a family member). If I don’t have time to sort a new task out or pick a date, the new task just ends up in an “inbox” that I constantly monitor. You can also dictate tasks into an Apple Watch which is the most non-disruptive way to get stuff out of my head. Which brings me back to my concept of getting everything written down. In my to-do list, there are items that are months, even years out. There are recurring items that I do every day or every week. That way, it has become natural to come back to the lists and actually use them. You can separate items out in projects, sections inside a project and then each task can have sub-tasks. Breaking larger tasks down to smaller items also needs to become a natural effort. If that super important task that will take months to complete is just one item on your list, it will not give you emotional gratification to complete it. But breaking it down into smaller items helps to make and see progress. You can also add files, comments, priorities & reminders. I don’t use all of them but I use some of them selectively. I have projects dedicated to “work”, “home”, “personal” and other larger efforts. I even have “template” projects that can be exported and imported. In each of my primary projects, I put a section for long-term as well as repeating tasks so they don’t show up all the time. They’ll only appear on the “today” or “soon” view that I really love in todoist. It’s an aggregated view across all projects. In one of the recent updates, Todoist also introduced a “boards” view which reminded me a lot of Trello boards – which is a great way to visualize tasks other than a list. For long-term motivation, Todoist also has a basic gamification feature called “karma” that tries to motivate you to complete e.g. at least 5 tasks each day or 30 tasks per week. Apple Watch ring completion fans know this helps 😉 Overall, Todoist has been really helpful… (If you feel inspired to use Todoist, I’d appreciate if you follow this referral link :-))

Which brings me to the last tool I’d like to highlight here: time tracking. Constant working from home sometimes feels like days are just passing. But how much am I actually working, how much time goes into meetings, how much time goes into email or self-education? It’s not about providing a timesheet to my boss. It’s about insights where my time goes. In general, “retrospectives” are a great way to better understand, learn and improve in the future. Doing retrospectives after projects but also individual meetings can be a great tool to constantly improve. But that’s a different topic. I didn’t want to rely on “feelings” or rough estimates alone. I wanted to see where my time is going. A couple of years ago, my wife was playing around with Timeular but back then it had some technical issues that made her return the device after a few days. But earlier this year, my colleague Robbie mentioned it as well and caught my attention. A few days later, I had my own Timeular device – which is basically a dice with eight sides. It connects via bluetooth to your computer or smart device and you can assign categories to each side. There are stickers to put onto the sides. You can also write on them. Or print your own labels.

As soon as you flip it to one side, the Timeular app picks up the signal and starts tracking the responding category/activity. The cool thing here is, you can enhance those categories with #-tags or e.g. @-mention of people. It took me a while and several iterations but I am happy with my categories and #-tags now (all brainstormed and documented on a Miro board :-)). The Timeular team also just added a cool new keyboard shortcut feature that allows you to start tracking without flipping the device (e.g. when you are not at your desk) or when you want to edit a running session’s category or hashtags without going to the Timeular app. Once you have tracked some efforts, you can interactively generate reports on any timeframe (last week, Sunday till Tuesday, last month, specific year) and see which categories or tags or people are taking what amount or percentage of your tracked time. So at the end of the week, you can see how much time you actually worked overall, how much time went into certain topics and if your “feeling” about a week is actually reflected in those numbers. It also gives you a good idea about the number of context-switches you do per day. Or when you typically start and finish tracking your day – all including trends over time.
I use the higher-level categories to structure my time tracking into “external facing” (presenting at events, customer or partner meetings, …), “internal in support of a customer” (preparing for a presentation, alignment meetings, …), “internal-internal” (teamcalls, …) but also e.g. “self-development” (product/company specific, skill development, …), “mentoring” and “PTO”. But I don’t track “breaks” during the day. I just put the tracker in the neutral position so it does not track at all. If I work with customer “ACME Corp”, I tag all work for that customer #acmecorp and Timeular autocompletes that hashtag. The hashtag is used across my “internal” and “external” activities but allows me to break-down activities easily in the interactive report. I think you get the idea. The categories don’t have to be static either. You can have more than 8 categories and only “enable” certain categories on the dice for a certain time. I also have a category for “travel” (well, for some day in the future). But in this case, I wouldn’t flip the dice during travel, this would simply not work. Instead, I can add timeslots in the app manually when the category is not reflected on the dice – or if I forgot to flip it. The physical device makes it very easy – it just sits on my desk and having it there is a constant reminder to actually flip it to the correct side. If you are interested, check out timeular.com (UPDATE Nov 16: you can also folly my referral link if you want 😉 …)

Note: I pay for the pro/premium plan of the services that I mentioned above. Some of the features might not be available in a free plan!

Work from home: my homeoffice setup and gadgets

I have been working from a homeoffice for over 10 years now. But when travel stopped due to COVID-19, lots of things have changed even for me. This article is the beginning of a short blog series where I’ll highlight some of the tools and practices that work for me.

While it was always an option, video conferencing and online collaboration over-night became the new default and it seems like these trends are not going away anytime soon. And given my role, I spend lots of time on meetings with customers, partners & colleagues. I remember the old days when “virtual meetings” were “conference calls” and audio quality was the least common denominator audio codec of the participants dialing-in to a bridge. I can’t imagine going through six months of crappy conference calls so I am very grateful for the reliable and high-quality platforms that Zoom and MS Teams have offered us in these difficult times. We left the dark ages of conference calls and audio is now typically transmitted over a broadband IP connection – so even when I join a Zoom meeting on my phone, I don’t even consider to click the “call-back” option to join the audio anymore. Even when being on the road, the Voice-over-IP stability and quality is outperforming traditional phone calls dramatically.

My personal experience has been that a better audio quality has a very positive impact on productivity & focus and also provides a more inclusive environment. If people have a hard time following a presentation or conversation, a virtual meeting can become more exhausting than necessary. And people with hearing issues might not even be able to fully participate in an active conversation with bad quality. Therefore, I consider it a courtesy to my fellow meeting participants to bring the best possible experience to the virtual conference table.

Before COVID-19, I used a pretty standard Jabra headset and audio quality was average. But I didn’t spent this much time on video conferences after all. So since I upgraded my homeoffice setup a few months ago, I received lots of positive feedback – and questions about the equipment I use. So here we are 🙂

Webcam: I am among the lucky ones that got a decent webcam when all this started. I use a Logitech Brio Ultra HD Pro WebCam that is mounted to the top of my monitor. It’s a decent device – even though I sometimes have the impression the camera has issues with focus.

Brio Ultra HD Pro WebCam
(Image from Logitech)

Light: my office has a decent sized window with lots of natural light coming in – but only on one side. So I put up pretty regular LED uplights in the other side of the room to get some better light coverage from both sides. And above my webcam & monitor, there is an Elgato Key Light Air because… well it’s there now and works. It fit nicely with my Elgato Stream Deck panel that I use for some desk automation – but that’s a different story.

Elgato Key Light Air
(Image from Elgato)

Audio: the audio setup has been a little more complicated. I experimented with a few things over time and looked e.g. at several Blue microphones but wasn’t 100% convinced. Coincidentially, there is this company named “Sennheiser” (you might have heard of them ;-)) which has their global HQ not too far away from where I live. And since Sennheiser equips lots of major opera houses, live broadcasting events and artists like Ed Sheeran with high quality microphones for decades now, I was sure they must have something for upping my Zoom calls as well. And what can I say? It’s been love at first sight.

So a Sennheiser Handmic Digital is now part of my homeoffice equipment and I mounted this into a standard microphone arm. What impressed me right away is the fact that it’s super easy to use – the “plug and play” promise is not just marketing. My MacBook recognized the device immediately and I have not configured anything special. It’s just a new audio device. The digital experts from Apogee are providing the technology for the digital audio converter and pre-amp that consolidates potentially multiple devices into a slick and all-metal body. It comes with USB as well as a Apple Lightning connectivity. My dear and beloved travel companion for more than 4 years, a Sennheiser PXC 550 Wireless, as well as a basic 2.1 Logitech speaker setup serve me well from an audio consumption aspect.

Microphone comparison: MacBook, Webcam, Jabra headset, Sennheiser
Microphone comparison: MacBook vs. Sennheiser
Sennheiser Handmic Digital (Picture from Sennheiser)
HAUEA Microphone Arm (Picture from Amazon)
Sennheiser PSX 550 Wireless (Picture from Sennheiser)

Thanks for reading! Feel free to reach out via Twitter for comments or discussions!

VMware Project Pacific – collection of materials

Blogposts:

VMworld US 2019:

VMworld Europe 2019 sessions:

  • HBI1452BE – Project Pacific: Supervisor Cluster Deep Dive – STREAM DOWNLOAD
  • HBI1761BE – Project Pacific 101: The Future of vSphere – STREAM DOWNLOAD
  • HBI4500BE – Project Pacific: Guest Clusters Deep Dive – STREAM DOWNLOAD
  • HBI4501BE – Project Pacific: Native Pods Deep Dive – STREAM DOWNLOAD
  • HBI4937BE – Introducing Project Pacific: Transforming vSphere into the App Platform of the Future – STREAM DOWNLOAD
  • KUB1840BE – Run Kubernetes Consistently Across Clouds with Tanzu & Project Pacific – STREAM DOWNLOAD
  • KUB1851BE – Managing Clusters: Project Pacific on vSphere & Tanzu Mission ControlSTREAM DOWNLOAD

Podcasts:

Labs / Hands-On:

  • HOL-2013-01-SDC – Project Pacific – Lightning Lab: https://labs.hol.vmware.com/HOL/catalogs/lab/6877

Other interesting sources:

Feel free to reach out if you are missing any interesting sessions here – happy to update this post anytime! @bbrundert

2019-05-30 – Cloud Native Short Takes

KubeCon + CloudNativecon Barcelona 2019 & related announcements

Other community updates