Tanzu Kubernetes Grid Service (TKGS) is the embedded Kubernetes runtime inside vSphere with Tanzu. It runs conformant Kubernetes clusters directly on ESXi through a Supervisor layer that turns vSphere into a Kubernetes control plane itself. This piece walks through the layers as a practitioner sees them during install, cluster provisioning, day-2 operations, and failure recovery.
The two-plane model
TKGS separates concerns into two planes that must be understood distinctly before anything else makes sense.
The Supervisor is a Kubernetes cluster whose control plane runs as three VMs on ESXi hosts inside a vSphere cluster that has Workload Management enabled. Its API server is what kubectl hits when you create namespaces, apply cluster manifests, or bind storage policies. The Supervisor is not for running application workloads. It is the management plane.
The TKG clusters (also called guest clusters or workload clusters) are the conformant Kubernetes clusters where application pods actually run. Each TKG cluster is provisioned by the Supervisor into a vSphere Namespace, using Cluster API under the covers. Each cluster is a set of Photon or Ubuntu VMs with its own etcd, its own API server, and its own kubelet on every node.
The important idea is that the Supervisor turns each vSphere Namespace into a Kubernetes namespace with resource quotas, storage policy bindings, and RBAC. A TKG cluster lives inside a vSphere Namespace and inherits its policies.
Supervisor control plane internals
When Workload Management is enabled on a vSphere cluster, vCenter deploys three Supervisor Control Plane VMs across the ESXi hosts. Each VM runs a modified Kubernetes control plane with a stacked etcd. A vSphere service called wcpsvc (Workload Control Plane) on vCenter is responsible for lifecycle of these VMs, certificate rotation, and version upgrades.
Two components are worth calling out because they explain most operational behaviour.
Spherelet is a kubelet variant that runs on ESXi hosts, not inside guest VMs. When you use vSphere Pods (the pod-per-VM model used mostly by the embedded Harbor registry and some legacy PodVM patterns), spherelet is what schedules and monitors them. TKG worker nodes are regular VMs and do not use spherelet at the pod layer, but the Supervisor still relies on it for host-level tasks.
NSX Container Plugin or the vSphere Networking stack provides pod networking for Supervisor workloads. The stack you pick at enable time (NSX-T versus vDS with HAProxy or Avi) fundamentally changes what the Supervisor can offer. NSX-T gives you distributed firewall, native LoadBalancer type services, and Namespace-level Tier-1 routers. The vDS path uses an external load balancer for API endpoints and hands pod networking to Antrea or Calico inside each TKG cluster.
Cluster provisioning flow
This is where the most confusion tends to happen. A TKG cluster is not created by a Helm chart or a CLI tool that talks to vCenter. It is created by applying a Kubernetes custom resource to the Supervisor API server. The Supervisor then reconciles that intent into VMs.
A few reconciliation nuances matter here. The TKG Service does not directly create VMs. It creates Cluster API objects, and CAPV (Cluster API Provider for vSphere) translates those into VirtualMachine CRDs that VM Operator understands. This layering is why you can independently upgrade the TKG Service, the CAPI providers, and the base VM images without rebuilding the Supervisor.
The kubeadm bootstrap sequence is standard. The difference is that the join tokens, TLS material, and node identity are generated by the Supervisor and injected via cloud-init on first boot. There is no manual kubeadm join anywhere.
Networking architecture
Networking is where deployment decisions bite hardest, so it deserves its own diagram.
The relationship worth memorising is that each vSphere Namespace gets its own Tier-1 gateway attached to the shared Tier-0. Each TKG cluster inside that Namespace gets a further Tier-1 for its own node segment and API VIP. This is why NSX-T deployments consume a non-trivial number of routing hops and why the Tier-0 sizing decision made on day one has downstream throughput impact.
On the vDS path, the story simplifies. You lose Namespace-scoped Tier-1s and distributed firewall, and you gain an external load balancer that must be sized for both the Supervisor VIP and every TKG cluster API VIP.
Storage architecture
Storage in TKGS is one of the cleaner parts of the design because it uses First Class Disks (FCDs) and the Cloud Native Storage (CNS) control plane in vCenter.
A vSphere storage policy is exposed inside the Supervisor as a Kubernetes StorageClass. When a workload in a TKG cluster creates a PersistentVolumeClaim, the CSI driver in the TKG cluster calls into a paravirtual CSI driver that forwards the request to the Supervisor CSI, which in turn calls CNS in vCenter to create an FCD on a datastore that matches the policy.
A subtle but important point. The FCD is not owned by the TKG cluster. It is owned by CNS in vCenter and is loaned to the worker VM through hot-add attach. If a worker VM dies, the FCD is detached and reattached to whichever worker the pod reschedules onto. This is what makes StatefulSets survive node loss without any external orchestration.
Authentication and RBAC
TKGS uses vCenter SSO as the identity source for the Supervisor. The kubectl vsphere login command exchanges vCenter credentials for a JWT that the Supervisor API server accepts through a custom authentication webhook. RBAC bindings on vSphere Namespaces map vCenter users and groups to Kubernetes roles.
Inside a TKG cluster, the picture is different. On first cluster creation, the user who created the TKC becomes cluster-admin via a bootstrap ClusterRoleBinding. From there, most teams either federate with an external OIDC provider through Pinniped or manage RBAC natively. There is no automatic identity federation from vSphere SSO into the TKG cluster unless you enable Pinniped explicitly through the addon system.
Lifecycle management and ClusterClass
Newer TKG Service versions have moved from the older TanzuKubernetesCluster v1alpha2 API to the Cluster API v1beta1 Cluster type driven by ClusterClass. This changes the operational model in a real way.
With ClusterClass, the Supervisor ships a set of blessed templates that define control plane and worker topology, machine images, and addon configurations. When you create a Cluster resource that references a class, you get variables instead of raw manifests. Upgrades become a matter of bumping the class version, which the TKG Service rolls out node by node using CAPI's rolling update logic.
Node image upgrades are content-library driven. New Photon or Ubuntu OVA versions land in the subscribed content library, VM Operator sees them, and rolling replacement happens by creating a new Machine, waiting for it to become Ready and drain the old one, then deleting the old Machine. Etcd is upgraded on control plane nodes one at a time with quorum preserved throughout.
High availability and failure domains
Supervisor control plane VMs are placed on three separate ESXi hosts using vSphere DRS affinity rules that TKGS creates on your behalf. Loss of any single host triggers HA restart of the CP VM on a surviving host, and the remaining two CP VMs continue serving the API during the outage because etcd quorum survives.
TKG cluster control planes follow the same pattern within their assigned Namespace. Failure domains can be defined at the Supervisor level to place worker nodes across vSphere zones, which correspond to distinct vSphere clusters or fault domains. This is how you build multi-AZ style topologies without leaving vSphere.
The failure mode worth planning for is loss of the Supervisor itself. If the Supervisor becomes unavailable, existing TKG clusters continue running workloads because their control planes are independent. What you lose is the ability to create, scale, or upgrade TKG clusters until the Supervisor is restored. This is why treating the Supervisor as production infrastructure with proper backup of the wcpsvc state on vCenter and etcd snapshots is not optional.
Where this leaves the practitioner
TKGS is not a wrapper around upstream Kubernetes with a vSphere theme. It is a Kubernetes-native control plane grafted onto vCenter, using CRDs to model vSphere primitives and Cluster API to bridge intent into VMs. The architectural payoff is that everything from namespace quotas to storage attach to node upgrades is reconciled the same way, through controllers watching desired state. The cost is that a mistake in Supervisor sizing, NSX-T topology, or content library subscriptions cascades into every TKG cluster downstream. Design the Supervisor once, and design it carefully.
