Building a Kubernetes Platform Without Vendor Lock-In

When an organization needs a new container platform, choosing a Kubernetes distribution is often one of the first decisions on the table. But what happens if you want to switch distributions, virtualization platforms, or even migrate the entire solution to the cloud in the future?

That was one of the central questions in an assignment where, acting as a solution architect, I helped an organization design a new, on-premises container platform.

From Development Platform to a Broader Container Service

The client already had an OpenShift-based container platform, primarily used by developers for in-house applications. Now, a new need had emerged. More and more commercial off-the-shelf applications were being delivered as containers, and the organization needed an environment where these could also be deployed and managed.

The existing platform was built around a large Kubernetes cluster where applications were isolated using namespaces. For third-party applications, requirements differed. Some required their own operators, specific configurations, or a higher degree of isolation than what the team wanted to handle within a shared cluster.

My assignment was to work alongside the client's technical team to create a solution design for a new, parallel platform. A key guiding principle was not to let the legacy setup dictate how the new one should be built.

We began by gathering requirements and discussing needs with various stakeholders. A few guiding principles quickly emerged: the platform had to be modular, automated, and portable. New clusters should be provisioned on demand without extensive manual intervention, and we wanted to avoid building in dependencies on any specific vendor or underlying infrastructure.

As the work progressed, a broader vision took shape. Instead of merely creating a siloed environment for third-party applications, we began to see the opportunity to build a unified container operations service. Over time, the development environment could also become just one of several clusters provisioned and managed through the same platform.

Choosing OpenShift While Avoiding Lock-In

One of the major challenges was selecting the Kubernetes distribution.

We evaluated several options. A Microsoft-based solution was considered for a time, but was ruled out because it could not meet the requirements for air-gapped operations without internet dependencies. We also explored using pure upstream Kubernetes.

The latter aligned well with the ambition to use open source and reduce vendor lock-in. At the same time, we had to account for the organization's current capabilities. Operating and maintaining Kubernetes requires deep expertise, and there was already in-house experience with OpenShift, as well as a requirement for an enterprise support model.

Ultimately, we landed on OpenShift Kubernetes Engine (OKE).

It was not the obvious choice given the initial desire to minimize vendor dependencies. However, we concluded that the priority was not necessarily to avoid a commercial Kubernetes distribution, but rather to avoid architecting the rest of the solution so tightly around it that it became locked in.

OKE would serve as the foundation, while automation, networking, storage, and other platform services would, wherever possible, leverage open and distribution-agnostic solutions.

If the organization later decided to replace OKE with another commercial distribution or upstream Kubernetes, the entire architecture would not need to be rebuilt.

Automation Made Multiple Smaller Clusters Feasible

Another critical part of the design was shifting how Kubernetes clusters were viewed.

Instead of one large shared cluster, we wanted the ability to spin up smaller clusters with dedicated roles. This provided better application isolation and made it easier to accommodate diverse technical requirements without impacting other systems.

However, such a model only works if creating and managing clusters is straightforward. Manually installing and configuring numerous Kubernetes environments would quickly become unsustainable.

The solution was to build on Infrastructure as Code and GitOps.

We designed a workflow where a new cluster could be provisioned based on a specification. Terraform was used to provision virtual machines on VMware, after which automation—primarily using Ansible—would install Kubernetes and the necessary platform components. Argo CD was included to manage configuration declaratively via GitOps.

By decoupling provisioning from OpenShift, we avoided locking cluster creation into the distribution's proprietary tooling. VMware was the initial virtualization platform, but the design allowed it to be replaced by another hypervisor or a public cloud provider without overhauling the entire process.

The architecture also included a dedicated management cluster for shared operational tooling such as GitOps, monitoring, and logging. Workloads would then run in dedicated application clusters tailored to their specific needs.

This provided a model where clusters could be treated as standardized resources rather than bespoke, handcrafted environments.

Proposed Workflow: From Specification to Kubernetes Cluster

1. Cluster requirement – Define the cluster in a specification.
2. Git / GitOps – Manage the specification as a declarative source of truth.
3. Terraform – Provision virtual machines.
4. Ansible – Install Kubernetes.
5. Argo CD – Configure platform services.

Target architecture: A management cluster for shared operational services and dedicated application clusters provisioned as needed. OKE on VMware initially. Portability prioritized by minimizing platform-specific dependencies.

Simplified target view of the proposed automation workflow.

The Same Mindset for Networking and Storage

The requirement for portability also shaped the choice of infrastructure components.

Storage was a prime example. The client had an existing enterprise SAN solution, but we wanted to avoid tightly coupling the Kubernetes platform's storage management to it.

For block storage, we chose Longhorn, which utilizes storage attached to the Kubernetes nodes and handles replication and persistent volumes directly within the cluster.

This does not mean the underlying storage disappears. The virtual machines still require disks, but Kubernetes does not need to be directly integrated with a specific SAN product. The exact same pattern can be applied whether the nodes run on VMware, another hypervisor, or in a cloud environment.

We also identified a need for S3-compatible storage among some of the prospective users. At the time, the organization lacked an established S3 service, so CubeFS was proposed as a potential option to provide this capability within the platform. This was put forward as a proposed solution rather than a finalized architectural decision.

Networking presented a similar challenge.

The existing OpenShift environment was tightly integrated with Cisco ACI. While functional within the existing infrastructure, it meant parts of the network management were locked into Cisco's platform.

In the new design, we chose Cilium as the CNI. This allowed us to manage Kubernetes network policies declaratively via GitOps and reduce dependencies on the underlying physical network fabric.

Furthermore, Cilium offered multi-cluster networking and policy management capabilities. This was particularly compelling since our architecture was based on multiple smaller clusters rather than a single large one.

This allowed us to establish a unified networking model without requiring all clusters to share the exact same physical network footprint.

Solution Architecture Is Also a Communication Challenge

A significant part of my role was developing and validating the design in close collaboration with the client's technical team.

My job was not to write all the Terraform or Ansible code myself, though I participated in technical discussions and worked closely with the engineers implementing the solution. The primary responsibility was ensuring that the technical decisions formed a cohesive whole that met the organization's requirements.

It was not always straightforward.

Requirements and the target vision evolved throughout the project, partly as we uncovered new needs and possibilities. We also had to justify why the organization should invest in an additional container platform when a working OpenShift environment was already in place.

Not all stakeholders shared the same technical background or had the same mental model of what Kubernetes was and what role the new platform was meant to play.

Being able to articulate the architecture across different levels of abstraction became just as critical as choosing the right technical components.

One of my key takeaways from the assignment was how much time you save by establishing a shared understanding early on—both of the immediate requirements and the long-term vision. At the same time, you have to be ready to revisit earlier decisions when circumstances shift.

A Platform That Can Evolve Over Time

The engagement resulted in a solution design for an automated, modular Kubernetes platform where multiple smaller clusters could be provisioned and operated through shared, repeatable processes.

The design accounted for the organization's existing skill sets and infrastructure, while actively avoiding letting those constraints dictate or restrict future technical paths.

This involved trade-offs. For example, we chose OpenShift for its enterprise support and existing in-house expertise, and VMware as the initial virtualization platform. But by relying on distribution-agnostic tooling and components, we minimized how much the rest of the stack depended on those choices.

To me, that is perhaps the most important lesson from this assignment. Vendor independence does not have to mean avoiding commercial products or trying to build everything in-house. It is about understanding the dependencies you introduce and ensuring they do not become harder to change than necessary.

A good architecture does not have to be independent of everything. But it should give the organization the flexibility to change direction without having to start from scratch.

Need Help Finding the Right Path for Your Container Platform?

We help organizations evaluate platform choices, design solution architectures, and plan for automated, secure, and portable container operations. Whether you are building greenfield or evolving an existing environment, we would be happy to discuss your prerequisites.