Crossplane Is a Control Plane, Not Terraform in a Pod

At 02:13, kubectl get managed shows a database stuck with SYNCED=False. The application team sees a Kubernetes object. The cloud account contains a half-created database, three security groups, and a subnet association nobody remembers requesting. Welcome to Crossplane operations.

This is not a criticism. Crossplane can be an excellent foundation for an internal platform, particularly when Kubernetes is already your operational centre of gravity. But treating it as Terraform with YAML misses the point and creates some nasty failure modes. Crossplane is a control plane. It continuously reconciles desired state, owns external resources, and exposes infrastructure through Kubernetes APIs. That changes how you design, secure, upgrade, and debug the system.

The useful bit is the API you build

Crossplane providers add custom resources for external systems. A cloud bucket, database, network, or identity policy becomes a Kubernetes object. With Crossplane v2, managed resources can be namespaced, and Compositions can create both cloud resources and ordinary Kubernetes resources.

A current installation starts conventionally:

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane crossplane-stable/crossplane \
 --namespace crossplane-system \
 --create-namespace

Providers are installed as packages. This example uses the AWS S3 provider because the resource is easy to recognise, not because the design depends on AWS:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
 name: crossplane-contrib-provider-aws-s3
spec:
 package: xpkg.crossplane.io/crossplane-contrib/provider-aws-s3:v2.0.0

Once credentials and a provider configuration exist, a bucket looks like this:

apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
 namespace: payments
 generateName: receipts-
spec:
 forProvider:
 region: eu-west-1
 providerConfigRef:
 name: payments-cloud
 kind: ProviderConfig

That is the lowest-level interface. Giving application teams direct access to every provider field is usually a mistake. You have merely moved the cloud console into kubectl, complete with hundreds of choices and several creative ways to run up a bill.

The better move is to define a small composite API that expresses what teams actually need. An application team should request something like PostgresInstance with a service tier, retention period, and owning team. The Composition decides engine settings, network placement, encryption, labels, backup policy, and which provider resource implements the request.

This is where Crossplane earns its keep among devops tools. The product is not the provider CRD. The product is your API and the guardrails behind it.

Reconciliation is both the feature and the trap

Crossplane continuously compares the managed resource with the external system. If someone changes a field in a cloud console, the provider normally changes it back to match spec.forProvider. That is useful automation, provided everybody understands who owns the resource.

During an incident, an engineer may make a console change to restore service. Minutes later, Crossplane reverses it. The controller is doing exactly what it was asked to do, with impeccable timing and no sympathy.

Write the ownership model down. Better still, enforce it. Restrict direct cloud changes, send audit events somewhere operators actually watch, and make the break-glass procedure include pausing reconciliation or updating the declared state. Do not leave people to discover the controller loop while the incident clock is running.

Deletion needs the same attention. Removing a managed resource usually tells Crossplane to delete the external resource. A careless Git merge can therefore become a cloud deletion workflow. Use protected branches, admission policies, and appropriate deletion policies. Test deletion in a disposable account before trusting it with databases or network foundations.

Crossplane also depends on finalizers. If a provider is broken, uninstalled, or unable to authenticate, deletion may stall. Manually removing a finalizer can orphan the cloud resource. Sometimes that is the least bad recovery option, but it should be a conscious decision followed by an inventory check, not a copied command from a chat thread.

Compositions need software engineering, not YAML stamina

A Composition is executable platform behaviour. Review it accordingly.

Crossplane v2 uses Composition Functions for transformation and orchestration. Functions are packages that run in a pipeline, receiving observed and desired state before returning the resources Crossplane should apply. They are more capable than a long pile of patches, and capability brings testing obligations.

Use the Crossplane CLI to render Compositions before they reach a cluster:

crossplane render xr.yaml composition.yaml functions.yaml

Commit representative inputs and rendered expectations. Exercise invalid tiers, missing fields, region restrictions, resource renames, and upgrades from the previous Composition revision. Schema validation catches malformed input. It does not tell you that changing a composed resource name will replace a production database.

Publish Compositions as versioned configuration packages, pin provider and function versions, and promote those packages through environments. Crossplane supports OCI packages and digest references, which makes releases repeatable. Floating tags belong in experiments, not production control planes.

The official Composition documentation also calls out an easy-to-miss detail: composing third-party Kubernetes resources requires explicit RBAC for the Crossplane service account. Treat that permission as production code. A wildcard ClusterRole added to make reconciliation turn green is still a wildcard ClusterRole after everyone forgets why it exists.

Operating it means operating another platform

Teams evaluating new tools often count the happy-path YAML and ignore the controller estate. Crossplane adds core controllers, provider controllers, functions, package revisions, credentials, CRDs or managed resource definitions, and external API dependencies. Each component has versions, logs, resource limits, and failure modes.

Monitor more than pod health. Alert on managed resources that remain unready or unsynchronised, provider health, reconciliation errors, authentication failures, and unusual reconciliation latency. Keep cloud-side monitoring too. A green Kubernetes controller cannot prove that an external dependency is useful to its consumers.

Start troubleshooting from both ends:

kubectl get managed
kubectl describe bucket.s3.aws.m.upbound.io -n payments
kubectl get providers
kubectl get providerrevisions
kubectl logs -n crossplane-system deployment/crossplane

Then inspect the relevant provider pod and the cloud audit trail. Conditions and events usually explain whether the failure is schema validation, credentials, quota, policy, dependency ordering, or an external API problem.

Provider upgrades deserve a staging cluster with realistic managed resources. CRDs change. API versions change. Provider behaviour changes. Crossplane v2 is compatible with legacy v1-style resources, but migration still requires deliberate testing. The v2 upgrade guidance explicitly recommends moving one minor version at a time.

Where I would use it

Crossplane fits when a platform team wants to offer opinionated, self-service infrastructure APIs and already knows how to operate Kubernetes controllers. It is particularly strong when one application request should produce cloud infrastructure, Kubernetes workloads, policies, and secrets through a single reconciliation model.

I would not introduce it merely to avoid learning another infrastructure language. For a small estate managed by one team, conventional infrastructure as code may be easier to review, plan, and recover. More tooling is not automatically more platform.

Being open source is useful here because providers, functions, and controller behaviour can be inspected. It does not remove the need to assess maintenance activity, supported resource coverage, upgrade history, and the security model of every package you install.

If you only do one thing this week

Take one proposed composite API and run a deletion and provider-failure exercise in a disposable account. Delete the claim, block provider credentials, change the external resource manually, and upgrade the provider. Watch what reconciles, what stalls, and what disappears.

If the team cannot explain each result from Kubernetes conditions and cloud audit logs, the API is not ready for production. Fix that before adding another Composition.

TecLeads helps engineering teams ship this kind of thing faster and more safely. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.