Moving an app onto single-node Kubernetes, no downtime

TimeTracer used to run as a handful of Docker containers behind nginx. It now runs on k3s on the same machine, which also hosts a mail server, a CI server and an image registry. The move happened without a maintenance window, and the interesting parts were not Kubernetes at all. They were the three decisions that had to be made before anything was applied, and the two secrets that could not be regenerated.

If you are considering the same move, this is the shape of the work.

Decide who owns port 443 before you install anything#

k3s ships Traefik and Traefik wants ports 80 and 443. On a box that already had nginx terminating TLS for a mail interface and a CI server, that is a fight, and it is a fight you want to resolve on paper rather than by discovering which one lost at install time.

We gave the ports to the host's nginx and moved Traefik to NodePorts, reachable on loopback only. nginx terminates TLS for everything, then proxies the application hostnames to Traefik's HTTP NodePort. Certificates stay with certbot on the host, which was already renewing them for the services that are not in the cluster.

The alternative — Traefik on 80 and 443, with everything else proxied through it — would have meant moving certificate issuance into the cluster and making the mail server depend on Kubernetes being healthy. The mail server is how we find out the cluster is unhealthy. That settled it.

The cost of our choice is one extra hop for every request and a second place where a timeout can be configured. Worth it.

Decide where images come from, and keep it off the network#

A single-node cluster pulling from a public registry on every rollout is slow and depends on something outside the box. We run a registry on loopback, with no TLS and no authentication, and tell k3s about it through /etc/rancher/k3s/registries.yaml.

That sounds careless written down, which is why it is written down. The registry binds to 127.0.0.1 only, and that was verified from outside rather than assumed. TLS on a loopback connection protects against an attacker who is already root on the box, at which point they have the images anyway. Authentication would mean a credential in the kubelet config to protect images that are already readable from the filesystem. Both would be ceremony.

The useful property is that every image is tagged with the commit SHA that built it, and nothing is tagged latest. A deployment therefore names exactly one immutable artefact, the cluster state says which commit is running, and a rollback is a tag you already have on disk.

Decide what cannot be regenerated#

This is the part that bites. Most secrets can be rotated at a cutover and nobody notices. Two of ours could not.

The token signing secret signs login sessions and also signs the time-limited URLs for screenshots. Regenerate it and every logged-in user is logged out, and every screenshot link in a cached page or an e-mail is invalid. Not a disaster, but an unnecessary and very visible one on the morning of a migration.

The web push key pair is worse, because it cannot be fixed afterwards. Browser push subscriptions are bound to the public key they were created with. Replace the pair and every existing subscription is silently dead: no error, no notification, no way to re-subscribe a user without them visiting the site again.

So those two came across unchanged, and everything else — database passwords, API tokens — was generated fresh. Writing that distinction down before the cutover is what made it a decision rather than an incident.

One more that belongs in the same category: the database's own privileges. The application authenticates as a user with read and write on one database and nothing else. The administrative account exists, but only the backup script and a restore use it. On the old setup the application had been connecting as the administrative user, which is the kind of thing a migration is a good excuse to fix, because you are rewriting the connection string anyway.

Then the boring Kubernetes part#

With those three decided, the manifests are unremarkable, with three details worth copying.

One replica, and say why. The API holds a WebSocket hub and several background workers in process. Two replicas would mean two reapers racing each other and clients connected to whichever pod the proxy chose. One replica is correct here, and the manifest has a comment saying so, because the next person's instinct will be to scale it.

maxUnavailable: 0 and maxSurge: 1. That is what makes a rollout zero-downtime on a single replica: the new pod must become ready before the old one is taken away. Without it, the default strategy will happily remove the only pod it has.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

On one replica the default maxUnavailable: 25% rounds down to zero, so writing it out looks redundant right up until somebody sets replicas: 2 and finds out it was not.

Tell the framework which proxies to trust. The API rate-limits logins by client address. Behind a proxy, every request arrives from the proxy, so without a trusted-proxy configuration the limit is shared by every client on the internet — one person getting their password wrong locks out everybody. The list is the pod network CIDR and loopback:

TRUSTED_PROXIES=10.42.0.0/16,127.0.0.0/8

This is the single easiest thing to miss when an application moves behind an ingress, because nothing breaks: the limit simply applies to the wrong thing.

The cutover#

The order we used, which is the order we would use again:

1. Build and push the images, tagged with the commit SHA. Nothing is serving them yet.
2. Create the namespace, the config and the secrets. Verify the two reused values are byte-identical to the old ones, because this is the last cheap moment to notice.
3. Restore the database into the cluster's instance from a fresh dump, and bring up the API with no ingress route pointing at it. Exercise it directly through a port-forward: log in, load a session, fetch a screenshot. A pod that passes its readiness probe has only told you that one HTTP handler returns 200.
4. Copy the uploads directory across and leave it syncing. Ours is a few hundred megabytes of screenshots, so the first copy is slow and the second is not.
5. Add the ingress rule. Switch the nginx server block from the old container to Traefik's NodePort and reload. This is the only step with a user-visible effect, and reloading nginx does not drop established connections.
6. Watch for a few minutes with the old stack still running and still able to serve. The rollback is one nginx reload for as long as you leave it in place.
7. Final sync of uploads, then stop the old containers.

We did not change DNS at all, which removed the entire class of problems that come with TTLs and caching resolvers. The hostname pointed at the same IP before and after; only what was listening behind nginx changed. If you can arrange your cutover that way, do.

What we would tell someone considering it#

Single-node Kubernetes on a box you share with other things is a real option, and the reason to do it is not scaling. It is that the manifests describe what is running, in a repository, in a form you can diff. Our docker-compose.yml had drifted from reality in several small ways that nobody had noticed, including a debug logging flag left on in production. Writing the manifests surfaced all of it, because you cannot write a manifest for something you do not understand.

The honest cost is that you now have a control plane to keep alive, and on one node there is no other node to move to. We accept that because the alternative we were comparing it against was not a managed cluster; it was the compose file that had quietly stopped describing the truth.