Upgrades¶

The controller¶

The controller follows a minor release line, such as 9.7. To pick up the latest patch release in that line, run this on the controller:

cstacks upgrade

It backs up the database first, then pulls the new image, runs any migrations, and restarts the controller. If the backup fails, the upgrade stops before changing anything.

Re-running the installer also upgrades the controller whenever the running image doesn't match the configured one.

To move to a new minor or major release, read that release's notes first. Then set controller_image_tag on the controller in hosts.yml, for example controller_image_tag: "9.8", and re-run the installer. This setting overrides the provisioner's default, so remove it once a provisioner update moves the default to that release or later.

SFTP containers after upgrading to 9.8.1¶

From v9.8.1, the default SFTP container image is ghcr.io/computestacks/cs-docker-bastion:v4. The older v2 image crash-loops on start on any controller from 9.6 onward, while the controller still shows the container as running. The upgrade moves the ComputeStacks Bastion Image setting from v2 to v4. If you had changed that setting yourself, it's left as it is; if it points at your own build, that build must read project metadata from the /v1/managed/ paths.

The image is read only when an SFTP container is created or rebuilt, so existing SFTP containers keep running v2 until you rebuild them. Restarting isn't enough. Rebuild each one from its power menu on the admin SFTP containers list (/admin/sftp). Rotating a project's SSH password also rebuilds its SFTP container.

Everything else¶

Every other component the provisioner installs, such as Docker, the ComputeStacks agent, borg, Prometheus, and Loki, is pinned to an exact version, and nothing upgrades on its own. The Docker, agent, HAProxy, and node exporter packages are held so that unattended upgrades can't move them.

To upgrade them, upgrade the provisioner and re-run it:

cd computestacks
git pull
make deps
make site ENV=prod

The installer converges every server to the pinned versions and restarts services as needed. Operating system security updates are installed automatically by unattended upgrades.

An agent upgrade restarts the agent, but not Docker or any container. From cs-agent 3.4.0, the agent's systemd unit only wants docker.service instead of requiring it, so stopping or restarting Docker leaves the agent running and answering the controller; anything that needs Docker fails and reports while it is down.

Docker on the nodes¶

Upgrading Docker restarts it, and customer containers have no restart policy, so a normal run doesn't upgrade Docker or reboot a live node. It leaves the node alone with a warning instead. Upgrade those nodes deliberately, in a scheduled customer maintenance window, after updating the provisioner:

make upgrade-docker ENV=prod                  # every node, one at a time
make upgrade-docker ENV=prod LIMIT=node1001   # one node, or resume a failed run

For each node in turn, the run places the node in maintenance and waits for its background work to drain, stops the running customer containers, upgrades Docker and any pending operating system updates, reboots if one is required, starts exactly the containers that were running, and releases maintenance. The first failure stops the run, and re-running it resumes. Before it touches any node, it checks that every targeted node keeps Docker and containerd on its data disk, and stops if one doesn't. It requires cs-agent 3.4.0 or later on every node and controller 9.8.0 or later, so upgrade the agent with a normal run first, before the maintenance window.

PostgreSQL 18¶

The provisioner installs PostgreSQL 18 on the controller and the nameservers, from the PostgreSQL project's own apt repository (apt.postgresql.org) rather than Ubuntu's. Servers installed when the provisioner used PostgreSQL 17 must be upgraded to 18 with pg_upgrade before you run the updated provisioner against them. The provisioner doesn't upgrade an existing database, so run pg_upgrade yourself first. This applies to the nameservers' PowerDNS database as well as the controller's. Back up the controller database first with cstacks database-backup.

Nodes without a data disk¶

The provisioner now requires a second, blank data disk on every container node, and keeps Docker and containerd on it. A node installed before this requirement has no data disk and can't be converted in place: a normal run, make upgrade-docker, and make validate all refuse it, and a normal run stops for every server, not just that node. Rebuild it with a data disk attached.

Environments installed with the v1 installer¶

The current provisioner doesn't upgrade an environment built with the earlier, Debian-based installer; it can only add regions to it. For what changed between the two, see Coming from v1 in the provisioner repository.