Operations

An installation is its database and its .env. Whoever backs up both can put it up again on another host. Beside that: what to watch, how to get back in when nobody receives mail any more, and what can go wrong day to day.

Backing up

Two things, every night: a dump of the database and a copy of the .env.

#!/bin/sh
# /etc/cron.daily/heinzel-pro-backup
set -eu
dir=/var/backups/heinzel-pro
install -d -m 700 "$dir"
cd /opt/heinzel-pro
docker compose exec -T db pg_dump -U heinzel -Fc heinzel_pro \
  > "$dir/heinzel-pro-$(date +%F).dump"
cp .env "$dir/env-$(date +%F)"
# Then copy both to where the host is not: another host, a bucket, a tape.

The file must be executable, and the line #!/bin/sh must come first, or cron skips it with "Exec format error". Check it once the way cron calls it:

$ sudo chmod 755 /etc/cron.daily/heinzel-pro-backup
$ sudo run-parts --verbose --regex '^heinzel-pro-backup$' /etc/cron.daily
$ ls -l /var/backups/heinzel-pro

With your own PostgreSQL the dump belongs in its backup; with the bundled one, the volume pgdata is the database, and the dump is the way, not a copy of the volume while it runs.

Restoring

On a new host, by the installation guide up to and including the compose file, then:

1. Put the saved .env at /opt/heinzel-pro/.env. With the same HEINZEL_VERSION the dump was taken at; an upgrade comes afterwards.

2. Start the database only and restore the dump.

   $ cd /opt/heinzel-pro
   $ sudo docker compose up -d db
   $ sudo docker compose exec -T db pg_restore -U heinzel -d heinzel_pro --clean --if-exists \
     < /var/backups/heinzel-pro/heinzel-pro-2026-10-24.dump

3. Start the application and check.

   $ sudo docker compose up -d
   $ curl -s http://127.0.0.1:4000/health

   Sign in, test a server: if Heinzel signs in, the keys are readable, and the restore is complete.

Point the DNS name at the new host, set up the proxy, and the VPN connections: by way 1 the other sides report back by themselves once the name resolves; by way 2 Heinzel calls the router.

Rotating the vault key

1. Generate a new key: openssl rand -base64 32
2. In the .env move the old key to HEINZEL_VAULT_KEY_RETIRED and set HEINZEL_VAULT_KEY_RETIRED_ID to the old id (so far 1); enter the new one as HEINZEL_VAULT_KEY with HEINZEL_VAULT_KEY_ID=2.
3. Restart, then re-encrypt every secret: sudo docker compose exec app bin/rotate-vault-key
4. Remove the RETIRED lines and start once more. Back up the .env anew.

Watching

GET /health returns status, version, rules_version, database and pending_migrations and answers with 200 only when the database is reachable and no migration is pending. A monitor that checks https://heinzel.example.com/health for 200 from outside every minute is enough for the question "is it running". Requests on localhost stay plain even with FORCE_SSL, for the probe from the host.

In the application, Installation, Log says the same and more in the box "Operation": license, mail delivery, emergency stop and the background jobs (waiting, scheduled, running, retrying, given up). A growing counter at "retrying" is a mail server or a webhook that does not accept.

The other way round, Heinzel Pro takes alerts from your monitoring and looks at the affected server, see Alerts from the monitoring.

Logs

sudo docker compose logs app shows the application's log; LOG_LEVEL in the .env turns it from info to debug or warning. What users do is not in it but in the audit log in the database: one chain per tenant, one of its own for the installation. Both export as CSV and verify:

$ sudo docker compose exec app bin/heinzel_pro eval HeinzelPro.Release.verify_audit

If the log is to go to a SIEM, an administrator of the installation gives a tenant an HTTPS endpoint under Settings, Integrations; every event then goes there within a minute as one line of JSON, with position and hash in the chain. Retention is set per tenant under Settings, General; deletion after the period is the only way rows are removed, and is logged itself.

Locked out

If no mail arrives any more, or nobody knows the only administrator's password, an operator with shell access on the host makes a sign-in code for any account:

$ sudo docker compose exec app bin/login-code admin@example.com

The code works once and for 15 minutes; type it with the address on the sign-in page. Then set up password or second factor anew under Account. If somebody lost the device with their passkey or authenticator app, an administrator of the installation resets that account's passkeys and second factor under Installation, Accounts or sends a sign-in code. There a person can also be put into a tenant whose only administrator is locked out.

Changing mail later

Mail is set up in the .env, because it has to work before anybody can sign in. A new way means: change the lines, sudo docker compose up -d, then a test mail under Installation, Mail delivery. The page shows what currently applies.

Every mail has the same frame: a greeting by name where the address belongs to an account, else "Hello", the sender name from MAIL_FROM_NAME as sign-off, and below a footer with the sender, the installation's address, why the mail comes, and the imprint. The imprint is entered on the same page in the field Imprint under every mail; the preview below it shows the test mail as it arrives. Left empty, the mail has no imprint.

The same page holds the Daily report: every morning after six (Berlin time) each administrator of the installation gets the day before in figures, in their language. Four tiles open it: accounts, tenants and servers with the day's new ones big and the total small beneath (and the deleted ones, if there were any), and the day's orders with what they bring in a month and the cancellations. Beneath the figures it names each person behind them, so a call the next day is answered from the mail: every new account with name and address, the time it signed up, how long it has been in the system, whether the address is confirmed, and each of its tenants with tariff, role, a model connected or not and the number of servers and sessions; every deleted account with the time, by whom and after how long; every tariff change from what to what, with who made it; and the tenants deleted. A line beneath shows how much work was done yesterday: active tenants, how many people were in the pages, the working time as the tariff counts it with the agent's share, sessions begun and commands run; beneath it a table of the ten tenants with the most working time in the last seven days, each with its tariff and its owner's address. The new accounts are a table too. It counts per tenant, never per person: who inside a tenant did how much is the tenant's business. The subject names only the day's new accounts and new orders with what they bring in a month (a yearly order counts a twelfth); the day is the mail's first line. It is on from the start and switched off there with a tick. A restart at that hour costs no report: the next hourly run makes it up.

The emergency stop

A tenant's administrators have it under Settings, Policy, the installation's under Installation, Tenants for all tenants at once. It stops every session, takes back every waiting command, aborts running commands on the hosts, and from then on nothing starts until it is lifted, not after a restart either. It asks for a reason, stands in the audit log, and a banner on every page shows that it is on.

What leaves the installation

An installation opens connections to these places and to no other. There is no telemetry, no usage figures and no license check to the outside, only the fetch of a renewed key; a test in the product's test suite fails as soon as the code opens a connection not listed here.

• Model providers: the address of every provider you create, preset api.anthropic.com or api.openai.com, or a server inside your own network.
• Your servers: SSH to the hosts you add, on the port you give.
• VPN connections: WireGuard to the peer from the block you pasted; a Tailscale connection talks to Tailscale's coordination server and relays or to your Headscale.
• Mail: the mail server or mail service you set up.
• Endpoints you enter: the team's webhook, the collector for the audit log, the monitoring that takes annotations.
• Single sign-on: the OpenID Connect issuer you set up.
• Web pages: a page the agent reads with fetch_url, fetched by the installation itself, and through the headless browser if one is set up.
• Rule set: api.github.com and github.com, hourly for the check for a newer rule set and when updating, without an account and without anything about the installation. Whoever switches the automatic update off and uploads the archive never opens this connection.
• License renewal: heinzel-pro.de, four times a day, only with a license key. It sends the key the installation holds and nothing else; Installation, Settings, License switches it off.

Docker itself fetches the image from ghcr.io when you pull it.

Removing Heinzel Pro

For an installation set up by the installation guide, say after the trial. The data sits in the pgdata volume; the servers it looked after keep Heinzel's access until you remove it there.

1. A last backup, if you want to keep the data. Run the script from Backing up once by hand, then copy dump and .env off the host. After this the audit log only exists in the dump.

2. Remove containers, database and images.

   $ cd /opt/heinzel-pro
   $ sudo docker compose down -v --rmi all
   $ cd / && sudo rm -rf /opt/heinzel-pro /var/backups/heinzel-pro
   $ sudo rm -f /etc/cron.daily/heinzel-pro-backup

   -v deletes the volume with the database, for good.

3. The proxy and the certificate. With nginx remove the site and delete the certificate, with Caddy the block in the Caddyfile:

   $ sudo rm /etc/nginx/sites-enabled/heinzel-pro /etc/nginx/sites-available/heinzel-pro
   $ sudo nginx -t && sudo systemctl reload nginx
   $ sudo certbot delete --cert-name heinzel.example.com

4. The firewall and mail. Close the rules from the installation again, and whoever connected Postfix on the host takes the Compose network out of mynetworks:

   $ sudo ufw delete allow 51900:51919/udp
   $ sudo ufw delete allow from 172.30.79.0/24 to any port 25 proto tcp
   $ sudo ufw delete allow from 172.30.79.0/24 to any port 11434 proto tcp   # only with Ollama on the host
   $ sudo postconf -e "mynetworks = $(postconf -h mynetworks | sed 's| 172.30.79.0/24||')"
   $ sudo systemctl reload postfix

5. On every server Heinzel looked after. Delete the key line ending in heinzel-pro from authorized_keys. Whoever created the user heinzel for it removes it with its sudo rule:

   $ sudo deluser --remove-home heinzel
   $ sudo rm -f /etc/sudoers.d/heinzel

   VPN connections on the other side's routers and servers stay until you delete them there.

6. Docker, if it was only there for Heinzel Pro. On Debian sudo apt purge docker.io docker-compose, then sudo apt autoremove --purge once apt-get -s autoremove has shown it takes only Docker's packages. Delete /var/lib/docker and /var/lib/containerd by hand. The firewall chains Docker created (DOCKER, DOCKER-USER) and the forwarding (net.ipv4.ip_forward) go only with a reboot of the host.

Check at the end that nothing listens any more: ss -lntu | grep -E ':4000|:519' stays empty.

Troubleshooting day to day

For everything else: support@heinzel-pro.de with the line from the log and the version from /health.