Schedules, findings and rollouts

A schedule lets Heinzel look after your servers at fixed times, with nobody there. What it finds and may not fix alone becomes a finding. And a task that is to do the same on many servers is a rollout.

Creating a schedule

The example's task:

Input: Install the pending security updates on all servers and check afterwards that every service is running. Whatever a server's journal excludes, leave in place and report it. No reboot.

Whoever changes level, task or model becomes the schedule's owner; it runs in that person's name and as far as a session of theirs could go alone. A schedule belongs to a group, a person or everybody; the people of that group change, pause and start it.

The list is built like the sessions list: tabs for All, Running, Last run failed and Paused, then one line per schedule with group, name and task, where it runs with its rights in words, and on the right its rhythm in words ("Sun at 04:00") over the cron fields, each named and the set ones marked, with the next run. Ordered by the next run, the list stands under headings: what runs several times a day first, then each day with its date, paused schedules last. The sign in front says how the last run went. Every head orders the list; the menu also orders by cron expression and rights. With the boxes you run, pause or enable several schedules at once. A click on the line opens the schedule's page: task, when, where and with which rights, the latest runs with their sessions, a run under way with its steps, and the buttons Run now, Pause, Edit, Remove and, while it runs, Stop.

What happens at night

A run is an unattended session per server, and it can ask nobody. Reading commands run. Changes run where the schedule stands on rwx and tenant, owner and server allow it. Everything else is skipped, named in the report under "Needs an operator" and kept for a week on the Approvals page, where somebody takes it over into a session of their own. A server group can carry a maintenance window (weekdays, from, until); outside it a run only reads.

In the morning a mail, as text and as HTML: first the hosts that need somebody, then one line per host where everything is in order.

The whole night with screens, as a recipe: Security updates every night.

Findings

What a run found to be a problem stands under Findings: one entry per server and problem. Seen again the next night, it stays one entry with a counter; if a later run of the same schedule no longer sees it, it closes by itself. You take a finding over, close it by hand or start a session from it: the proposed task is already in the form, read it before you start.

The page is laid out like the list of sessions: on the left one line per finding with its severity, the server, the finding, the schedule that saw it and when it first appeared; on the right the chosen finding with Where it comes from (schedule, run, server), Why it is here (new, worse than before, back after somebody closed it, or reminded again on request), What the check saw and its suggestion. With the boxes you close several at once.

Rollouts: the same on many servers

New session, "The same on many": write the task once, choose the servers, pick the level. The level holds for every session of the rollout; at "Changes alone" none of them waits for an approval. At most 50 servers.

The servers stand in a list as under Servers, with their load now and the day of their last session. A click on a row chooses the server. The search takes names, regular expressions and #tag; "Add all matches" or the box in the bar chooses everything that matches. The bar counts the servers chosen, and "Clear the choice" lets go of all of them. With "full" in the column head the servers are named in full instead of by their first label.

"How do they run?" has three answers:

Under "When?" a rollout starts now or at a time, as a single session does. Until then it stands in the sessions list with its first server; stopping it calls it off.

After the go of "First one, then all", only those commands run on the other servers without asking that a person approved on the first and that succeeded there, none of them irreversible, as the same text and in the same order, each only once the one before succeeded. Everything else asks as always. Every server keeps its own session, its own record and its own stop button.

The rollout's page shows its sessions in the same rows as the sessions list: task, server and last change, with a sign for the state in front. A row opens the session, a column head orders the list. Below them, numbered, stand the servers still waiting, in the order they get their turn.

Alerts from the monitoring

Let your monitoring send an alert to Heinzel Pro, and Heinzel looks at once: under Settings, Integrations you generate a token; the monitoring sends it as a bearer token to /api/alerts/<tenant>. Alertmanager's webhook is accepted as it is; any other system sends JSON with host, title and text.

$ curl -X POST https://heinzel-pro.de/api/alerts/example \
  -H "Authorization: Bearer $HEINZEL_ALERT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"host": "web03.example.com",
       "title": "Disk /var 92 % full",
       "text": "Threshold 90 % exceeded for 10 minutes"}'

If the host is one of your servers, a session starts there that only reads and reports cause and proposed fix; the answer to the call holds the session's address. The same alert finds its session again within an hour, and at most twenty sessions start per hour. The other way round, Heinzel Pro marks changes in the monitoring: with an endpoint that takes annotations like Grafana, every executed command appears in the graphs.

What notifies you

Under Settings, Alerts: mail addresses and a webhook (Slack, Teams, Mattermost, generic JSON), and a box per event: approval needed, command failed or denied, host key changed, scheduled run finished, repeated failed sign-ins, user created, an approval waiting too long, a model the provider retires, and a weekly mail of what was left undone. Events an upgrade brings are off until you switch them on.

The API

A pipeline that has Heinzel check a deployment, a script that creates servers, a report fetched every morning: under Administration, API tokens an administrator generates a token with the scopes read, sessions and servers. It acts in the name of the person who created it and never may more than they. And it never approves a command: it sees what waits, and it can deny; an approval is a person in the browser.

# Start a session
$ curl -X POST https://heinzel-pro.de/api/v1/example/sessions \
  -H "Authorization: Bearer hzp_…" \
  -H "Content-Type: application/json" \
  -d '{"server": "web03.example.com",
       "task": "Check why nginx has been returning 502 since the deploy at 14:02"}'

# Read its state and the last answer
$ curl -H "Authorization: Bearer hzp_…" \
  https://heinzel-pro.de/api/v1/example/sessions/0199e2c4-…

# The change record as Markdown
$ curl -H "Authorization: Bearer hzp_…" \
  https://heinzel-pro.de/api/v1/example/sessions/0199e2c4-…/record

An unknown tenant and a wrong token both get 401 with the same body; a call outside the scope 403; one that cannot be done as things stand (a session that is busy) 409. 120 calls per minute and address are the default. The script heinzel from the repository is a shell wrapper around these calls.