The TYPO3 deployments kept their state in etcd — stopped, deploying, deployed, installing, installed, starting, started. Useful, but invisible unless you query etcd by hand.
This exporter reads that state per project and stage, including required versus running services, and exposes it as Prometheus metrics — so dashboards and alerts could finally answer "is the deploy done, and is everything up?".
| Fact | Detail |
|---|---|
| Source | etcd keys for project + stage |
| Output | Prometheus metrics |
| Tracks | Deployment status, required vs. running services |
| Stack | Node (express, node-etcd, prom-client) |
| Started | February 2018 |
What's in it
Each deployment state maps to an ordered value, so alerts can fire on "stuck before installed".
Environment-driven configuration keeps it reusable across projects and stages.
In detail
What is actually inside it
TYPO3_SERVICES and TYPO3_REQUIRED_SERVICES are read separately, so the exporter can tell "this container is still starting" from "this deployment is missing a service it declared".
PROJECTKEY, STAGE, and an etcd endpoint built from MACHINEID: one exporter container serves every project and stage without per-project code.
Mapping an enum to a monotonically increasing number is a small decision that makes dashboards and alert rules dramatically simpler.
| Detail | Value |
|---|---|
| Source | etcd keys per project + stage |
| Output | Prometheus metrics (express + prom-client) |
| States | stopped → deploying → deployed → installing → installed → starting → started |
| Started | February 2018 |
Technical notes
The decisions behind it
Why expose etcd state through Prometheus at all?
What did alerts look like?
installed for 15 minutes" — possible because the status is a scale rather than a set of labels.Reading the state
One key per service, one endpoint per machine
The state lives in etcd as one path per service, and exactly three environment values appear in it:
/ruhmesmeile/projects/typo3/${TYPO3_STAGE}/${TYPO3_PROJECTKEY}/status/${service}/current
The value of that key is the state name. The TYPO3_STAGE segment started as a hardcoded review literal and became an environment variable in commit 6f086fc5. The endpoint is derived as well: https://${MACHINEID}.etcd.services.ruhmesmeile.local:2379. An earlier revision targeted ${MACHINEID}.ttt.roles.addresses.services.ruhmesmeile.local and was changed because the hostname has to be one the client certificate was issued for; the certificate is read at process start from three fixed paths — /etc/ssl/etcd/ca.pem, /etc/ssl/etcd/calculonc.pem, /etc/ssl/etcd/calculonc-key.pem — so a single project's certificate name sits inside the exporter that serves every project.
TYPO3_SERVICES and TYPO3_REQUIRED_SERVICES do different jobs. A service in TYPO3_SERVICES gets a watcher on current plus one initial GET, and therefore a metric. A service that is only required is seeded in etcd instead: current = stopped is written with prevExist: false, then a sibling stopped marker key is touched. That precondition is why a project whose keys already exist logs a 412 on every exporter restart instead of having its state reset. One asymmetry is easy to miss: a service listed only in TYPO3_REQUIRED_SERVICES gets an etcd path but no series, because the export loop iterates TYPO3_SERVICES alone.
| State | Value |
|---|---|
| `stopped` | 0 |
| `deploying` | 1 |
| `deployed` | 2 |
| `installing` | 3 |
| `installed` | 4 |
| `starting` | 5 |
| `started` | 6 |
The metric contract
Two series, no project label
prom-client gets exactly two series. typo3_current_status is a Gauge labelled service only, set to the ordinal from the table above. typo3_status_counter is a Counter labelled service and status. The scale is a public contract rather than an internal detail: the deploying and deployed states were added on 2018-03-03 and every later value shifted up by two, so work that had encoded installing = 1 was afterwards reading deploying. Ordering is what makes the numbers cheap to alert on; it is also what makes insertions expensive.
The increment is conditional, so the counter means less than its name suggests. Before counting, the exporter fetches the sibling key .../status/<service>/<state> and only proceeds if that key exists — the marker keys are the deploy side's record that a state was reached, and the exporter creates none of them itself apart from the stopped seed for required services. The counter therefore counts status changes the pipeline left a marker for, not transitions observed; and because it lives in the process, it restarts at zero with the container while the etcd keys survive. Its help text still reads Counter for number of status changes, timestamped, although the timestamp was dropped on 2018-03-20: the marker key's value used to be passed as a timestamp argument (set(value, ts) / inc(1, ts), seconds × 1000) and the leftover callback parameter is still named timestamp, now unused.
Neither metric carries the project, stage or machine. Series identity comes from the Prometheus target, which forces a scrape config per deployment; the payoff is that alerts stay one line. Because the scale is monotone and the pipeline walks it in order, "is this deploy progressing?" is a comparison rather than a state check: typo3_current_status{service="typo3"} < 4 held for fifteen minutes covers every stall between deploying and installed without naming them. The cost is that 0 / stopped is both the healthy idle value and the value a deploy that never got anywhere sits at, so no instantaneous rule can separate idle from broken — only duration can.
Where it bites
Failure modes of a 116-line exporter
What happens when etcd returns a state the scale does not know?
STATUS[getStatusName(value)] is undefined, and Gauge.set(undefined) in prom-client 10.2.2 throws TypeError: Value is not a valid number: undefined. The throw happens inside an etcd callback, there is no try/catch and no uncaughtException handler, so the exporter process exits and the container restarts — taking every counter with it. The only guard is getStatusName(): a value containing stopped or installing is folded back onto those two names, everything else is looked up verbatim.Does the watcher survive an etcd outage?
node-etcd's watcher retries on its own: on a connection error it emits reconnect and re-watches after 2^attempts * 300 ms plus jitter, resuming from the last seen modifiedIndex, and the 401 resync path handles an index the cluster no longer has. The exporter subscribes only to change, so none of that is visible — during a reconnect the gauge silently keeps its last value, and a dead watch looks exactly like a steady deployment. node-etcd also emits error for an unexpected response, and an error event with no listener on an EventEmitter is a thrown exception, which is another restart.Is shutdown graceful?
SIGTERM handler calls server.close(...), but server is never assigned — app.listen(PORT, HOST) discards its return value — so the handler dies on ReferenceError: server is not defined before watchers.forEach(... stop()) runs, the intended process.exit(0) is never reached and the process ends with exit code 1. The same file has no defaults for its configuration: process.env.TYPO3_SERVICES.split(',') runs at load, so an unset variable is a boot-time crash, and the value "typo3, mail" produces a service name with a leading space whose key lookups all fail and are only logged.Links
Where to find this one