Before containers-as-a-service were a given, we ran our own cluster — and this repository described it as code: machines, services, environments and templates, plus the binaries and utilities to provision them.
It was the operational counterpart to the frontend work: the same instinct, that the description of a system belongs in version control and should be reviewable.
| Fact | Detail |
|---|---|
| Scope | Machine, service and environment definitions |
| Stack | Shell, CoreOS tooling |
| Started | December 2022 (repository) |
What's in it
Separate machine/service definitions per environment, with shared templates.
The quickstart in the internal wiki was enough to bring a new service up.
In detail
What is actually inside it
binaries/*-exporter/ each have a Dockerfile and a README and the binary: monitoring agents were versioned with the infrastructure that needed them, not fetched at boot.
configs/calculon1…calculon5 and configs/momcorpas/xtradb-node — per-machine configuration directories (yes, the machines were named after Futurama robots), with the Percona XtraDB node carrying its own clustercheckcron and entrypoint.sh.
A new service was a directory with a service definition and a template, which is what made the setup reviewable by people who were not on call that week.
| Detail | Value |
|---|---|
| Describes | machines, services, environments, templates |
| Operators | confd, fleetctl + mysqld/process/proxysql exporters |
| Repository | December 2022 |
Technical notes
The decisions behind it
Why hand-commit binaries?
confd and fleetctl were part of the deployment contract: pinning them in the repository meant a machine could not silently get a different version mid-incident. It is not elegant by today's standards, and it worked.What was the hard part?
Machine × service
A host and a unit, joined by a metadata string
Placement is not configured in a scheduler. It is declared twice, in two files, and matched by a string.
A machine is one machines/cloud-config.<host>.yaml — 25 of them. The file is a coreos-cloudinit document: it sets up flannel, pins etcd to v2.3.7, writes /opt/bin/fleet-wrapper, opts into reboot-strategy: etcd-lock, and starts fleet.service with the machine's identity in the environment:
Environment="FLEET_METADATA=role=calculatron,instance=1"
Environment="FLEET_METADATA=role=uhat"
A service is one services/<name>@.service — 174 unit files, 110 of them templated. The [X-Fleet] block at the bottom is the placement rule, and in the common case the instance suffix is the role:
[X-Fleet]
MachineMetadata=role=%i
So mysql-xtradb-node@uhat.service or grafana@hq.service resolve onto exactly one host each, while Global=true copies a unit to every member. That is how cadvisor, grafana and nginx-gen — and the SkyDNS/registrator units, which live inside the cloud-configs rather than in services/ — reach all 25 machines.
Two less obvious keys carry real intent: Conflicts=mysql-xtradb-node.service on the ProxySQL unit keeps the proxy off any host running a database node, and MachineOf=mysql-xtradb-backup@%i.service pins the backup @.timer to whichever machine its service landed on, so the timer cannot fire on a host that has no backup unit.
Provisioning is a script, not a tool. util/provision-machine.util.sh mints a client certificate for the new host with cfssl — the SANs include <host>.etcd.services.ruhmesmeile.local and 127.0.0.1 — copies the etcd keys into place, drops a node_exporter into /opt/bin, installs fleetctl v0.11.8, and reboots. fleetctl then reaches the cluster over SSH on port 2222, not 22: environments/fleet.env sets FLEETCTL_TUNNEL=momcorpfdc.ruhmesmeile.solutions:2222, and util/set-fleettunnel.util.sh re-points that tunnel at the momcorpttt1..3 machines when you need to act on a different part of the fleet.
| Selector | Effect |
|---|---|
| MachineMetadata=role=%i | the @instance suffix names the machine role |
| MachineMetadata=role=fdc | fixed to the single ProxySQL host |
| Global=true | one copy of the unit per member machine |
| Conflicts=mysql-xtradb-node.service | proxy never co-locates with a DB node |
| MachineOf=…@%i.service | timer runs on its service's machine |
| FLEETCTL_TUNNEL …:2222 | fleetctl reaches the cluster over SSH |
Agents you can pin
Why five binaries are checked into the repository
Five executables are committed under binaries/: confd (21.2 MB), fleetctl (5.8 MB), node_exporter (14.5 MB), mysqld_exporter (10.5 MB) and proxysql_exporter (8.4 MB). Nothing in the repository fetches them at boot — the unit files reference /opt/bin/<name> or /home/julrich/confd, and provisioning copies the binary in.
The three exporters are built by a Dockerfile that starts from fedora:25 and installs Go 1.7.4 via gimme, because Fedora 25 shipped the same systemd version (231) as CoreOS did at the time. The comment in all three files is explicit about the reason: "if in question: better be safe than sorry." util/update-mysqld-exporter-binary.util.sh drives the whole loop — build the image, clone the upstream Go source into the context, run go get -t -v ./..., go test -v ./... and go build inside the container, then copy the resulting binary back into the repository. The build recipe itself is credited to the journalbeat project's CI, which is where the approach came from.
The process exporter is the one that did not survive. Its README states it "is currently not running, because we can't fetch the percona/processexporter repo, because it doesn't exist anymore"; there is no process_exporter binary here, no unit in services/, and `binaries/process-exporter/processexporter-src is a git submodule reference with no .gitmodules beside it — a fresh clone gets an empty directory and the build script cannot run. cadvisor` took over the container-metrics role it was meant to fill.
Discovery by DNS, config by template
What makes a new container visible to monitoring
Every machine runs its own SkyDNS with the etcd cluster as backend — a Global=true fleet unit defined inside the cloud-configs, not in services/. The etcd key space is the registry: /skydns/local/ruhmesmeile/services/<service>/<host>.
Two things write into it. Containers are registered by ruhmesmeile/registrator:skydns with -internal -cleanup -resync 600 -ttl 90 -ttl-refresh 30 — a custom image, because stock registrator speaks Consul rather than the skydns2 etcd schema. Host-level agents register themselves: node-exporter's ExecStartPost writes {host: <flannel IP>, port: 9100} to /skydns/.../node-exporter/%H after each start, so a machine's address is published by the agent on it rather than by an operator.
Out of 174 unit files, one uses confd. prometheus-confd.service invokes /home/julrich/confd -onetime -backend etcd -node https://127.0.0.1:2379 -confdir=/var/prometheus/confd with the etcd client certificates, against a prometheus.toml that lists 19 etcd keys and a single reload_cmd = "docker kill --signal=HUP prometheus". The unit carries its own admission that reading a binary from a home directory is wrong: # TODO: do not use /home/julrich anymore.
A companion timer re-runs the whole render every five minutes (OnCalendar=*:0/5, MachineOf=prometheus-confd.service), so an exporter that registers itself at 10:03 is being scraped by 10:05 without anyone editing a configuration file.
prometheus.yml.tmpl builds every target list with gets "/skydns/.../<job>/*" inside a dns_sd_configs block and relabels __meta_dns_name back into an instance label; it also scrapes etcd itself over TLS.
The MySQL exporter is reused three times at different resolutions — mysql-hr on /metrics-hr every 10s, mysql-mr on /metrics-mr every 20s with a 5s timeout, and mysql-lr on /metrics-lr every 60s with a 20s timeout. The unit that feeds them passes about 21 -collect.* flags, and it has a real defect: the line -collect.perf_schema.eventsstatements=true is missing its trailing backslash, so the shell ends the docker run command there and the last six collector flags never reach the exporter.
The file also records the cost of the TYPO3 v7 migration — seven typo3-v7-* blocks sit next to their non-v7 twins, and one fully commented-out typo3-solr-exporter job is a dead end kept for reference.
The database stack
Where the pattern stopped being enough
How does a database node find its cluster?
Not through an orchestrator — through etcd's raw v2 API. mysql-xtradb-node@.service passes DISCOVERY_SERVICE=%H.etcd.services.ruhmesmeile.local:2379, and the container's entrypoint.sh does the rest: POST /v2/keys/pxc-cluster/queue/rm-mysql to announce itself, read the queue back, drop its own address, and build --wsrep_cluster_address="gcomm://<remaining>". It then publishes ipaddr and hostname under /v2/keys/pxc-cluster/rm-mysql/<ip>/ with ttl=30.
clustercheckcron — Percona's clustercheck rewritten as a five-second loop — keeps that registration honest. Each pass it reads wsrep_local_state and wsrep_cluster_status; only a Primary node in state 4 (or a donor, when that is accepted) refreshes its keys with ttl=30, and anything else -XDELETEs its own directory. Registration is readiness: a node that falls out of sync stops being a discovery candidate within 30 seconds, with no separate health endpoint involved. The three nodes are @as, @ofro and @uhat; ProxySQL runs on fdc, and Conflicts= keeps those roles apart.
What happens after a full reboot or a lost quorum?
safe_to_bootstrap: 1 in /var/xtradb-node/data/grastate.dat on the chosen node, export XTRADB_NODE_OPTS=--wsrep-new-cluster, then fleetctl start mysql-xtradb-node@ofro.service, wait five minutes, and repeat for @uhat and @as. Afterwards ProxySQL has to be restarted and add_cluster_nodes.sh re-run inside the container. The unit ends with the honest note that the nodes "shouldn't lose data" but that the cluster they form "will be a new one" — a fresh Galera cluster rather than a recovered one.What does the database stack need beyond the generic pattern?
ProxySQL (ruhmesmeile/proxysql-docker:latest) serves 3306 in front of the nodes with its admin interface on 6032, pinned to server_version="5.7.19", threads=4 and max_connections=2048 in its own proxysql.cnf. It needs proxysql-admin --node-check-interval=5000 --mode=loadbal --enable to attach the cluster, a proxysql_galera_checker row in its scheduler to keep node status current, and — per the unit's own comment — max_allowed_packet reset to 256 MB after a restart before large imports work again. That is why the unit carries roughly a hundred lines of commented runbook for exactly those three operations, plus the query-logging switch and a sysbench benchmark.
node.cnf sizes the node for the workload: innodb_buffer_pool_size=6G, innodb_log_file_size=512M, innodb_file_per_table=1, innodb_autoinc_lock_mode=2, binlog_format=ROW (what Galera requires), max_allowed_packet=256M, max_connections=1024. Two of its lines are dead weight — wsrep_cluster_name=Theistareykjarbunga and wsrep_sst_auth="root:" are both overridden by the --wsrep_cluster_name=rm-mysql and --wsrep_sst_auth arguments the entrypoint passes to mysqld.
What was still unfinished?
TODO.md is committed next to the code, so the answer is documented. Its first item is finishing the TYPO3 v7 retirement: delete the duplicated -v7 units and the fallback blocks they left in prometheus.toml and prometheus.yml.tmpl. The largest theme is secrets — the root, clustercheck, xtrabackup and ProxySQL passwords are literal strings in the unit files and proxysql.cnf, and the list wants them, the Atlassian database connections, the AWS root key referenced under environments/ (only fleet.env remains there now) and the per-machine SSH access keys moved into Vault. Backups are unfinished in a more concrete way: a note says the backup script "only runs when there's terminal activity?!", rolling backups to Backblaze are still a ticket, and replacing calculon1 along with setting up an SSH bastion host is on the list. The one genuine open question is whether Solr on MomCorpFDC loses its data on every restart, and why.Links
Where to find this one