Incident report INC-2026-031: nightly database backup silently failing

Summary

The nightly PostgreSQL dump that is pulled from the database host to off-site storage failed for three consecutive nights after an SSH key rotation. Nothing was lost. For three days the newest restorable backup was up to 72 hours old instead of 24. The alert that caught it was the backup-age check, not the job itself, which is the real finding.

Timeline (UTC)

Impact

No data loss and no downtime. Recovery point objective of 24 h was not met for three days (actual: up to 72 h). Had the database failed on 09-13, up to two days of writes would have been lost.

Root cause

A process gap, not a technical fault. The rotation checklist listed hosts, not credentials and their consumers. A credential whose consumer lives outside the customer's own hosts was invisible to it. The job's silence made a one-night problem a three-night problem.

What changes so it does not recur

1. Every scheduled job now alerts on its first failure, not only when its output goes stale. Done 09-14, all customers.
2. Backup-age threshold lowered from 72 h to 30 h. Done 09-14, all customers.
3. Credential inventory rewritten per credential with its consumer host and the service that uses it, so rotation starts from consumers, not from hosts. Done 09-16.
4. Rotation checklist (RB-021) now ends with a forced run of every backup and sync job that uses a rotated key. Done 09-16.

Runbook entry RB-044 added. Verified by on-call lead 2026-09-16.

Change request CR-2026-142: raise memory limit of the inference container

Request, as received

Our inference API restarted about 70 times on Sunday and latency spiked each time. Can you look at it today? (customer CTO, shared channel, 10-06 09:14)

Findings

The container runs a 7B model server with an 8 GB memory limit. Sunday's traffic included long prompts; at a context of 8192 tokens resident memory reaches about 9.4 GB and the kernel kills the process. The service restarts automatically, which is why nobody noticed before Monday. 71 kills in the log between 10-05 03:10 and 10-05 21:48.

Plan

1. Raise the container memory limit from 8 GB to 14 GB. The host has 52 GB free; no other service is affected.
2. Set the model server's context size explicitly to 8192 so memory use is bounded and predictable, instead of growing with the longest request.
3. Add an alert on container restarts: more than 2 in 10 minutes pages the on-call engineer.
4. Restart the container once. Expected interruption: about 40 seconds while the model loads; the load balancer returns 503 for that window.

Rollback: set the limit back to 8 GB and restart. Nothing else is changed. Not in this change: moving the service to a second container for zero-downtime restarts; proposed separately as CR-2026-143.

Review

Checked the numbers against the kill log: anon-rss at kill was 9.3 to 9.9 GB, so 14 GB leaves headroom. Confirm after restart that the model loads on the GPU, not on the CPU fallback; last month's driver update changed the default on one host. Otherwise approved. (second engineer, 10-06 11:02)

Customer confirmation

Confirmed, go ahead in the 22:00 window. (customer CTO, 10-06 11:30)

Execution log (UTC)

22:00:12  limit 8G -> 14G applied to container inference-01
22:00:40  server config: ctx-size=8192 set, checked into customer's infra repo, commit 4e1f2a9
22:01:05  restart issued
22:01:47  health check green, model on GPU 0 (confirmed via device query), 14.9 GB VRAM in use
22:02:30  restart alert rule deployed, test-fired, page received
22:03:00  ten test requests at ctx 8192: p50 410 ms, p95 890 ms, no restarts
22:25:00  25 minutes of production traffic, zero restarts, resident memory steady at 9.6 GB

Verification and close

No restarts in the following 48 h (checked 10-08 22:00). Change closed. Customer informed with this log attached.

Monthly cost review: AWS bill for September 2026

Where the money goes

Recommended, in order

Not recommended

Moving RDS to a single availability zone would save $390 a month and is not worth the exposure for a production database. Spot instances for inference would save more than the savings plan but would cause interruptions your API cannot absorb today.

If the inference boxes moved to our servers

The two inference instances cost $3,490 a month on demand. The equivalent on our hardware, managed, is $3,400 a month for two dedicated GPUs with more memory each, and the NAT and egress lines shrink with them. The saving is small; the gain is the named engineer, which is why we recommend doing items 1 to 5 first and deciding on the move with real numbers in November.

Next step

Items 3, 4 and 5 need only your written confirmation and are done the same day. Items 1 and 2 need the attached lists signed off. All five are reversible except the deletions in item 2.

Onboarding checklist: first fourteen days

Monthly summary: October 2026

In one paragraph

A quiet month after the inference fix. One incident, no customer-facing downtime. Five changes, all confirmed and verified. Cloud spend down 9% on September. The open item that matters is the single point of failure in the message queue, which we propose to remove in November.

Availability

Incidents

INC-2026-038, 10-19, severity 3. Worker queue backed up for 65 minutes after the queue broker's disk filled with retained messages. Caught by the disk alert at 80%, not by a customer. Fixed by purging a dead-letter queue nobody consumed and setting a retention limit. Report sent 10-19. Runbook RB-051 added.

Changes

Spend

Open items and November plan

1. Queue broker is a single instance on one disk. Proposal CR-2026-155: two-node cluster, half a day, one planned failover test. Needs your written go-ahead.
2. Savings plan for two instances (item 1 of the September cost review) still awaits your sign-off. $1,650 a month from the day it is signed.
3. Quarterly restore test due 11-14; we will schedule it in the channel.