apt-mirror: Internal Package Mirror on EKS

Cutting Package Updates Off the Internet

Every internal server that ran apt upgrade was opening its own outbound connection to deb.debian.org and archive.ubuntu.com. That is an internet dependency on every host in the estate, none of it under our control, and it is the kind of dependency that quietly becomes a vulnerability the moment egress is throttled, firewalled, or blocked.

The fix is to move the fetch inside the perimeter. Debian and Ubuntu mirrors now run in-cluster, so a package update is a local call to infrastructure we operate — and it keeps working when egress is throttled or blocked. Only the sync tier talks to the public archives, which turns N uncontrolled outbound dependencies across the fleet into one controlled, auditable egress path.

The archive is not a cache that quietly rots — it is a deliberate, verifiable distribution point holding ~118 GiB of Debian 12 and ~259 GiB of Ubuntu 24.04 on EFS, served from a single nginx tier behind the existing ALB edge, with every release file GPG-verified before it is served.

Architecture

Terraform builds the VPC, the EKS cluster, and the EFS filesystem with one access point per distribution. On top of that the system splits cleanly into a sync tier that writes and a serving tier that only reads — and that split is what makes the isolation work. The sync Deployments are the only workloads that ever reach the public archives, and they are the only ones that need egress at all. Everything a client touches, including the Grafana observability stack, is served from inside the perimeter.

Each distribution gets an independent sync Deployment, trigger CronJob, ConfigMap, and EFS access point, and all of them share a single mirror-nginx replica that mounts each PVC read-only and aliases the archive path per distro.

The Asymmetry Trap

The single most important thing to know about this system is that the two distributions are not symmetric. Debian keeps its security suite under a separate archive root; Ubuntu keeps it under the same one:

List bookworm-security under the Debian main root and the suite 404s — which does not degrade gracefully, it aborts the entire sync. This is not hypothetical: it is why the first mirror image never completed a run. It is also why the serving layer aliases each distro separately instead of sharing one rule.

Signature Verification, Not Trust

Pulling packages over the network is exactly the situation where “it came from the archive” is not good enough, so the mirror refuses to serve anything it cannot verify. apt-mirror checks InRelease and Release with gpgv. Inside a container ~/.gnupg is empty, so without an explicit keyring every distribution fails with “Can't check signature: No public key”. Each distro's apt config therefore sets signed-by to a keyring the image actually ships — debian-archive-keyring or ubuntu-keyring — plus xz-utils for the compressed release files. Clients still verify signatures independently on their own side; the mirror simply never publishes an archive it failed to verify, so the trust decision is made once, in one place, instead of on every host.

Snapshot-Pinned Patch Sets

A mirror that syncs nightly is only useful for patch management if a client can be pinned to a known-good package set — one that stays known-good after the live mirror moves on. That needs a point-in-time copy, retained for a while, served from the same endpoint. Snapshots use cp -al (archive + hardlink) into snapshots/<timestamp>/ on the same EFS volume.

Retention is capped at the newest 3, and pruning is guarded by a name pattern — only ^[0-9]8-[0-9]6Z$ may ever be deleted. That rail is what stops a bug in the prune step from removing LATEST, or the live mirror. Clients pin a snapshot by pointing their sources list at it:

deb http://<mirror>/snapshots/20260928-140036Z/debian \
  bookworm main contrib non-free non-free-firmware

Single-Writer Discipline

The spool is single-writer. Two apt-mirror processes on one EFS tree fight over the lock and can interleave a half-written pool/, so both syncs use strategy: Recreate — a rolling update would briefly run two pods, which is the exact failure being prevented. Snapshots serialise against syncs with flock on the spool lock file. Two details are easy to get wrong and both were hit:

• •The sync takes the lock in a subshell around apt-mirror only. The container then idles for ~24h holding the EFS mount warm; holding the lock across that idle period would block every snapshot indefinitely.
• •The lock file lives at the spool root, not var/ — var/ does not exist on a fresh access point, and flock cannot create a lock file in a missing directory.
• •Exiting is not an option. The sync pod idles rather than exits; exiting makes the Deployment restart it in a tight loop and re-pull the EFS mount every minute for nothing.

Security Engineering

A package mirror is a distribution point, so the endpoint and the pipeline both get treated as infrastructure that can be attacked.

The One That Took the Most Debugging

Dynamic EFS provisioning cannot work on this cluster, and the error message does not say so. The EFS CSI driver is installed as an EKS addon and authenticates through addon-owned Pod Identity associations, which take precedence over the service account's IRSA annotation. The addon role lacks elasticfilesystem:DescribeAccessPoints, so every provisioningMode: efs-ap attempt fails and the PVC sits Pending forever. A correctly-permissioned IRSA role exists in the account and is never used.

The fix was to stop asking: every mirror PVC binds to a static PV with an explicit volumeHandle, one per access point. A static PV never reaches the EFS control plane, so provisioning needs no IAM at all. The knock-on consequences are recorded rather than rediscovered — the StorageClass becomes decorative, and PVC-reported capacity is cosmetic because EFS has no quota, so du -sh on the mount is the only honest number.

The Cluster Scales to Zero

The node group terminates at 17:58 UTC and returns at 10:00, with KEDA scaling apps up at 10:05. The mirror answers 502 overnight by design, and a sync interrupted by the drain resumes the next morning rather than restarting, because apt-mirror keeps its state in the spool. Schedules are placed against that window — Debian syncs at 10:00, Ubuntu at 12:30 so the two do not contend for the same node bandwidth, and snapshots run at 12:30 well clear of the drain.

Cost shaped the architecture in other ways too. CI still applies manifests inside the off-hours window but skips waiting for rollout, so git can never silently diverge from the cluster. The node group sits at its pod ceiling — 5 nodes × 11 = 55 allocatable slots — which is why the serving tier and Grafana run a single replica and why prometheus, loki and blackbox roll with maxSurge: 0: a default 25% surge would leave the rollout unschedulable. Ubuntu carries only main and restricted because adding universe and multiverse would roughly double the archive for packages a controlled internal estate rarely needs.

Evidence From a Real Browser

Diagrams only prove that someone drew a diagram. The pages below were captured by a headless Chrome loading each URL as an unauthenticated client — what an apt client on the internet actually receives, not what curl from inside the cluster returns. Every page returned 200 with secureContext: true, which is the browser's own verdict on the wildcard certificate after the TLS handshake succeeded — a stronger claim than a status code. Zero console errors and zero failed requests; the single 404 is /favicon.ico, which a package mirror has no reason to serve.

The capture is regenerable and exits non-zero on any real error, which is the part that makes it worth more than a picture. It is also deliberately honest about its limits: it covers page load only. It does not prove a package installs, that signature verification passes end to end, or that every file in the pool is intact — those are separate claims and they need apt-get against the mirror to check. The cluster state was likewise captured verbatim from the API server, including the one command that failed.

Documentation as Infrastructure

This system has an unusually large blast radius for a “small” service, so the reasoning is a deliverable rather than a comment. Six ADRs record each decision with its context and consequences; an operating guide names the failure modes that are easy to reintroduce — the nginx regex-before-prefix trap that 404s every package, ContainerPort having a name field rather than containerName (which --dry-run=client will not catch), KEDA silently overwriting Deployment.replicas every morning, and dists/ 404ing during a first sync because apt-mirror stages into skel/ before promoting.

The manifests themselves are deliberately comment-free— 507 lines of commentary were moved into the docs, with a block-scalar-aware stripper so the decision stays auditable, and every ConfigMap's data verified byte-identical across the change. The rule is that reasoning lives in docs/, and # lines that ship inside data payloads are left alone because they are read at runtime.

Skills & Technologies