Reproducible builds and kernel tracing with the frozen Reference System

The helloscope archive and the ELF Atlas fixtures are byte-canonical: every disassembly row, every build-id, every source line is pinned to one exact userland. reference-system/ freezes that userland (glibc r22 build-id 020d6f7c, sha256 27d2232b; GNU objdump 2.46.0; gcc 16.1.1; pyelftools 0.33) anchored to the public Arch Linux Archive snapshot 2026/05/02. reference-system/prove.sh has already shown, rootlessly, that this env regenerates the committed archive: 402/402 stage-0 disassembly rows byte-identical.

This guide is about using that frozen env for new work — building binaries for rooms and experiments, and running kernel-level traces — on either machine (this Linux box or the MacBook Pro) and on Azure.

1. One frozen environment, two delivery forms

The freeze ships in two shapes because two different jobs need two different things:

The spine of everything below is that distinction:

• A build only touches userland. gcc and objdump never care which kernel is booted, so a container — which shares the host kernel — reproduces builds perfectly. This is why the Docker image is enough, and why it works identically on Linux and macOS.
• A trace touches the kernel. Instruction tracing / Intel PT / perf / eBPF read CPU and kernel facilities that a container cannot provide — a container has no kernel of its own; it borrows the host's. So tracing needs a VM whose kernel is itself pinned to 7.0.10. That is the Azure VM.

Keep this in mind whenever you reach for the freeze: need a reproducible binary or fixture → local Docker; need a real kernel under the program → Azure VM.

2. Reproducible builds locally (works today, either machine)

2.1 Get the image

# From the repo root. Builds helloscope-refsys if absent, then proves it.
reference-system/prove.sh
# or just build it without the proof:
docker build -f reference-system/Dockerfile -t helloscope-refsys .

The image is archlinux:latest downgraded in one transaction to the ALA anchor 2026/05/02, plus the frozen toolchain and pyelftools 0.33. It carries /opt/helloscope/frozen-identities.txt for offline inspection.

On Apple-Silicon Macs (arm64), pass --platform linux/amd64. This is an x86-64 image producing x86-64 binaries — the target the archive is built for. Docker Desktop runs it under emulation (Rosetta/qemu inside its Linux VM): the emulated x86-64 gcc still emits x86-64 machine code, and Docker does not transpile the produced binaries — it emulates the compiler, which computes the same output bytes it would on a native x86-64 host. So a build on an arm64 Mac is byte-for-byte identical to a build on this x64 box, just slower. Reproducibility is fully preserved; the Mac is simply a slower builder.

docker build --platform linux/amd64 -f reference-system/Dockerfile -t helloscope-refsys .
docker run --platform linux/amd64 -v "$PWD":/work helloscope-refsys …

2.2 The build recipe

Mount the repo at /work (the image's WORKDIR) and run the compiler inside:

REPO=/home/jflam/src/helloscope        # on the Mac: your clone path
docker run --rm -v "$REPO":/work helloscope-refsys \
  bash -c 'cd /tmp && cp /work/exhibit-001-elf-atlas/src/hello.c . &&
           gcc -g -O0 -fPIE -pie hello.c -o hello &&
           ls -l hello && size hello'

To keep the artifact, build into a mounted output dir instead of /tmp:

mkdir -p "$REPO/scratch"
docker run --rm -v "$REPO":/work -w /work/scratch helloscope-refsys \
  gcc -g -O0 -fPIE -pie ../exhibit-001-elf-atlas/src/hello.c -o hello-debug
# hello-debug now sits in $REPO/scratch on the host.

16 GB RAM is ample. hello and coreutils builds are light — a build peaks well under a couple of GB; there is no memory pressure to plan around.

2.3 The numbers this env produces (measured — cite, don't re-derive)

Building the museum's hello inside the frozen container:

gcc -g -O0 -fPIE -pie hello.c        # debug:   16,936 B, .text 275 B, main @ 0x1139, full DWARF
gcc -O2 -fPIE -pie hello.c && strip  # release: 14,472 B, .text 281 B, no DWARF

The debug build's 16,936 B and main source-mapped to 0x1139 match the museum's committed hello (16,984 B) — same program, same layout.

Pedagogical nuance worth stating out loud: the release .text is larger (281 > 275 B — -O2 codegen), yet the file is smaller (14,472 < 16,936 B). Stripping did not shrink the code; it removed six debug sections plus the symbol table. Optimization and stripping pull in opposite directions on different parts of the file, and this pair shows exactly that.

2.4 Platform-split payoff: fixtures on the Mac

Per CLAUDE.md, byte-canonical fixture extraction is Linux + GNU objdump only — macOS's LLVM objdump emits different disassembly and would corrupt the fixtures. The frozen image is a Linux + GNU-objdump 2.46.0 environment, so it satisfies that guard on any host. The Mac can therefore do byte-canonical fixture work it otherwise couldn't, simply by running the extractor inside the container:

docker run --rm -v "$REPO":/work -w /work helloscope-refsys \
  python exhibit-001-elf-atlas/tools/extract_fixture.py --help

(extract_fixture.py's own objdump --version "GNU objdump" check passes inside the image.) This is the same path prove.sh uses for its byte-equality proof.

3. The worked scenario: the debug/release × hello/true build matrix

The first experiment the reference env is for is a 2×2 matrix — two programs, each built debug and release — because the four builds, compared pairwise, teach distinct lessons.

Two axes, two stories:

• Down a column isn't the point; across a row is. debug-vs-release of the same program isolates what stripping and optimization actually change (see §2.3 — code slightly grows, file shrinks, DWARF vanishes).
• debug-hello vs debug-true (across the top row) is the "model GNU command" story. hello is a lone main; true is a real coreutils command.

3.1 hello — both builds

cd /tmp && cp /work/exhibit-001-elf-atlas/src/hello.c .
gcc -g  -O0 -fPIE -pie hello.c -o hello-debug        # 16,936 B, DWARF present
gcc -O2 -fPIE -pie hello.c -o hello-release && strip hello-release   # 14,472 B, stripped

3.2 true — needs the coreutils 9.11 source

/usr/bin/true is not a toy. coreutils src/true.c is ~80 lines (~50 of code), yet the shipped binary is 43,248 B because true.c links gnulib scaffolding common to every GNU command: version_etc, close_stdout, usage, and the setlocale / bindtextdomain / textdomain i18n setup. hello has none of that; true is the smallest program that still carries the full GNU-command skeleton, which is exactly why it is the interesting second room.

To build it in the frozen env you need the coreutils 9.11 source tree (configure + make), then compile just the true target:

# Inside the container (docker run -v "$REPO":/work -w /work/scratch ... bash):
curl -fSLO https://ftp.gnu.org/gnu/coreutils/coreutils-9.11.tar.xz
tar xf coreutils-9.11.tar.xz && cd coreutils-9.11

# Two gotchas, both verified: (1) coreutils refuses to ./configure as root
# (the container's default user) unless you set FORCE_UNSAFE_CONFIGURE=1;
# (2) `make src/true` ALONE fails — the lib/ build generates configmake.h and
# other headers src/true.c needs, so you must run the full `make` first.
export FORCE_UNSAFE_CONFIGURE=1        # only needed when building as root

# debug true: un-stripped, full DWARF → source-maps into the coreutils/gnulib tree
./configure --quiet CFLAGS='-g -O0'
make -j"$(nproc)"                      # builds lib/ (generates headers) then src/
cp src/true /work/scratch/true-debug

# release true: optimized; strip to approach the shipped /usr/bin/true
make clean && ./configure --quiet CFLAGS='-O2' && make -j"$(nproc)"
strip src/true && cp src/true /work/scratch/true-release

Measured (built in the frozen container, coreutils 9.11):

Two things this proves. First, the rebuild is real: strip the debug build and you land at 43,736 B — within ~1% of the shipped 43,248 B. Same source, same frozen toolchain, essentially the same binary; the ~500 B and the .text difference are just -O0 vs the distro's -O2 codegen. Second, the debug build carries 207 named functions where the shipped binary has none — that is the legibility the room needs.

And here is the payoff, from the debug build's DWARF (objdump -dl, instruction-lines attributed per source file):

The program itself — true.c — contributes 29 of the mapped instruction-lines; quotearg.c alone contributes 591. The 80-line source is a rounding error in its own binary. Nearly all of true is gnulib's "be a model GNU command" scaffolding — and with the debug build you can name every piece of it by source file, which the stripped release makes impossible.

Honesty note: a rebuilt true is not byte-identical to the distro /usr/bin/true — it gets its own build-id (different build path, build time, and configure environment). The point is not to reproduce that binary bit-for-bit; it's to have the same source made legible — a true you compiled, with DWARF you can walk, that the debug build maps straight back to true.c and the gnulib sources. The stripped release, by contrast, source-maps to nothing, exactly like the shipped one.

4. Azure utility build+trace VM — agent-executable runbook

Audience: a future Claude running on the MacBook Pro, where az login already works. This box (the Linux origin box) has no az, no passwordless sudo, no qemu-img — so it cannot drive Azure. The Mac can.

All scripts live under reference-system/azure/. Run commands from the repo root unless noted.

Step 0 — the one honesty caveat: build-vhd.sh needs root on a Linux builder

build-vhd.sh uses pacstrap, loop devices, and mount — it requires root and a Linux host. macOS cannot run it directly. Two workable paths from the Mac:

• (recommended) Build the VHD on a throwaway Azure Linux VM. Spin up a small Arch/Ubuntu VM, install the builder deps, run the script there, then run deploy-azure.sh from that same VM (it already has az, or install it).
• Or run it on any Linux box where you do have root (not this origin box).

# On the throwaway Linux builder (as root), with the repo checked out:
pacman -S --needed arch-install-scripts qemu-img gptfdisk dosfstools   # Arch builder
# (Ubuntu builder: apt-get install arch-install-scripts qemu-utils gdisk dosfstools)

sudo WORKDIR=/var/tmp/hs SIZE_GB=8 reference-system/azure/build-vhd.sh

build-vhd.sh pins pacman to the ALA anchor, pacstraps the frozen toolchain, pulls kernel + headers 7.0.10.arch1-1 from the permanent package pool, configures Hyper-V initramfs modules + serial console, and prints the in-rootfs frozen identities.

Manual step the script deliberately leaves to you (documented, not a bug). build-vhd.sh creates the raw disk (step 4) but stops before partitioning — it prints "complete GPT/ESP/grub install per README". You must run the disk- assembly block from reference-system/azure/README.md before the VHD is bootable. Run it as root on the same builder, right after the script:

RAW=/var/tmp/hs/helloscope-refsys.raw
LOOP=$(losetup --find --show "$RAW")
sgdisk -og "$LOOP"
sgdisk -n1:0:+512M -t1:ef00 -c1:ESP  "$LOOP"      # UEFI system partition
sgdisk -n2:0:0     -t2:8300 -c2:root "$LOOP"
partprobe "$LOOP"
mkfs.fat -F32 "${LOOP}p1"; mkfs.ext4 "${LOOP}p2"
mount "${LOOP}p2" /mnt; mkdir -p /mnt/boot; mount "${LOOP}p1" /mnt/boot
cp -a /var/tmp/hs/rootfs/. /mnt/
genfstab -U /mnt >> /mnt/etc/fstab
arch-chroot /mnt grub-install --target=x86_64-efi --efi-directory=/boot \
    --removable --bootloader-id=GRUB
arch-chroot /mnt sed -i \
  's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 earlyprintk=ttyS0"/' \
  /etc/default/grub
arch-chroot /mnt grub-mkconfig -o /boot/grub/grub.cfg
umount -R /mnt; losetup -d "$LOOP"

# Then convert raw → fixed Gen2 VHD (Azure rejects dynamic VHDs):
qemu-img convert -f raw -O vpc -o subformat=fixed,force_size \
  "$RAW" /var/tmp/hs/helloscope-refsys.vhd

Note: build-vhd.sh's step 5 also emits a qemu-img convert, but on a run where you did the partitioning by hand the raw disk isn't finalized until the block above completes — so run this convert after it. If both fire, the last conversion wins; harmless.

Step 1 — deploy to Azure (needs az login)

reference-system/azure/deploy-azure.sh /var/tmp/hs/helloscope-refsys.vhd \
    helloscope-refsys-rg eastus helloscope-refsys

This uploads the VHD as a page blob, creates a Gen2 managed image, and boots a Standard_D4s_v5 (Intel — exposes hardware PMU / Intel PT for instruction tracing) with --security-type Standard. Override with VM_SIZE=.... It prints the VM's public IP on completion.

Step 2 — verify the freeze on the live VM

IP=<public-ip-from-step-1>
scp reference-system/azure/verify-on-vm.sh reference-system/freeze.env azureuser@$IP:
ssh azureuser@$IP 'bash verify-on-vm.sh'

Expect FREEZE VERIFIED: glibc / binutils / gcc / kernel versions, libc + ld build-ids, libc sha256, and objdump version all matching freeze.env. The kernel check confirms both the package (pacman -Q linux) and the running (uname -r) kernel are 7.0.10.

Step 3 — use the VM as a build+trace box

Once verified, the VM is a full frozen host — same userland as the container, plus the real 7.0.10 kernel. Build exactly as in §2–§3, and trace on top:

ssh azureuser@$IP
# On the VM:
git clone <this-repo>            # or: scp -r the sources up
cd helloscope
gcc -g -O0 -fPIE -pie exhibit-001-elf-atlas/src/hello.c -o hello-debug

# Kernel-level tracing the container could not do:
sudo pacman -S --needed perf strace     # if not already present
perf record -e intel_pt//u ./hello-debug    # Intel PT instruction trace
strace -f ./hello-debug                      # syscall trace

# Bring artifacts / trace data back to the Mac:
# (from the Mac)
scp azureuser@$IP:~/helloscope/hello-debug ./
scp azureuser@$IP:~/perf.data ./

Because the VM's userland is byte-identical to the archive's, a trace of the frozen hello (or /usr/bin/true) maps 1:1 onto the committed fixtures and rooms.

Step 4 — tear down

az group delete -n helloscope-refsys-rg --yes --no-wait

Why builds "just work" either way: the VM's kernel and userland are pinned to the same freeze as the container. So a build driven on the Mac, on this box's container, or by ssh into the Azure VM produces the same bytes — the frozen environment, not the laptop, determines the output.

5. The mental model, stated plainly

• A persistent Azure utility VM gives John reproducible builds and kernel traces regardless of which laptop he is on — it's the single frozen kernel + userland, reachable over ssh from anywhere. Stand it up once; az vm deallocate when idle to stop compute charges, az vm start to resume.
• The local Docker image is the zero-cost path for anything that doesn't need a real kernel — every build, every fixture extraction, on either machine, free.

Decision table:

6. Honest boundaries

• This repo's automation builds and proves the Docker image today (rootless, done: 402/402 rows byte-identical). It cannot run Azure from this box — there is no az, no passwordless sudo, no qemu-img here. The Mac has az; the VHD build needs root on a Linux builder (§4 Step 0).
• Local full-VM tracing on this box would need qemu + KVM + root, none of which is currently available. So kernel tracing = Azure for now. If a rootful Linux host with KVM becomes available later, the same VHD (or a plain qemu-system-x86_64 boot of the rootfs) would run traces locally too — the freeze is host-agnostic; only the privileges are missing.
• build-vhd.sh intentionally stops before disk partitioning; the GPT/ESP/grub block in §4 Step 0 (mirrored from reference-system/azure/README.md) is a required manual step, not an omission. </content> </invoke>