leadbybuild.ingArchive

LeadByBuilding Newsletter · 7 min read

Plan around the failure domain, and trust what the cloud runs over what the repo says

Two AI picks: Meta's 99% cut in GPU cluster interruptions through failure-domain planning, and Shopify's LLM-judge-trained scorers for cross-store product matching. Two engineering picks: Shopify's exposure scanner built on cloud inventory instead of config files, and checkpointed steps for multi-step Bedrock pipelines on AWS Lambda.

AI & Agent Engineering

AI infrastructurevendor-reported

Meta says it cut interruptions on its Prometheus GPU cluster by 99% by planning capacity around a 72-GPU failure domain

Meta's account of its Prometheus supercluster says interruptions peaked near 1,000 events per 8K GPUs daily in mid-2025, then fell 99%. Each NVL72 rack runs a 64-GPU slice with 8 in-rack spares, plus about 2% spare racks. It pairs replica-level fault tolerance with frequent checkpointing, and reports over 85% goodput for pre-training and over 90% for RL.

Why it matters

NVL72 widens the blast radius of one fault 9x versus NVL8, so the rack becomes the unit to plan capacity around. Spares cost about 13% extra GPUs for a 32,768-GPU run, mostly from slice geometry. These are Meta's own figures for a frontier-scale cluster.

Do this week

For any long job, write down its failure domain and what happens on one fault. Decide your spare ratio and checkpoint interval from that, then run a drill that kills one worker and time the recovery.

Meta Engineering →
LLM at scaleverified

Shopify matches products across stores by training cheap pair scorers on labels from an LLM judge checked against humans

Shopify's cross-store clustering asks whether two listings are the same product instead of asking an LLM for a global label. It finds candidates with embeddings and approximate nearest-neighbour search, scores pairs with three scorers (L1, L2, L3), then builds clusters from a graph. An LLM judge labels samples, validated against human annotations, to train cheaper matching models.

Why it matters

Pairwise judgments can be non-transitive, so the system avoids simple connected components and weighs evidence per merge. Using expensive reasoning to make training data, not to score every pair, is the cost lever. Retrieval can also miss true matches, which stay unmerged until a later build.

Do this week

If an LLM sits in a high-volume loop, use it to label a sample and validate that sample against humans. Then train or pick a cheaper model for the bulk, and keep the expensive one for the hard cases.

Shopify Engineering →

Engineering

Security engineeringverified

Shopify rebuilt its exposure scanner on Cloud Asset Inventory and cut inconclusive Ingress results by about 80%

Shopify's first scanner read Kubernetes files in a repository and gave false alarms, misses and blind spots. Overexposed now lists admitted Ingresses and Cloud Run services from GCP Cloud Asset Inventory, sends one anonymous probe to each, and records one of six verdicts with evidence. It replays rule changes on saved scans and checks the scanner against SSRF.

Why it matters

A repository shows intent, while the cluster shows what is enforced. Its rule that uncertainty does not mean protected keeps unclear results visible. It also notes that a 404 on Cloud Run can come from Google's front end before the app runs.

Do this week

Compare your list of public endpoints from the cloud inventory with the one in your repos this week. Treat any endpoint you cannot classify as inconclusive, not safe, and give it an owner.

Shopify Engineering →
Serverless workflowsvendor-reported

AWS shows checkpointed Lambda durable functions so a throttled Bedrock call retries one step, not the whole pipeline

An AWS Compute Blog tutorial builds a five-step pipeline (fetch, score, analyze, publish, record) as one Lambda durable function. Each step's output is persisted before the next, and the function suspends between steps without holding an environment open. Scoring is deterministic Python, and Amazon Bedrock only writes the prose analysis at temperature 0.

Why it matters

Keeping the deterministic score separate from the model call means a retry never re-spends inference or changes the number. The example is a docs-scoring demo, not a production case study, so treat the cost claims as AWS's.

Do this week

Find one multi-step job where an LLM call sits after cheap deterministic work. Split it into checkpointed steps, keep scoring outside the model, and check that a forced failure at the last step re-runs only that step.

AWS Compute Blog →

Worth experimenting with

Run a one-worker failure drill and time the recovery

Pick one long-running job or agent pipeline, kill a single worker mid-run, and record what restarts, what is lost since the last checkpoint, and who notices.

Why. Meta's reliability gains came from naming the failure domain and the recovery path for each fault. A small drill shows whether yours is defined.

Measure. Minutes from fault to resumed progress, and the amount of work redone since the last checkpoint.

Classic / essential read

Bounded Context

Martin Fowler · 5 min

Fowler explains why one unified model of a large domain is not feasible, using the word meter meaning different things inside one utility. The same confusion shows up when teams share a Customer or Product type.

One lesson. When one word means different things to two teams, draw a boundary and map between the models instead of forcing one shared definition.

Read the original →

Both themes this week are about trusting enforcement over intent: size capacity by the failure domain you actually have, and take inventory from the cloud rather than from config. Pick one place where your system relies on what a file says should happen, and check what actually runs.

Get this every morning.

Free. Same issue for everyone. One click to unsubscribe.

Subscribe