leadbybuild.ingArchive

LeadByBuilding Newsletter · 8 min read

Verify the layer you assume: certificates, caches and agent output

An urgent TLS trust-store incident leads today. Cloudflare also shows how a Postgres-era caching assumption broke MySQL, Meta shows agents gated on tests rather than compilation, CNCF explains scheduler and autoscaler gaps for GPUs, and tsrs offers an early Rust TypeScript compiler.

AI & Agent Engineering

Agentic developmentvendor-reported

Meta built Forum with an agent in the loop and gated every task on tests, not compilation

Meta describes building its Forum app with an agent running plan, build, test and correct cycles while engineers set goals and approve results. A task counts as done only after automated tests and visual checks pass. Meta says Forum reached feature parity with Facebook Groups in eight weeks.

EngineerPlanBuildTestTests and visualchecks pass?Donesets goalpassfail:correct,rebuildapproves the result
Why it matters

The gains Meta credits came from changing the codebase and tooling, such as giving the agent tools to read the live view hierarchy, rather than from better prompts. Timing and memory figures are Meta's own, from one app.

Do this week

Pick one recurring agent mistake this week and fix it in the environment: add a rule, a hook that runs the linter after each agent turn, or a tool that lets the agent check state directly.

Engineering at Meta →
Kubernetes for AIverified

CNCF post: AI accelerators expose a gap between the Kubernetes scheduler and the autoscaler

Two AWS authors writing on the CNCF blog say GPU pods can stay pending while capacity exists, because the node the autoscaler adds is not one the scheduler will use. They cover Dynamic Resource Allocation claims and the Workload and PodGroup APIs, which are beta and off by default.

Why it matters

Gang scheduling is handled on the scheduler side only; the autoscaler still reasons pod by pod, so it can add nodes for a job that never starts. Support for DRA in autoscalers varies by implementation.

Do this week

When a GPU pod stays pending, compare what the scheduler and the autoscaler each consider a viable node before assuming capacity is gone. For gang jobs, look at Kueue with a provisioning request.

CNCF Blog →

Broader Engineering

Security incidentverified

Eight eMudhra intermediate certificates compromised; check the trust stores your services use

Cloudflare says that on October 19, 2026 it learned private keys for eight eMudhra intermediate certificates were compromised, letting an attacker forge certificates for any domain. It lists fixed versions, including cloudflared 2026.10.2, and says it found no active exploitation on its services.

Updating these may change nothingYour service codeSDK or client libraryThe trust store is in one of theseHTTP libraryLanguage runtimeBase image or OSFind which one, patch it, redeploy.
Why it matters

The exposure sits in certificate verification, not TLS encryption. Cloudflare advises looking beneath the SDK, since an HTTP library, runtime or OS may hold the trust policy, and a passing browser test does not cover background services.

Do this week

List where your services verify server certificates: base images, language runtimes, bundled trust stores. Patch, redeploy, then test through each app's real TLS path. Never disable verification to clear errors.

Cloudflare Blog →
Debuggingverified

Cloudflare traced rare empty MySQL results to a cache that stored bytes in two different formats

Hyperdrive returned empty results about 0.001% of the time, only with caching on. The Client node cached bytes translated to the Worker driver's capabilities, while the Endpoint expected Hyperdrive's own, so a shared datacenter cache served mismatched bytes. The fix caches one canonical form.

BEFOREClientwrites driver'sformatCacheEndpointexpects its ownformatEmpty result, about 0.001% of queriesAFTERClientwrites canonicalformCacheEndpointreads canonicalformClient success 99.99%, as Cloudflare reports
Why it matters

An assumption from Postgres, that you can cache exactly the bytes written to the client, failed once MySQL capability negotiation differed. Cloudflare reports Client success reached 99.99% after the change.

Do this week

If a cache or proxy sits between protocols, check that every writer stores the same canonical form. Add a test that has one component write and another read the same key.

Cloudflare Blog →
Developer toolingvendor-reported

Cloudflare and VoidZero start tsrs, a Rust port of TypeScript, citing up to 8x speed

Cloudflare says tsrs is up to 8x faster and uses about 70% less memory than TypeScript 7.0 with the same number of type-check workers, and that TypeScript 7.1 beta is up to 2.5x faster than 7.0. It is an experimental npm package, built with the TypeScript team, not a fork.

Type-check speed, 7.0 = 1x (higher is better)TypeScript 7.01x7.1 betaup to 2.5xtsrsup to 8xMemory, 7.0 = 100% (lower is better)TypeScript 7.0100%tsrsabout 30%
Why it matters

These are vendor benchmarks run with the TypeScript team on matched settings. The post also says AI models can now do large ports in days, which is why parity must be kept in sync with upstream by agents and tests.

Do this week

Time your own type-check in CI and note memory. Then try npx tsrs on one repository, treating it as experimental, and compare diagnostics with tsc before drawing any conclusion.

Cloudflare Blog →

Worth experimenting with

Map where each service verifies TLS certificates

Pick three services. For each, record which library, runtime or OS store validates outbound certificates, how that store is updated, and whether a base image bakes it in. Then run one request through the real client path.

Why. The eMudhra guidance says updating an SDK may not change the underlying trust policy. A small map shows where patches actually land.

Measure. Count services whose trust store source you could identify within 30 minutes, and note how many needed a redeploy to pick up a patch.

Classic / essential read

Strangler Fig

Martin Fowler · 5 min

Fowler explains why like-for-like rewrites of serious systems usually fail and why gradual replacement, starting with new features beside the legacy code, is the approach he and colleagues prefer.

One lesson. Before a modernization, agree the outcomes you want, then break the work into parts you can deliver one at a time, and change the organization so that can continue.

Read the original →

Commute watch · optional

Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — BraintrustAI Engineer · 18:15AWS Distinguished Eng: Learning From 3000 Incidents And How Engineering Is Changing | Marc BrookerRyan Peterman · 71:31

Each story today comes down to checking the layer you assumed was fine: who your services trust, what your cache stored, and whether an agent's output was verified. Pick one and test it this week.

Get this every morning.

Free. Same issue for everyone. One click to unsubscribe.

Subscribe