leadbybuild.ingArchive

LeadByBuilding Newsletter · 8 min read

Plan for metered AI pricing, test risky diffs with generated tests, and find the shared facility behind your redundancy

Three AI picks: Meta's LLM-generated tests that fail on risky diffs, LeadDev on preparing for metered AI pricing, and Spring AI 2.1 letting apps bring their own embeddings. Two engineering picks: Cloudflare's Q3 DDoS data and its Q3 Internet disruption review, where shared facilities were a recurring weak spot.

AI & Agent Engineering

AI testingvendor-reported

Meta's Catching JiTTests: LLM-generated tests that fail on risky diffs, with a 60% true-positive rate

Meta says Catching JiTTests are now part of its infrastructure. Overnight, it picks the previous day's risky diffs and generates tests meant to fail on them, via intent-aware and dodgy-diff workflows. An LLM-as-Judge ensemble and a rule-based filter (RubFake) rank catches before engineers are contacted. 589 engineers (64% of outreach) engaged; Meta reports a 60% true-positive rate and 74 serious privacy bugs caught at the diff stage in H1 2026.

Why it matters

The design avoids per-item human review: engineers set criteria and look at outcomes in aggregate. A deterministic filter sits next to a non-deterministic judge. All numbers are Meta's own, from one very large codebase.

Do this week

Pick one risk class, such as permission checks. Generate a failing test per risky diff for a month, filter with a fixed rule set, and track how many reach an engineer and how many are real.

Engineering at Meta →
Engineering leadershipverified

LeadDev: per-seat AI plans may end, so make token use visible and match model and effort to the task

James Stanier argues that per-seat AI plans are subsidised and that metered pricing is spreading. He cites Simon Willison's estimate of $2,180 at per-token rates against $200 paid, and says metered cost would be roughly ten times. He advises showing consumption to engineers and managers, never ranking people by tokens, and testing lower reasoning effort. AT&T cut some task costs by up to 56% for a 2% quality drop via a router.

Why it matters

Engineers who hit limits may move to personal plans, so company code goes through accounts nobody controls. The AT&T figure depends on having an internal quality measure, which is extra engineering work. The ten-times figure is the author's estimate.

Do this week

Pull a month of usage per team from your admin dashboard and share the anonymised spread. Then have one team run a routine task at lower reasoning effort and compare cost and quality.

LeadDev →
Agent frameworksverified

Spring AI 2.1 adds upsert and query-vector search so apps can supply their own embeddings

Spring AI 2.1 adds VectorStore upsert, which takes documents paired with application-supplied vectors, and a similaritySearch overload that accepts a query embedding. The store's embedding model is not called in either case. The post uses image embeddings from a multimodal model for a product catalogue. It warns that store dimensions must match the model, for example 1408, and that queries must use the same model that created the vectors.

Why it matters

This lets you reuse embeddings from a provider batch API, another team or a migration without re-embedding through add(). The risk is silent mismatch: a different model with the same dimension count gives meaningless results.

Do this week

If you re-embed existing data, test upsert on one collection. Add a check that stores the embedding model name with each vector and fails queries made with a different model.

Spring Blog →

Engineering

Securityvendor-reported

Cloudflare Q3 DDoS report: 96% of attacks ended in under 10 minutes and DNS amplification rose 215%

Cloudflare's Q3 2026 report says it mitigated 18.2 million network-layer DDoS attacks. 96% ended in under 10 minutes and 98% stayed under 1 Gbps. Attacks above 1 Tbps fell from 805 in Q2 to 251, which it links to law enforcement action against four IoT botnets. CLDAP floods collapsed by 99.5% while DNS amplification rose 215% and other UDP floods rose to 31.31% of network-layer attacks.

Why it matters

Most attacks end before a person could respond, so manual mitigation does not fit. The vector mix shifts quarter to quarter, so static assumptions about the common attack go stale. The data comes from Cloudflare's own network.

Do this week

Check that your DDoS protection acts without a human and that any rate limits or alerts on UDP and DNS traffic fit a short, low-bandwidth burst. Confirm you run no open resolver.

Cloudflare Blog →
Resiliencevendor-reported

Cloudflare Radar Q3: a power failure at one Honduran interconnection facility cut national HTTP traffic 63%

Cloudflare Radar recorded 49 Internet disruptions in Q3 2026: 19 government-directed shutdowns (18 of them exam shutdowns in Iraq), 14 power outages and five natural disasters. On September 10 a power failure at an interconnection facility in Honduras affected at least 11 major providers; HTTP traffic fell 63% for about three hours. On August 13 a cooling failure at a Phoenix data center nearly stopped traffic from Namecheap.

Why it matters

Shared facilities hide single points of failure: providers that look independent can sit in one building. Power and cooling were the cause behind many of these events, not software.

Do this week

List where your critical services and providers physically sit. Ask each host and CDN vendor about power and cooling redundancy at the facility level, and exercise one regional failover this quarter.

Cloudflare Blog →

Worth experimenting with

Share anonymised AI usage with one team and test lower effort

Pull one month of AI usage per engineer for one team and show the team the anonymised spread, with names visible to the manager only. Then pick one routine task and run it at a lower reasoning effort or a smaller model.

Why. LeadDev's piece says teams that understand consumption before a metered bill adapt faster. A small test shows what the cheaper setting costs in quality.

Measure. Tokens or cost per completed task and the share of outputs needing rework at each setting. If rework does not rise, the cheaper setting is a candidate default.

Classic / essential read

Feature Toggles (aka Feature Flags)

Pete Hodgson · 30 min

Hodgson groups toggles into release, experiment, ops and permissioning types, and says each should be managed differently. He warns that toggles add validation complexity and carry a cost like inventory.

One lesson. Treat flags as inventory with a carrying cost: give each an owner and removal date, and consider a cap on how many can exist at once.

Read the original →

Commute watch · optional

How to Evaluate LLM Agents & Build Enterprise GuardrailsAgileWoW · 33:29An Incident Post-Mortem • Paul Adams • GOTO 2017GOTO Conferences · 31:07

Make AI spend visible before a bill forces it, test risky changes with checks that can fail, and look for the shared facility or default your redundancy quietly depends on.

Get this every morning.

Free. Same issue for everyone. One click to unsubscribe.

Subscribe