Assume the agent and the inference stack will misbehave, and build the boundary around them
Two AI picks: Cloudflare explains how a rejected request leaked other users' KV cache, and Netflix wraps an LLM in deterministic log diagnosis. Two engineering picks: AWS ships recommended fault-injection tests and puts governance context on S3 objects.
AI & Agent Engineering
AI securityverified
Cloudflare traces cross-request data exposure in GLM-5.3 on Workers AI to a rejected request that decode still trusted
Cloudflare says 155 responses to prompts over 681,280 tokens contained other customers' data between Aug 27 and Sep 9. The prefill KV pool held 681,280 tokens against a 1,048,576-token context. SGLang marked an over-length request rejected but still sent a placeholder KV transfer, so decode read stale pages. Fixes are in SGLang v0.5.22; Cloudflare found no evidence of deliberate exploitation.
Why it matters
The failure came from a config mismatch the team knew about, plus an engine path that reported a rejected request as live. Pool size alone is fragile, since quantization or parallelism changes can shift it, so Cloudflare also made the rejection path safe.
Do this week
If you self-host disaggregated inference, upgrade SGLang to v0.5.22 or later. Add a deploy check that KV pool size covers the full context length, and test that an over-length prompt returns a client error.
Netflix puts an LLM around deterministic rules to cover the half of job failures its regexes classified as unknown
On peak days roughly 100,000 Netflix workloads fail. Legacy Pensive used rules and made about 50% of failures self-service, but about half of roughly seven million classifications over three months were unknown. Pensive AI numbers each log line and has the model return only start and end line numbers, then rebuilds the text from its own copy. Calls share a deadline inside Maestro's 90-second timeout, and late calls are dropped.
Why it matters
Returning two integers cuts output tokens and stops the model rewriting a stack trace. Netflix reports AI extraction now covers 100% of failures. Its 321% figure for fewer manual debug actions is observational, not causal.
Do this week
When adding an LLM to an existing pipeline, keep the rules, run the model beside them under a hard deadline, and have it return references into the source instead of copied text.
AWS Resilience Hub now generates four fault-injection tests from your resilience policy and scores them against your RTO
Recommended resilience testing builds AWS Fault Injection Service experiments from what Resilience Hub knows about a service. Four tests ship: Availability Zone recovery, dependency validation, Multi-Region isolation and Multi-Region recovery. A recovery test passes when success criteria return to healthy within the policy's RTO and stay healthy. AWS advises starting with a single soft dependency.
Why it matters
Hand-built experiments are hard to design and drift as architecture changes. AWS says to choose success criteria such as order success rate, not resource state, because the impaired resources are expected to be unhealthy.
Do this week
Pick one soft dependency of a service and block it in a non-production run. Define pass as a customer-facing rate staying above a threshold, and note any dependency that turns out to be hard.
Amazon S3 annotations let governance context travel with an object version without rewriting the object
S3 annotations attach context such as classification, retention policy, residency and a PII inventory to an object version. One version can hold up to 1,000 annotations, each 1 byte to 1 MB. Updating one does not rewrite the object or change its ETag, and copies and replicas carry them. Annotations stay readable in Glacier classes without a restore, and an S3 Metadata annotation table makes them queryable in Athena.
Why it matters
Governance labels usually sit in a separate database that can drift from the data. Keeping them on the object removes one sync job. A single record versus many granular annotations is a design choice with trade-offs in overwrite scope and query granularity.
Do this week
Check which governance fields in your catalog could live on the object. Prototype one, such as a retention class, on a versioned test bucket and query it through the annotation table.
Check that an over-length prompt returns an error, not a response
On a staging model endpoint, send prompts just under, at and just over the configured context limit. Record the HTTP status and whether the response arrives with a plausible body.
Why. Cloudflare's leak depended on an over-length request returning success. A boundary test shows whether your own stack fails closed.
Measure. Status code for each size, plus a canary string planted in an earlier request to see whether it ever appears in a later response.
Classic / essential read
Test Pyramid
Martin Fowler · 5 min
Fowler argues for many more fast, low-level tests than brittle end-to-end tests through a GUI, and warns teams against mixing up end-to-end, UI and customer-facing tests.
One lesson. Count your tests by layer this week and ask which slow end-to-end test could be replaced by a faster test through the service layer.