leadbybuild.ingArchive

LeadByBuilding Newsletter · 7 min read

Autonomous attack frameworks, graders you can trust, and a stale leader behind GitHub's write outages

Two AI picks: Cloudflare's data on LLM-driven penetration testing frameworks and the credential habits they exploit, and Anthropic's recommendations for building evaluations with reliable graders. Two engineering picks: GitHub's October availability report, including how a replacement machine with outdated records caused two write outages, and application-level health checks reported by EC2 itself.

Top news

  1. 01Microsoft 365 outage that began Thursday continues into Friday; Microsoft blames servers failing automated health checks

    A health check that takes servers out of service can itself become the capacity problem. Review what your checks do under load and how quickly you can override them.

    Ars Technica · 13 Nov 2026
  2. 02FAA issues a nationwide ground stop after a communication issue affecting all Air Route Traffic Control Centers

    Officials say no cause has been ruled out. Know which of your services share one communications dependency and what the safe degraded mode is when it fails.

    The Verge · 13 Nov 2026
  3. 03Jensen Huang resigns as Nvidia CEO; CFO Colette Kress is named interim CEO

    TechCrunch reports the exit raises questions about Nvidia's support for open models. Teams that depend on its open-weight work should watch for changes in direction.

    TechCrunch · 12 Nov 2026
  4. 04Researcher discloses SignDrop, a Windows zero-day that gives local users System rights on some versions of Windows

    Microsoft says it is investigating. Teams with Windows 11 26H2 machines should track the fix, since the researcher says the technique can turn off antivirus and EDR.

    Ars Technica · 13 Nov 2026
  5. 05Uber notifies drivers that license information was stolen in an October breach at its provider Fireworks AI

    Credentials stolen from a third-party AI provider were used to reach Uber's data. Check what data your AI vendors hold and how narrowly their access is scoped.

    TechCrunch · 13 Nov 2026

AI & Agent Engineering

Securityverified

Cloudflare: LLM penetration testing frameworks have multiplied, and static CI/CD tokens are the way in

Cloudflare reports that open source LLM penetration testing orchestrators have spread since the ARTEX repository was taken down on October 8. It tracks more than 50 such agents and says daily requests from these tools have grown 30x since October 1. In incidents it investigated, attackers compromised static, non-expiring tokens in CI/CD pipelines that gave broad cross-environment access. It recommends six controls.

Why it matters

These frameworks run every stage of an attack with parallel agents, so small configuration mistakes are found systematically and one operator can target many organizations. The figures are Cloudflare's own and the six controls map to its products, but the root cause it names, long-lived broad tokens, applies to any stack.

Do this week

List the API tokens your CI/CD pipelines use. For each, record whether it expires, whether it is restricted by IP, and how many environments it reaches. Replace the broadest non-expiring one this week.

Cloudflare Blog →
Evaluationverified

Anthropic's guide to evaluations: pass or fail whole conversations, and test the grader itself

Anthropic published recommendations for building safety evaluations. In its multi-turn evals an expert writes a specification, a separate Claude model plays the user, and a grader model scores each transcript; one bad turn fails the conversation. For graders it advises atomic yes or no criteria, a different model from the one tested, flagging broken items, and validation against human labels.

Expert writesspecificationFirst user turnis fixedClaude modelplays the userScenario runsseveral timesGrader scoreseach transcript
Why it matters

The post is about safety, but the grader advice fits any evaluation that uses an LLM judge. Graders can break and still tally scores, and the post says they favor longer responses. If conclusions change when the grader changes, you measured the grader, not the model.

Do this week

For one eval that uses an LLM judge, have two people label a sample of transcripts without seeing the judge's verdicts. Compare their agreement with each other to their agreement with the judge, and revisit the rubric if the judge is the outlier.

Anthropic →

Engineering & Leadership

Reliabilityverified

GitHub's October report: a replacement machine with outdated records caused two write outages

GitHub's October availability report says 140 of 151 production services and 32 of 50 MySQL primaries now run in Azure Central US. On October 7, a replacement machine that manages where database writes are sent started with outdated records, was elected leader, and routed writes to retired servers. The settings passed validation. A maintenance task then ignored a pause and repeated the outage. Request failures peaked at 8.9% and 9.3%.

Routing machinesreplacedNew machine:outdated recordsElected leaderWrites go toretired serversEstablishedmachine leadsMaintenanceignorespause
Why it matters

Three ordinary gaps lined up: retired servers left in records, a new node eligible for leadership before its state was checked, and automation that did not honor a pause. Other incidents in the report show the same shape, such as calls that waited for timeouts instead of failing fast, and simultaneous retries that saturated a network link.

Do this week

Pick one system that elects a leader or publishes routing configuration. Check whether a newly added node can take over before its state is verified, whether a wrong configuration would be rejected, and whether every automated job honors a maintenance pause.

GitHub Blog →
Cloud operationsverified

EC2 application status checks: application health as an instance signal, with a staged rollout

An AWS Compute Blog walkthrough covers EC2 application status checks. EC2 evaluates an application endpoint every 60 seconds and reports the result through the DescribeInstanceStatus API and CloudWatch, next to instance, system and EBS checks. EC2 Auto Scaling can replace an instance whose application status is impaired. A check can be excluded from the aggregated status while it is validated, and reporting can be paused for maintenance.

Why it matters

Instance and system checks pass while a web server has stopped accepting requests. This gives fleets a built-in application signal without a custom health metric. The post is also clear about the risk: without a pause, planned maintenance can get a healthy instance replaced. The checks incur charges.

Do this week

If you run EC2 fleets where a hung process goes unnoticed, add one check in excluded mode and watch its results for a week before letting Auto Scaling act on it. Set the grace period to at least your application's start time.

AWS Compute Blog →

Worth experimenting with

Find your broadest long-lived pipeline token

Take one CI/CD system and list every API token or key it stores. For each, note whether it expires, whether its use is restricted by IP, and which environments it can reach. Pick the one with the widest reach and no expiry, and replace it with a scoped, expiring credential.

Why. Cloudflare's incident work found attackers repeatedly compromising static, non-expiring tokens in CI/CD pipelines that granted broad cross-environment access. Automated frameworks make use of such a token quickly.

Measure. Count tokens with no expiry before and after. Record how many environments the replaced token could reach and how many its replacement can reach.

Classic / essential read

Leader and Followers

Unmesh Joshi · 2 min

This short pattern summary from Patterns of Distributed Systems explains why replicated systems select one server as leader: it takes decisions on behalf of the entire cluster and propagates them to the other servers. GitHub's October 7 outage shows the cost when the elected leader holds the wrong state.

One lesson. A single leader simplifies consistency, but everything it publishes is trusted by the rest of the cluster. Treat who may become leader, and with what state, as a safety property.

Read the original →

Check the things you trust by default. A long-lived pipeline token, an LLM grader, a newly elected leader and a health check all act with authority, so verify each one before it is allowed to act alone.

Get this every morning.

Free. Same issue for everyone. One click to unsubscribe.

Subscribe