


Copilot Platform @GitHub. Formerly Observability @Honeycombio and F# lang/.NET tools @Microsoft


I’ve been using Fusion as a daily driver for a while now: - Define signals for production SLOs - Detect bad microVM states - Find what some unattributed IDs are across hundreds of telemetry tables Plus it does well on some benchmarks I guess 🙂 Always on Max mode.

A good write up and another example of how.complexsystems.fail/
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below. We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations. The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services. First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments. The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores. Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts. Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own. Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States. When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive. Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally. Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway. Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change. After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region). The final Post-Incident Review (PIR) will provide more details later this week.

fwiw the overall outcome wouldn’t have been that terrible in this case, but it’s definitely the sort of thing you’ll show your whole ass on down the road in some review where someone is caremad about measuring the right stuff

Almost fooled myself today how good Opus is for writing verifiably correct code and context switching over to SLO definitions and success/counter metrics. The silly thing tried to jam two independent metrics together and say to measure that aggregate!

Follow @wormandwalnut for actually good and very strange AI art

This is how we do a substantial amount of work these days on the copilot team. I still prefer the richer desktop app experience when I’m focused on my own thing, but when collaborating with others this is how it’s done
Both Slack and Teams surfaces are GA and on a weekly release cadence. Keep your team in flow 🏎️ github.blog/changelog/2026-1…

Both Slack and Teams surfaces are GA and on a weekly release cadence. Keep your team in flow 🏎️ github.blog/changelog/2026-1…

Lots of moral and economic reasons to curtail betting markets, but maybe my personal pet peeve is that they’re objectively no better than traditional polls and are likely just a downstream poll indicator. Now that we’ve libbed out can we stop with them?

This thread is incredible. Anyways, there's a reason why my laptop has a sticker that says "AI at the keyboard, human at the deposition".
Hey @Railway, is there a reason employee's of yours are trying to breach my infrastructure? Your employee Jake Runzer, just attempted to tonight.

FWIW I think many many engineers will overindex on the latter and believe it can improve the former, and they will be proven wrong about that

I think the near-ish term for software will look like: - Genuinely net-new feature work still bottlenecked on vetting ideas, SDLC, internal processes, etc. - Radically faster time to improve performance metrics for that feature with a small agent fleet shortly afterwards

It was super cool building and scaling our cloud agents and enabling consistent experiences across web, mobile, desktop app, CLI, slack, and teams to all GitHub Copilot customers. Once you start doing work with cloud agents, it’s hard to go back. Excited to bring the next wave soon! github.blog/changelog/2026-1…

Calling it now. AI is going to usher in a whole new era of software development. I’m calling it Software as a Service. People will want to use these tools but they’ll realize they don’t want to spend the time to maintain and update them over time. This will create a whole new category of businesses and entrepreneurs that can take on that task for businesses that would rather work on their core products. AI is changing everything. This is a whole new paradigm of thinking about the way we interact with software.
i have no time to maintain all my vibe coded tools we need a business model where someone else maintains the code and i just pay a monthly fee

@github Copilot in the cloud is now one connected experience across the web, mobile app, CLI, VSCode, and the Copilot app. It has more tools, connectors, and faster compute to help you get more done. github.blog/changelog/2026-1…

Scaling cloud agents 10x in the middle of a conf —> now this rocks
Not a flashy announcement, but today we rolled out some new platform foundations for Copilot running agents to everyone right as Datadog had a big incident, and it all went off without a hitch. Very excited to scale agents in the cloud further and further