How GitHub's Telemetry Pipeline Hits 1.6 Million Events a Second

GitHub is enormous, and it keeps growing. A new developer joins the platform roughly every second, which means the internal systems watching it never get quieter. One of those systems is a telemetry and security-logging pipeline that swallows about 700 TB of data per day, used to spot malicious behavior and troubleshoot the platform.

The ingestion layer that fed it was a set of Java binaries layered on Azure Event Hubs, sitting between telemetry producers and the consumers that process the data. At up to 460 GB of events per day it was already "reaching its design limits," and availability was starting to slip.

The pain was operational. Every consumer needed its own environment and manual, time-consuming tuning to perform well, the Java code was prone to breakage and hard to debug, and the compute bill kept climbing. As GitHub's Stephan Miehe put it, the internal system was limiting the company's growth.

The actual job here is simple: accept events and hand them on, closer to an HTTP POST than a complex service. GitHub wanted a serverless platform with minimal overhead so the team could stop babysitting environments, and since they were already on Event Hubs they went looking through Azure-native options and landed on Azure Functions.

They joined the private preview of a new plan called Flex Consumption, which offers large scale-out, long execution times, private networking, instance-size selection, and concurrency control. The new app is written in Python, triggered directly from Event Hubs.

The flexibility comes with a catch that Miehe named directly: Flex Consumption gives you "a lot of knobs to turn," and there is a balance between that flexibility and the effort of setting it up right. The next three slides are about which knobs actually mattered.

The single biggest throughput lever was batching. Instead of shuttling events one at a time, the function consolidates large numbers of messages into one larger message before handing off to the consumers that do the real processing. Fewer, fatter handoffs mean the per-invocation overhead stops dominating.

Batch size is a genuine sweet spot, not a formula. Too large and an instance runs out of memory to process the batch; too small and you spawn so many executions that overhead swamps the work. GitHub settled on 5,000 messages per batch through experimentation, with each function execution finishing in about 100 to 200 milliseconds.

It is worth sitting with how ordinary this is. The clever part of a millions-per-second pipeline turned out to be picking one integer well, because that integer decides whether memory or overhead becomes the bottleneck first.

Azure Functions uses target-based scaling here. Rather than waiting to notice rising latency and reacting after the fact, it looks at how many unprocessed messages are sitting on the event hub and, given the batch size and partition count, jumps straight to an appropriate instance count.

The ceiling is structural: the app scales out to at most one instance per event hub partition, up to 1,000 instances for the largest deployments. Partition count becomes a hard planning decision. Miehe's warning is blunt: be deliberate about how many partitions you provision, because if you don't have enough, you constrain consumption no matter how elastic the compute is.

This is the sense in which the system "refuses to scale further": elasticity is real, but it is capped by a number you chose up front. Under-provision partitions and Functions will happily stop adding instances well before your traffic does.

With the knobs set, a single Flex Consumption app sustained 1.6 million events per second against a network-restricted event hub, and production peaks run above 1 million events per second. Underneath, a new Microsoft PaaS backbone called Legion gets fresh compute onto the private network fast: outbound VNet connectivity in 624 ms at p50 and 1,022 ms at p90, with cold-start latency down about 53% platform-wide over 18 months.

The most telling decision was the cutover. GitHub deliberately ran the old and new pipelines side by side to find where the ceiling is before trusting it, rather than assuming the numbers held in production. The selection goal was headroom: as Miehe framed it, they didn't want to migrate again in six months because they'd hit another wall.