OpenAI's engineers recently encountered a complex bug in their C++ data infrastructure service, Rockset, which was causing mysterious crashes. The team initially struggled to identify the root cause, as they were dealing with two unrelated bugs that coincidentally manifested simultaneously. This led them to adopt an innovative approach, which they dubbed 'epidemiological debugging'.
Instead of focusing on individual crashes, they built a pipeline to analyze every production core dump from the past year, looking for patterns at a population level. By using ChatGPT to write a script that extracted and labeled data from these core files, they were able to identify two distinct crash populations. One set of crashes, known as 'misaligned-stack' crashes, was traced to a single physical host in an Azure region, where the CPU was silently producing incorrect results.
The other set of crashes, 'return-to-null' crashes, was linked to a race condition in the GNU libunwind library's Ux8664_setcontext function, which had been present for 18 years. This function updates the stack pointer (%rsp) before reading the instruction pointer (%rip), creating a narrow window where a signal could corrupt the instruction pointer. OpenAI's Rockset, with its frequent signal delivery events, triggered this race condition, leading to production crashes.
The engineers fixed the issue by reordering the instructions in the Ux8664_setcontext function, ensuring that %rip is read before %rsp is updated. They also upstreamed a self-contained reproducer to GNU libunwind, demonstrating that other unwinders, like libgcc, do not have the same problem.
This case study highlights the importance of building a high-quality dataset for debugging production crashes. By analyzing population-level data, the team was able to distinguish between two distinct bugs, leading to a more accurate understanding of the problem. This approach not only helped them resolve the issue but also provided valuable insights into the importance of data-driven debugging techniques in complex systems.
