I much prefer faking to mocking, because that still preserves a lot of the real-world behavior relevant to prod bugs.
A full prod reproduction would be a holy grail, but probably only attainable for complex distributed systems if we have access to a prebuilt staging-like environment.
We're traversing the bridge between mocks, fakes and staging at this time.
Of course, we did not send the PRs to the repos seeing how they're already overloaded with them.
Nginx for example has 181 open PRs right now, but they only merge 2 or 3 in a day.
OSX is the best, we use the in-built (seatbelt) sandbox via sandbox-exec.
For Windows, we use WSL containers when available.
By default, if a safe sandbox environment is not available, we inform the user that a repro is not possible in the current conditions.
We do a lot of AST parsing - for both code and build configuration languages. Even then, we still have to rely on the LLM to figure out a lot of the details.
Making this work reliably for non-frontier models and codebases that don't have existing test harnesses is where a lot of the design work goes in.
This is essentially a (RCA <-> Repro test case) loop until we're recreated the bug. If our attempts are not converging and we’re on the wrong track, we ask for human input.
are you using any AI tools to debug productions issues at this time?
FixBugs was built because while investigating prod incidents, I had an epiphany.
SWEs build tools to solve all types of problems, but we ourselves use the flakiest tools.
While working at Google for example I was surprised GDB support for any type of binary debugging was almost non-existent.
This also allows us to inject various types of faults, which is helpful with debugging more complex systems.
I think FixBugs is most useful during high volume bug triage. This is where having a low false-positive async debugging agent is most helpful.