Years and years ago I had the opportunity to give Intel processor designers (the time of the 386!) requests for features.
I requested a system tick timer for stamping logs (they did that), bus mask and value registers that triggered a debug interrupt on a match (they did that).
And a jump source history. Maybe 10 jumps back. So on a breakpoint you could figure out how you got there. A time travelling debug feature.
At this point Intel sold an expensive debug probe for recording the bus, you plugged this insane cable into the processor socket and it actually executed in their external hardware, recording every instruction.
My jmp history would have replaced much of that, obviating it's need for the vast majority of users.
Ah well, it didn't happen. So now we all rebuild code 'debug' so we can add tracing and tracking, disrupting the execution path, changing timing and code size and on and on.
I always regretted not getting that.
[1] https://stackoverflow.com/questions/14670586/what-is-the-ove...
[2] The MSRs for LBRs (MSR_LASTBRANCH_*_{TO,FROM}_IP) and BTS (IA32_DS_AREA) are described in Volume 4, Section 2.2 of the SDM: "MSRS IN THE INTEL® CORE™ 2 PROCESSOR FAMILY". Core 2 was launched in 2006.
Though they are sufficient to do what the person you responded to asked for which is just execution trace.
Has anyone here tried using this with Noir yet? I'm curious about the performance overhead of the tracing mechanism, especially for longer-running programs. Also wondering if there are plans to support JavaScript/TypeScript for web development use cases.
It's not ready yet, and it might be proprietary: it would be great if we can open source it, if we find a sustainable business model for that
Why and How zk-SNARK Works (2019)
Nim and some Python are used for our closed source rr backend currently, and the frontend is written in Nim (compiling to JavaScript).
The backend for blockchain and scripting language which is open sourced, is newer and we used Rust there for several reasons. One of them is related to the fact that many blockchain languages are implemented in Rust and this makes it easier to interoperate/contribute. There are other aspects as well: both languages have pros and cons.
Some pros of Nim are e.g. it's metaprogramming support; the ability to share easily code/types between backend and frontend(it's an alternative to both e.g. C++/Go and TypeScript for us).
We're thankful to both language communities!
Unlike the existing platforms, which capture only message flows and require you to make educated guesses when some anomaly is observed, our system will let you accurately replay the processing code for each message to quickly identify the root cause for the anomaly.
This would rely on our ability to jump to the specific moment in time when a certain incoming message starts being processed. This moment can be identified either by a log line with a specific format or by a call to some special tracking function (e.g. track_incoming_message(request_id)).
For the system languages, the RR[1] recordings try to be practical by capturing only the non-deterministic events in the program execution. You can pair this with a ring buffer that discards the data after a certain retention period.
For the scripting languages(or any implementation using the db-like traces) we might add some advanced record filtering options.
(But maybe we are misunderstanding the question?)
You can not replay execution without a known state followed by all non-determinism after that state which is most easily done by starting from the initial state. To discard data, you need to manifest a state snapshot corresponding to that time to enable forward reconstruction from that state.
[0] https://github.com/metacraft-labs/runtime_tracing#format
However, some of the important optimizations, that we're preparing are not related so much to the format, but to record more specific things and reconstruct more in the postprocessing.
This would be also great for LLM to give some context via MCP server or even let LLM pick what variables history wanna see instead of giving full recording file.
Also nice would be some recording filtering that you wanna e.g. pick few variables and display history during whole execution maybe with some specific formating and maybe even for some numeric variables or like tensors, images, etc pass to rerun for visual debugging so you can see plot it
The Python initial prototype is not yet finished. It's easy to play with, so anyone interested can actually work on it! Currently, in the experimental tracers, Ruby is usable for smaller programs, so one can try Ruby immediately.
I do plan on improving some of the prototypes, and on adding additional ones: for Lua, but JavaScript: e.g. v8 is also a good target. Scripting language users that find it useful, are welcome to discuss/chat with us, or even directly contribute or propose support for new languages.
A form of record filtering is planned indeed.
We have experimented with automatic chart visualizations of some things, we've planned custom visual representation as well, great to see interest in those
There’s Replay for browsers and Wallaby for Node.
I've been searching for something like this, so my question is
I have almost identical program in version 1 and 1.01 and I need to find how their behaviour changed
So, I run both of them ./binary1.exe input.txt ./binary2 input.txt and record their execution with your tool
And now, I'd want to extract such data from your tool:
Visited functions and how locals were changing. e.g
int test(int n) {
n++;
std::cout << n << std::endl
n += 15;
if (n > 22)
{
n--;
}
return n + 1;
int main(int argc, char* argv) {
auto result = test(argv[1]);
std::cout << result << std::endl
Visited Function: main with arguments (argc: 2, argv ["path", 7])
Visited Function test with arguments (n: 7)
test: N set to 8
test: N set to 23
test: Entered If (n > 22):
N set to 22
Exit If (n > 22)
test: N set to 23
Exit function test
Main: result set to 23
Exit function main
Can I achieve it with your tool / recording data format?
I don’t like that the headline is “designed to support multiple languages” but it only actually supports an obscure language I’ve never heard of. Feels like a bait and switch.
I am sorry if the headline felt misleading or the current support disappointing: we do have experimental Ruby support, that you can try right now if you record a `<somepath>.rb` program.
We do design the frontend, trace format/lib and backends to support multiple languages. Ruby is already having experimental support, and we try to keep various other languages/usecases in mind. We hope to find a model that lets us work more on supporting many more scripting languages. We'd also love contributors/the community adding support for languages or codetracer itself!
We also do have a closed source backend based on RR[1] that has partial C/Rust/Nim support, but it is not yet ready. It might be released as a proprietary solution. (However if we find an alternative sustainable business model, it would be great to be able to open source it.)
The scripting/blockchain languages backend is more db-like: it collects a trace by hooking in tracing API-s or instrumenting/patching vm-s (the trace is later postprocessed before replay).
The system languages backend is based on RR[1] recordings currently.
We'd be happy to discuss more usecases or languages!
Though if it uses rr it won’t be able to run on macOS. Bummer, macOS seems to get harder and debug on. Luckily lima vms make it easy to remote :/
We've researched possible alternative approaches/tools as well, especially keeping in mind Windows/Mac support.
The traces for Noir and the scripting languages work in a completely different way, capturing all the relevant data which is later indexed into a db-like structure. With some future optimizations this can be very useful for various shorter programs in scripting languages, and generally for blockchain languages(as the running time there is usually low) and we hope that eventually with flexible record filtering it can be practical even for capturing important segments/aspects of long-running real world projects.
One question I have is, how exactly does it record and what are the boundaries of the recording?
For example does it only record the userland execution of a single process, or does it have broader boundaries like including kernel code and/or execution of multiple processes? How does it handle shared memory regions that may be modified outside of the recording?
For the scripting languages and smart contract/ZK languages, we instrument the interpreters using high level hooking API-s or direct patches, and we produce a trace.
For system languages, we directly build on top of RR[1] recordings for now: RR can record multiple processes, and it works in userland. IIRC it doesn't support modifications of shared memory outside of the recording. It's very well documented in their paper: Engineering Record And Replay For Deployability: Extended Technical Report[2].
I loved debugging with that when I was working on react web app.
Wish there was something like this for react native :(
Fray is a controlled concurrency testing tool for the JVM that supports record and replay. It could be a perfect backend for codetracer. (I'm the author of Fray)