23.8.1. Reverse debug the emulator
While step debugging any complex program, you always end up feeling the need to step in reverse to reach the last call to some function that was called before the failure point, in order to trace back the problem to the actual bug source.
While GDB "has" this feature, it is just too broken to be usable, and so we expose the amazing Mozilla RR tool conveniently in this repo: https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242
Before the first usage setup rr with:
echo 'kernel.perf_event_paranoid=1' | sudo tee -a /etc/sysctl.conf sudo sysctl -p
Then use it with your content of interest, for example:
./run --debug-vm-rr --userland userland/c/hello.c
This will:
-
first run the program once until completion or crash
-
then restart the program at the very first instruction at
_start
and leave you in a GDB shell
From there, run the program until your point of interest, e.g.:
break qemu_add_opts continue
and you can now reliably use reverse debugging commands such as reverse-continue
, reverse-finish
and reverse-next
!
To restart debugging again after quitting rr
, simply run on your host terminal:
rr replay
The use case of rr
is often to go to the final crash and then walk back from there, so you often want to automate running until the end after record with --debug-vm-args
as in:
./run --debug-vm-args='-ex continue' --debug-vm-rr --userland userland/c/hello.c
Programs often tend to blow up in very low frames that use values passed in from higher frames. In those cases, remember that just like with forward debugging, you can’t just go:
up up up reverse-next
but rather, you must:
reverse-finish reverse-finish reverse-finish reverse-next