Linux Kernel Module Cheat

3.9. GDB step debug multicore userland

For a more minimal baremetal multicore setup, see: Section 33.10.3, “ARM baremetal multicore”.

We can set and get which cores the Linux kernel allows a program to run on with sched_getaffinity and sched_setaffinity:

./run --cpus 2 --eval-after './linux/sched_getaffinity.out'

Source: userland/linux/sched_getaffinity.c

Sample output:

sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

Which shows us that:

initially:
- all 2 cores were enabled as shown by sched_getaffinity = 1 1
- the process was randomly assigned to run on core 1 (the second one) as shown by sched_getcpu = 1. If we run this several times, it will also run on core 0 sometimes.
then we restrict the affinity to just core 0, and we see that the program was actually moved to core 0

The number of cores is modified as explained at: Section 24.3.1, “Number of cores”

taskset from the util-linux package sets the initial core affinity of a program:

./build-buildroot \
  --config 'BR2_PACKAGE_UTIL_LINUX=y' \
  --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
;
./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'

output:

sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

so we see that the affinity was restricted to the second core from the start.

Let’s do a QEMU observation to justify this example being in the repository with userland breakpoints.

We will run our ./linux/sched_getaffinity.out infinitely many times, on core 0 and core 1 alternatively:

./run \
  --cpus 2 \
  --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \
  --gdb-wait \
;

on another shell:

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity.out" main

Then, inside GDB:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c

and we observe that info threads shows the actual correct core on which the process was restricted to run by taskset!

We should also try it out with kernel modules: https://stackoverflow.com/questions/28347876/set-cpu-affinity-on-a-loadable-linux-kernel-module

TODO we then tried:

./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'

and:

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out"

to switch between two simultaneous live threads with different affinities, it just didn’t break on our threads:

b main_thread_0

Note that secondary cores in gem5 are kind of broken however: gem5 GDB step debug secondary cores.

Bibliography: