Need help understanding the output of strace

Operating Systems & Open Source Linux
1

I’m trying to understand how things work with sudo and have traced through the steps of what happens after I intentially gave a wrong password. I can roughly understand the sequence of events by going down the lines and googling what each line does. However, I’m having trouble understanding how the lines are linked.

Below are shows some output from strace -f -p <PID of sudo>. I picked them to help with my questioning.

1   sendto(5, "<85>Jun 16 20:10:50 sudo: pam_un"..., 102, MSG_NOSIGNAL, NULL, 0) = 102
2   openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 6
3   lseek(6, 0, SEEK_CUR)                   = 0
4   fstat(6, {st_mode=S_IFREG|0644, st_size=3328, ...}) = 0
5   read(6, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 3328
6   close(6)                                = 0 
7   openat(AT_FDCWD, "/etc/shadow", O_RDONLY|O_CLOEXEC) = 6
8   lseek(6, 0, SEEK_CUR)                   = 0
9   fstat(6, {st_mode=S_IFREG|0640, st_size=1720, ...}) = 0
10  read(6, "root:!:18503:0:99999:7:::\ndaemon"..., 4096) = 1720
11  close(6)                                = 0
12  brk(0x55b06cff6000)                     = 0x55b06cff6000
13  access("/var/run/utmpx", F_OK)          = -1 ENOENT (No such file or directory)
14  openat(AT_FDCWD, "/var/run/utmp", O_RDONLY|O_CLOEXEC) = 6
15  lseek(6, 0, SEEK_SET)                   = 0
16  alarm(0)                                = 0

At line 12, there is a brk() syscall. Who called it? Is it the previous process close()? Who called access() in line 13? Line 13 is the line I’ve tracked to show any indication that the password was incorrect, but I can’t trace who/what called it and why. I have looked at the sudo source code, however I think the syscall span across one source code and i need help knowing where to look and how to trace.

2

The thing is, strace simply lists system calls, it’s not a general profiler/tracer/debugger; It shows all interactions between the kernel and the Userland, NOT everything a program does.

The question of "Who called " can’t be answered by strace, as a function call in a C application doesn’t require any kernel interaction.

Also keep in mind that most syscalls are made by LibC functions:
If you’re looking through the sudo, looking for syscalls you hopefully don’t find many.
Syscalls are not guaranteed to be portable between different kernel versions, and especially different kernels(e.g. FreeBSD).

Take a look at GDB.
GDB is a proper debugger, and can be used to analyze a running process more thoroughly, including:
dumping stack/call frames(“call trace/local variables”),
single-stepping(“run one line of code at a time”),
peeking/poking memory(“read/write process RAM”), and
breakpoints(“When this line is reached, pause the program and show where it’s currently at, what it’s state is”).

You might need to recompile sudo, or install debug symbols from your distro’s repositories for all symbols etc. to be resolved(although you can use GDB without debug symbols).

https://sourceware.org/gdb/wiki/GDB%20Front%20Ends
You should check this list to see if your IDE/editor has support for GDB(you probably can get cool features like clicking a line in the source code to set a breakpoint, inspect local variables, etc. - makes using GDB way more painless.

3 Likes
3

What I’m trying to achieve is understand how the “sudo” process works, which actually involves other modules like pam. I suppose I need to compile my own copy of these other modules…wonder how deep that rabbit hole goes.

I’ve never touched kernel code before, any things I should do while compiling/adding prints/debug?

4

Well, you can try to understand the sudo process on multiple levels.

There is a “part” of sudo that is handled by the kernel, but that’s mostly related to filesystem permissions, the SUID bit, and permissions on Unix in general.

$ ls -lah /usr/bin/sudo
-rwsr-xr-x 1 root root 179K Feb 27 09:28 /usr/bin/sudo

As you can see, in addition to the regular file permissions rwx, the sudo file has an additional permissions bit set, indicated by an “s” in the output above(it’s also colored special on a real terminal) - the setuid bit.

If this bit is set, an application(like sudo) is allowed to change it’s own UID to the owner of the file(also in the ls output above, in this case root).

If you’re thinking about sudo now, it’s job seems somewhat simpler:
Verify that the user running the sudo command is in the sudoers file, then allow/deny access.
It runs completely in userspace.

libPAM is also not that difficult: It’s just a collection of shared libraries(the Linux equivalent of .dll, aka. a list of callable C functions and data, basically an application without an entry point) that can look up your login information somewhere and can return success/fail.

There should be no need to touch kernel code, but you’re of course welcome to look up how setuid etc. works on the kernel side as well - be prepared though, kernel land is strange: You don’t have a regular libc, instead you need to use the special klibc. Memory allocations works way different. Less/other protections(don’t relay on them), etc.

If you’re simply looking to upgrade your knowledge regarding sudo, and it’s interactions with libPAM, /etc/passwd, /etc/shadow you should stick with analyzing source code, using GDB with debug symbols and strace.
(As long as you grasp what setuid does; look it up if you don’t know!)

A “fun” thing to do is implement your own “sudo”. Doesn’t even need to elevate to root, could just elevate to your own user.
You don’t need to read /etc/passwd or /etc/shadow, you can just hardcode password(-hashes) for experimentation.
Depends on how deep you want to get though, and your previous C/Unix programming experience(this is probably not a good first-time C experience).

EDIT: Don’t use that for anything serious though. sudo does a lot more than just giving you permissions; It makes sure that your environment variables are clean, that your terminal is secure and enabled for sudo, and loggs commands etc. to your system log for audit purposes.

1 Like
5

Thanks!

This deviates quite a bit from the original title. Let me know if I should setup a new thread. I will give that a try and come back should I run into any problems. Any other steps or gotchas I need to observe when compiling/running modified (additional prints…if such a thing is possible) kernel code in userland?

6

Sorry, I just saw your post right now.

Again, usually you shouldn’t need to worry about what the kernel does.
You definitely however can add some debug print statements to the kernel. It’s some effort(mostly re-compiling your kernel), but not too bad.
For better debugging consider not booting the kernel on your machine with your regular Linux distro, but running in in QEMU with a minimal initramfs(busybox usually is enough for this; You can a slo use a statically compiled bash).

https://www.kernel.org/doc/html/latest/core-api/printk-basics.html

Don’t forget to set an appropriate Log level.

Pro tip: You can redirect your console to serial, and let QEMU connect serial to your terminal. This has the advantage that you can copy-paste kernel output, and have a nice scrollback buffer from your terminal emulator.

Also of use if you’re kernel hacking:
https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html

But again, not strictly needed to figure out how sudo works, or at least not specifically sudo(The mechanisms in the kernel have nothing to do with sudo, only with general UNIX-y filesystem and user permissions.)

Sorry for the late answer :stuck_out_tongue: