In ex1 and the lectures, we learned about strace and some of the things that it can do to help us understand what’s going on in a process. As I understand daemon processes are like background processes but I am not exactly sure what they do.
How do daemon processes work and can strace be used on these processes to figure out what they do?
So-called "daemons" work like any other processes in Linux, with a few
notable differences I will describe below. This means they can be investigated via the
/proc file system and commands that use it, they can be straced, and you can even attach
gdb to them. However, as they are often run either as the root user or under a dedicated
account, regular users are prevented from doing so. It requires superuser permissions.
Where daemons differ is that they typically do not have a controlling terminal so they are not subject to
terminal management (they do not receive SIGINT for instance if someone types Ctrl-C), etc.
As a brief case study, let's look at the cupsd daemon on a Ubuntu 18.04 machine.
This daemon has process id 32069:
# ps ax | grep cupsd | grep -v grep
32069 ? Ss 0:00 /usr/sbin/cupsd -l
The ? denotes that it lacks a controlling terminal.
We can examine its file descriptors:
# ls -l /proc/32069/fd
total 0
lr-x------ 1 root root 64 Feb 11 10:08 0 -> /dev/null
lrwx------ 1 root root 64 Feb 11 10:08 1 -> 'socket:[2352462]'
l-wx------ 1 root root 64 Feb 11 10:08 10 -> 'pipe:[2350417]'
lr-x------ 1 root root 64 Feb 11 10:08 11 -> 'pipe:[2350418]'
l-wx------ 1 root root 64 Feb 11 10:08 12 -> 'pipe:[2350418]'
l-wx------ 1 root root 64 Feb 11 10:08 14 -> 'pipe:[2350419]'
lrwx------ 1 root root 64 Feb 11 10:08 15 -> /var/log/cups/access_log
lrwx------ 1 root root 64 Feb 11 10:08 2 -> 'socket:[2352462]'
lrwx------ 1 root root 64 Feb 11 10:08 3 -> 'socket:[2353206]'
lrwx------ 1 root root 64 Feb 11 10:08 4 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 Feb 11 10:08 5 -> 'socket:[2345942]'
lrwx------ 1 root root 64 Feb 11 10:08 6 -> 'socket:[2350414]'
lrwx------ 1 root root 64 Feb 11 10:08 7 -> 'socket:[2350415]'
lrwx------ 1 root root 64 Feb 11 10:08 8 -> 'socket:[2350416]'
lr-x------ 1 root root 64 Feb 11 10:08 9 -> 'pipe:[2350417]'
Standard output and standard error refer to a socket:[2352462] object, which is a
Unix domain socket.
A Unix socket works similar to a pipe, except that it is bidirectional and that it
is tied to a path in the filesystem that is used to refer to it.
We can use the ss(8) to find out the processes connected to the endpoints of this.
# ss -x | grep 2352462
u_str ESTAB 0 0 /run/systemd/journal/stdout 2351222 * 2352462
u_str ESTAB 0 0 * 2352462 * 2351222
Here, we learn that cupsd standard output is connected to a Unix socket at
/run/systemd/journal/stdout.
systemd-journald
reads from this socket and stores the output in its logs.
The journalctl command can be used to retrieve what cupsd output to its standard output or error.
Here's the last line:
# journalctl -u cups | tail -1
Feb 11 10:08:02 gback-N501VW systemd[1]: Started CUPS Scheduler.
It is also possible to attach strace to the running cupsd daemon:
# strace -p 32069
strace: Process 32069 attached
epoll_wait(4,
Here, we see cupsd in the BLOCKED state inside a epoll_wait system call referencing file descriptor 4. This file descriptor refers to
a so-called epoll set
lrwx------ 1 root root 64 Feb 11 10:08 4 -> 'anon_inode:[eventpoll]'
cupsd maintains multiple file descriptors at once (see the listing above) and it needs
to be prepared to read from any of them, but it cannot use read(2) on any individual file
descriptor for this purpose.
If it used read() on one file descriptor and were blocked it couldn't then read from another
file descriptor until after that first file descriptor had data available, which prevented
it from reacting to input.
Instead, it implements a method to multiplex multiple file descriptors at once.
Further investigation would reveal the other processes with which cupsd communicates,
such as /usr/lib/cups/notifier/dbus which interacts with the D-Bus.