This FAQ answers questions students have had in past semesters related to the 'esh' project.
Click here to show all answers
No. Built-in commands are executed by the shell without forking a new process.
Moreover, some built-in commands, notably cd, but also any command related to
environment variables (such as bash's export command), must be executed
in the shell's process or else their effect would vanish after the shell's child
process exited.
No. You may assume that your shell is started in the foreground (i.e., has ownership of its terminal) and is already in its own process group. This will be the case if you test your shell by starting it from bash (type "./esh"), and it will also be the case if the shell is started from the Python test harness.
Read the manual (man 3p exec) to learn about the different versions.
You may use any of them; a recommended function is execvp().
When a foreground process exits, the shell needs to reacquire the terminal (by making its own process group
the foreground process group of its controlling terminal.) This is done via tcsetpgrp().
Since the process issuing the tcsetpgrp() is not in foreground group, the tcsetpgrp() call will result in the delivery
of SIGTTOU. (The same rules that apply to all processes apply to your shell, too.) In this case, this
creates a catch-22: the shell itself needs to be the foreground process group to call readline() to receive
the next input from the user, but the call needed to make the shell's process group the terminal's
foreground process group requires that the calling process already is part of the foreground process group.
POSIX resolves this dilemma by stipulating that the
shell should block SIGTTOU during the call to tcsetpgrp(), and that if it does, SIGTTOU will not be delivered.
You can achieve this via the provided
esh_signal_block()/esh_signal_unblock() functions, as shown in
this this code snippet.
We highly recommend you read pages 18-21 of the POSIX 1003.1 Rationale. The entire document is available here. (SLO Login required).
These functions provide convenient wrappers for the signal masking and handling functions you need,
as well as functions to save/restore the state of the terminal. They use the POSIX API as discussed
in the book in Section 8.5.5. There is no need to use old-style, pre-POSIX signal handlers
(e.g., using signal() or the book's Signal() function, as used in the
examples in 8.5.1-8.5.4.) The functions in esh-sys-utils.c are used in my sample solution.
By sending SIGCONT to the job's process group. If the job is also made the foreground job, SIGCONT must be sent after ownership of the terminal has been transferred.
strace shows the pipe() and dup2() calls
I'd expect. It seems the child process never exits, so my shell is stuck in waitpid()
Check that the shell closes its copy of the pipe file descriptor(s) after the fork().
If it doesn't, since there's still an open file descriptor referring to the write end,
the pipe itself will not be closed, and programs
that read until they exhaust their standard input (such as cat, grep and
so on) will be stuck attempting to read from their stdin. Use ls -l /proc/NNNNN/fd
where NNNNN is the pid of your shell to examine the state of the shell's file descriptors.
struct list as a
parameter to a function?
The provided list class uses head and tail sentinels that are stored in each struct
list instance. The last element in the list contains a 'next' pointer to the tail sentinel, and
the first element contains a 'prev' pointer to the head sentinel. If you copied the struct list
instance, those pointers would not be updated, rendering the copied struct list instance useless.
Always pass pointers to a struct list to functions, as in struct list *.
void myfunction(struct list *l) { // ok
}
void dontdeclareafunctionlikethis(struct list l) { // will not work
}
To avoid other list-related pitfalls, particularly pitfalls related to deleting elements and iterating over lists, you must read list.c.
The only use for it that I could see is if you're implementing sophisticated built-in commands
with many switches/options. Otherwise, probably not. Note that the provided parser code prepares
each command line already in a form that can be passed to execvp().
No. Although theoretically possible, this is not a recommended approach. But if you're familiar with yacc/lex and wish to add additional features (say redirecting stdout/stderr separately), you're welcome to extend it.
Yes. Some processes place the terminal in a special, so-called 'raw' state. For instance, vi does that.
In 'raw' state, the program receives characters as they are typed. (By contrast, in the 'cooked' state,
programs normally read lines of input.) If a foreground job is suspended and later resumed, you need to save
and restore whatever state it placed the terminal in. Use the provided functions in esh-sys-utils.c
for this purpose, which wrap the tcsetattr()/tcgetattr() functions.
Note that you do not need to understand what exact state the job put the terminal in. On a related note, if a
process that places the terminal in the raw state is stopped, the shell must restore the original state of
the terminal before issuing the next prompt. To that end, save the initial state of the terminal
when the shell is first started and restore this state whenever the shell outputs its prompt and
waits for user input.
The rationale for this approach is that a job control shell must allow the execution of programs that are themselves job control unaware, i.e., programs that assume they have exclusive control of the terminal. See here.
You may use this code snippet. See also Question 4.
The only signal that you must catch is SIGCHLD. When SIGCHLD arrives, you need to reap any children whose status may have changed and update the shell's data structures accordingly.
Optionally, you may catch SIGINT (^C). Catching SIGINT could be used, for instance, to allow a user to abort a partially completed command line, like bash does. You would need to use sigsetjmp()/siglongjmp() to that end, as shown on a lecture slide. Note that when the user types ^C, SIGINT will reach the shell only when it is in the foreground process group, which is the case only while no other foreground job is running - namely, while the shell waits for user input.
When a foreground job is running, it will have ownership of the terminal. This means that the keys ^Z and ^C cause the OS to send SIGTSTP and SIGINT directly to the processes in the foreground process group, and *not* to the shell. So, for SIGTSTP and SIGINT to stop/terminate your foreground process, you don't have to do any signal handling!
A second signal you may optionally catch or ignore is SIGTSTP. Like ^C, ^Z will reach the shell only while waiting for user input. You could ignore SIGTSTP so that the shell isn't suspended itself when the user types ^Z instead of typing a command. This is what most shells do, as you can observe by typing ^Z on a bash prompt.
Because we've seen students too many times wasting their time and ours trying to debug obviously wrong code.
Building the project on any environment other than our lab machines is not officially supported. That said, the code should build on most recent Linux distributions (including Debian and Redhat-based ones.) Feel free to share any problems you encounter on the Piazza forum!
These terms are often used synonymously, but they describe different aspects. A pipeline is a sequence of one or more user commands separated by the pipe (|) symbol, which will result in the execution of one or more processes. Each pipeline forms a job. A new process group is created for each job. By convention, the first process of each pipeline becomes the leader of the job's process group, the other processes join the leader's process group.
So, put briefly, each pipeline is a job, and each job has its own process group.
Think about what could happen in each case. You need to avoid a situation where the launched program performs some action that requires ownership of the terminal, but tcsetpgrp() hasn't been called. In addition, when you call tcsetpgrp(), the process group you specify must have been created also. Read pages 18-21 of the POSIX rationale document.
No. You will receive notifications for each process that's part of a pipeline/job. They may exit in any order. Your shell must manage this: for instance, if a pipeline/job that consists of multiple processes is running in the foreground, the shell should not prompt for new input until after all processes that are part of the job have been reaped (and your shell received confirmation from the OS that they've terminated.)
Note that one call to waitpid() yields only one notification, even if you've specified waitpid(-1,..) to accept notifications for any of your children. This is so that the status of each child can be reported, separately. You will thus need to call waitpid() repeatedly.
strace -ff shows that writing
to file descriptor 1/reading from file descriptor 0 by the child processes
results in -1 EBADF.
You probably confused the read and write ends of the pipe.
This may have several reasons. A common one is that the output of your shell does not match what the test script expects, and/or that you haven't adapted the file describing the output format of your shell accordingly. Among others, the test harness expects that the name of a job can be extracted via a regular expression from the job list. In addition, it expects that the shell echoes the command line of a job when it executes the 'fg' command. (Like bash does.)
An object can only be in one list at a time. If you're placing an esh_pipeline object into a new list without first removing it from the old list, you're corrupting the old list's next/prev pointers. This will lead to undefined results if you're doing this while iterating over the old list. Read list.c for techniques of how to avoid this. If you need to keep an object in multiple lists simultaneously, you must provide separate list_elem fields.
A common mistake is to misunderstand the effect of fork(). When the shell calls fork(), a new child process is created. The child process will eventually call exec() to load the program the user desired while the parent process will continue to implement the shell functionality - wait for foreground jobs, implement job control etc. The child process obtains a copy of all data from the parent at the time fork() is called. "All data" here means all heap objects, all global, and all local variables. This copy will have the same values that it had before the fork(), but any modifications affect only the child's copy. When the child execs, all of its data is wiped and replaced with the data of the new program. So, code like this
/* WRONG CODE */
if (fork() == 0) {
/* in child */
cmd->pipeline->status = FOREGROUND; /* update child's status */
/* Above statement is wrong and pointless - it updates the
child's copy of the 'pipeline' object only.
All of the child's data, including everything on its heap, is about to
be discarded by the subsequent execvp call we are about to execute. */
execvp(cmd->argv[0], cmd->argv);
}
is utterly wrong. Make sure that anything related the management of job lists, job status, etc. is done before the fork(), or after the fork() in the context of the parent (shell) process. Be aware that while you can use local/global variables and heap objects to pass information from the parent to the child when it is forked, you cannot use those variables to pass information back from the child to the parent, or from the parent to the child after the child has been forked.
When you code them. An all-too-frequent mistake is for students to waste hours or even days debugging their code's logic when they could have found the error immediately or in short time simply by checking which system calls fail, and why.
The strategy for using waitpid() correctly is a bit tricky. waitpid() must be called on three code paths: when waiting for a process started as a foreground process to terminate, when implementing the 'fg' command, and also from inside the SIGCHLD handler.
You should call waitpid() in all of these situations with -1 as the first argument and using the WUNTRACED option; when calling it in the SIGCHLD handler, also specify WNOHANG using the bitwise | operator.
To avoid race conditions, keep SIGCHLD blocked when calling waitpid() while waiting for a foreground job. (Note that SIGCHLD is blocked inside the SIGCHLD signal handler already.) But do not keep SIGCHLD blocked all the time - when the shell sits at the command prompt, waiting for user input, SIGCHLD must be unblocked so that the shell can reap terminating children.
Since you are passing -1, it may happen that while waiting for a foreground job waitpid() returns to tell you about a background job that terminated. In that case, you must record this information in your job list (since the OS will tell you only once about it!), and then call waitpid() again. Here is a code snippet you can use to implement this approach.
Note also that even though the waitpid() call issued by the shell when waiting for a foreground job will reap any children that have exited or changed status, a SIGCHLD signal, once pending, will still be delivered as soon as you unblock SIGCHLD. In that case, the OS has nothing to report when you call waitpid(). That's why you must use the WNOHANG flag in the SIGCHLD handler and that's why you should ignore if waitpid() returns -1 there.
The verb "to block" is used in at least two different meanings in operating systems. A process may "block" when it encounters a situation in which it cannot make progress. For instance, a process may wait for some event (such as a read() operation to complete, or a child process to exit, or simply for some time to pass). In this case, we say the process is blocked. Blocked processes do not use CPU time. They are unblocked when what they're blocked on finally happens. At that point, they become ready and the scheduler will eventually resume them as soon as a CPU/core becomes available.
Another meaning of "to block" refers to "blocking a signal." Blocking here means to delay the delivery of a pending signal until some later point in time. This is accomplished using an appropriate system call (sigmask(2) or sigblock(2)). It is needed when there is a potential for race conditions between the signal handler and the main control flow of a program.
\r\n) - I thought
the Unix line termination character was LF (\n)?
The regular expressions in eshoutput.py refer to the information that is output
the master side of the pseudo terminal, and the logfile logs everything that is input into
the master side of the pseudo terminal. Your shell, on the other hand, is connected to
the pseudo terminal's slave end. The pty driver performs the same operations a regular
terminal driver would. It can also be customized in the same way.
For instance, if the flag ONLCR is set, it will transform a NL into a
CRLF (See termios(3); you can run stty -a to see which flags are enabled
by default for the pseudo terminal used by ssh.). Similarly, if ^Z is sent to the
master end of the terminal, SIGTSTP will be sent to the terminal's foreground process job.
pty(7) summarizes the functionality provided by pseudo terminals:
A pseudo-terminal is a pair of virtual character devices that provide a bidirectional communication channel. One end of the channel is called the master; the other end is called the slave. The slave end of the pseudo-terminal provides an interface that behaves exactly like a classical terminal. A process that expects to be connected to a terminal, can open the slave end of a pseudo-terminal and then be driven by a program that has opened the master end. Anything that is written on the master end is provided to the process on the slave end as though it was input typed on a terminal. For exam- ple, writing the interrupt character (usually control-C) to the master device would cause an interrupt signal (SIGINT) to be generated for the foreground process group that is connected to the slave. Conversely, anything that is written to the slave end of the pseudo-terminal can be read by the process that is connected to the master end. Pseudo-terminals are used by applications such as network login services (ssh(1), rlogin(1), telnet(1)), terminal emulators, script(1), screen(1), and expect(1).
('expect(1)' is a program that performs similar functionality as the pexpect library Patrick extended for our testing framework.)
You need to read the fine print in the man page for these functions:
To get the prototypes under glibc, define both _XOPEN_SOURCE and
_XOPEN_SOURCE_EXTENDED, or use "#define _XOPEN_SOURCE n" for some integer
n larger than or equal to 500.
So, use:
#define _XOPEN_SOURCE 500 #include <unistd.h>
The reasons for this are historical. If you look at the Unix family tree you'll see two major branches: systems deriving from System V on the left, and systems in the BSD line in the center. Linux, on the right, now attempts to provide support for both lines; sometimes, there are functions with the same name, but different signatures (for instance, setpgrp()). Then you have to tell it which one you want by defining certain pre-processor constants. In this assignment, you should not need to use any such functions. Use setpgid() instead of setpgrp(), and the only place where you would need to learn a process's process group is to identify the shell's own process group. Use getpgrp() for this purpose, as suggested in setpgid(2). There is no need to ever call getpgid() on a process pid other than your own (that is, the shell's).
When POSIX signal handling is used, the OS will pass along additional information to the signal handler in arguments #2 and #3. Argument #2 is of type siginfo_t. See sigaction(2). However, since traditional Unix signals are not queued, si_pid for a SIGCHLD will contain the pid of the first child that caused SIGCHLD to become pending. Therefore, it cannot be used to learn which children need to be reaped. You must use waitpid(-1, ...). Study sigchlddoesnotqueue.c for an example.
Keep in mind that such messages are printed only if a (working) shell detects that one of its child processes terminated with SIGSEGV. Your shell will be the child process of your login shell (typically bash); so if the segmentation fault occurs in your shell process, you'll likely see the error message printed by bash. However, if your shell crashes on the code path between when it forks a new process (about to exec() a command), then it would be your shell's responsibility to print this error message after waiting for the child. If it does not do that yet, you won't see anything. valgrind is of great help here: it follows both the parent and the child process and reports segmentation faults in either.
sa_sigaction_t is a convenience function pointer type that is declared in esh-sys-utils.h like so:
/* Signal handler prototype */ typedef void (*sa_sigaction_t)(int, siginfo_t *, void *);
Any void-returning function with a signature of (int, siginfo_t *, void *) can be substituted where a variable of type sa_sigaction_t is expected. This means you should declare your signal handling routine to have this signature, then you can pass it to esh_signal_sethandler which provides a convenient wrapper for the underlying sigaction(2) system call that installs this signal handler. I recommend using this function instead of the outdated signal(2) function, because it allows more fine-grained control over the signal processing semantics. For instance, you can specify that system calls that are interrupted by a signal be automatically restarted; you can specify that additional information be sent along to the signal handler routine (such as in the case of SIGSEGV, where a program faulted), and you can specify exactly which additional signals, if any, should be blocked during the execution of the signal handler.
In my opinion, there's rarely a reason to use signal(2) anymore; however others have pointed out that signal(2) is part of ISO C and may be preferred for portability in general. However, as shown in this table, SIGCHLD is part of POSIX only and not part of ISO C, given its Unix-specific semantics. This differs from such signals as SIGINT or SIGSEGV, which can be meaningfully implemented in any C language environment.
Please see also Q. 28
Recall that a process can dictate, for each signal with the exception of SIGKILL, what it wants to do if that signal is raised (handle, ignore, or take the default action which may be termination). In addition, for signals that it handles, it can temporarily delay them by blocking and later unblocking them.
If a process sets up any special handling for signals and/or blocks signals, any children of the process will inherit such actions and will inherit the parent's signal mask (a bitmask describing which signals are blocked, if any). So if your parent shell blocks SIGCHLD when calling fork(), SIGCHLD will be blocked in the child. (Note, however, that the child is its own separate process for the purpose of signal delivery - the reasons for why it may receive SIGCHLD are completely separate from why the parent process may receive SIGCHLD, or any of the other signals. On fork(), pending signals that haven't been delivered to the parent yet are not pending in the child so that the child doesn't accidentally receive signals not intended for it.)
If a process executes a new program, the signal handling is also largely unaffected, with one notable exception: any signal that is handled will be reset to the default action - this is clear since the signal handler itself belongs to the old program being replaced by the exec() call. However, exec() does not affect if signals are ignored or blocked - those signals will remain ignored or blocked.
Your shell must make sure that new programs are started with reasonable signal settings for each signal. That means that all signals have their default actions, and no signal is blocked. Consequently, if your shell blocked any signal (such as SIGCHLD) in the parent, the signal should be unblocked in the child before the exec() call. Similarly, if you decide to ignore any signals (such as SIGINT, perhaps), reinstall the default behavior before calling exec(). You may call esh_signal_sethandler(SIGINT, (sa_sigaction_t) SIG_DFL); to that end. Otherwise, the child process would ignore SIGINT, and you couldn't terminate it with ^C. You do not need to reset signals you handle, such as SIGCHLD, to their default action - exec() will do that for you.
For example, when I start a process in the background like so:
$ ./esh esh> sleep 30 & [1] 1818 esh> jHere is where the shell is stuck. I can't type anything anymore.
In addition, when I log onto the same machine on another terminal, I see my esh process consuming 100% of the CPU (using top).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1805 cs3214 20 0 107m 1212 948 R 71.8 0.1 26:01.35 esh
When I use strace to see what it's doing, I seeing constantly repeating output like this:
$ strace -p 1805
.... lots of output like this ....
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
When I attach gdb, I see that it's calling tcsetattr() from readline(), which is called in my main() function:
(gdb) attach 1805 (gdb) bt #0 0x0000003b0f0d8f78 in tcsetattr () from /lib64/libc.so.6 #1 0x0000003b1041979f in ?? () from /lib64/libreadline.so.6 #2 0x0000003b10419b60 in rl_deprep_terminal () from /lib64/libreadline.so.6 #3 0x0000003b10426ada in rl_cleanup_after_signal () from /lib64/libreadline.so.6 #4 0x0000003b10426f89 in ?? () from /lib64/libreadline.so.6 #5 0x0000003b1042919e in rl_getc () from /lib64/libreadline.so.6 #6 0x0000003b104296c0 in rl_read_key () from /lib64/libreadline.so.6 #7 0x0000003b1041550f in readline_internal_char () from /lib64/libreadline.so.6 #8 0x0000003b10415a55 in readline () from /lib64/libreadline.so.6 #9 0x0000000000406552 in main (ac=1, av=0x7fff7ef59348) at esh.c:495
Answer: You do not have ownership of the terminal when you call readline(). Readline() assumes it has ownership so it can safely call, for instance, tcsetattr() - which is a function that requires such ownership. If that's not the case, readline() will act in undefined ways (apparently blindly retrying over and over after SIGTTOU is sent.)
In the example shown above, the most likely reason is that you wrongly gave ownership of the terminal to the 'sleep 30 &' background job the user started. If the user starts a job in the background, that job must not be given ownership of the terminal.
In order to not have to separately handle the case where a foreground job (which must be given ownership of the terminal!) terminates or exits, it may be useful to simply also reacquire ownership before calling readline() (or even better, before printing anything, including the prompt).
Note that you still must avoid giving terminal ownership to the background job to avoid the race condition where the background job (in the child process, before the exec()), snaps away ownership from the shell that's already returned to the prompt.
This occurs when the process group to which you are trying to add the process no longer exists, for instance, because all the processes in it have already terminated. This can occur if you erroneously wait for the first child in a pipe to finish before forking the second child.
Waiting for the first child also breaks a pipe's built-in flow control mechanism. If the first child outputs more than the pipe's capacity, it will block forever since you never start the second child that would have otherwise drained the pipe and consumed the data produced by the first child. In current Linux, pipe capacity is only 64K. In addition, you're losing parallelism.
The Linux man page for setpgid() is incomplete in that it does not list the case where pgrp no longer exists. Even though EPERM seems unusual here - ESRCH would perhaps have been more clear to the user, POSIX 1003.1/2004 actually calls for EPERM here:
[EPERM]
The value of the pgid argument is valid but does not match the process ID
of the process indicated by the pid argument and there is no process with
a process group ID that matches the value of the pgid argument in the same
session as the calling process.