All the broken
Too many broken
In our shellmounds
— Grayceon, Shellmounds
This is the material for a series of workshops I ran at my workplace on how to write a Unix shell.
The focus is slightly more on building an interactive shell than a scripting-oriented shell, only because I think this is more gratifying, even if it's less useful.
Be warned that some of the suggestions and discussion make opinionated choices without discussing equally-valid alternatives.
This is a work in progress and there may remain many infelicities. Patches Thoughtfully Considered. Feel free to report issues via Github.
The shell is at the heart of Unix. It's the glue that makes all the little Unix tools work together so well. Understanding it sheds light on many of Unix's important ideas, and writing our own is the best path to that understanding.
This workshop has three goals:
(some of this rationale is expanded on in my blog post, Building shells with a grain of salt)
I've tried to break this up into progressive stages that cover mostly orthogonal topics. Each stage contains a description of the facilities that will be discussed, a list of manpages to consult, and a set of tests. I've tried to also hint at some functionality that is fun but not necessary for the tests to pass.
In the root of this repository, there is a script called
you can run all the tests against your shell-in-progress by specifying
the path to your shell's executable, like this:
$ ./validate ../mysh/mysh
It should tell you what stage you need to implement next.
To run the tests, you will need
expect, which is usually in a
expect, and a C compiler. The way the tests are
implemented is less robust than one might hope, but should suffice for
our pedagogical goals.
The tests assume you will be implementing a vanilla Bourne-flavored
shell with some ksh influences. Feel free to experiment with
alternate syntax, but if so, you may need to adjust the tests. Except
where specifically noted,
ksh) should pass all the
tests, so you can "test the tests" that way. (Try
./validate /bin/bash; likewise,
cat should fail all the tests. Originally, I
/bin/sh, but I decided the material in stage 5 was
In which we discuss the basics of Unix processes, write the simplest possible shell, and then lay the foundations for the rest of the steps.
In which we add pipes and fd redirection to our shell.
In which we discuss signals and add support for ever-helpful chords
In which we discuss environments, variables, globbing, and other oft-misunderstood concepts of the shell.
In which we apply some polish to our shell to make it usable for interactive work.
In which I prompt you to go further.
I'll link to some of the shells that were written as a result of this workshop here shortly, including a couple I wrote to serve as examples of different approaches.
I wrote this workshop partially because I felt other tutorials don't go far enough, but all of these are worth reading, especially if you're having trouble with a stage they cover:
Although there is an elegant relationship between C and Unix which makes it attractive to write a shell in the former, to minimize frustration I suggest trying a higher-level language first. Ideally the language will have good support for:
Languages that provide a lot of their own infrastructure with regards signals or threads may be much more difficult to use.
You will probably run into issues related to the JVM, particularly with signals and forking, but as a starting point, you could do worse than loading libc with JNA.
There's also jtux.
There are a variety of approaches, but ljsyscall looks promising.
perlfunc(3perl); all the functions we want are at hand, usually
with the same name.
The implementation seems a little too heavy to do this conveniently, but see the Scheme section below for alternatives.
Process has most of what you need. You can use
Shellwords but you
decide if it's cheating or not.
Although we use few enough calls that you could just create bindings directly, either to libc with the FFI or by directly making syscalls, for just getting something working, the nix-rust library should provide all the necessary facilities.
Although core Tcl doesn't provide what's necessary,
does. For example, Tcl doesn't have a way to
exec, but expect
overlay to do this.