BEST·BOOKS
+ MENU
Books Authors Tags Endorsers Sources
← Back to Software Tools

AI Study Notebook AI-generated

Software Tools

Brian Kernighan and P. J. Plauger

Key points Not available
On this page

Software Tools — Chapter-by-Chapter Outline

Author: Brian W. Kernighan and P. J. Plauger First published: 1976 Edition covered: First edition, Addison-Wesley, 1976 (ISBN 9780201036695). A companion volume, Software Tools in Pascal (1981, ISBN 9780201103427), presents the same program sequence in Pascal; the core chapter structure is identical, but the language differs. This outline covers the original 1976 Ratfor edition.

Central thesis

Software Tools argues that the way to write reliable, maintainable, and composable programs is to build a collection of small tools, each doing one job well, and then assemble them into larger solutions by connecting their inputs and outputs. The authors demonstrate this philosophy not by describing it abstractly, but by actually constructing—in working, tested code—a sequence of programs of increasing sophistication: from character-counting utilities, through sorting and pattern matching, to a full text formatter, a macro processor, and finally the structured-language preprocessor (Ratfor) in which all the earlier programs were themselves written.

The book made two claims that were radical for 1976. First, that structured programming—replacing GOTO-based spaghetti with loops, conditionals, and subroutines—is not a style nicety but a practical engineering requirement for readable and correct code. Second, that portability is achievable: by writing programs that consume only a standard character stream and produce only a standard character stream, you decouple the tool from any particular machine or operating system. The combination of these two claims produces the central commitment: a programmer's primary investment should be in building and refining a toolkit, because tools, unlike bespoke programs, compound in value.

How do you write programs that are small, correct, composable, portable, and self-improving?

Chapter 1 — Getting Started

Central question

What does a "good" tool look like at its simplest, and how should the practice of building tools begin?

Main argument

The copy program as the minimal tool. The chapter opens with copyin and copyout — programs that simply relay characters from standard input to standard output unchanged. These trivial programs establish the template every subsequent tool will follow: read from a character stream, process, write to a character stream. The authors introduce a key primitive set (a small library of I/O functions named getc, putc, getline, putline, etc.) that hides operating-system specifics. Every program in the book is written against these primitives, not against any real OS's system calls, which is the first act of portability discipline.

Counting tools: charcount, linecount, wordcount. Three short programs—charcount, linecount, and wordcount—demonstrate that useful tools can be extremely brief. charcount accumulates a count of every character read and prints one number. linecount increments a counter each time a newline arrives. wordcount requires a slightly subtler state machine: it must track whether the previous character was whitespace to detect word boundaries. The authors are deliberate about the definition of "word" — a maximal sequence of non-whitespace characters — showing that even a simple tool demands a precise specification.

The detab program and tab-stop discipline. detab converts tab characters to the appropriate number of spaces, given that tab stops fall every N columns (the chapter uses every 4). This requires maintaining a column counter as characters flow through. The program introduces the idea that a filter may carry internal state from character to character, and that the state must be correctly maintained across all input variations (tabs at the start of a line, multiple consecutive tabs, tabs after other content).

Documentation and testing as first-class concerns. After each program, the authors provide brief but structured documentation — the program's purpose, its input format, its output format, and any assumptions it makes. They also insist that every program be tested against edge cases before moving on. This methodological discipline — write the spec, write the code, test immediately — is as much a lesson as the code itself.

Key ideas

  • Every program in the book is written against a small set of I/O primitives, not OS system calls, so that it runs unchanged on any system that implements those primitives.
  • A character stream (standard input → process → standard output) is the universal interface: any tool that obeys this contract can be combined with any other tool.
  • Even trivial programs benefit from an explicit specification and test — precision about what "word" means, for instance, prevents later confusion.
  • Internal state (e.g., the column counter in detab) must be accounted for carefully; it is the source of most bugs in filter programs.
  • Tools are not demonstrations: every program in the book was extracted from real software production use.
  • Structured programming (no GOTOs; use if/else, while, for) makes even short programs significantly more readable and debuggable.

Key takeaway

Even the simplest useful tools require precise specifications, careful state management, and portable I/O discipline — habits established here that every subsequent chapter depends on.

Chapter 2 — Filters

Central question

How do you build programs that transform a character stream — transliterating, compressing, or rearranging its content — without knowing anything about where the stream comes from or where it goes?

Main argument

The filter as the fundamental Unix abstraction. This chapter formalizes the concept of a filter: a program with no knowledge of files, terminal types, or callers, that reads characters from standard input, transforms them, and writes to standard output. Filters compose freely because their interface is uniform. The authors use the metaphor of LEGO blocks: any filter snaps onto any other.

entab: the inverse of detab. Where detab expands tabs to spaces, entab compresses runs of spaces back into tabs, producing a more compact representation. Building entab requires the same column-counting state machine as detab but driven in reverse: the filter accumulates spaces, waits to see whether they align with a tab stop, and emits a tab character only when the run ends exactly at a tab boundary. The symmetry with detab is a lesson in invertibility: good tools often come in pairs.

compress and expand: run-length encoding. compress encodes runs of repeated characters using a simple notation: a run of N identical characters c is written as -Nc. expand decodes this back to the original. The pair demonstrates a general principle about lossless encoding: the representation must be self-delimiting (the - prefix and the count field must be parseable unambiguously) and the encoding must handle all special cases including the encoding character itself appearing in the input.

echo: command-line to output. echo copies its command-line arguments to standard output as a single line, with spaces between arguments. Although tiny, echo introduces the concept that tools can accept parameters at invocation time rather than only reading from standard input. This foreshadows the more powerful parameter mechanisms of later chapters.

translit: character transliteration. translit is the chapter's most complex tool. It accepts two arguments — a source character set and a destination character set — and maps every character in the source set to the corresponding character in the destination set. The argument language supports ranges (a-z), escape sequences (\n, \t), and negation (with ^), making it a small domain-specific language for character mapping. The authors build a parser for this mini-language, which foreshadows the more complete parsing work in the macro and Ratfor chapters. translit implements what later became the Unix tr utility.

overstrike: printer overprinting. overstrike processes backspace sequences embedded in its input and reorganizes the text into separate passes so that characters meant to overstrike each other land on the same position. This tool is specific to old line-printer hardware but illustrates an important generalization: a filter can do multi-pass processing by buffering a line internally, even though its external interface is still a single-pass character stream.

Key ideas

  • Filters have no idea of their environment: the same filter program can read from a file, a pipe, a terminal, or a network socket without modification.
  • Invertibility (compress/expand, detab/entab) is a design goal worth pursuing: if a transformation can be reversed, it should be.
  • A mini-language for specifying parameters (as in translit's range syntax) is often more powerful than a long option list and foreshadows proper language processing.
  • Run-length encoding illustrates that even simple compression schemes require careful attention to edge cases and self-delimiting representations.
  • Internal line buffering is compatible with the filter model: a filter may accumulate an entire line before writing anything, as long as eventual output is correct.
  • Composing filters via pipes produces compound tools without writing a single new program.

Key takeaway

Filters are the atomic units of Unix-style software; building them correctly means imposing a single interface discipline that makes composition free.

Chapter 3 — Files

Central question

How do you build tools that operate on named files rather than a single undifferentiated character stream, while preserving portability and simplicity?

Main argument

Moving beyond stdin/stdout. The first two chapters work entirely with standard input and output. Chapter 3 confronts the reality that most useful programs need to read from or write to named files, open multiple files, concatenate streams, and compare contents. The challenge is to do this without tying the tools to a specific operating system's file API. The authors extend their primitive library with file-opening and file-closing operations that map onto the host OS but present a uniform interface to the tool.

print and the include directive. print is a program that reads one or more named files and copies them to standard output. It introduces a feature that grows in importance: an include directive — a line beginning with #include "filename" — causes the named file's contents to be spliced in at that point. This is the earliest appearance in the book of the idea that a tool can preprocess its own input, a thread that culminates in the macro processor and Ratfor translator of the later chapters.

concat: file concatenation. concat (the ancestor of Unix cat) appends named files end-to-end and writes the result to standard output. The interesting design question is error handling: what should concat do if a named file does not exist? The authors argue for clear, immediate error messages to standard error (never swallowing errors silently) and continuation with the remaining files. This is an early articulation of the robustness principle.

compare: file differencing. compare reads two files in parallel and reports the first line where they differ. This is a simpler precursor to Unix diff. Its implementation requires opening two files simultaneously and keeping their line-read positions synchronized — more complex than anything in the filter chapters. The tool introduces the notion of exit status: a program that finds a difference returns a non-zero exit code, making it useful in scripts.

Building the file-primitive layer. The chapter closes by consolidating the file-access primitives into a clean library. The authors discuss the tradeoff between exposing raw OS file descriptors and hiding them behind opaque handles, arguing for the latter on portability grounds. This layered design — primitives hiding OS details, tools built on primitives — is a recurring architectural pattern throughout the book.

Key ideas

  • Named-file access requires extending the primitive layer while keeping tools themselves OS-agnostic.
  • The include mechanism shows that a tool can be its own preprocessor, a pattern that scales all the way to Ratfor.
  • Error reporting must go to standard error, not standard output, so that error messages do not corrupt a tool's output when it is used in a pipeline.
  • Exit status is a tool's contract with its caller: a non-zero status signals failure and allows shell scripts to branch on success or failure.
  • Handling multiple files cleanly requires thinking about resource limits (how many files can be open simultaneously?) and explicit cleanup.
  • Even simple tools like compare raise design questions about what counts as "meaningful" output and what information the caller needs.

Key takeaway

Moving from streams to files requires only a thin new primitive layer; the tool-building discipline of composability and error transparency applies equally to file-oriented programs.

Chapter 4 — Sorting

Central question

How do you build a general-purpose sorting tool that handles arbitrary text lines, supports multiple sort keys, and is fast enough for real use?

Main argument

Why sorting is a core tool. Sorting appears trivially in many programs — counting word frequencies, generating indexes, preparing merge input — but a correct, general sort is surprisingly non-trivial. The authors argue that rather than re-implementing sorting inside each program that needs it, you should build one excellent standalone sort tool and invoke it as a pipeline stage.

Internal sort: Shell sort. The chapter begins by implementing an in-memory sort using the Shell sort algorithm (after Donald Shell, 1959). Shell sort is a practical choice: it is significantly faster than insertion sort for realistic input sizes, requires no extra memory (it sorts in place), and is compact enough to present completely. The authors show the nested-loop structure and explain the diminishing-gap sequence that makes Shell sort work. The trade-off versus more complex algorithms (quicksort, merge sort) is discussed: Shell sort's simplicity and in-place operation make it preferable when memory is scarce or code size matters.

From internal sort to pipeline sort. An in-memory sort works only if the entire input fits in available memory. The authors extend the tool to an external sort that writes sorted runs to temporary files and then merges them — the classic sort/merge pattern. This requires implementing a merge that reads from multiple sorted streams simultaneously and selects the smallest element at each step.

Key fields and comparison functions. A general sort must support sorting by a specified field (column) rather than the whole line, numeric sorting as well as lexicographic sorting, and sort order reversal. The authors design a comparison function abstraction: the sort engine calls a user-replaceable comparison function, so that sorting by different keys requires only swapping the comparison function, not rewriting the engine. This early use of a function pointer for customization prefigures today's qsort-style API.

The unique flag. Sorting is often combined with deduplication: output only unique lines (the Unix sort -u behavior). The authors implement this as an option on the sort tool, showing how a simple post-processing step — suppress a line if it equals the previous output line — integrates cleanly into the sort's output phase.

Key ideas

  • Building one correct, fast, general sort tool is worth far more than embedding ad-hoc sorting in each program that needs it.
  • Shell sort is a practical workhorse: simple, in-place, fast enough for many real-world inputs, and small enough to present completely.
  • External sort (sort runs, then merge) handles inputs larger than available memory; the merge phase requires reading multiple files simultaneously.
  • A comparison-function abstraction decouples the sort engine from the sorting criterion, enabling reuse across different key types without code duplication.
  • Exit status and error handling matter for sort: attempting to sort an unreadable file should report an error and exit non-zero.
  • Sorting and deduplication are so frequently combined that offering a unique option in the sort tool itself is better interface design than requiring a separate uniq pass.

Key takeaway

A sorting tool built around a clean comparison-function abstraction and an external merge strategy covers essentially all real-world sorting needs from a single, tested implementation.

Chapter 5 — Text Patterns

Central question

How do you build a tool that searches text for lines matching a pattern, where "pattern" means something richer than a literal string?

Main argument

The find program (precursor to grep). The chapter's central tool is find — a program that reads lines from standard input and prints those that match a given pattern. This is the direct ancestor of the Unix grep utility. The challenge is defining what "pattern" means in a way that is powerful enough to be useful but simple enough to implement in a few hundred lines.

Regular expressions and metacharacters. The authors introduce a simplified regular expression language. The metacharacters are:

  • . (dot): matches any single character
  • *: matches zero or more occurrences of the preceding element
  • ^: anchors the match to the start of a line
  • $: anchors the match to the end of a line
  • [...]: matches any character in the specified set
  • [^...]: matches any character not in the set

The chapter demonstrates how each metacharacter extends the class of patterns expressible, and why each is a natural generalization of the literal-match case.

The pattern-matching engine. The engine is structured as a recursive descent over the pattern string. A key design decision is to represent the compiled pattern as an array of (type, value) pairs — a small bytecode — rather than directly interpreting the pattern string character by character. This representation makes the matching loop clean and extensible: adding a new metacharacter requires only a new type code and a new case in the match loop.

grep-like variants: find lines not matching, find line numbers. The authors show how minor variations on find — inverted match (print lines that do not match), line-number output — can be added as flags without restructuring the core. This illustrates that a well-factored tool accommodates natural variations cheaply.

Pattern matching as a reusable component. Because the matching engine is encapsulated behind a clean interface (compile(pattern) / match(compiled, line)), it can be reused in the editor of Chapter 6 and the macro processor of Chapter 8 without modification. The authors are explicit about this reuse: the pattern-matching code is written once and linked into multiple tools.

Key ideas

  • Regular expressions are the natural language for specifying text patterns: they extend literal search to a useful class of structural patterns while remaining compact enough to type on a command line.
  • Compiling the pattern to an intermediate representation (type/value pairs) separates the parsing of the pattern from the matching of text, making both cleaner.
  • Recursive descent over the compiled pattern handles * (repetition) by trying the greedy match first and backtracking if it fails.
  • The ^ and $ anchors are the simplest mechanism for context-sensitive matching; they require only that the engine know whether it is at the start or end of a line.
  • Writing the pattern engine behind a stable interface means the same code serves in find, in the editor, and in the macro processor.
  • A tool with a clean exit status (0 if any match found, non-zero otherwise) integrates naturally into shell scripts.

Key takeaway

A simple regular expression engine — a dozen metacharacters, a compiled intermediate form, and a recursive match loop — is enough to power grep, an interactive editor search command, and macro pattern matching from a single body of code.

Chapter 6 — Editing

Central question

How do you build a line-oriented text editor — a program that allows a user to navigate a file, find text by pattern, and make changes — from the tools and techniques developed so far?

Main argument

A full interactive tool, not just a filter. The previous chapters built filters and batch-processing tools. An editor is the book's first interactive program: it reads commands from the user rather than data, and it maintains persistent state (the contents of the buffer being edited) across many command-response cycles. This changes the architecture significantly.

The ed model. The authors implement an editor closely resembling Unix ed. The editor works on a line-addressed buffer: an array of lines in memory. Commands take address arguments (line numbers, ranges, or regex addresses) and operate on the addressed lines. The core commands are:

  • a (append): insert lines after the addressed line
  • d (delete): remove addressed lines from the buffer
  • p (print): display addressed lines
  • s/pattern/replacement/ (substitute): replace the first (or all) occurrences of a pattern in addressed lines with a replacement string
  • r/w (read/write): read a file into the buffer or write the buffer to a file
  • q (quit): exit

Address parsing and line ranges. The editor must parse address specifications before dispatching commands. Addresses can be absolute line numbers (5), relative offsets (+3, -1), pattern searches (/regex/), or symbolic addresses (. for current line, $ for last line). Ranges are written as start,end. The address parser is one of the more complex pieces of code in the chapter; the authors present it as a recursive descent parser that returns a pair of line numbers.

The substitute command and the pattern engine. The s command reuses the pattern-matching engine from Chapter 5. The replacement string supports & (insert the matched text) and back-references, demonstrating that a well-factored component can be extended without modification. The authors discuss the tricky edge case of substitutions that produce the empty string and substitutions of patterns that match the empty string.

Global commands. The g/pattern/command form applies a command to every line matching a pattern — the source of the g in grep (global regular expression print). This generalization turns the editor into a batch-processing tool as well as an interactive one; a sequence of g commands can serve as a structured text transformation script.

Design discipline for interactive programs. The authors observe that interactive programs require more careful attention to error messages, default behaviors, and undo (the editor provides no undo in this implementation, which they acknowledge as a limitation). The command loop — read a command, parse it, execute it, loop — is a simple pattern, but getting error recovery right (do not exit on bad input; print a helpful message and continue) requires deliberate design.

Key ideas

  • An interactive tool maintains persistent buffer state; its main loop reads commands rather than data, inverting the filter pattern.
  • Line addressing (by number, relative offset, regex, or symbolic name) is a uniform mechanism that makes all commands compositional: any command can be applied to any addressable range.
  • Reusing the Chapter 5 pattern engine inside the editor illustrates the payoff of the tool-building philosophy: the matching code cost was paid once and is now free.
  • The g/pattern/cmd mechanism generalizes from interactive editing to scripted batch transformation — a single command form serves both use cases.
  • Error handling in interactive programs must be non-fatal: a bad command should produce a message and return to the prompt, not terminate the program.
  • The substitute command's replacement language (with & for the matched text) is a small but powerful DSL embedded within the larger command language.

Key takeaway

An editor is a persistent-buffer program driven by a small command language; building it from the pattern-matching engine and file-primitive library already developed demonstrates that well-factored components compose into large, useful programs with surprisingly little new code.

Chapter 7 — Formatting

Central question

How do you build a text formatter — a program that takes marked-up text and produces beautifully laid-out output with line filling, justification, headers, footers, and page breaks?

Main argument

The gap between text and typeset output. Plain text files do not automatically produce well-formatted documents: lines may be too long or too short, paragraphs are not filled to the page width, there are no headers or page numbers. A formatter reads a source file annotated with formatting commands and produces output that fills these gaps. The authors' formatter is a simplified version of Unix roff (the earliest Unix text formatter, written by M. D. McIlroy), which is itself a descendant of the MIT CTSS runoff program.

Command syntax. A formatting command is a line beginning with a period, followed by a two-letter command name and optional arguments. Examples:

  • .pl N — set page length to N lines
  • .ll N — set line length to N characters
  • .sp N — insert N blank lines
  • .bp — begin a new page
  • .ce N — center the next N lines
  • .in N — indent N characters from the left margin
  • .ju / .nj — enable / disable right-justification
  • .he "text" — set page header
  • .fo "text" — set page footer

All other lines are text to be formatted.

The fill-and-justify engine. The core of the formatter is the fill-and-justify engine: it accumulates words from the input until the current output line would overflow, then either breaks the line (no-justify mode) or stretches it by distributing extra spaces between words (justify mode). The justify step requires careful arithmetic: compute the number of extra spaces needed, distribute them as evenly as possible left-to-right (placing any remainder spaces at the beginning of the line to avoid a ragged right edge).

Page management. The formatter must count output lines and insert page breaks with headers and footers at the appropriate positions. This requires a small page-state machine: track how many lines have been output on the current page, trigger a new page when the count reaches the page length minus the footer space, and automatically emit headers on each new page. The page-number substitution in headers and footers (replacing % with the current page number) is a simple but practical feature.

Centering and indenting. .ce centers lines by computing the padding needed to place the text in the middle of the line width. .in shifts the left margin for the indented region. Both require adjusting the effective line length used by the fill engine.

Limitations the book acknowledges. The formatter handles only a fixed character width and does not support proportional fonts, multiple columns, or footnotes. The authors explicitly note these limitations and point toward the later macro processor (Chapter 8) as the mechanism for extending the formatter's command set without modifying the formatter itself.

Key ideas

  • A formatter separates content (text lines) from presentation (dot commands), allowing the same source to be reformatted by changing command parameters.
  • The fill-and-justify engine is the core of all text formatting; its output quality depends on the accuracy of the space-distribution arithmetic.
  • Page management requires a small state machine tracking line count, page count, header/footer content, and margins.
  • Even-distribution of extra spaces in justification is a simple algorithm but has a non-trivial edge case: the last line of a paragraph should never be justified.
  • The formatter reads its own output format: dot commands must be distinguishable from text, which requires a convention (period in column 1) that the authors chose for readability.
  • Extending the formatter's command set via a macro processor (the next chapter) is a cleaner design than hard-coding every possible formatting operation.

Key takeaway

A text formatter is a fill-and-justify engine driving a page-state machine; building it correctly requires careful attention to justification arithmetic, page boundary detection, and a clean command syntax.

Chapter 8 — Macro Processing

Central question

How do you build a general-purpose macro processor — a program that expands symbolic names into arbitrary text, supports arguments, and can be used to extend any other tool?

Main argument

What a macro processor does. A macro processor reads text and replaces occurrences of defined names with their expansion strings. The simplest form is a symbolic constant: define(MAXLINE, 1000) causes every subsequent occurrence of MAXLINE to be replaced by 1000. The more powerful form supports parameterized macros: define(square, ($1 * $1)) turns square(x) into (x * x). The authors argue that a macro processor is a multiplier: by layering a macro processor over the formatter of Chapter 7, you can add new formatting commands (like .tl for title) without touching the formatter's source.

The define mechanism. Macros are stored in a symbol table as (name, body) pairs. When the input scanner encounters a name followed by (, it recognizes a macro call, collects the arguments (handling nested parentheses by counting depth), substitutes $1, $2, etc. in the body with the actual argument texts, and re-scans the expanded text. Re-scanning means that a macro body may itself contain macro calls, which are expanded in turn — a powerful recursive capability.

Built-in operations. Beyond define, the macro processor provides built-in operations:

  • ifdef(name, text) — emit text only if name is defined
  • ifelse(a, b, yes, no) — conditional expansion
  • incr(name) — increment a numeric macro
  • eval(expression) — evaluate an arithmetic expression and return the result as text

These built-ins turn the macro processor into a rudimentary programming language.

Design challenges: quoting and re-scanning. The most subtle aspect of macro processing is quoting. Without a quoting mechanism, it is impossible to pass a string that looks like a macro call without expanding it, or to include a comma in an argument. The authors implement a quoting convention (text between backtick and single-quote is not expanded), explaining the tradeoffs between different quoting disciplines. Re-scanning — expanding the result of an expansion — is powerful but can cause infinite loops if a macro expands to itself; the authors show how to detect and break such cycles.

The macro processor as a meta-tool. The chapter's conclusion argues that the macro processor is qualitatively different from the earlier tools: it does not process text in a domain (files, patterns, formatting) but extends the language of other tools. By sitting in a pipeline before the formatter, it can add arbitrarily many new commands. By sitting before the Ratfor translator (Chapter 9), it can add new syntax. This meta-level capability is the highest expression of the tool-building philosophy.

Key ideas

  • A macro processor implements name-to-text substitution with argument replacement, turning symbolic names into an extension mechanism for any downstream tool.
  • Re-scanning the expansion allows macros to call other macros, enabling composable abbreviations but requiring cycle detection.
  • Built-in operations (ifdef, ifelse, eval) turn the macro language into a programmable meta-language, capable of conditional and arithmetic computation at expansion time.
  • Quoting is essential for any macro processor: without it, you cannot pass a macro call as an argument or include special characters in text.
  • The macro processor's architecture — a scanner that recognizes calls, an argument collector that respects nesting depth, a substituter, and a re-scanner — is a clean four-stage pipeline.
  • The macro processor described here directly inspired Dennis Ritchie's m3 and, subsequently, Kernighan and Ritchie's m4 macro processor, which became a Unix standard.

Key takeaway

A macro processor is a meta-tool: rather than processing content directly, it extends the command language of downstream tools, making the whole toolkit extensible without modifying individual tool source code.

Chapter 9 — A Ratfor-Fortran Translator

Central question

How do you build the very preprocessor in which all the book's programs are written — a translator that adds structured control flow to Fortran 66 while remaining entirely expressible in Fortran 66 itself?

Main argument

The problem with Fortran 66. All programs in the book are written in Ratfor — Rational Fortran — a structured language that looks like C but compiles to standard Fortran 66. Fortran 66 has no if/else, no while, no for, and no block structure: control flow is expressed entirely through GOTO statements and statement labels. This produces code that is notoriously difficult to read, modify, and debug. The solution is a preprocessor that translates the structured forms into the equivalent GOTO-based code, allowing programmers to write in a clean language without sacrificing compatibility with existing Fortran compilers.

Ratfor's structured control constructs. The Ratfor language adds:

  • if (cond) stmt and if (cond) stmt else stmt — structured conditionals
  • while (cond) stmt — condition-tested loop
  • for (init; cond; incr) stmt — C-style counted loop
  • do ... until (cond) — post-test loop
  • break and next — exit or restart the innermost loop
  • { stmts } — statement grouping into blocks

Each construct is translated into a labeled GOTO sequence. For example, while (a > b) { ... } becomes:

label1 CONTINUE
IF(.NOT.(A.GT.B)) GOTO label2
...
GOTO label1
label2 CONTINUE

The translator generates fresh label numbers to avoid collisions.

The translator's architecture. The Ratfor translator is a recursive descent parser over lines of input. It reads one logical statement at a time (handling line continuation), recognizes the leading keyword (if, while, for, etc.), and emits the appropriate Fortran. Non-Ratfor lines (ordinary Fortran statements, comments, data declarations) pass through unchanged. The translator is therefore a classic source-to-source compiler with a very simple grammar.

Self-hosting: the most important bootstrap. The Ratfor translator is itself written in Ratfor. This means the translator processes its own source code. The first time it was compiled, it must have been bootstrapped through a Fortran compiler by hand or via an earlier, cruder translator. The existence of this bootstrapping pathway — and the fact that the authors make it explicit — is a profound demonstration of the tool-building philosophy's ultimate destination: tools that make other tools, including themselves. The book's ninth and final tool is the language in which the first eight tools were written.

Relationship to the macro processor. Ratfor does not include a macro facility; that is left to the macro processor of Chapter 8. The intended usage pattern is to run the input through the macro processor first and then through Ratfor, with each tool doing one job: the macro processor handles name substitution and conditional compilation; Ratfor handles structured control flow.

Legacy. Ratfor influenced the design of the C preprocessor (which inherited its #define and #include model from the combination of Ratfor and the macro processor). It also directly influenced the Fortran 77 standard, which added structured control flow constructs closely resembling those Ratfor had shown were both implementable and beneficial.

Key ideas

  • Ratfor adds if/else, while, for, break, next, and block structure to Fortran 66 via a preprocessor that translates to labeled GOTOs.
  • The translator is a recursive descent parser, the simplest architecture for a language with well-factored, unambiguous grammar.
  • Non-Ratfor lines pass through unchanged; the translator need only recognize the structured constructs, leaving everything else alone.
  • Self-hosting — writing the translator in the language it translates — is the book's ultimate demonstration that tools compound in value.
  • The separation of concerns between the macro processor (name substitution) and the Ratfor translator (control flow) allows each to be simpler than a combined tool would be.
  • Ratfor's influence on Fortran 77 and on the C preprocessor shows that a tool designed for one immediate context can reshape a language ecosystem.

Key takeaway

The Ratfor translator — a source-to-source compiler written in Ratfor — closes the book's argument: by building tools that build tools, programmers can elevate their language, their environment, and ultimately their own productivity.

The book's overall argument

  1. Chapter 1 (Getting Started) — Establishes the foundational discipline: write tools against portable I/O primitives, give each tool a precise specification, test immediately, and use structured code without GOTOs; the five programs in this chapter are the atoms from which all later tools are assembled.
  2. Chapter 2 (Filters) — Formalizes the filter abstraction — uniform stdin-to-stdout programs that compose freely via pipes — and shows that this single interface discipline is sufficient to build six genuinely useful transformation tools including the ancestor of Unix tr.
  3. Chapter 3 (Files) — Extends the model to named files by adding a thin file-access primitive layer, introduces the include mechanism as the first example of a tool preprocessing its own input, and insists on error reporting to stderr and non-zero exit codes for failure.
  4. Chapter 4 (Sorting) — Demonstrates that one general, correct sorting tool with a comparison-function abstraction and an external sort/merge strategy serves all sorting needs without ad-hoc re-implementation, and introduces the unique flag as a natural pipeline composition.
  5. Chapter 5 (Text Patterns) — Builds a regular expression engine with a compiled intermediate representation and reusable interface, showing that a single body of pattern-matching code can power standalone search, in-editor search, and macro-pattern matching.
  6. Chapter 6 (Editing) — Shows how tools and components from prior chapters (the file primitives, the pattern engine) assemble into the book's first interactive program: a line-addressed editor with a small command language and a global-command mechanism that generalizes editing to scripted batch transformation.
  7. Chapter 7 (Formatting) — Builds a complete text formatter by combining a fill-and-justify engine with a page-state machine, demonstrating that even a complex, multi-mode program decomposes cleanly when each subsystem has a single responsibility; explicitly acknowledges that the macro processor will extend the formatter's command set without modifying the formatter itself.
  8. Chapter 8 (Macro Processing) — Introduces the meta-tool: a macro processor that extends the command language of any downstream program, demonstrating recursive expansion, built-in conditional and arithmetic operations, and the power of re-scanning; directly precursors the Unix m4 macro processor.
  9. Chapter 9 (A Ratfor-Fortran Translator) — Closes the argument by building the very language in which the book was written, achieving self-hosting and demonstrating the ultimate compounding value of tool-building: tools that make tools, including themselves.

Common misunderstandings

Misunderstanding: This is a book about Fortran programming.

The programs are written in Ratfor, not standard Fortran. Ratfor is a structured language that happens to compile to Fortran; the book could equally have been subtitled "how to do structured programming in any language." The Fortran connection is a historical artifact of the available portable language in 1976, not the point of the book. The principles apply in any language.

Misunderstanding: The tools described are outdated because Unix already provides grep, sort, and ed.

Many of the tools the book builds are now standard Unix commands precisely because the Unix designers absorbed this book's philosophy. The value of the book is not the tools themselves but the methodology of building them: starting from primitives, building incrementally, testing at each step, reusing components. That methodology is as relevant today as in 1976.

Misunderstanding: The book teaches software tools in the sense of IDEs, build systems, or debugging tools.

The title uses "tools" in the narrower Unix sense: filter programs and text-processing utilities that compose via pipes. The book does not cover debuggers, profilers, build systems, or version control in any depth.

Misunderstanding: The book advocates for small programs as an end in themselves.

The book advocates for small, focused programs because they are easier to test, easier to combine, and easier to replace than large, monolithic programs. Smallness is a means to correctness and composability, not a virtue for its own sake. The book explicitly acknowledges that the formatter and editor are sizeable programs.

Misunderstanding: Portability means the programs will run anywhere without modification.

Portability, as the book defines it, means that the programs depend only on a small primitive layer that can be re-implemented for each target environment. The programs themselves need not change; only the primitive layer changes. This is a disciplined, layered approach to portability, not magic platform independence.

Central paradox / key insight

The book's deepest insight is a form of bootstrapping paradox:

To write better programs, you need better tools. But better tools are themselves programs. Therefore, the act of writing programs is also the act of improving the tools you write programs with.

The final chapter makes this explicit: Ratfor, the language in which every program in the book is written, is itself one of those programs. The book's ninth tool is the language used to write the first eight. This is not a circular argument but a virtuous spiral: each tool you build raises the level at which you can build the next one.

A corollary, less often noticed, is that this philosophy has an enemy: the one-off program. Every time a programmer writes a sorting routine inside an application rather than using a standalone sort tool, they spend debugging budget on something that has already been debugged. Every time they write a pattern-matching function rather than linking in the shared engine, they risk divergent behavior. The tool-building philosophy is also, implicitly, an argument against duplication of effort — and for the discipline of recognizing when you are solving a general problem versus a specific one.

Important concepts

Ratfor (Rational Fortran)

A structured programming language implemented as a preprocessor for Fortran 66. Ratfor adds if/else, while, for, break, next, and block-grouping { } to standard Fortran syntax. A Ratfor translator reads Ratfor source and emits equivalent Fortran 66 GOTO/label code. All programs in the book are written in Ratfor. First described by Kernighan in a 1975 paper in Software — Practice and Experience.

Filter

A program that reads from standard input, transforms the data, and writes to standard output, with no dependency on any named file, terminal type, or operating system. Filters compose freely via pipes because their interface is uniform. detab, entab, translit, and compress/expand are all filters.

Standard input / standard output / standard error

Three universal I/O channels available to every program. Standard input delivers data; standard output delivers results; standard error delivers diagnostic messages. Keeping error output on a separate channel ensures that errors do not corrupt the data stream when a tool is used in a pipeline.

Pipe / pipeline

A mechanism (provided by the shell) that connects the standard output of one program to the standard input of the next. Pipelines allow complex transformations to be expressed as a sequence of simple tools without creating intermediate files. The filter discipline is the prerequisite for pipelines to work correctly.

Primitive layer

A small, OS-specific library (implementing getc, putc, open, close, etc.) that hides the details of a particular operating system's file and I/O API. All tools in the book are written against the primitive layer, not against any OS directly. Porting the book's programs to a new system requires only re-implementing the primitive layer.

Regular expression

A compact notation for specifying a class of strings. The book's regular expression language includes . (any character), * (zero or more repetitions), ^ and $ (line anchors), and [...] (character classes). A compiled regular expression is an array of (type, value) pairs; the match function walks this array against an input string.

Exit status

An integer returned by a program to its caller (the shell or a parent program). By convention, zero means success and non-zero means failure. Tools with meaningful exit statuses — compare returns non-zero if files differ; find returns non-zero if no lines matched — integrate cleanly into shell scripts that branch on success or failure.

Line addressing (in the editor)

A scheme for identifying lines in an editor buffer using absolute line numbers, relative offsets, regex searches (/pattern/), the current-line marker (.), or the last-line marker ($). Line addresses can be combined into ranges. The addressing scheme is uniform across all editor commands, so any command can be applied to any addressable range.

Macro expansion

The process by which a macro processor replaces a symbolic name (and optionally its arguments) with a defined expansion string. Expansion may be recursive: the expanded text is re-scanned for further macro calls. The $1, $2, ... placeholders in a macro body are replaced with the actual argument texts at each call site.

Structured programming

A programming discipline that eliminates unstructured GOTO statements in favor of structured control constructs: if/else, while, for, break, and return. Structured programs are demonstrably easier to read, reason about, and test. The Ratfor preprocessor implements structured programming on top of a language (Fortran 66) that did not natively support it.

Self-hosting

A translator or compiler that is written in the language it translates or compiles. The Ratfor translator is written in Ratfor. The first bootstrap required a hand-translated or otherwise obtained Fortran version; thereafter, the Ratfor translator can compile itself. Self-hosting is the ultimate demonstration of a tool's maturity: the tool is good enough to build itself.

Primary book and edition information

Background and overview

Ratfor

The tools in modern context

The macro processor heritage

Additional chapter summaries and study resources

These are secondary resources and should be used alongside, rather than instead of, the original book.