Work: Building Stuff

By the time you read this, tulips will be blooming while the last snow melts, students at the university will have shed their down vests in favor of shorts, the Boulder Creek bicycle path will once again be impassable at lunch: Spring will have arrived in the Rockies. And with the arrival of spring, a young consultant's fancy turns to... a new way to build programs.

We'd promised two months ago to finish our exploration of how to build documents that can be formatted by both Web browsers and traditional paper formatters. In the verbal tradition of Alan Winston, system manager at Stanford's Synchrotron Radiation Laboratory, we've gotten disracted, and said ``push!'' in the middle of a paragraph. In this case, the thing in our new stack space was the atchange utility we wrote for Tom ``pop'', we wanted to pass on some comments from Tom, now that he's been using atchange for a while, and to finish some thoughts that atchange raised around the water cooler. Besides, we've interrupted our flow of thought in every series we've written for this magazine, so now it's a tradition.

Fan mail from some flounder.
We had a nice note from Tom after we sent off last month's article commenting on our text. Tom rejected our suggestion that he switch from the C shell to the Korn shell because he's already lost enough research time by being locked out of his government lab. Tom also wondered why we were twitting him about continuing to program in Pascal: after all, he'd rather keep up with his real research than rewrite an existing 200k line code base. (JLC sympathizes: he's got a vital utility program that's been used constantly since 1986, but he doesn't have the week to convert it from Pascal to C++, which is what it should be written in.) Tom also suggested that we add more comments in our Perl programs to help folks like him who are still learning the language. We'll try.

Tom pointed out an interesting feature of atchange: if you set atchange to watch a directory, you can trigger an action when any file in the directory changes. For example, if you're working on multiple utilities with multiple interacting sources in a single directory, atchange can fire off the make command to re-build all the pieces in the directory.

Children of make.
Speaking of make, the old UNIX standby has been a mainstay of development work for nearly two decades. It's a useful tool: before its invention by Mike Lesk you had to keep track of what needed to be recompiled every time you edited a source file. Our amp;pre- work around was to write scripts that just rebuilt everything, so we wouldn't screw up. It was not only wasteful of programmer time, but also profligate of CPU power (remember, this was back in the days of 64k main memory, when the bytes were still sometimes core memory!).

(Push! JLC had a bottle of real memory cores on his desk at Interactive Systems Corporation for a number of years. Finally, one day someone asked what they were, which lead to the exercise of walking around the office asking 50 or so technical people -- folks who'd been programming for most of their professional lives, if not most of their chronological lives -- what these little metal donuts were. Two people knew: Ted Dolatta and Marv Rubenstein, both of whom had written code for machines with tubes, a distinction neither of us can claim. Pop!)

Anyway, make changed the way we built programs. Now, every directory of source has a makefile in it. In fact, because the makefile is not really well suited to interactions with source management tools, the System V.3 source tree, as delivered by AT&T, contains a small directory for each utility -- awk, cat, ls, and so on -- most of which contain a makefile and a single source file. We've noted the problem ourselves: we have a directory of little utilities, which in the normal course of events contains a makefile and an RCS directory. We check out a utility to work on, and end up typing make foo a lot, because the default target in the makefile is to rebuild everything. Worse, we can't rebuild everything, because most of the sources are safely locked up in their RCS archives.

Well, in one of those cool ideas in the ``now that you've told me, it's just bloody obvious'' category, back in 1989, David MacKenzie at Rockefeller University published a program in comp.sources.misc (volume 6, issue 9, at ftp://ftp.uu.net/usenet/comp.sources.misc/volume6/xc.Z) that solved the problem. His program, the idea for which says he stole from a similar program at St Olaf College, simply assumes that the command to build a simple, one module utility is buried in header comments of the program source itself. For example, an ls -CF -R of our small sources directory looks like this:

Makefile        RCS/            SHAR/           jfold.c         volume.c

./RCS:
Makefile,v         date-diff.c,v      pr.c,v             trunc.c,v
a2lf,v*            duplex,v*          precol,v*          uncram,v*
a2ps,v*            grepmail,v*        psfixtoc,v*        unvt100.c,v-DEAD
backup,v*          headers.c,v        purge,v*           unvt100.l,v
backup-full,v*     jfold,v*           soelim.c,v         uucat.c,v
booksort,v*        jfold.c,v          sound,v*           uudecode.c,v
cram,v*            pagecnt.ps,v       tex,v*             uuencode.c,v
daily,v*           pclbreak.c,v       texit,v*           volume.c,v
daily-disk,v*      pclmend.c,v        tps,v*             wdiff,v*

./SHAR:
date-parse.jlc   date-parse.sh    rel.shar

static char id[] = "$Id: ...";
/* CMD: $(CC) $(CFLAGS) -o rdr rdr.c */

Notice that we're using something that looks a lot like the syntax out of our makefile.

How do we process this line? We'll break tradition and show you a shell script, rather than our usual Perl program, in the next section.

The build program.
We'll run through the actual code for the program, a paragraph at a time. We begin with the normal header declarations, including the magic cookie to declare this a shell script.

#! /bin/sh
# Basic build script
# $Id: bld.sh,v 1.2 1997/02/14 18:00:07 jeff Exp $
#  Expects a comment line in the .C file like:
#    /* CMD: $(CC) $(CFLAGS) -o foo foo.c

Next, we've saved the last source file name in $HOME/.bldlog, so we can use it as the default source file on this invocation. This means that after we've started, instead of typing make, and relying on the makefile to contain the appropriate default rules, we type bld and rely on our log of recent sources for a default.

We could just save the name of the last source file, but we're smarter than that: we tag each saved file name with the directory, so we can work on several files in parallel, each in a different directory. (This is probably overkill, but it's the strategy we use for our front-end shell function for vi: in that case, we really do want to have a separate target for each directory.)

##  first, find the current default
##    file for this directory:
where=`pwd`
[ -r $HOME/.bldlog ] || touch $HOME/.bldlog
file=`sed -n "s;^$where ;;p" $HOME/.bldlog`
#  use the file named on the command line,
#    if we have one...
[ -n "$1" ]  &&  [ -r "$1" ]  &&  file=$1  &&  shift

We also want to derive the target name from the source name. Between source and target, we now have the contents of the make variables $< and $@, which we can go ahead and use in our command lines.

#  we set the default target, too
target=`expr $file : '\(.*\)\.[^.]*'`

We can always have screwed up, and not have put a file name on the command line:

##  check for error
[ -z "$file" ]  &&  echo no file specified?!  &&  exit
[ ! -r "$file" ]  &&  \
 echo "can't read source file $file"  &&  exit

Now that we know what the source file is, we need to extract the build rule from the file:

##  now extract the command line from the
##   from the source file
sed -n "s;/\* *CMD: *\(.*\)\*/;\1;p" $file >/tmp/$$

And similarly, if there isn't a command line, we create a default line.

#     we may need the default command line,
#     if what we initially extracted was null:
[ ! -s /tmp/$$ ]  &&
 echo '${CC} ${CFLAGS} -o $@ $< ${LIBS}' >/tmp/$$

Now comes a slightly tricky bit. We're allowing ourselves to use environment variables in the command line. This is actually important: like in a regular makefile, we want to allow ourselves the ability to change the default compiler by changing a definition. This is particularly important when moving sources from machine to machine: we run the Sun SparcWorks compiler on our SunOS boxes, but the FSF Gnu compiler on our Solaris machines. We do this by having three sets of definitions, which are just shell environment variable sets: The first is hard- wiring for the default values of compiler and flags. The second is the a global per-user file of definitions. Both of those can be over-ridden by definitions in the local directory. (Why several levels? Suppose, for example, that I have some software in a pc subdirectory that I compile with a DOS cross- compiler, or I want to set LIBS=-lm in my math utility subdirectory.)

We're in effect creating a shell script to run our compile, so these definition files are pretty straight- forward. Our $HOME/.bldrc is just shell environment sets:

CFLAGS=-g
CC=gcc
LIBS=

##  we have defaults and then
##    global and local setup files
( echo CC=cc CFLAGS=-O LIBS=
  [ -r $HOME/.bldrc ]  &&  cat $HOME/.bldrc
  [ -r ./.bldrc ]  &&  cat ./bldrc

As a check, we turn on echo for the command line itself. This allows us to see the compile line that's begin run, just like our old, familiar make.

#   turn on expanded echo
  echo set -x

We have our build command squirrelled away in /tmp/$$, and we should be able to add it to the stream for our output pipe directly. No such luck: We've got two variables we can't use shell environment variables for, $@ and $<, so we do the substitution by directly editing the build command.

##  now make substitutions for the $@ and $<,
##  which we can't get from the shell.
##  also, translate $(CC) to ${CC} to avoid shell burp
  sed -e "s/\$@/$target/g" \
 -e "s/\$</$file/g" \
 -e "s/(\([^)]*\))/{\1}/g" /tmp/$$) |
sh

What happens to the output we've pushed down this pipe? It goes to a very simple destination: The Bourne shell, sh

We're almost done. We've got to save the name of the source we just built in our log for later, and do a little cleanup.

##  log the file for later use
( echo $where $file;
  egrep -v "^$where " $HOME/.bldlog) >/tmp/$$
mv /tmp/$$ $HOME/.bldlog

Voila: we're done.

Next month, we'll plan to get back to HTML. Until then, happy trails.

Further information about atchange is on the atchange page.

Schneider Lab.
origin: 1997 January 7
updated: 2012 Mar 08