BDUMP, Automatic dump

What is BDUMP?

BDUMP is a utility to do automated filesystem dumps on unix systems. It uses the fstore utility to send the dump images to a central archiving repository. Statistics on how much each system has dumped are available. This is a table showing the number of gigabytes that were in use two weeks ago, followed by the number of gigabytes added to that over the following week and then the amount added on each of the last four days.

The bdump utility performs its functions in four phases:

  1. Schedule new dumps.
  2. Perform dumps.
  3. Prune dumps.
  4. Generate offsite copies.

1) Scheduling dumps.

The first phase is to schedule new dumps. The command to do this is bdump1. Bdump1 executes the "fstore -list" command and looks at the /etc/dumpdates command to determine the last dump state of each of its (the current system's) native filesystems. The dump schedule for each of the native filesystems is found in /usr/local/bdump/sched. A typical file for golding may appear as: 
  #
  #  Bdump schedule file.
  #
  #  Syntax is <delim><pat><delim>{<white><type>"="<hours>"/"<pct>}
  #
  #  Where delim  Is some delimiter that does not appear in the pattern.
  #        pat    Is a regular expression to match the filesystems for this
  #               dump schedule (see ed(1)).
  #        white  Is space or tab characters.
  #        type   Is a single letter, usually F for full and I for incremental.
  #        hours  Is the maximum time to elapse before doing another dump at
  #               at this level.  Specifying hours of -1 requests no dump.
  #        pct    Maximum fraction of the size of the previous dump at this
  #               level that the level n+1 dump can be before doing another 
  #               dump at this level.
  #
  #  The schedule on the first pattern matching the filesystem will be used.
  #  The type=hours/pct fields are repeated for each dump level desired.  The
  #  last entry's fraction must be 1.0.

  '^/tmp'         F=-1/1.0
  '^/crash'       F=-1/1.0
  '^/mnt'         F=-1/1.0
  '^/inst.images' F=2880/0.01     I=12/1.0
  '^/$'           F=1440/0.3      I=12/1.0
  '^/usr$'        F=1440/0.3      I=12/1.0
  '^/var$'        F=1440/0.3      I=12/1.0
  '^/usr/local$'  F=1440/0.3      I=12/1.0
  '^/home$'       F=1440/0.3      I=12/1.0
  '^/wgt'         F=-1/1.0
  '^/wg'          F=1440/0.3      I=168/.9        I=12/1.0
  '^/sy'          F=1440/0.3      I=12/1.0
  '^/tulsa$'      F=1440/0.3      I=12/1.0
  '^/tulsa/src$'  F=1440/0.3      I=12/1.0
In the above, any filesystem starting with "/wgt" will not be dumped. The "/wg" filesystems will get a full dump at least every 1440 hours (60 days) or when the previous incremental level 1 dump is over 30% the size of the last full. A level 1 dump will be done at least every 168 hours (1 week) or when the previous incremental level 2 dump is over 90% the size of the last incremental level 1 dump. A level 2 dump will be done every day (as long as the previous level 2 dump was done at least 12 hours earlier).

Dumps will also be scheduled if the information in the /etc/dumpdates file doesn't match the information gleened from the fstore listing. This can happen if the /etc/dumpdates file gets destroyed or a filesystem gets moved from one device to another.

The process of scheduling a dump consists of simply putting a file into /usr/local/bdump/todo. The name of the file indicates the mount point of the filesystem to be dumped (with slashes replaced by underscores). The contents of the file consists of three fields: an integer specifying the level of the dump, the block device of the filesystem and the rest of the line is a comment to be put on the fstore command. A typical file might consist of: 

    1 /dev/lv00 I-Aged From 94.04.01.17
Indicating that a level one dump of /dev/lv00 should be done and be given a comment of "I-Aged From 94.04.01.17". The format of the comment above is the "type" field from the sched file (being "I" for incremental) followed by a reason of: 
    Aged: Indicating that the maximum time between dumps had been 
	  exceeded.

    Size: If the latest level+1 dump exceeded the size threshold.

    Need: If there was no dump existing at this level.

    Sync: If the last dump at this level did not match /etc/dumpdates.
and the time of the "parent" dump in yy.mm.dd.hh format.

Dumps can also be schedule manually with the "bdump" command. The syntax for this command is: 

    bdump [-q] [level] mountpoint ["comment"]
When bdump1 runs, it will not replace a previously scheduled dump that has a level less than or equal to the level it would have scheduled. Thus a level zero quarterly dump can be scheduled when convenient. The -q option with no comment specified can be used to generate an appropriate comment for a quarterly dump.

2) Performing dumps.

The second phase is to do the actual dumps. The command to do this is bdump2. The bdump2 command must be run from the /bdump/todo directory and the parameters should be a list of the files in the directory to process. Multiple bdump2 commands can be run at the same time as they'll skip any files that are locked or nonexistant. When bdump2 completes processing a file it will remove it if it was successful or leave it behind if not. The script "bdump.sh" which performs the dumps via cron should do something like: 

     bdump1
     cd /usr/local/bdump/todo
     bdump2 *
Bdump2 gets its commands to dump the filesystems from the file /usr/local/bdump/cmds. The fields from the todo file are passed to these commands via environment variables. A typical file for golding might contain: 
  #
  #  Bdump commands file.
  #
  #  Syntax is <key>=<rc>:<cmd>
  #
  #  Where key    Is one of the keywords DUMP, ARCHIVE, COMMIT.
  #        rc     Is the expected successful exit code from the command.
  #        cmd	  Is the body of the command to be executed.  Environment
  #               variables that can be used in the command are:
  #
  #               DEV:     The block device for the filesystem.
  #               FS:      The mount point for the filesystem.
  #               LFN:     The filename for the dump image.
  #               COMMENT: The comment to accompany the dump image.
  #               LEVEL:   The level of the dump.
  #

  DUMP=0:/usr/local/etc/dwrap -"$LEVEL"uf - $DEV
  ARCHIVE=0:/etc/fstore -dump $LFN -comment "$COMMENT"
  COMMIT=0:/etc/fstore -commit $LFN -comment "$COMMENT"
The successful return code for the dump command is zero on AIX systems and it's one on Ultrix. Most other commands have a successful return code of zero. The dwrap utility is a wrapper for the standard dump utility and is described below. To do a dump, bdump2 executes the DUMP command above with the output piped to the ARCHIVE command above. If both of these commands terminate with a success indication, the COMMIT command above is executed. Bdump2 executes these commands in children processes which set their process group accordingly and then "execve" a command of the form "/bin/sh -xc <string>". If, say, the ARCHIVE command should terminate with a failure status, the child executing the DUMP command and everything else in its process group will be killed.

3) Pruning the dumps.

The third phase of the dump process is to prune out old dump images that are no longer required. This is done by the bdump3 utility. Bdump3 gets a list of all the dumps in the silo and then deletes the ones that are no longer necessary based on the schedule defined in its configuration file, /usr/local/bdump/prune. The idea behind the pruning is that you want to be able to restore files from the past. As the distance into the past that you must reach grows, the less particular you are about getting a dump from a specific date. Thus if you want to restore a file that was deleted today, you want to be able to find it on last night's dump, but if you want to restore a file that was deleted last week, chances are that any day in the previous week would suffice. If you have to go back a month, any dump made that quarter might be okay. A typical pruning file for golding might be: 
  #
  #  Bdump pruning file.
  #
  #  Syntax is <delim><pat><delim>{<white><days>":"<keep>}
  #
  #  Where delim  Is some delimiter that does not appear in the pattern.
  #        pat    Is a regular expression to match the filesystems for this
  #               dump schedule (see ed(1)).
  #        white  Is space or tab characters.
  #        days   Is an integer specifying a count of days prior to now.
  #        keep   Is an integer specifying the maximum number of days one 
  #               should go back before finding a dump of the specified
  #               file system.
  #
  #  The schedule on the first pattern matching the filesystem will be used.
  #  The days:keep fields are repeated.
  #  


  '^/$'                   7:1     30:7
  '^/usr$'                7:1     30:7
  '^/var$'                7:1     30:7
  '^/usr/local$'          7:1     30:7
  '^/inst.images'         2:1     30:7
  '^/home$'               7:1     30:7
  '^/wg'                  7:1     14:3    30:7	60:30   370:100
  '^/sy'                  7:1     14:3    30:7	60:30
  '^/tulsa$'              7:1     14:3    30:7	60:30
  '^/tulsa/src$'          7:1     14:3    30:7	60:30
For the /wg disks, if you want to restore a file from 0-7 days ago you should only have to go back one day. If you're looking for a file that was deleted 8-14 days ago, you should be able to find it within 3 days. One that was deleted within the last two months (60 days) should be found on a dump no more than one month old (30 days). Anything up to 370 days ago will be found on the quarterly dumps which are made every 90 days or so. It is assumed that nothing older than 370 days will need to be restored.

The pruning process will retain approprate dump images to do the above restores plus any parent dumps of those dumps which are assumed to be the most recent n-1 level dump done prior to dump retained. Thus if a full dump of /usr was made in August and daily incrementals were done after that, at the end of December we would have the last seven daily dumps, one dump for each of the preceeding three weeks and the full dump that was made in August.

Bdump3 writes a set of "fstore -erase" commands to its stdout and exits with a zero to indicate success. The bdump.sh script that gets invoked by cron is expected to do something like: 

   bdump3 > /tmp/bdump3.$$
   case $? in
   0) sh /tmp/bdump3.$$;;
   esac
   rm -f /tmp/bdump3.$$
to perform the actual pruning of the dump images.

4) Offsite tapes.

If an atomic bomb is dropped on the University, we will need to get the systems up as soon as possible with all the student's GIF files intact. To achieve this feat, a copy of the latest dump is sent down to the 3737 building to safeguard it against this possibility. Hopefully if someone nukes the admin building the blast won't take out both the 4545 building and the 3737 building. Yeah, right! No, the real reason is to guard against an earthquake. The theory is that when the big one hits it will be satisfied with taking out the copy in the 3737 building and leave the 4545 building alone. Anyway, a copy is sent down to the 3737 building.

If a disk dies or a file simply needs to be restored, it is usually vital enough that we don't want to wait for the tapes to be brought up from the offsite storage space. Rather than sending a recent dump offsite, then, a copy of it is made and the copy is sent offsite. Once we see that the copy has made it offsite (well, at least out of the silo) the copy of the previous dump that was sent offsite is deleted.

The schedule of which filesystems are sent offsite and how often is defined in the /usr/local/bdump/offsite file. A typical offsite file for golding might look like: 

  #
  #  Bdump offsite schedule file.
  #
  #  Syntax is <delim><pat><delim>{<white><days>":"<class>}
  #
  #  Where delim  Is some delimiter that does not appear in the pattern.
  #        pat    Is a regular expression to match the filesystems for this
  #               dump schedule (see ed(1)).
  #        white  Is space or tab characters.
  #        days   Is an integer specifying maximum days before new offsite.
  #        class  Is the class to give offsite class images.
  #
  #  The schedule on the first pattern matching the filesystem will be used.

  '^/tmp'                 -1:NONE
  '^/crash'               -1:NONE
  '^/mnt'                 -1:NONE
  '^/a1$'                 -1:NONE

  '^/$'                   14:X3737
  '^/usr$'                14:X3737
  '^/var$'                14:X3737
  '^/usr/local$'          14:X3737
  '^/usr/local/prod$'     14:X3737
  '^/cluster$'            14:X3737

  #
  #  UA Galaxy disks
  #
  '^/wgt'                 -1:NONE
  '^/wg'                  14:X3737
  '^/ux'                  14:X3737
  '^/sx'                  14:X3737
  '^/sy'                  14:X3737
  '^/www'                 14:X3737
The above schedule indicates that most of the filesystems will have a copy of a dump sent offsite that is no more than 14 days old. When the latest dump image that was sent offsite is more than 14 days old, a copy of the latest image will be requested. If the parent of the dump that is copied already has a copy offsite, that copy will be retained, otherwise a copy of the necessary parent dumps will also be requested. Any class that starts with a capital X is sent offsite, X3737 in particular goes to the 3737 building.

The bdump4 command actually generates a script with the "fstore -dcopy" commands required to get the offsite copies generated. "Dcopy" stands for "delayed copy". The copy is not made immediately, but merely scheduled. When the backup daemon is relatively idle it will process pending delayed copies and request that the tapes be ejected.

Appendix A: Dwrap

The dump utility was designed with the mentality that automatic dumps are not desired and assumes that it will be run from an interactive terminal. Thus if it has problems it will open /dev/tty and prompt the terminal directly for assistance. This doesn't work too well if the dump process is invoked by a cron process which has no controlling tty. The dwrap utility gets around this by allocating a dummy tty for the dump utility to use and then spawns the dump command. Dwrap watches any output that the dump utility may issue and when it asks a question such as "Do you want to abort the dump?" it answers "yes".

Dwrap figures out what to do based on a configuration file dwrap.cf which it currently looks for in either the current directory or in /adump (dwrap has not yet been moved to the bdump product). A typical dwrap.cf file for golding might appear as: 

  #
  #  The following defines the dump command that dwrap should execute.
  #
  @/dev/ad?+/etc/pnd_dump:/dev/vp?+/etc/pnd_dump:/etc/backup
  #
  #  The following is the list of messages we look for coming out of the
  #  dump utility on this system.  The syntax of this file is:
  #
  #    delimiter, regular expression, delimiter, response, delimiter
  #
  #  If the regular expression matches, the response is sent to dump's
  #  pseudo tty.  Be sure to put \n in to indicate a newline in the
  #  response.
  #
  :Do you really want to abort dump:yes\n:
The rather cryptic syntax on the dump command to execute is designed to handle the 7051's requirement that a different dump command be used based on what type of filesystem is being dumped. The '@' indicates that this is the line that contains the dump command (rather than one of the messages to look for). The question mark-colon syntax is based on the C construct. In the above, if the device being dumped starts with "/dev/ad" or "/dev/vp", the dump command to use is "/etc/pnd_dump". Otherwise it's /etc/backup. Luckily the parameters on these two commands is the same except that pnd_dump doesn't want a leading minus sign on its options. The leading "+" on the selected command tells dwrap to eat the minus sign on the parameters prior to invoking the command. It was initially anticipated that dwrap would have a whole repertoire of questions that it would have to parse such as answering "yes" to "Do you want to abort?" and answering "no" to "Do you want to continue?", but during the life of adump, we have not run into a situation where it has had to answer any questions. If fstore dies you'd expect dump to get a pipe error and ask for "another tape", but it just seems to hang without saying anything. Adump was unable to kill these hung dwrap processes, but bdump uses the process group mechanism which successfully cleans things up when this happens.

Appendix B: File Naming

The naming convention that bdump uses for dump images is: 
	host.fs.lev.yy.mm.dd.hh
Where: 
	host: Name of the host making the dump.
	fs:   Mount point of the filesystem.
        lev:  Single digit for dump level, 0=full, 1=inc, 2=...
	yy:   Year,
        mm:   Month,
        dd:   Day
        hh:   and Hour of the dump start time.
A list of available dumps can be retrieve with the fstore -list command which allows an ed(1) style regular expression: 
   #golding> fstore -list wg08
   Connected to archive3.u.washington.edu.
   COX     1637.86  golding.wg08.0.93.12.19.22    Q-Winter
   COX     1751.27  golding.wg08.0.94.03.27.22    Q-Spring
   COX     1658.85  golding.wg08.0.94.06.18.18    Q-Summer
   COX     1615.62  golding.wg08.0.94.09.19.19    Q-Fall
   C        156.74  golding.wg08.1.94.10.14.21    I-Aged From 94.09.19.19
   C        233.23  golding.wg08.1.94.11.07.18    I-Aged From 94.09.19.19
   C       1440.64  golding.wg08.0.94.11.19.22    F-Aged
   CGX     1440.64  golding.wg08.0.94.11.19.22A   X-3737
   C         89.42  golding.wg08.1.94.11.24.01    I-Size From 94.11.19.22
   C        119.74  golding.wg08.1.94.12.01.20    I-Aged From 94.11.19.22
   C        195.35  golding.wg08.1.94.12.09.01    I-Size From 94.11.19.22
   C         46.74  golding.wg08.2.94.12.10.19    I-Need From 94.12.09.01
   C         83.05  golding.wg08.2.94.12.13.17    I-Aged From 94.12.09.01
   C        222.26  golding.wg08.1.94.12.16.20    I-Aged From 94.11.19.22
   CGX      222.26  golding.wg08.1.94.12.16.20A   X-3737 From 94.11.19.22
   C       1533.57  golding.wg08.0.94.12.18.21    Q-Winter
   C         34.71  golding.wg08.1.94.12.19.22    I-Sync From 94.12.18.21
   C         47.68  golding.wg08.1.94.12.21.19    I-Size From 94.12.18.21
   C         33.15  golding.wg08.2.94.12.24.01    I-Need From 94.12.21.19
   C         36.48  golding.wg08.2.94.12.24.21    I-Aged From 94.12.21.19
   C         40.42  golding.wg08.2.94.12.26.18    I-Aged From 94.12.21.19
The first column is the status of the dump. C = complete; O = out of the silo; G = Offsite, g = Offsite pending; X = eject requested. The second is the size of the dump in megabytes.

To restore a file modified on Nov 28th and destroyed on Dec 17th, one should be able to go to any one of the dumps: 

  	golding.wg08.1.94.11.24.01
	golding.wg08.1.94.12.01.20
	golding.wg08.1.94.12.09.01
	golding.wg08.1.94.12.16.20
  or	golding.wg08.1.94.12.16.20A
To restore the file from one of these you would use the command: 
  AIX:    #golding> fstore -restore golding.wg08.1.94.12.01.20 | restore -Biv
  Ultrix: #stein1>  fstore -restore golding.wg08.1.94.12.01.20 | restore ivf -

To restore this filesystem if it were destroyed at noon on the 27th, one would use the following: 

        #golding> newfs /dev/whatever
        #golding> mount /dev/whatever /mnt
	#golding> cd /mnt
        #golding> fstore -restore golding.wg08.0.94.12.18.21 | restore -Br
	#golding> fstore -restore golding.wg08.1.94.12.21.19 | restore -Br
	#golding> fstore -restore golding.wg08.2.94.12.26.18 | restore -Br
	#golding> cd /
	#golding> umount /mnt
	#golding> mount /dev/whatever /wg08
To restore the filesystem back to its state as of 6pm the previous night.

Appendix C: Reports

The bdump3 utility can produce (with the -r option) a report indicating how well or how poorly the backup situation is for each filesystem. This is a rather cryptic report, but conveys quite a bit of information in a condensed format if you know how to read it. A typical report for golding's disks might look like: 
                    Ap                Ma    FJDNOcSAJJMA
filesystem          222222222211111100322110223330000000
----------          987654321098765494050505890011123344
/                   aaaaaaaagfedcbafad***yyxxurmrn
/home               aaaaaaaagfedcbaaIdPlgbjep
/inst.images        aaaaaaaafedcbaQQQPPplgb
/sy00               aaaabaaba*****************yvsp
/sy01               aaaabaabacbacbabQQPppkfam*yvsp
/sy02               aaaabaabacbaa*************yvsp
/sy03               aaaabaabacbacbacUTTSSRRQQcyvsp
/sy04               aaaabaabacbaFEdc**********yvsp
/sy05               aaaabaabacba**************wtqi
/sy06               aaaabaababa**************yvsp
/sy07               aaaabaabacbaba***********xurm
/sy08               aaaabaabacbaEdcb******YYXurp
/sy10               aaaabaabacbacbachcPplgbppc
/sy11               aaaabaabacbacbacgbPppmhcpc
/tulsa              aaaabaabacbacbafacbqpppniprkb
/tulsa/src          aaaabaabacbacbabfadppmhcp
/usr                aaaaaaaagfedcbaacdPppojevsp
/usr/local          aaaaaaaagfedcbaacdPppoje**xurp
/var                aaaaaaaadcbagfedfa##
                    Ap                Ma    FJDNOcSAJJMA
filesystem          222222222211111100322110223330000000
----------          987654321098765494050505890011123344
/wg00               aaaaaaaacbacbabaIdcbpojeUrp#########
/wg01               aaaaaaaacbacbabaIdebpojeprp#########
/wg02               aaaaaaaacbacbabaIdeppnidpro#########
/wg03               aaaaaaaacbacbabacdeppnidUrovsp######
/wg04               aaaEDCbacbacbacbhceppplgbqjurp######
/wg05               aaaaaaaacbacbabaccdRRQQppqj#########
/wg06               aaaaaaaacbacbabahcdpppkfarmurp######
/wg07               aaaaaaaacbacbababcdRRQQpprmurp######
/wg08               aaaaaaaacbacbacbhccppnidprkurp######
/wg09               aaaaaaaacbacbacbhccppnidUrmwtqa#####
/wg10               baaaaaaacbacbacbhccepnidprmvspa#####
/wg11               baaaaaaacbacbacbhcdpplgbprkwtqa#####
/wg12               baaaaaaacbacbacbeaNidmhcUrkvspa#####
/wg13               babaaaaacbacbacbJeamhcpnirkwtqa#####
/wg14               babaaabacbacbacbhcNidmhcTqjrmqa#####
/wg15               babaaaabacbaaIHgbcIdojeUTqgwtqa#####
/wg16               baabaaabacbaacbagbNidmhcTqgvspa#####
/wg17               baaaaaabacbacbabgbfappojeqjsp#######
/wg18               aaaaaaabababacbagbPojekfaqjqj#######
/wg19               aaaabaabababacbagbPppmhcTqitqga#####
/wg20               aaaabaabacbabacbfadppkfaTqh#########
/wg21               aaaabaabacbaabagbbcWVVUUTqh#########
/wg22               aaaabaabacbaabafabWWVVUUTqh#########
/wg23               aaaabaabacbabacbaeWWVVUUTqh#########
/wg24               aaaabaabacbabacbacWWVVUUTqh#########
The header of the report indicates the date represented by each column. The first column of letters is April 29th. This shows the current day and each of the previous 14 days back to April 9th. From there it shows every 5th day back to 60 days before the current date, February 28th, and from there every 30th day back to 400 days ago, April 4th, 1993. The letters in the columns indicate how many days back you would have to go to restore a file lost on that date. The smaller the letter, the less you'd have to go back. The letters indicate the following number of days: 
   a =  1    g =  7    m = 13    s = 50    y = 110
   b =  2    h =  8    n = 14    t = 60    z = 120
   c =  3    i =  9    o = 15    u = 70    * = 121+
   d =  4    j = 10    p = 20    v = 80    # = N/A
   e =  5    k = 11    q = 30    w = 90
   f =  6    l = 12    r = 40    x = 100
A lowercase letter indicates that the dump is within the scheduled window as defined by the prune file. An uppercase letter indicates that you'd have to go back further than you should to get a file restored. The pound sign indicates that there are no dumps prior to that date, but the prune file indicates that there should be. This would be normal if the filesystem wasn't that old. A short line indicates that there's no dump, but none was expected. Thus, we see above that /wg04 missed some dumps on April 26, 25, 24 and 23rd. Of course, that was before bdump was running. It's then fine back to p = 15-20 days prior to October 1st (September 15th) when the Fall quarter dumps were done. /wg04 didn't exist prior to that, so we don't have any dumps of it.