KArc - Ken's Archive Utility

Server host preference

There should be one server entry for each address of each server that this client should attempt to access in order to retrieve or store dump images. The preference field specifies an integer value indicating the initial base preference for a particular server. Each address listed is sent a query asking how busy it is. The response from this query is added to the base preference to get an overall preference value. The address with the highest overall value is the one used when establishing the control channel connection. All dumps will be sent to that server if that server is active. On a restore, if the server chosen doesn't have the requested image, it will forward requests to the proper server. The actual data will not make the extra hop to the chosen server, but activity on the control channel will.

COSId id

The COSId entry in the configuration file establishes the HPSS Class Of Service Identifier for all dump images data and index files. The COSId statement only needs to be on server hosts, this parameter is ignored by client utilities.

DATId id

The DATId entry specifies the class of service for the dump image data file. It only needs to be specified on servers where the index and data files are written with different class of service identifiers.

DSKId value

The DSKId entry specifies whether this server supports an extra disk copy of the index files. A value of 0 indicates no, non-zero indicates yes. An extra disk copy is only needed if access to the backend storage subsystem would be too slow.

Encrypt modes [ key ]

Specifies the default encryption mode for traffic between the kdump utility and the karcd servers. The modes parameter can be one or more comma delimited tokens as follows:

None:	No encryption is performed. The data and index file are sent to the karcd dæmons in clear text. This option should be used only if you're comfortable that there are no network sniffers running or you don't care that people may see the contents of your files (including your karc and ssh private keys if you happen to dump the filesystem that those are stored on).
Private:	Files that are protected with no world (others) read permission are encrypted when sent across the network. This includes files in directories that do not contain an execute permission (traverse permission) for others.
Public:	All files, whether all users on the local system can read them or not, are encrypted when sent across the network.
Index:	The index file is encrypted when sent across the network. If the index file is not encrypted, sniffers can discover the names of the files that are being dumped.
All:	The All option is the same as public,index.

If private or public is specified, the encryption key must also be named for kdump. This key is stored in the dump_keys file or can be stored on the server for global keys that must be accessed by multiple hosts.

The configuration file just specifies the default for kdump and karc. These defaults can be overridden by options on the respective utility's command line.

NDXId id

The NDXId entry specifies the class of service for the dump image index file. It only needs to be specified on servers where the index and data files are written with different class of service identifiers.

Ordinal ordinal

The Ordinal entry is only used by the karcd server, it is ignored by other clients. Each karcd server has a unique ordinal from 0 to 2.

Port port

The Port entry specifies the server TCP/IP port number that the karcd server listens to.

karc -command arg [options]

Commands:

-commit name

Commits the specified dump image to permanency. The expected usage of the karc commit function is to pipe the output of the standard Berkeley dump utility into a karc -dump name command. If the dump and the karc utility complete successfully they'll have a zero exit status. If that's the case, the karc -commit name command can be used. Dump images that have not been committed will be deleted periodically.

-dump name

Write stdin to dump image data file. The dump image name is created and karc simply transfers whatever it finds on stdin to the data file for the dump image. The karc utility will not create an index file, the kdump utility creates that itself through the karc API.

A script that archives the contents of a subdirectory might do something like:

NAME=`date '+subdir.%Y.%m.%d.%H.%M'`
tar -cf - subdir | karc -dump $NAME -comment "dump of subdir"
case $? in
  0) karc -commit $NAME;;
  *) echo "Oh no, the tar piped to karc failed";;
esac

-erase name

Remove dump image. The data file, the index file and the name are permanently removed from existence.

-list pattern

Display dump image status. Existing dumps matching the specified pattern are displayed. The pattern consists of an ed(1) regular expression including an optional group and member in the syntax: group/member/name. If an alternate group or member is specified, the security level must be high enough to permit that or else karc will get a "No Authorization" error.

-newkey file

Establishes a new LSC key. The expected procedure for setting up a new client is:

	client% cd /usr/local/lib/karc
	client% lsc_keygen kar4_GRP_`hostname` priv_keys newkey

        admin% rcp client:/usr/local/lib/karc/newkey newkey
	admin% karc -key 7 -newkey newkey

The data encryption keys can also be stored on the server if desired. In this case, only the public half of the key is used, both for the encryption and decryption, so the private half may be discarded.

-read file

The read command reads additional requests from the named file or stdin if a single dash is specified. Directives on this input file look like complete karc commands that the shell would interpret. The benefit of using this option rather than submitting multiple commands is that karc will make one connection to the server which can greatly reduce the time required to, say, erase several dozen dump images.

-restore name

Read dump image data file to stdout. The expected usage of this function is to pipe a dump created with karc -dump into the Berkeley restore utility or a derivitive thereof. The karc utility will not access an index file. The krest utility accesses that directly through the karc API.

A session to retrieve a file from the subdir archive created above might look something like:

frodo% karc -list subdir
C-d-  subdir.2000.10.20.15.01    20.02   dump of subdir
frodo% karc -restore subdir.2000.10.20.15.01 | tar -xf - subdir/lost_file

-set name

Set new comment.

-comment 'text'

Specify dump image comment. This option can be used on the -dump or -set commands.

-format 'format'

Specify list format string. The list format string allows you to tailor the output of the -list command how you want it. The format string is not unlike a C language printf format with the following percent characters defined:

	C	Comment associated with dump image.
	F	File name. This is the full name of the dump image including the group and member fields.
	G	Group name. The group name that the dump image belongs to.
	H	Host name. The member name that the dump image was created under.
	L	Local file name. This is the image name without the group and member fields prepended.
	S	Status. This a four character field giving the status of the dump image. The first is C for committed, p for partial or i for initial. The second either e for erased or dash for not. You'll see erased files if not all copies of the file have been erased from all servers. The third is a d if there's a data file or dash if not. The fourth is an i if there is an index file or dash if not.
	T	Time. Creation time in seconds since the beginning of the epoch.
	c	Creation time in standard unix ctime(3) format.
	d	Size of the data file in floating point megabytes.
	f	Flags associated with dump as a hexidecimal number. The flag bits are defined as KAR_S_xxxx in the kar.h include file.
	i	Size of the index file in floating point megabytes.
	m	Modification time in ctime(3) format.
	s	Size of the data file plus the size of the index file in floating point megabytes.
	t	Creation time in ctime(3) format (same as 'c').

Each of these fields can be modified as "%[-]w.f+nX" where w is the minimum width of the field (leading minus sign for left justification), f is the maximum width for a string or the digits right of the decimal point for the floating point fields and n is an offset to the first displayed character of the resulting string. Thus %.12+4c will get the month, day of month, hours and minutes of the creation time without the leading day of the week or trailing colon, seconds and year. The default format string is: "%-5.5S %9.2s %-36L %C"

-key level

Specify alternate key level. By default the karc utility will use the most recently created key in the priv_keys file. An alternate key can be selected by specifying the single digit level with the -key option.

-port port

The karcd server runs on the port specified in the configuration file. The -port option can select a test server running on an alternate port.

-server host

The karc utility normally connects to the servers in the order specified in the configuration file. An alternate server can be selected with the -server option.

-sort field

Sort list by Creation, Name or NOne. Normally the list command sorts its output by creation time. This can be overridden with the -sort option. The NONE sort option yields an apparent random order (sorted by the KNS server's internal hashing function).

Access cn[+] group/host/name

Assigns the existing name to the connection number. The plus following the cn requests control access -- otherwise it's a read-only access and no changes are allowed.

Close cn

Closes the current stream associated with the cn (see Open). The threads transfering the data are shut down and their status is and the total number of bytes transferred is returned.

Comment cn text

Sets the comment associated with the dump image to the specified text.

Commit cn

Sets the committed flag on the dump image.

Create cn group/host/name

Creates the specified name in the KNS name space and assigns the connection number to the new dump image.

Erase cn

Removes the dump image and deletes the name from the KNS name space.

List pattern

Retrieves entries matching the pattern from the KNS name space.

Open cn type

Establishes a pipeline to transfer the specified dump image data, type=d, or index, type=i, file across the network. Multiple threads are started to perform the transfer. One communicates with a network partner by opening a socket to which the client is instructed to connect. Another communicates with the data store. The reading thread fills buffers while the writing thread(s) flush them out until end of data.

Read cn start length

If the transfer is from the data store to the network client (a restore operation by the client), read instructions are necessary to tell the data store reading thread what portions of the data are required. The reading thread will seek to the appropriate place, fill the buffers with the data and then proceed to the next read instruction. The start and length values are in 64 bit values as hexidecimal strings high+low. An asterisk for the length field requests a read to end of data. An octothorpe (pound sign) for the start field indicates end of read requests. The buffers won't be flushed unless they're filled or the end of read request is encountered, so all read requests should be sent prior to blocking on reading the data socket or in a separate thread.

Report cn rate

The report request is used to provide the karcd dæmon with the transfer rates for the client's portion of the transfer. The transfer rate is used to identify bottlenecks in the system.

When a copy completes, the next file in the stack is handed off to the idle child until the stack is empty. The requests are processed LIFO to increase the chances that the dump image is still in the local disk cache and doesn't need to be retrieved from tape.

Dec 2005

The broker dæmon processes the following commands:

Create group/host/name

Allocates a new location in the broker database. The create request is the first activity on a particular chunk. All flags are zeroed and the update/create time is set to the current clock. In 24 hours the entry will be recycled if not updated.

Delete group/host/name

Releases all information about the specified entry and recycles the space it used. This call should be made after all instances of the file have been removed from the disk cache and back end store.

Modify group/host/name +/-flags

Sets or clears the indicated flags for the particular chunk.

Commit group/host/name size

Signifies that the chunk is complete and is ready to be flushed to the back end server and/or remote servers. The size parameter indicates the final segment size. The broker will spawn off the karcf helper processes to deal with the flush requests.

Stage group/host/name [offset length]

Stages the specified chunk from the back end store into the disk cache. If the offset and length are specified, control returned after that portion has been staged in. The actual staging process is handled by karcf helper child processes.

Dec 2005

Flush prefix group/host/name length

Copies the specified chunk to TSM. The prefix describes which disk pool the segment currently resides in and the length is in bytes modulo 4GB.

List [pattern]

Generates a list of files in the broker's namespace that match the given pattern. By default all files are listed. This is used by the reconciliation utilities to align the broker's view with TSM's view and the current disk cache.

TBD group/host/name TBD

Copies the specified chunk to a remote server.

Stage prefix group/host/name [offset length]

Copies the entire chunk from TSM to the disk cache. The prefix describes the intended disk pool for the segment. When the optional indicated chunk is disk resident a partial reply is returned.