Compressed CKD Dasd Emulation

Contents

Introduction

Using compressed CKD files, or cckd files, you can significantly reduce the file space required for emulated CKD dasd and possibly gain a performance benefit because less physical i/o occurs. Using the shadow function, you can minimize the amount of data loss in the event of file corruption.

A cckd file contains track images, which may be compressed or uncompressed, and overhead blocks which are headers, lookup tables, and free space. Compressed track images may be compressed by zlib or bzip2.

Track images are addressed by track number using a two table lookup method: track number divided by 256 (trk >> 8) indexes into the primary lookup table, which contains the file offset to the secondary lookup table; the remainder of track number divided by 256 (trk & 0xff) indexes into the corresponding secondary lookup table, which contains the offset and length of the track image.

There is a single primary lookup table and a variable number of secondary lookup tables. The maximum number of secondary lookup tables is the number of tracks for the device type divided by 256, rounded up. For example, a 3390-3 contains 50085 tracks and would require at most 196 secondary lookup tables.

A regular CKD file contains a 512 byte header followed by track images, each taking the same amount of space: the maximum track size. The offset of a track image can be readily calculated by the track number:

offset = 512 + trk * maxtrksz

A cckd file can take significantly less file space than a regular CKD file because

Performance improvements may also occur because less data is read and written to the hard drive.
However, the lookup tables must be accurately maintained, track images must be compressed and uncompressed, and free space must be kept track of, and dealt with by a garbage collector. This results in a more complicated file structure, more CPU activity, and the possibility of file corruption due to program failure or bug. The introduction of shadow files, however, reduces the impact of possible file corruption.

Shadow Files

Malcom Beattie originally introduced the concept of shadow files in a post to the newsgroup 8 December 2000. The function is actually implemented as a kind of snapshot, where a new shadow file can be created on demand. A CKD emulated dasd is represented by a base file and 0 or more shadow files. The base file can be either a regular CKD file (with some restrictions) or a cckd file. All files are opened read-only except for the current file, which is opened read-write.

Shadow files are implemented using the same file structure as base cckd files. By default, there can be up to 8 shadow files in use at any time for an emulated CKD device. The base file is designated file [0] and the shadow files are files [1] up to file [8]. The highest numbered file in use at a given time is the current file, where all writes will occur. Track reads start at the current file and proceed down until a file is found that actually contains the track image.

A shadow file, then, contains all the changes made to the emulated CKD dasd since its creation, until the creation of the next shadow file. The moment of the shadow file's creation can be thought of as taking a snapshot of the current emulated CKD dasd at that time, because if the shadow file is later removed, then the emulated CKD dasd will revert to the state it was at when the snapshot was taken.

Using shadow files, you can keep the base CKD file on a read-only device such as cdrom, or change the base CKD file attributes to read-only, ensuring that this file can never be corrupted.

Hercules console commands are provided to add a new shadow file, remove the current shadow file (with or without backward merge), and display the shadow file status and statistics.

CCKD File Structure

Like a regular CKD emulation file, the first 512 bytes of a compressed or shadow file contains a CKDDASD_DEVHDR block. The eye-catcher at the beginning is different to distinguish the file: The next 512 bytes contain a compressed device header or CCKDDASD_DEVHDR block. This contains the version-release-mod level of the file, options, space statistics, and total number of cylinders for the device. Next is the primary lookup table or the L1TAB. Each 4 byte entry in the L1TAB contains the file offset to a secondary lookup table (or L2TAB) or 0x00000000 (indicating that the secondary lookup table is null), or 0xffffffff (indicating that the previous file should be searched instead).
The size of the L1TAB is dependent on the number of tracks on the emulated device.

CKDDASD_DEVHDR
CCKDDASD_DEVHDR
L1TAB

.  .  .

Following the L1TAB, in no particular order, are L2TABs, compressed track images, and free spaces.

L2TABs contain 256 8-byte entries,and each are, consequently, 2048 bytes in length. Each entry contains the offset and length of a track image. If the offset is 0x00000000 then the track image is null; if the offset is 0xffffffff then the previous file should be searched instead.

L2TAB entry
offset
4 bytes		 length
2 bytes		 [unused]
2 bytes

A compressed track image contains the following two fields:

HA
5 bytes		 track image (compressed or uncompressed)
length-5 bytes

The HA contains 0CCHH, that is, a byte of zeroes, 2 bytes indicating the cylinder of the track, and 2 bytes indicating the head of the track on the cylinder. Both CC and HH are stored in big-endian byte order. The track is computed by

trk = (((CC[0] << 8) + CC[1]) * trks_per_cyl) + (HH[0] << 8) + HH[1]

Since the first byte of the HA is always 0x00 (at least in emulated CKD files), this byte as stored in the file actually indicates the compression algorithm used for the remainder of the track image (0 = no compression, 1 = zlib compression, 2 = bzip2 compression).

Free space contains a 4-byte offset to the next free space, a 4-byte length of the free space, and zero or more bytes of residual data.

Free Space entry
offset
4 bytes		 length
4 bytes		 residual
(length - 8) bytes

The minimum length of a free space is 8 bytes. Since free space is ordered by file offset and no two free spaces are adjacent, offset in the free space entry is always greater than the current free space offset + the current free space length, unless the offset is zero, which indicates the free space chain is terminated.
The free space chain is read when the file is opened for read-write and written when the file is closed; while the file is opened, the free space chain is maintained in storage.

Methodology

This section is tedious; you probably want to skip to the next section unless you are genuinely curious as to how cckd actually works. This section is for my edification as much as anything...

Initialization

When a CKD dasd emulation file is initialized, function ckddasd_init_handler in ckddasd.c is called. After ckddasd_init_handler has completed its initialization, if the file is a cckd file, function cckddasd_init_handler in cckddasd.c is called.

cckddasd_init_handler obtains a cckd extension and stores its address in field cckd_ext in the DEVBLK (the control block that represents the device).

The compressed device header (CCKDDASD_DEVHDR) and L1TAB for the file is read.

If shadow files exist, they are opened and their CCKD_DEVHDRs and L1TABs are read. If the last file opened could only be opened read-only, then a new shadow file is created.

The basic point is that the CCKD_DEVHDR and the L1TAB for the base file and each shadow file is read and stored in an array in the cckd extension, and each file is opened read-only, except the current file, which is opened read-write.

File I/O

In the course of executing a channel program, routines in ckddasd.c will call the generic routines read_track and update_track. These generic routines have their addresses stored in the DEVBLK fields ckdrdtrk and ckdupdtrk. For regular ckd dasd images these are ckd_read_track and ckd_write_track and for cckd dasd images these are cckd_read_track and cckd_write_track.

The read_track routine is called when a track switch occurs. A track switch can switch back to the same track. The routine is responsible for scheduling the previous active track to be written, if it has been modified and the write has not already been committed. The new track is read (or retrieved from cache), and the buffer address containing the track image is placed in the DEVBLK field buf. DEVBLK field bufoff is set to the offset in the buffer of the track image and DEVBLK field bufoffhi is set to the end offset of the image in the buffer. In this way, buf can contain multiple track images. Both ckd_read_track and cckd_read_track place a single track image in the device buffer.

The update_track routine is called when the contents of the track image buffer is updated, and passes the offset into the buffer where the update began and its length. The update_track routine may either immediately write the updated contents of the buffer or remember the portion of the buffer that has been updated and write later. The ckd_update_routine will write the updated data immediately if the options nofulltrk or nolazywrite was specified for the device, otherwise, the updated data will be written at the next track switch. The cckd_update_track routine will schedule the track image to be written sometime after a track switch, or 5 seconds.

These routines alone are responsible for reading and writing to the ckd disk image, either compressed or uncompressed.

Quick Start

The ckd2cckd utility can be used to create a new compressed CKD file from a regular CKD file. Your disk images can be a combination of regular CKD files and compressed CKD files. Simply specify the names of your new compressed ckd files in hercules.cnf in place of the regular CKD file names. You can also use the cckddump program on an os/390 system to build a compressed CKD file from a real disk that can be transferred to your Hercules machine and used right away.

Using Shadow Files

Shadow files enable you to make updates to cckd emulation files and not worry about possibly corrupting your entire disk image. I strongly urge those of you who use cckd to start using shadow files immediately and change your base file to read-only. This, in turn, reduces the amount of data you have to back up, increasing the amount of file savings cckd has to offer. You can even change shadow files to read-only, as long as a new shadow file can be created. You can also use shadow files for regular (non-cckd) files.

Shadow files are automatically enabled for cckd files; you must explicitly enable them for regular CKD files. To enable shadowing for a CKD device, specify

sf=shadow_file_name

on the device statement in the hercules.cnf file. shadow_file_name should include a spot in the file name, similar to multiple CKD dasd files, that can be used as a sequence number, for example, sf=../mvs/shadows/mvsres_1.500. The naming convention substitutes the shadow file number (1 thru 8) on the character preceding the period after the last slash, or the last character if no period follows the last slash. Example

0500 3390 ../mvs/disks/mvsres.500 sf=../mvs/shadows/mvsres_1.500

If you did not specify sf= for a cckd file, or you wish to change the shadow file name for a cckd or regular file, but no shadow files are in use, then you can issue the following command on the Hercules console:

sf=xxxx   shadow_file_name

where xxxx is the device unit address. For example, sf=0500 ../mvs/shadows/mvsres_1.500.

Specifying a shadow_file_name does not explicitly create a shadow file if the base file or current shadow file is able to be opened read-write. Otherwise, if the base file and all existing shadow files (if any) can only be opened read-only, then a new shadow file is created.

To explicitly create a new shadow file, issue the following command on the Hercules console:

sf+xxxx

where xxxx is the device unit address or * (for all eligible units). For example, sf+0500. All updated track images that haven't been written are written and the current file is hardened. Note that if a lot of write activity is ocurring at the time the sf+ command is entered, then the exact state of the hardened file can not be predicted. A new shadow file is created and all new writes are directed to it.

To remove the current shadow file, issue either of the following commands on the Hercules console:

sf-xxxx
sf-xxxx nomerge

where xxxx is the device unit address or * (for all eligible units). For example, sf-0500. If nomerge was not specified, then the current shadow file contents are merged into the preceding shadow file or base file. The current shadow file is deleted and the preceding shadow file or base file is made the current file. If the preceding file is read-only, then an error message is issued. If possible, you can make the preceding file read-write and re-issue the command. Note that if merge is specified or implied, then the command may take some amount of time depending on the size of the old shadow file.
[hmmm... note to myself -- if sf-xxxx nomerge was specified and preceding file is read-only, then delete the current file and recreate it ??]

To compress the current shadow file issue the following command on the Hercules console:

sfcxxxx

where xxxx is the device unit address or * (for all eligible units). For example, sfc0500.

To display the status and statistics for a shadow-enabled file, issue the following command on the Hercules console:

sfdxxxx

where xxxx is the device unit address or * (for all eligible units). For example, sfd0500. This command displays status and statistics for the base file and all shadow files representing the emulated dasd. The following data is displayed:
  size The total size of the file
  free The amount of free space in the file as a percentage of the file size
  nbr The number of free spaces in a file
  st File open status - ro=read-only; rd=read-only, but can be opened read-write; rw=read-write
  reads Number of times cckd_read_trkimg performed physical read i/o
  writes Number of times cckd_write_trkimg performed physical write i/o
  l2reads Number of times a secondary lookup table was read
  hits Number of times cckd_read_trk found a track image in the track cache when called by the i/o thread
  switchesNumber of times cckd_read_trk was called by the i/o thread (cckd_lseek)
  readaheadsNumber of track images read by the readahead threads
  misses Number of track images read by the readahead threads that were never referenced when the track cache entry was stolen

CKD Options

In this section I will attempt to document all the options that can be specified for a CKD file (regular or compressed) in the hercules.cnf file (or on the attach panel command).
RegularcckdFunction
syncio
syio
nosyncio
nosyio
X X Specifies whether or not synchronous I/O will be attempted for the device. For synchronous I/O, the channel program will be executed within the scope of the SIO or SSCH instruction as long as all data referenced by the channel program is already cached. If a ccw attempts to reference data that is not cached, then the channel program is restarted asynchronously at that ccw. Synchronous I/O reduces threading overhead, which may resut in a performance boost. The default is syncio for cckd files and nosyncio for regular ckd files.
lazywrite
nolazywrite		X   Data written to a cached track image will not be immediately written, but will be written when a track switch occurs. Thus, only one write will occur for a track image while it is the active image. nolazywrite, the default, specifies that all writes are performed when requested.
fulltrackio
fulltrkio
ftio
nofulltrackio
nofulltrkio
noftio		X   Specifies whether or not a full track will be read when a track switch occurs. Subsequent reads to this track image will not cause any physical I/Os. Turning on fulltrackio can considerably enhance CKD device response time. However, if you are sharing CKD disk images with more than 1 instance of Hercules at the same time when writes could occur, you should specify nofulltrackio. The default is fulltrackio.
readonly
rdonly
ro
X X Causes the CKD file image to be opened read-only. Attempts to write to the emulated device will cause an I/O error unless option fakewrite is also specified. If readonly is specified for shadowed file images, then the base file will be opened readonly and a shadow file will be created if one doesn't exist.
fakewrite
fakewrt
fw
X X Writes to a readonly file will be considered successful even though no write actually occurred. This option is only meaningful if readonly is also specified. Fakewrite is ignored for shadowed file images.
cache=nXX Specifies the number of track images that will be cached. The default is the number of tracks per cylinder for the device. [For cckd files, the default is the number of tracks per cylinder plus the number of readahead threads]. If nofulltrackio is specified for a regular CKD file, then no caching occurs. Caching always occurs for cckd files, although you can set the cache value to 1.
sf=file_nameXX Specifies the name of the shadow file(s) for the emulated device. The name should have a spot where the shadow file number can be inserted into the name (see above).
l2cache=n* X Specifies the number of Secondary Lookup Tables (l2tabs) that will be cached for the cckd or shadowed device. (Each l2tab is 2048 bytes). The default is 32.
dfwq=n* X Specifies a threshold for the size of deferred-write-queue where processing will be throttled if the size exceeds this number. Each entry in the deferred- write-queue contains a pointer to a buffer whose size is max-track-size. The default is 64.
wt=n*X Specifies the time in seconds that an updated track image will be written after its last reference. The garbage collector is responsible for scheduling these old track images to be updated. The default is 60 seconds.
ra=n*X Specifies the number of readahead threads (and number of tracks to be read ahead) when sequential access to the emulated device is detected. That is, each track that is read ahead of time is read by a different thread. A value between 0 and 9 can be specified. Currently, readahead should be disabled for Windows32 due to an unknown error involving the pthreads implementation. Default for WIN32 is 0 otherwise the default is 2.
dfw=n*X Specifies the number of deferred write threads. A number between 1 and 9 may be specified; the default is 1. It has not been shown that specifying a greater number results in any performance improvements.
* Option is only applicable if shadowing is active for the regular CKD file.

Generally, the defaults for all options (except sf=) should not be changed unless there is an explicit reason for doing so. If you use cckd files, then I strongly recommend that you start using shadow files. If you use regular CKD files, then you can use shadow files if you want to gain the snapshot benefit .

Utilities

  • ckd2cckd [options] source-file target-file
    • Description Copies a regular CKD Dasd emulation file to a compressed CKD Dasd emulation file. The target file cannot previously exist. If the emulated Dasd device is in more than 1 file then specify the first file. After the copy completes, the target file contains no free space, imbedded or otherwise.
    • Options
      • -compress n
        Compression Algorithm
        • 0 don't compress
        • 1 compress using zlib
        • 2 compress using bzip2
      • -dontcompress n
        Same as -compress 0
      • -maxerrs errs
        Maximum number of errors that can occur before the copy is terminated; if 0 then errors are ignored. Default is 5.
      • -nofudge
        [deprecated]
      • -quiet
        Quiet mode; don't display status
      • -z parm
        Parameter passed to compression

        zlib compression level:
        0 = no compression
        1=fastest ... 9=best

        bzip2 blockSize100k value:
        1=fastest ... 9=best
  • cckd2ckd [options] source-file target-file
    • Description Copies a compressed CKD Dasd emulation file to a regular CKD Dasd emulation file. The target file cannot previously exist. More than 1 target file may be created.
    • Options
      • -cyls n
        Number of cylinders to copy if the entire file isn't to be copied. If 0 then only the number of cylinders in use are copied.
      • -maxerrs errs
        Maximum number of errors that can occur before the copy is terminated; if 0 then errors are ignored. Default is 5.
      • -quiet
        Quiet mode; don't display status
      • -validate
        Validate track images [default]
      • -novalidate
        Don't Validate track images
  • cckdcdsk [-level] file-name
    • Description Performs compressed or shadowed CKD Dasd emulation file integrity verification and recovery and repair.
    • Options
      • -level
        A digit 0, 1 or 3 that specifies the level of checking. The higher the level, the longer the integrity check takes.
        • 0 Minimal checking. Device headers are verified, free space is verified, primary lookup table and secondary lookup tables are verified.
        • 1 Same checks as level 0 plus all 5-byte track headers are verified.
        • 3 Same checks as level 1 plus all track images are read, uncompressed and verified.
  • cckdcomp [-level] file-name
    • Description Removes all free space from a compressed or shadow CKD Dasd emulation file. (Compresses or compacts a cckd file ... your choice!). If level is specified, then cckdcdsk is called first with the specified level; this is a short-hand method to call both functions in one utility call.
    • Options
      • -level
        A digit 0, 1 or 3 that specifies the level of checking. The higher the level, the longer the integrity check takes.
        • 0 Minimal checking. Device headers are verified, free space is verified, primary lookup table and secondary lookup tables are verified.
        • 1 Same checks as level 0 plus all 5-byte track headers are verified.
        • 3 Same checks as level 1 plus all track images are read, uncompressed and verified.
  • cckdfix file-name
    • Description This is a skeleton program that is not compiled during make. It can be edited to change/repair the device headers.
    • Compiling Enter `cc -o cckdfix -DARCH=390 cckdfix.c' to compile and link the edited program.
  • cckddump

    FAQ

    Q. What devices are supported ?
    A. 2311, 2314, 3330, 3340, 3350, 3375, 3380, 3390 and 9345.

    Q. Is a 3390 model 9 supported ?
    A. Yes, maybe. A 3390-9 is a little over 8G in size. A cckd file cannot exceed 2G on a system that does not support large files, otherwise it cannot exceed 4G. If the data on the 3390-9 compresses to below these limits then the answer is Yes.

    Q. How can I get rid of the free space in my files ?
    A. Once the total amount of free space falls below 6% of the total file size, the garbage collector is not very aggressive about eliminating free space. To remove all free space from the file while Hercules is running use the sfc console command. See Using Shadow Files above. Otherwise, you can use the cckdcomp utility. See Utilities above.

    Q. How can I display the space statistics for a compressed file ?
    A. The statistics are displayed when the compressed file is opened. Currently, there is no supplied method to display these statistics at any other time. However, it shouldn't be too hard to write a shell script (similar to dasdlist) to display these statistics. The statistics are contained in the CCKDDASD_DEVHDR which is at offset 512 in the compressed file; the header is mapped in hercules.h.

    Q. What is a "null track" anyway ?
    A. The term "null track" is just something I made up. It is what is returned when a zero offset is found in either the primary or secondary lookup table for the track. It contains the folllowing fields:
    0CCHHHome address
    CCHH0008 00000000standard R0
    CCHH1000end-of-file marker
    ffffffffend-of-track marker
    When a null track is written, space previously occupied by the track is freed and the offset in the secondary lookup table is set to zero. If all offsets in the secondary lookup table are zero, then the secondary lookup table is freed and the primary lookup table entry is zeroed.

    Q. I want to try bzip2 but I'm getting compiler errors. What am I doing wrong ?
    A. Probably bzip2 is not installed or is not installed properly. You can obtain bzip2 from here. If bzip2 is installed, then you need to find the directory where bzlib.h is installed and the directory where libbz2.a is installed. You can then add "-I bzlib.h-directory" to the CFLAGS in the make file and add "-L libbz2.a-directory" to the LFLAGS.

    Q. Which is better, zlib or bzip2 ?
    A. This is a religious question. I have no actual preference, I just wanted to make a choice available.

    Q. Can other compression programs be used ?
    A. Yes. The program is architecturally structured so that other compression algorithms can be added rather painlessly. This will require, of course, an update to the source.

    Q. Can this compression scheme be used for FBA devices too ?
    A. I have not worked with FBA devices for over 20 years. However, it seems to me that a similar program for FBA devices should be simpler than this program for CKD devices (none of those count/key/data fields mucking everything up). Since an FBA block is 512 bytes, it might not be efficient to have each block compressed individually; it might be better to compress blocks in 32K or 64K chunks. If someone asks very nicely, I may consider looking into it;-)

    Changes

    BUGS

    This code is absolutely bug free; if you encounter any problems then it must be a personal problem and you've done something wrong. Also, there are no enhancements that can be made because I've already thought of them all and implemented them. By the way, I have some prime soon to be ocean front property in Tennessee to sell to the highest bidder;-)

    cckddump os/390 hlasm program

    The cckddump program (supplied in file cckddump.hla) is an os/390 assembler language program that creates a compressed CKD Dasd emulation file from a real DASD volume. This program must be APF-authorized since it modifies the DEB to be able to read all tracks from the real device. The program executes 16 or so instructions while in supervisor state/key 0; otherwise the program runs entirely in problem state/key 8. It is not the prettiest assembler language program I've ever written, and there are plenty of enhancements that I originally intended to put into the program that I haven't yet; once I got the program working good enough, I spent the rest of my time writing the fun stuff, the Hercules part.

    The real CKD Dasd volume that is dumped must be an ECKD device (ie support 'Locate Record' and 'Read Track' CCWs); this shouldn't be a problem because I don't think any os/390 release supports a non-ECKD device. The output file must be a DASD file; its characteristics are LRECL=4096, BLKSIZE=4096, RECFM=F. The program only dumps allocated tracks (plus track 0) and only dumps tracks up to DS1LSTAR for DSORG=PS and DSORG=PO files. The program will call zlib to compress the track images if the zlib routines have been linked with the program; however, I don't think the program will be advantageous if it can't call zlib.

    Preparing zlib

    Assemble and linkedit cckddump

    Executing cckddump

    Make the file available to Hercules

    Feedback

    Questions ?? Problems ?? Comments ?? Suggestions ?? Corrections ?? Bugs ??
    Let me know at gsmith@nc.rr.com

    greg smith

    Last updated 1 May 2002