mirror of
https://github.com/SDL-Hercules-390/hyperion.git
synced 2026-05-10 20:31:55 +02:00
37387ec2f6
30 May 2002 Various dasd fixes - Greg Smith git-svn-id: file:///home/jj/hercules.svn/trunk@905 956126f8-22a0-4046-8f4a-272fa8102e63
1097 lines
54 KiB
HTML
1097 lines
54 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 3.0//EN" "html.dtd">
|
|
<HTML>
|
|
<HEAD><TITLE>
|
|
Hercules: Compressed CKD Dasd Emulation</TITLE>
|
|
<LINK REL=STYLESHEET TYPE="text/css" HREF="hercules.css">
|
|
</HEAD>
|
|
<BODY BGCOLOR="#ffffcc" TEXT="#000000" LINK="#0000A0"
|
|
VLINK="#008040" ALINK="#000000">
|
|
<h1>Compressed CKD Dasd Emulation</h1>
|
|
<hr noshade>
|
|
<h2>Contents</h2>
|
|
<ul>
|
|
<li><a href="#introduction"> Introduction </a>
|
|
<li><a href="#shadowfiles"> Shadow Files </a>
|
|
<li><a href="#filestructure"> File Structure </a>
|
|
<li><a href="#methodology"> Methodology </a>
|
|
<li><a href="#quickstart"> Quick Start </a>
|
|
<li><a href="#usingsfiles"> Using Shadow Files </a>
|
|
<li><a href="#options"> Options </a>
|
|
<li><a href="#utilities"> Utilities </a>
|
|
<li><a href="#faq"> FAQ </a>
|
|
<li><a href="#changes"> Changes </a>
|
|
<li><a href="#bugs"> Bugs </a>
|
|
<li><a href="#cckddump"> cckddump os/390 hlasm program</a>
|
|
</ul>
|
|
|
|
<hr noshade>
|
|
<h3><a NAME="introduction">Introduction</a></h3>
|
|
Using compressed CKD files, or <b>cckd</b> files, you can significantly
|
|
reduce the file space required for emulated CKD dasd and possibly gain
|
|
a performance benefit because less physical i/o occurs. Using the
|
|
<i>shadow</i> function, you can minimize the amount of data loss in the
|
|
event of file corruption.
|
|
<p>
|
|
A <b>cckd</b> file contains <i>track images</i>, which may be <i>compressed</i>
|
|
or <i>uncompressed</i>, and overhead blocks which are <i>headers</i>,
|
|
<i>lookup tables</i>, and <i>free space</i>. Compressed track images
|
|
may be compressed by
|
|
<a href="http://www.info-zip.org/pub/infozip/zlib/"><b>zlib</b></a> or
|
|
<a href="http://sourceware.cygnus.com/bzip2/"><b>bzip2</b></a>.
|
|
<p>
|
|
Track images are addressed by <i>track number</i> using a two table
|
|
lookup method: <i>track number</i> divided by 256 (<code>trk >> 8</code>)
|
|
indexes into the <i>primary lookup table</i>, which contains the file offset
|
|
to the <i>secondary lookup table</i>; the remainder of <i>track number</i>
|
|
divided by 256 (<code>trk & 0xff</code>) indexes into the corresponding
|
|
<i>secondary lookup table</i>, which contains the <i>offset</i> and
|
|
<i>length</i> of the track image.
|
|
<p>
|
|
There is a single <i>primary lookup table</i> and a variable number of
|
|
<i>secondary lookup tables</i>. The maximum number of <i>secondary lookup
|
|
tables</i> is the number of tracks for the device type divided by 256, rounded up.
|
|
For example, a 3390-3 contains 50085 tracks and would require at most 196
|
|
<i>secondary lookup tables</i>.
|
|
<p>
|
|
<a NAME="formula1">
|
|
A regular CKD file contains a 512 byte <i>header</i> followed by <i>track
|
|
images</i>, each taking the same amount of space: the maximum track size.
|
|
The offset of a track image can be readily calculated by the track number:
|
|
<center>
|
|
<code>offset = 512 + trk * maxtrksz </code>
|
|
</center>
|
|
</a>
|
|
<p>
|
|
A <b>cckd</b> file can take significantly less file space than a regular
|
|
CKD file because
|
|
<p>
|
|
<ul><li>A track image only occupies its <i>length</i> in the file and
|
|
not the <i>maximum track size</i>
|
|
<li>A track image may be compressed, reducing its <i>length</i>
|
|
<li>Unused or <i>null</i> tracks do not occupy any space at all
|
|
</ul>
|
|
Performance improvements may also occur because less data is read and
|
|
written to the hard drive.
|
|
<br>However, the <i>lookup tables</i> must be
|
|
accurately maintained, <i>track images</i> must be compressed and
|
|
uncompressed, and <i>free space</i> must be kept track of, and dealt
|
|
with by a <i>garbage collector</i>. This results in a more complicated
|
|
file structure, more CPU activity, and the possibility of file corruption
|
|
due to program failure or bug. The introduction of <i>shadow files</i>,
|
|
however, reduces the impact of possible file corruption.
|
|
|
|
<p>
|
|
<hr noshade>
|
|
<p><h3><a NAME="shadowfiles">Shadow Files</a></h3>
|
|
|
|
Malcom Beattie originally introduced the concept of <i>shadow</i>
|
|
files in a <a href="http://www.egroups.com/message/hercules-390/6695">
|
|
post</a> to the <a href="http://www.egroups.com/community/hercules-390">newsgroup</a>
|
|
8 December 2000. The function is actually implemented as a kind of
|
|
<i>snapshot</i>, where a new shadow file can be created on demand.
|
|
A CKD emulated dasd is represented by a <i>base</i> file and 0 or more
|
|
shadow files. All files are opened <i>read-only</i>
|
|
except for the <i>current</i> file, which is opened <i>read-write</i>.
|
|
<p>
|
|
Shadow files are implemented using the same file structure as base <b>cckd</b> files.
|
|
By default, there can be up to 8 shadow files in use at any time for an
|
|
emulated CKD device. The base file is designated file <b>[0]</b> and
|
|
the shadow files are files <b>[1]</b> up to file <b>[8]</b>.
|
|
The <i>highest</i> numbered file in use at a given time is the <i>current</i>
|
|
file, where all writes will occur. Track reads start with the <i>current</i>
|
|
file and proceed down until a file is found that actually contains the track
|
|
image.
|
|
<p>
|
|
A shadow file, then, contains all the changes made to the emulated CKD dasd
|
|
since its creation, until the creation of the next shadow file. The moment
|
|
of the shadow file's creation can be thought of as taking a <i>snapshot</i>
|
|
of the current emulated CKD dasd at that time, because if the shadow file is
|
|
later removed, then the emulated CKD dasd will revert to the state it was at
|
|
when the <i>snapshot</i> was taken.
|
|
<p>
|
|
Using shadow files, you can keep the base CKD file on a read-only device
|
|
such as cdrom, or change the base CKD file attributes to read-only,
|
|
ensuring that this file can never be corrupted.
|
|
<p>
|
|
Hercules console commands are provided to add a new shadow file, remove
|
|
the current shadow file (with or without backward merge), compress the
|
|
curent shadow file, and display the shadow file status and statistics.
|
|
|
|
<p>
|
|
<hr noshade>
|
|
<p><h3><a NAME="filestructure">CCKD File Structure</a></h3>
|
|
|
|
Like a regular CKD emulation file, the first 512 bytes
|
|
of a compressed or shadow file contains a <code>CKDDASD_DEVHDR</code>
|
|
block. The eye-catcher at the beginning is different to distinguish
|
|
the file:
|
|
<ul><li><b>CKD_P370</b>    Regular CKD file
|
|
<li><b>CKD_C370</b>    Compressed CKD file
|
|
<li><b>CKD_S370</b>    Shadow CKD file
|
|
</ul>
|
|
The next 512 bytes contain a compressed device
|
|
header or <code>CCKDDASD_DEVHDR</code> block. This contains the
|
|
version-release-mod level of the file, options, space statistics,
|
|
and total number of cylinders for the device. Next is
|
|
the primary lookup table or the <code>L1TAB</code>. Each 4 byte
|
|
entry in the <code>L1TAB</code> contains the file offset to a secondary
|
|
lookup table (or <code>L2TAB</code>) or <code>0x00000000</code> (indicating that
|
|
the secondary lookup table is <i>null</i>), or <code>0xffffffff</code> (indicating
|
|
that the previous file should be searched instead).<br>
|
|
The size of the <code>L1TAB</code> is dependent on the number
|
|
of tracks on the emulated device.
|
|
<p>
|
|
<center>
|
|
<table border=1>
|
|
<tr><td align="left"><code>CKDDASD_DEVHDR</code><br><br><br><br></td>
|
|
<tr><td align="left"><code>CCKDDASD_DEVHDR</code><br><br><br><br></td>
|
|
<tr><td align="left"><code>L1TAB</code><br><br>
|
|
<center>.  .  .</center><br><br></td>
|
|
</table>
|
|
</center>
|
|
<p>
|
|
Following the <code>L1TAB</code>,
|
|
in no particular order, are <code>L2TAB</code>s, compressed track
|
|
images, and free spaces.
|
|
<p>
|
|
<code><b>L2TAB</b></code>s contain 256 8-byte
|
|
entries,and each are, consequently, 2048 bytes in length. Each entry
|
|
contains the <i>offset</i> and <i>length</i> of a track image. If
|
|
the <i>offset</i> is <code>0x00000000</code> then the track image is <i>null</i>;
|
|
if the <i>offset</i> is <code>0xffffffff</code> then the previous file should be
|
|
searched instead.
|
|
<p>
|
|
<center><code><b>L2TAB</b></code> entry
|
|
<table border=1>
|
|
<tr><td><code><b>offset</b></code><br>4 bytes</td>
|
|
<td><code><b>length</b></code><br>2 bytes</td>
|
|
<td><code>[unused]</code><br>2 bytes</td>
|
|
</table>
|
|
</center>
|
|
<p>
|
|
A <b>compressed track image</b> contains the following two fields:
|
|
<ul>
|
|
<li> A <i>track header</i>, also called a <i>home address</i> or a <i>track index</i>.
|
|
The track header is <b>never</b> compressed.
|
|
<li> The track image, beginning with the <i>R0 count</i> and ending with the
|
|
<i>end-of-track</i> marker, which is a count field containing all hex 0xff's.
|
|
The track image may or may not be compressed.
|
|
</ul>
|
|
<center><table border=1>
|
|
<tr><td><code><b>HA</b></code><br>5 bytes</td>
|
|
<td><b>track image</b> (compressed or uncompressed)</br><i>length</i>-5 bytes</td>
|
|
</table></center>
|
|
<p>
|
|
The <b>HA</b> contains <code>0CCHH</code>, that is, a byte of zeroes, 2 bytes indicating
|
|
the <i>cylinder</i> of the track, and 2 bytes indicating the <i>head</i> of the track
|
|
on the cylinder. Both <code>CC</code> and <code>HH</code> are stored in
|
|
<i>big-endian</i> byte order. The track is computed by
|
|
<br><center><code>
|
|
trk = (((CC[0] << 8) + CC[1]) * <i>trks_per_cyl</i>) + (HH[0] << 8) + HH[1]
|
|
</code></center><br>
|
|
Since the first byte of the <code>HA</code> is always 0x00 (at least in emulated
|
|
CKD files), this byte as stored in the file actually indicates the compression
|
|
algorithm used for the remainder of the track image
|
|
(<b>0</b> = no compression, <b>1</b> = zlib compression, <b>2</b> = bzip2 compression).
|
|
The hi-order bit (<b>0x80</b>) may also be on, indicating the track image was written
|
|
under <i>stress</i>.
|
|
<p>
|
|
<b>Free space</b> contains a 4-byte <i>offset</i> to the next free space,
|
|
a 4-byte <i>length</i> of the free space, and zero or more bytes of residual data.
|
|
<p>
|
|
<center><b>Free Space</b> entry
|
|
<table border=1>
|
|
<tr><td><code><b>offset</b></code><br>4 bytes</td>
|
|
<td><code><b>length</b></code><br>4 bytes</td>
|
|
<td><code><b>residual</b></code><br>(<i>length</i> - 8) bytes</td>
|
|
</table>
|
|
</center>
|
|
<p>The minimum length of a free space is 8 bytes.
|
|
Since free space is ordered by file offset and no two free spaces are adjacent,
|
|
<i>offset</i> in the free space entry is always greater than the current free space
|
|
offset + the current free space <i>length</i>, unless the <i>offset</i> is zero,
|
|
which indicates the free space chain is terminated.<br>
|
|
The free space chain is read when the file is opened for read-write and
|
|
written when the file is closed; while the file is opened, the free space chain is
|
|
maintained in storage.
|
|
|
|
<p>
|
|
<hr noshade>
|
|
<p><h3><a NAME="methodology">Methodology</a></h3>
|
|
This section is tedious; you probably want to skip to the
|
|
<a href="#quickstart">next section</a> unless you are genuinely
|
|
curious as to how cckd actually works. This section is for my edification
|
|
as much as anything...
|
|
<p>
|
|
<b>The interface</b>
|
|
<p>
|
|
All Hercules emulated devices have standard interface routines described
|
|
by the <code>DEVHND</code> structure. These routines are
|
|
<table>
|
|
<tr><td><b>init</b></td><td>Device initialization (attach)</td>
|
|
<tr><td><b>exec</b></td><td>Device execute channel command</td>
|
|
<tr><td><b>close</b></td><td>Device close (detach)</td>
|
|
<tr><td><b>query</b></td><td>Device query</td>
|
|
<tr><td><b>start</b></td><td>Device channel program start</td>
|
|
<tr><td><b>end</b></td><td>Device channel program end</td>
|
|
<tr><td><b>resume</b></td><td>Device channel program resume</td>
|
|
<tr><td><b>suspend</b></td><td>Device channel program suspend</td>
|
|
</table>
|
|
<p>
|
|
Emulated CKD DASD devices have two additional routines:
|
|
<table>
|
|
<tr><td><b>ckdrdtrk</b></td><td>Read CKD track image</td>
|
|
<tr><td><b>ckdupdtrk</b></td><td>Update CKD track image</td>
|
|
</table>
|
|
<code>ckdrdtrk</code> is called whenever a <code>SEEK</code>
|
|
type channel command is executed and <code>ckdupdtrk</code> is
|
|
called whenever any of the various <code>WRITE</code> channel
|
|
commands modifies the current track image.
|
|
<p>
|
|
The <b>cckd</b> code implements the following routines:
|
|
<table>
|
|
<tr><td><code>cckddasd_init_handler</code></td>
|
|
<td>cckd device initialization (attach)</td>
|
|
<tr><td><code>cckddasd_close_device</code></td><td>cckd device close (detach)</td>
|
|
<tr><td><code>cckd_start</code></td><td>cckd channel program start/resume</td>
|
|
<tr><td><code>cckd_end</code></td><td>cckd channel program end/suspend</td>
|
|
</table>
|
|
<p>
|
|
<b>The cache</b>
|
|
<p>
|
|
The implementation is <i>cache</i> table driven. The cache table contains
|
|
16 to 1024 entries, each representing a track image, and is shared by <em>all</em>
|
|
cckd devices. Each entry contains a pointer to a track image buffer which
|
|
is 64K bytes. That is, every 16 entries represents 1M of buffer space.
|
|
The size of the cache can be changed by a configuration file statement
|
|
or by a hercules console command.
|
|
<p>
|
|
Each entry can be in one (or certain combinations) of the following states:
|
|
<table>
|
|
<tr><td><b>active</b></td><td>A channel program is active for the device
|
|
and the track image is the last <code>SEEK</code>ed
|
|
track image.</td>
|
|
<tr><td><b>updated</b></td><td>The track image has been updated by a
|
|
<code>WRITE</code> command.</td>
|
|
<tr><td><b>write</b></td><td>The track image is not <b>active</b> and has
|
|
been <b>updated</b> and is eligible to be written.</td>
|
|
<tr><td><b>reading</b></td><td>The track image is currently being read.</td>
|
|
<tr><td><b>writing</b></td><td>The track image is currently being written.</td>
|
|
<tr><td><b>inactive</b></td><td>The track image is not <i>busy</i> and is eligible
|
|
to be <i>stolen</i></td>
|
|
</table>
|
|
When the <code>ckdrdtrk</code> routine is called, and the track to be
|
|
read is not the <em>active</em> track, then a <em>track switch</em> event
|
|
occurs. The cache is scanned to see if it contains the track image. If
|
|
it does, then a <em>cache hit</em> occurs else a <em>cache miss</em>
|
|
occurs.
|
|
<p>
|
|
In the case of a <em>cache hit</em>, if the entry status is <em>reading</em>
|
|
or <em>writing</em> then the code waits for the read or write to complete.
|
|
If the entry is <em>write</em>-pending, then the entry is changed back
|
|
to <em>updated</em>.
|
|
<p>
|
|
Otherwise, for a <em>cache miss</em>, the oldest <em>inactive</em> cache
|
|
entry will then be <em>stolen</em>.
|
|
If no inactive cache entry was found then the cache is <em>flushed</em>
|
|
and the code will have to wait for an inactive cache entry to become
|
|
available. This is called a <em>cache wait</em> event. When the cache
|
|
is flushed, all cache entries that have been <em>updated</em> and are
|
|
not <em>active</em> are set to <em>write</em>-pending and the <b>writer</b>
|
|
thread is signalled. After an entry has been stolen the track image will
|
|
then be read and uncompressed.
|
|
If sequential access is detected then track <em>readaheads</em> will be
|
|
scheduled.
|
|
<p>
|
|
<b>L1 tables</b>
|
|
<p>
|
|
The L1 tables, or <i>primary lookup</i> tables, contain the file offsets
|
|
to each <b>L2 table</b>, or <i>secondary lookup</i> table. Each L1 table
|
|
4-byte entry represents 256 track images and there is an L1 table for the base
|
|
file and each opened shadow file (if any). These tables are memory resident.
|
|
<p>
|
|
<b>The L2 cache</b>
|
|
<p>
|
|
The L2 tables, or <i>secondary lookup</i> tables contain the file offsets
|
|
and lengths of 256 track images. Each table is 2K in length. These tables
|
|
are cached in the <em>l2cache</em>, which is shared for all cckd devices.
|
|
The cache can contain 128 (512K) entries to 1024 entries (2M). Each cckd
|
|
device has one active l2cache entry implying that, architecturally speaking,
|
|
there can be no more than 1024 cckd devices. If this is a problem for
|
|
you then I'll buy you a beer ;-)
|
|
<p>
|
|
The size of the l2cache can be specified by a configuration statement or
|
|
changed by a hercules console command. If a cache miss occurs and no
|
|
inactive entries are found, then the cache is automatically extended
|
|
(unless the 1024 entry barrier would be crossed).
|
|
<p>
|
|
L2 tables are read but never written. Instead, as track images are
|
|
written, the individual 8-byte L2 entries are updated in the file.
|
|
<p>
|
|
<b>Writing</b>
|
|
<p>
|
|
Whenever the <code>ckdupdtrk</code> is called, indicating the active
|
|
track image has been updated, the <em>updated</em> flag bit is turned
|
|
on for the cache entry. However, the updated cache entries are never
|
|
scheduled to be written until a <em>cache flush</em> occurs (routine
|
|
<code>cckd_flush_cache</code>). A cache flush occurs whenever a cache
|
|
entry couldn't be <em>stolen</em> during read or at the beginning of
|
|
the <em>garbage collection</em> cycle.
|
|
<p>
|
|
When a <em>cache flush</em> occurs, all cache entries that are <em>updated</em>
|
|
(but not <em>active</em>) get the <em>write</em>-pending bit turned on
|
|
and one of the <b>writer</b> threads is signalled (or created).
|
|
<p>
|
|
The <em>writer</em> thread selects the oldest cache entry with the
|
|
<em>write</em>-pending bit on and changes the status to <em>writing</em>.
|
|
If there are other writes pending and other writer threads are waiting then
|
|
they are signalled. Otherwise if there are writes pending and the maximum
|
|
number of writer threads hasn't been created yet, then a new writer thread
|
|
is created.
|
|
<p>
|
|
The writer thread, because it performs track image compression, is cpu
|
|
intensive. Therefore, the writer thread resets its priority to 1 below
|
|
the cpu thread(s) (except under Cygwin (Windows)). In periods of
|
|
<em>write stress</em> then the level of compression is downgraded.
|
|
Write stress occurs when more than a quarter of the cache entries are
|
|
pending write, or when there are threads waiting for a cache entry, or
|
|
when a thread is waiting for this track image to be written. When a
|
|
track image is written under stress, a bit is turned on in the track
|
|
image so that the <em>garbage collector</em> can later rewrite the
|
|
track image properly compressed during a period of no stress.
|
|
<p>
|
|
A track image is always written to a new location in the file.
|
|
After a successful write, its L2 entry is updated and then the
|
|
space it previously occupied is freed.
|
|
<p>
|
|
The <em>writer</em> threads
|
|
<p>
|
|
<b>The Garbage Collector</b>
|
|
<p>
|
|
For some reason, this seems to be everyone's favorite routine.
|
|
For me, it has to do with the fact that the garbage man only has
|
|
to apparently work on Fridays ;-).
|
|
<p>
|
|
Simply, the garbage collector moves occupied spaces (track images
|
|
and level 2 tables) towards the beginning of the file while moving
|
|
free spaces towards the end of the file. When a free space reaches
|
|
the end of the file, then the file is <em>truncated</em>, that is,
|
|
reduced in size. Another important job the garbage collector performs
|
|
is to combine free spaces to make a larger free space that will be
|
|
more likely to satisfy a <i>get space</i> request. Finally, the
|
|
garbage collector will cause track images that were written under
|
|
stress to be rewritten (with optimal compression).
|
|
<p>
|
|
The garbage collector runs in a separate thread. There is a single
|
|
garbage collector for all cckd devices. The collector waits for
|
|
some number of seconds and then performs collection for each cckd
|
|
device in turn. The wait interval and the aggressiveness parameter
|
|
can be specified by a configuration file statement or by a hercules
|
|
console command.
|
|
<p>
|
|
The garbage collector algorithm simply selects an occupied space after
|
|
some free space, obtains a new space for that space, writes it to the
|
|
new space, updates the meta-data for that space to point to the new
|
|
space, frees the previously occupied space, and repeats the process
|
|
until some number of bytes have been moved. In other words, the collector
|
|
shifts the spaces around <i>coherently</i> such that in the event of
|
|
a crash then the space occupies at least one space in the file
|
|
(the original and possibly the new).
|
|
<p>
|
|
The garbage collector on each interval (or round ;-) flushes the cache
|
|
and waits for all pending writes to complete. Therefore, the collector
|
|
doesn't run while <em>write stress</em> is occurring.
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="quickstart">Quick Start</a></h3>
|
|
|
|
The <i>ckd2cckd</i> utility can be used to create a new compressed CKD file from a
|
|
regular CKD file. Your disk images can be a combination of regular CKD files and
|
|
compressed CKD files. Simply specify the names of your new compressed ckd files in
|
|
<code>hercules.cnf</code> in place of the regular CKD file names.
|
|
<p>
|
|
You can also use the <i>cckddump</i> program on an os/390 system to build a
|
|
compressed CKD file from a real disk that can be transferred to your Hercules machine
|
|
and used right away.
|
|
<p>
|
|
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="usingsfiles">Using Shadow Files</a></h3>
|
|
|
|
Shadow files enable you to make updates to <b>cckd</b> emulation files and
|
|
not worry about possibly corrupting your entire disk image. I strongly urge
|
|
those of you who use <b>cckd</b> to start using shadow files immediately and
|
|
change your base file to read-only. This, in turn, reduces the amount of
|
|
data you have to back up, increasing the amount of file savings cckd has to
|
|
offer. You can even change shadow files to read-only, as long as a new shadow
|
|
file can be created. You can also use shadow files for regular (non-cckd)
|
|
files.
|
|
|
|
<p>
|
|
|
|
Shadow files are automatically enabled for <b>cckd</b> files; you <em>must</em>
|
|
explicitly enable them for regular CKD files. To enable shadowing for a CKD
|
|
device, specify
|
|
<p><center>sf=<i>shadow_file_name</i></center><br>
|
|
on the device statement in the <i>hercules.cnf</i> file. <i>shadow_file_name</i>
|
|
should include a spot in the file name, similar to multiple CKD dasd files,
|
|
that can be used as a sequence number, for example, <b>sf=../mvs/shadows/mvsres_1.500</b>.
|
|
The naming convention substitutes the shadow file number (1 thru 8) on the
|
|
character preceding the period after the last slash, or the last character
|
|
if no period follows the last slash. Example <p>
|
|
<b>0500 3390 ../mvs/disks/mvsres.500 sf=../mvs/shadows/mvsres_1.500</b>
|
|
<p>
|
|
If you did not specify <i>sf=</i> for a cckd file, or you wish to change
|
|
the shadow file name for a cckd or regular file, but no shadow files are
|
|
in use, then you can issue the following command on the Hercules console:
|
|
<br><center>sf=<i>xxxx</i>   <i>shadow_file_name</i></center><br>
|
|
where <i>xxxx</i> is the device unit address. For example,
|
|
<b>sf=0500 ../mvs/shadows/mvsres_1.500</b>.
|
|
<p>
|
|
Specifying a <i>shadow_file_name</i> does not explicitly create a shadow file
|
|
if the base file or current shadow file
|
|
is able to be opened read-write. Otherwise, if the base file and all existing
|
|
shadow files (if any) can only be opened read-only, then a new shadow file is created.
|
|
<p>
|
|
To explicitly create a new shadow file, issue the following command on the Hercules console:
|
|
<p><center>sf+<i>xxxx</i></center><br>
|
|
where <i>xxxx</i> is the device unit address or <b>*</b> (for all eligible units).
|
|
For example, <b>sf+0500</b>.
|
|
All updated track images that haven't been written are written and the <i>current</i>
|
|
file is hardened. Note that if a lot of write activity is ocurring at the time the
|
|
<b>sf+</b> command is entered, then the exact state of the hardened file can not
|
|
be predicted. A new shadow file is created and all new writes are directed to it.
|
|
<p>
|
|
To remove the <i>current</i> shadow file, issue either of the following commands
|
|
on the Hercules console:
|
|
<p><center>sf-<i>xxxx</i></center>
|
|
<center>sf-<i>xxxx</i> nomerge</center><br>
|
|
where <i>xxxx</i> is the device unit address or <b>*</b> (for all eligible units).
|
|
For example, <b>sf-0500</b>.
|
|
If <b>nomerge</b> was not specified, then the <i>current</i> shadow file contents
|
|
are merged into the preceding shadow file or base file. The <i>current</i> shadow
|
|
file is deleted and the preceding shadow file or base file is made the <i>current</i>
|
|
file. If the preceding file is read-only, then an error message is issued.
|
|
If possible, you can make the preceding file read-write and re-issue the command.
|
|
Note that if <b>merge</b> is specified or implied, then the command may take some
|
|
amount of time depending on the size of the old <i>shadow</i> file.
|
|
<br>[<i>hmmm... note to myself -- if sf-xxxx nomerge was specified and preceding file is
|
|
read-only, then delete the current file and recreate it ??</i>]
|
|
<p>
|
|
To compress the current shadow file
|
|
issue the following command on the Hercules console:
|
|
<p><center>sfc<i>xxxx</i></center><br>
|
|
where <i>xxxx</i> is the device unit address or <b>*</b> (for all eligible units).
|
|
For example, <b>sfc0500</b>.
|
|
<p>
|
|
To display the status and statistics for a shadow-enabled file,
|
|
issue the following command on the Hercules console:
|
|
<p><center>sfd<i>xxxx</i></center><br>
|
|
where <i>xxxx</i> is the device unit address or <b>*</b> (for all eligible units).
|
|
For example, <b>sfd0500</b>.
|
|
This command displays status and statistics for the base file and all shadow files
|
|
representing the emulated dasd. The following data is displayed:
|
|
<table>
|
|
<tr><td>  <td valign="top"><b>size</b> <td>The total size of the file
|
|
<tr><td>  <td valign="top"><b>free</b> <td>The amount of free space in the file as a percentage
|
|
of the file size
|
|
<tr><td>  <td valign="top"><b>nbr</b> <td>The number of free spaces in a file
|
|
<tr><td>  <td valign="top"><b>st</b> <td>File open status - <b>ro</b>=read-only;
|
|
<b>rd</b>=read-only, but can be opened read-write;
|
|
<b>rw</b>=read-write
|
|
<tr><td>  <td valign="top"><b>reads</b> <td>Number of times <i>cckd_read_trkimg</i> performed
|
|
physical read i/o
|
|
<tr><td>  <td valign="top"><b>writes</b> <td>Number of times <i>cckd_write_trkimg</i> performed
|
|
physical write i/o
|
|
<tr><td>  <td valign="top"><b>l2reads</b> <td>Number of times a secondary lookup table was read
|
|
<tr><td>  <td valign="top"><b>hits</b> <td>Number of times <i>cckd_read_trk</i> found a track
|
|
image in the <i>track cache</i> when called by the
|
|
i/o thread
|
|
<tr><td>  <td valign="top"><b>switches</b><td>Number of times <i>cckd_read_trk</i> was called by
|
|
the i/o thread (<i>cckd_lseek</i>)
|
|
<tr><td>  <td valign="top"><b>readaheads</b><td>Number of track images read by the <i>readahead</i>
|
|
threads
|
|
<tr><td>  <td valign="top"><b>misses</b> <td>Number of track images read by the <i>readahead</i>
|
|
threads that were never referenced when the
|
|
<i>track cache</i> entry was <i>stolen</i>
|
|
</table>
|
|
|
|
<p>
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="options">CKD Options</a></h3>
|
|
|
|
In this section I will attempt to document all the options that can be
|
|
specified for a CKD file (regular or compressed) in the <code>hercules.cnf</code>
|
|
file (or on the <i>attach</i> panel command).
|
|
|
|
<table border=1>
|
|
<th><td>Regular<td>cckd<td>Function
|
|
|
|
<tr><td valign="top" align="left"> syncio<br> syio<br>
|
|
nosyncio<br>nosyio<br><td align="center">X<td align="center"> X
|
|
<td>Specifies whether or not synchronous I/O will be attempted for the device.
|
|
For synchronous I/O, the channel program will be executed within the scope
|
|
of the <b>SIO</b> or <b>SSCH</b> instruction as long as all data referenced
|
|
by the channel program is already cached. If a ccw attempts to reference data
|
|
that is not cached, then the channel program is restarted asynchronously at
|
|
that ccw. Synchronous I/O reduces threading overhead, which may resut in a
|
|
performance boost. The default is <i>syncio</i> for cckd files and <i>nosyncio</i>
|
|
for regular ckd files.
|
|
<tr><td valign="top" align="left">lazywrite<br>nolazywrite<td align="center">X<td>  
|
|
<td>Data written to a cached track image will not be immediately written,
|
|
but will be written when a track switch occurs. Thus, only one write
|
|
will occur for a track image while it is the <i>active</i> image.
|
|
<i>nolazywrite</i>, the default, specifies that all writes are performed
|
|
when requested.
|
|
<tr><td valign="top" align="left"> fulltrackio<br> fulltrkio<br>ftio<br>
|
|
nofulltrackio<br>nofulltrkio<br>noftio<td align="center">X<td>  
|
|
<td>Specifies whether or not a full track will be read when a track switch
|
|
occurs. Subsequent reads to this track image will not cause any physical I/Os.
|
|
Turning on <i>fulltrackio</i> can considerably enhance CKD device response time.
|
|
However, if you are sharing CKD disk images with more than 1 instance of Hercules
|
|
at the same time when writes could occur, you should specify <i>nofulltrackio</i>.
|
|
The default is <i>fulltrackio</i>.
|
|
<tr><td valign="top" align="left">readonly<br>rdonly<br>ro<br><td align="center">X<td align="center"> X
|
|
<td>Causes the CKD file image to be opened <i>read-only</i>. Attempts to write to
|
|
the emulated device will cause an I/O error unless option <i>fakewrite</i> is
|
|
also specified. If <i>readonly</i> is specified for shadowed
|
|
file images, then the base file will be opened readonly and a shadow file will be
|
|
created if one doesn't exist.
|
|
<tr><td valign="top" align="left">fakewrite<br>fakewrt<br>fw<br>
|
|
<td align="center">X<td align="center"> X
|
|
<td>Writes to a <i>readonly</i> file will be considered successful even though no write
|
|
actually occurred. This option is only meaningful if <i>readonly</i> is also specified.
|
|
<i>Fakewrite</i> is ignored for shadowed file images.
|
|
<tr><td valign="top" align="left">cache=<i>n</i><td align="center">X<td align="center">X
|
|
<td>Specifies the number of track images that will be cached. The default is the number
|
|
of tracks per cylinder for the device. [For <b>cckd</b> files, the default is the
|
|
number of tracks per cylinder <i>plus</i> the number of readahead threads]. If
|
|
<i>nofulltrackio</i> is specified for a regular CKD file, then no caching occurs.
|
|
Caching <em>always</em> occurs for <b>cckd</b> files, although you can set the cache
|
|
value to 1.
|
|
<tr><td valign="top" align="left">sf=<i>file_name</i><td align="center">X<td align="center">X
|
|
<td>Specifies the name of the shadow file(s) for the emulated device. The name should have
|
|
a spot where the shadow file number can be inserted into the name (see
|
|
<a href="#usingsfiles">above</a>).
|
|
<tr><td valign="top" align="left">l2cache=<i>n</i><td align="center">* <td align="center">X
|
|
<td>Specifies the number of Secondary Lookup Tables (l2tabs) that will be cached for the
|
|
<b>cckd</b> or <i>shadowed</i> device. (Each l2tab is 2048 bytes). The default is 32.
|
|
<tr><td valign="top" align="left">dfwq=<i>n</i><td align="center">* <td align="center">X
|
|
<td>Specifies a threshold for the size of <i>deferred-write-queue</i> where processing will be
|
|
<i>throttled</i> if the size exceeds this number. Each entry in the <i>deferred-
|
|
write-queue</i> contains a pointer to a buffer whose size is <i>max-track-size</i>.
|
|
The default is 64.
|
|
<tr><td valign="top" align="left">wt=<i>n</i><td align="center">*<td align="center">X
|
|
<td>Specifies the time in seconds that an updated track image will be written after its
|
|
last reference. The <i>garbage collector</i> is responsible for scheduling these
|
|
old track images to be updated. The default is 60 seconds.
|
|
<tr><td valign="top" align="left">ra=<i>n</i><td align="center">*<td align="center">X
|
|
<td>Specifies the number of <i>readahead</i> threads (and number of tracks to be read ahead)
|
|
when sequential access to the emulated device is detected. That is, each track that
|
|
is read ahead of time is read by a different thread. A value between 0 and 9 can be
|
|
specified. Currently, readahead should be <em>disabled</em> for Windows32 due to
|
|
an unknown error involving the pthreads implementation. Default for WIN32 is 0
|
|
otherwise the default is 2.
|
|
<tr><td valign="top" align="left">dfw=<i>n</i><td align="center">*<td align="center">X
|
|
<td>Specifies the number of <i>deferred write</i> threads. A number between 1 and 9
|
|
may be specified; the default is 1. It has not been shown that specifying a greater
|
|
number results in any performance improvements.
|
|
</table>
|
|
<font size=-1><b> * </b>Option is only applicable if shadowing is active for the regular CKD file.
|
|
</font>
|
|
<p>
|
|
Generally, the defaults for all options (except <b>sf=</b>) should not be changed unless there
|
|
is an explicit reason for doing so. If you use <b>cckd</b> files, then I strongly recommend
|
|
that you start using <i>shadow</i> files. If you use regular CKD files, then you can use
|
|
<i>shadow</i> files if you want to gain the <i>snapshot</i> benefit .
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="utilities">Utilities</a></h3>
|
|
|
|
<a NAME="ckd2cckd">
|
|
<li><b>ckd2cckd</b> <i>[options] source-file target-file</i>
|
|
<ul><li><small><b>Description</b></small> Copies a regular CKD Dasd emulation
|
|
file to a compressed CKD Dasd emulation file. The target
|
|
file cannot previously exist. If the emulated Dasd device
|
|
is in more than 1 file then specify the <em>first</em> file.
|
|
After the copy completes, the target file contains no
|
|
free space, imbedded or otherwise.
|
|
<li><small><b>Options</b></small>
|
|
<ul><li><b>-c</b>ompress <i>n</i><br>Compression Algorithm
|
|
<ul><li><b>0</b> don't compress
|
|
<li><b>1</b> compress using zlib
|
|
<li><b>2</b> compress using bzip2
|
|
</ul>
|
|
<li><b>-d</b>ontcompress <i>n</i><br>Same as <i>-compress 0</i>
|
|
<li><b>-m</b>axerrs <i>errs</i><br>Maximum number of errors
|
|
that can occur before the copy is terminated;
|
|
if 0 then errors are ignored. Default is 5.
|
|
<li><b>-n</b>ofudge<br>[deprecated]
|
|
<li><b>-q</b>uiet<br>Quiet mode; don't display status
|
|
<li><b>-z</b> <i>parm</i><br>Parameter passed to compression
|
|
<br>
|
|
<br>zlib compression level:
|
|
<br>0 = no compression
|
|
<br>1=fastest ... 9=best
|
|
<br>
|
|
<br>bzip2 blockSize100k value:
|
|
<br>1=fastest ... 9=best
|
|
</ul>
|
|
</ul>
|
|
</ul>
|
|
<a NAME="cckd2ckd">
|
|
<li><b>cckd2ckd</b> <i>[options] source-file target-file</i>
|
|
<ul><li><small><b>Description</b></small> Copies a compressed CKD Dasd emulation
|
|
file to a regular CKD Dasd emulation file. The target
|
|
file cannot previously exist. More than 1 target file may
|
|
be created.
|
|
<li><small><b>Options</b></small>
|
|
<ul><li><b>-c</b>yls <i>n</i><br>Number of cylinders to copy
|
|
if the entire file isn't to be copied. If <b>0</b>
|
|
then only the number of cylinders in use are copied.
|
|
<li><b>-m</b>axerrs <i>errs</i><br>Maximum number of errors
|
|
that can occur before the copy is terminated;
|
|
if 0 then errors are ignored. Default is 5.
|
|
<li><b>-q</b>uiet<br>Quiet mode; don't display status
|
|
<li><b>-v</b>alidate<br>Validate track images [default]
|
|
<li><b>-n</b>ovalidate<br>Don't Validate track images
|
|
</ul>
|
|
</ul>
|
|
</ul>
|
|
<a NAME="cckdcdsk">
|
|
<li><b>cckdcdsk</b> <i>[-level] file-name</i>
|
|
<ul><li><small><b>Description</b></small> Performs compressed or shadowed CKD Dasd emulation
|
|
file integrity verification and recovery and repair.
|
|
<li><small><b>Options</b></small>
|
|
<ul><li>-<i>level</i><br>A digit 0, 1 or 3 that specifies
|
|
the level of checking. The higher the level, the
|
|
longer the integrity check takes.
|
|
<ul><li><b>0</b> Minimal checking. Device headers are verified,
|
|
free space is verified, primary lookup table and secondary
|
|
lookup tables are verified.
|
|
<li><b>1</b> Same checks as level 0 plus all 5-byte track headers
|
|
are verified.
|
|
<li><b>3</b> Same checks as level 1 plus all track images are
|
|
read, uncompressed and verified.
|
|
</ul>
|
|
</ul>
|
|
</ul>
|
|
</ul>
|
|
<a NAME="cckdcomp">
|
|
<li><b>cckdcomp</b> <i>[-level] file-name</i>
|
|
<ul><li><small><b>Description</b></small> Removes all free space from a compressed
|
|
or shadow CKD Dasd emulation file. (Compresses or compacts a cckd
|
|
file ... your choice!).
|
|
If <i>level</i> is specified, then <b>cckdcdsk</b> is called first
|
|
with the specified level; this is a short-hand method to call both
|
|
functions in one utility call.
|
|
<li><small><b>Options</b></small>
|
|
<ul><li>-<i>level</i><br>A digit 0, 1 or 3 that specifies
|
|
the level of checking. The higher the level, the
|
|
longer the integrity check takes.
|
|
<ul><li><b>0</b> Minimal checking. Device headers are verified,
|
|
free space is verified, primary lookup table and secondary
|
|
lookup tables are verified.
|
|
<li><b>1</b> Same checks as level 0 plus all 5-byte track headers
|
|
are verified.
|
|
<li><b>3</b> Same checks as level 1 plus all track images are
|
|
read, uncompressed and verified.
|
|
</ul>
|
|
</ul>
|
|
</ul>
|
|
</ul>
|
|
<a NAME="cckdfix">
|
|
<li><b>cckdfix</b> <i>file-name</i>
|
|
<ul><li><small><b>Description</b></small> This is a skeleton program that is
|
|
not compiled during make. It can be edited to change/repair
|
|
the device headers.
|
|
<li><small><b>Compiling</b></small> Enter `<i>cc -o cckdfix -DARCH=390 cckdfix.c</i>'
|
|
to compile and link the edited program.
|
|
</ul>
|
|
<li><b>cckddump</b>
|
|
<ul><li><small><b>Description</b></small> This is an os/390 hlasm (High Level
|
|
Assembler) program that will create a compressed CKD emulation file
|
|
from an actual CKD device. See <a href="#cckddump">below</a> for
|
|
a description on how to build and run this program.
|
|
</ul>
|
|
</ul>
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="faq">FAQ</a></h3>
|
|
<table>
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
What devices are supported ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
2311, 2314, 3330, 3340, 3350, 3375, 3380, 3390 and 9345.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
Is a 3390 model 9 supported ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
Yes, maybe. A 3390-9 is a little over 8G in size.
|
|
A cckd file cannot exceed 2G on a system that does
|
|
not support large files, otherwise it cannot exceed
|
|
4G. If the data on the 3390-9 compresses to below
|
|
these limits then the answer is Yes.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
How can I get rid of the free space in my files ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
Once the total amount of free space falls below 6% of
|
|
the total file size, the garbage collector is not very
|
|
aggressive about eliminating free space. To remove
|
|
all free space from the file while Hercules is running
|
|
use the <b>sfc</b> console command. See
|
|
<a href="#usingsfiles">Using Shadow Files</a> above.
|
|
Otherwise, you can use the <b>cckdcomp</b> utility.
|
|
See <a href="#utilities">Utilities</a> above.
|
|
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
How can I display the space statistics for a compressed
|
|
file ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
The statistics are displayed when the compressed file
|
|
is opened. Currently, there is no supplied method to
|
|
display these statistics at any other time. However,
|
|
it shouldn't be too hard to write a shell script
|
|
(similar to <code>dasdlist</code>) to display these
|
|
statistics. The statistics are contained in the
|
|
<code>CCKDDASD_DEVHDR</code> which is at offset 512
|
|
in the compressed file; the header is mapped in
|
|
<code>hercules.h</code>.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
What is a "null track" anyway ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
The term "null track" is just something I made up. It is
|
|
what is returned when a zero offset is found in either the
|
|
primary or secondary lookup table for the track. It contains
|
|
the folllowing fields:
|
|
<table>
|
|
<tr><td><code>0CCHH</code></td><td>Home address</td>
|
|
<tr><td><code>CCHH0008 00000000</code></td><td>standard R0</td>
|
|
<tr><td><code>CCHH1000</code></td><td>end-of-file marker</td>
|
|
<tr><td><code>ffffffff</code></td><td>end-of-track marker</td>
|
|
</table>
|
|
When a null track is written, space previously occupied by
|
|
the track is freed and the offset in the secondary lookup table
|
|
is set to zero. If all offsets in the secondary lookup table
|
|
are zero, then the secondary lookup table is freed and the
|
|
primary lookup table entry is zeroed.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
I want to try bzip2 but I'm getting compiler errors.
|
|
What am I doing wrong ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
Probably bzip2 is not installed or is not installed
|
|
properly. You can obtain bzip2 from
|
|
<a href="http://sourceware.cygnus.com/bzip2/">here</a>.
|
|
If bzip2 is installed, then you need to find the directory
|
|
where <code>bzlib.h</code> is installed and the
|
|
directory where <code>libbz2.a</code> is installed.
|
|
You can then add "-I <i>bzlib.h-directory</i>" to the
|
|
CFLAGS in the make file and add "-L <i>libbz2.a-directory</i>"
|
|
to the LFLAGS.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
Which is better, zlib or bzip2 ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
This is a religious question. I have no actual preference,
|
|
I just wanted to make a choice available.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
Can other compression programs be used ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
Yes. The program is architecturally structured so that other
|
|
compression algorithms can be added rather painlessly. This
|
|
will require, of course, an update to the source.
|
|
<br><br>
|
|
|
|
<tr><td valign="top"><b>Q.</b><td>
|
|
Can this compression scheme be used for FBA devices too ?
|
|
<tr><td valign="top"><b>A.</b><td>
|
|
I have not worked with FBA devices for over 20 years.
|
|
However, it seems to me that a similar program for FBA
|
|
devices should be simpler than this program for CKD devices
|
|
(none of those count/key/data fields mucking everything
|
|
up). Since an FBA block is 512 bytes, it might not
|
|
be efficient to have each block compressed individually;
|
|
it might be better to compress blocks in 32K or 64K chunks.
|
|
If someone asks very nicely, I may consider looking into it;-)
|
|
<br><br>
|
|
|
|
</table>
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="changes">Changes</a></h3>
|
|
<ul>
|
|
<li>0.2.0
|
|
<ul>
|
|
<li>This release greatly enhances the stability of cckd files.
|
|
<li>Free spaces are read at file open time, maintained in storage,
|
|
and written at file close time. If a file is not successfully
|
|
closed, then free spaces are easily recovered by the chkdsk
|
|
function when the file is opened next.
|
|
<li>Imbedded free space is deprecated. Because the free space chain
|
|
is now in storage, the overhead required to keep an updated track
|
|
image at its same location in the file is no longer necessary;
|
|
the penalty for traversing the free space chain to find a new location
|
|
is greatly reduced. Also, because a free space header is no longer
|
|
written at the beginning of a freed space when it is freed, that space
|
|
is eligible for recovery in the event of a failure.
|
|
<li>Garbage collection is more efficient because it can combine the
|
|
most free spaces per iteration now that the penalty for scanning
|
|
free space is gone.
|
|
<li>Secondary lookup tables (or l2tabs) are now cached. Most overhead
|
|
I/O for cckd files is now eliminated.
|
|
<li>Support for read-only files is added.
|
|
<li>Utility <i>cckdcomp</i> is provided to remove all free space from
|
|
a cckd file.
|
|
</ul>
|
|
<li>0.2.1
|
|
<ul>
|
|
<li>The concept of shadow files is implemented, which logically performs
|
|
a snapshot function.
|
|
<li>Windows 32 support is enabled.
|
|
<li>Seems like there should be more but I can't mush my brain much farther.
|
|
Some new options have been added so #defines in hercules.cnf don't have
|
|
to be changed; overflow tracks are supported (I hope) -- Thanks Valery!!
|
|
</ul>
|
|
</ul>
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="bugs">BUGS</a></h3>
|
|
This code is absolutely bug free; if you encounter any problems then
|
|
it must be a personal problem and you've done something wrong. Also,
|
|
there are no enhancements that can be made because I've already thought
|
|
of them all and implemented them. By the way, I have some prime soon to be
|
|
ocean front property in Tennessee to sell to the highest bidder;-)
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="cckddump">cckddump os/390 hlasm program</a></h3>
|
|
The cckddump program (supplied in file <code>cckddump.hla</code>) is an
|
|
os/390 assembler language program that creates a compressed CKD Dasd emulation
|
|
file from a real DASD volume. This program must be APF-authorized since
|
|
it modifies the DEB to be able to read all tracks from the real device.
|
|
The program executes 16 or so instructions while in supervisor state/key 0;
|
|
otherwise the program runs entirely in problem state/key 8.
|
|
It is not the prettiest assembler language program I've ever written, and
|
|
there are plenty of enhancements that I originally intended to put into the
|
|
program that I haven't yet; once I got the program working good enough, I
|
|
spent the rest of my time writing the fun stuff, the Hercules part.
|
|
<p>The real CKD Dasd volume that is dumped must be an ECKD device (ie support
|
|
'Locate Record' and 'Read Track' CCWs); this shouldn't be a problem because
|
|
I don't think any os/390 release supports a non-ECKD device. The output file
|
|
must be a DASD file; its characteristics are LRECL=4096, BLKSIZE=4096, RECFM=F.
|
|
The program only dumps allocated tracks (plus track 0) and only dumps tracks up
|
|
to DS1LSTAR for DSORG=PS and DSORG=PO files. The program will call zlib
|
|
to compress the track images if the zlib routines have been linked with the
|
|
program; however, I don't think the program will be advantageous if it can't
|
|
call zlib.
|
|
<p>
|
|
<h4>Preparing zlib</h4>
|
|
<ul>
|
|
<li>zlib can be obtained from
|
|
<a href="http://www.info-zip.org/pub/infozip/zlib/"><b>here</b></a>
|
|
<li>Copy or ftp the *.c files to a LRECL=255,RECFM=VB partitioned dataset;
|
|
here we will call the dataset <i>prefix</i>.ZLIB.C
|
|
<li>Similarly, copy or ftp the *.h files to a LRECL=255,RECFM=VB partitioned
|
|
dataset; we'll call it <i>prefix</i>.ZLIB.H
|
|
<li>Edit member <i>prefix</i>.ZLIB.H(ZCONF). Near the bottom, before the 2nd
|
|
to last <code>#endif</code>, add the following lines:<br><pre>
|
|
# pragma map(compress,"COMPRESS")
|
|
# pragma map(compress2,"COMPRES2")
|
|
# pragma map(uncompress,"UNCOMPRE")
|
|
</pre>
|
|
<li>Allocate an object partitioned dataset <i>prefix</i>.ZLIB.OBJ;
|
|
LRECL=80,BLKSIZE=3200,RECFM=FB.<br>Submit the following job to compile zlib:
|
|
<pre>
|
|
// JOB
|
|
//CC JCLLIB ORDER=(CBC.SCBCPRC)
|
|
//*
|
|
//ADLER32 EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(ADLER32)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(ADLER32),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//COMPRESS EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(COMPRESS)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(COMPRESS),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//CRC32 EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(CRC32)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(CRC32),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//DEFLATE EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(DEFLATE)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(DEFLATE),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//EXAMPLE EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(EXAMPLE)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(EXAMPLE),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//GZIO EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(GZIO)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(GZIO),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//INFBLOCK EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(INFBLOCK)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(INFBLOCK),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//INFCODES EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(INFCODES)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(INFCODES),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//INFFAST EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(INFFAST)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(INFFAST),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//INFLATE EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(INFLATE)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(INFLATE),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//INFTREES EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(INFTREES)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(INFTREES),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//INFUTIL EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(INFUTIL)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(INFUTIL),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//TREES EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(TREES)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(TREES),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//UNCOMPR EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(UNCOMPR)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(UNCOMPR),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
//*
|
|
//ZUTIL EXEC EDCC,INFILE='<i>prefix</i>.ZLIB.C(ZUTIL)',
|
|
// CPARM='RENT,LIST,SOURCE,LONGNAME,AGG,OPT(2)',
|
|
// OUTFILE='<i>prefix</i>.ZLIB.OBJ(ZUTIL),DISP=SHR'
|
|
//USERLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.H
|
|
</pre>
|
|
<li>Prelink zlib using the following job:
|
|
<pre>
|
|
// JOB
|
|
//PLKED EXEC PGM=EDCPRLK
|
|
//SYSMSGS DD DISP=SHR,DSN=CEE.SCEEMSGP(EDCPMSGE)
|
|
//SYSLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ
|
|
// DD DISP=SHR,DSN=CEE.SCEEOBJ
|
|
//SYSOUT DD SYSOUT=*
|
|
//SYSPRINT DD SYSOUT=*
|
|
//SYSIN DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(ADLER32)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(COMPRESS)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(CRC32)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(DEFLATE)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(GZIO)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(INFBLOCK)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(INFCODES)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(INFFAST)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(INFLATE)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(INFTREES)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(INFUTIL)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(TREES)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(UNCOMPR)
|
|
// DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(ZUTIL)
|
|
//SYSMOD DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ(ZLIB)
|
|
</pre>
|
|
</ul>
|
|
|
|
<h4>Assemble and linkedit cckddump</h4>
|
|
<ul>
|
|
<li>Allocate partitioned dataset <i>prefix.cckddump.source</i>;
|
|
LRECL=80,RECFM=FB and copy or ftp file cckddump.hla
|
|
<li>Submit the following job:
|
|
<pre>
|
|
// JOB
|
|
//C EXEC PGM=ASMA90
|
|
//SYSLIB DD DISP=SHR,DSN=SYS1.MACLIB
|
|
// DD DISP=SHR,DSN=SYS1.MODGEN
|
|
//SYSPRINT DD SYSOUT=*
|
|
//SYSIN DD DISP=SHR,DSN=<i>prefix.cckddump.source</i>(CCKDDUMP)
|
|
//SYSUT1 DD UNIT=SYSDA,SPACE=(CYL,(1,1))
|
|
//SYSLIN DD DISP=(,PASS),DSN=&&OBJ,UNIT=SYSDA,SPACE=(CYL,(1,1))
|
|
// LRECL=80,BLKSIZE=3200,RECFM=FB
|
|
//L EXEC PGM=HEWL
|
|
//SYSPRINT DD SYSOUT=*
|
|
//SYSUT1 DD UNIT=SYSDA,SPACE=(CYL,(1,1))
|
|
//SYSLIB DD DISP=SHR,DSN=CEE.SCEESPC
|
|
// DD DISP=SHR,DSN=CEE.SCEELKED
|
|
//ZLIB DD DISP=SHR,DSN=<i>prefix</i>.ZLIB.OBJ
|
|
//SYSLMOD DD DISP=SHR,DSN=<i>apfauth.load</i>
|
|
//SYSLIN DD DISP=(OLD,DELETE),DSN=&&OBJ
|
|
// DD *
|
|
INCLUDE ZLIB(ZLIB)
|
|
INCLUDE SYSLIB(EDCXHOTL)
|
|
INCLUDE SYSLIB(EDCXHOTU)
|
|
INCLUDE SYSLIB(EDCXHOTT)
|
|
ORDER MAIN(P)
|
|
ENTRY MAIN
|
|
SETCODE AC(1)
|
|
NAME CCKDDUMP(R)
|
|
</pre>
|
|
<li>The assemble step (C) should complete with condition code 4.
|
|
This is a `feature' due to the way IBM macro IECSDSL1 is coded.
|
|
The linkedit step (L) should complete with condition code 0.
|
|
</ul>
|
|
|
|
<h4>Executing cckddump</h4>
|
|
<ul>
|
|
<li>The volume to be dumped is identified by the SYSUT1 DD statement;
|
|
the output compressed CKD Dasd emulation file is identified by the
|
|
SYSUT2 DD statement.
|
|
<li>Submit a job similar to the following:
|
|
<pre>
|
|
// JOB
|
|
//S1 EXEC PGM=CCKDDUMP
|
|
//STEPLIB DD DISP=SHR,DSN=<i>apfauth.load</i>
|
|
//SYSPRINT DD SYSOUT=*,RECFM=VB,LRECL=255,BLKSIZE=4096
|
|
//SYSUT1 DD DISP=OLD,UNIT=SYSDA,VOL=SER=<i>volser</i>
|
|
//SYSUT2 DD DISP=(,CATLG),DSN=<i>prefix.volser.cckd</i>,
|
|
// UNIT=SYSDA,SPACE=(TRK,(7500,1500),RLSE),
|
|
// LRECL=4096,BLKSIZE=4096,RECFM=F
|
|
</pre>
|
|
</ul>
|
|
|
|
|
|
<h4>Make the file available to Hercules</h4>
|
|
<ul>
|
|
<li>Copy or ftp <i>prefix.volser.cckd</i> in <b>binary</b> mode
|
|
to your platform running Hercules.
|
|
</ul>
|
|
|
|
<hr noshade>
|
|
<p><h3><a NAME="feedback">Feedback</a></h3>
|
|
Questions ?? Problems ?? Comments ?? Suggestions ?? Corrections ?? Bugs ??<br>
|
|
Let me know at <a href="mailto:gsmith@nc.rr.com"><em>gsmith</em>@<em>nc.rr.com</em></a>
|
|
|
|
<p>
|
|
greg smith
|
|
|
|
<p><small>Last updated 1 May 2002</small>
|
|
</BODY>
|
|
</HTML>
|