Programs related
to the Fictionmags Data Format
Background
A variety of programs related to
the Fictionmags Data Format have been written for various purposes, some intended
for "general use" and some more to meet local requirements.
The majority of the programs support
the use of a Windows environment variable LOCUSDIR which can be used to specify
a string of folder names (separated with semi-colons) where relevant files may
be located. An example might be:
f:\books\control\;f:\books\data\;f:\books\data\afi\;f:\books\data\cfi\;f:\books\data\fmi\;f:\books\data\gfi\;f:\books\data\sfi\;f:\books\data\wfi\
Unless specified otherwise, all programs
are DOS console programs that are run from the DOS command line. Where appropriate
they will accept either a single file name or a control file containing a list
of file names (preceded by an @) as in:
validate mags.asf
validate @00000.sfi
Control files can be nested (up to
a nesting level of 10) and can also contain comments and control flags, as follow:
- Anything following a tab and a
'!' character (and any preceding spaces or tabs) is ignored; if a line starts
with a '!' character then the whole line is ignored
- If a filename is followed by a
'~' character and an index abreviation (e.g. "mags.hav~AFI") then
it means that the file concerned technically belongs to a different group
and will be ignored by all programs except IdxGen
- If a line starts with a '-' character
it indicates a control flag. The "rule" is that all programs should
check for control flags relevant to themselves and ignore all others so that
control flags can be easily added without disruption. The flags currently
in use are:
- -V:n - sets the default validation
level for the following files and is used by Validate
and Xvalidate
- -X:filename - indicates a
cross-reference exceptions file, used by Xvalidate
- -I:x and -J:x - used by ChkCovers
to distinguish the different categories of index
The programs themselves fall into
the following xx categories:
There are also a couple of shareware
programs that assist the above programs in various ways.
Note that extensive use is made of
shared sets of routines that attempt to isolate
core functionality to avoid re-inventing the wheel and to simplify(!) the main
programs.
Main Programs
These are the main programs that
are used for maintaining the Big List and the indexes:
- CvtCovers - Converts a list of
cover scan files into entries in COVERS.CVT
- CvtSkel - Converts a set of 'E'
records into skeleton magazine issues
- GenAttrib - Generates a new version
of ATTRIB.TXT and ALL.TXT
- IdxGen
- Generate a suite of Index files
- MagGen - Generate the GCP website
files from the database populated by MagPop
- MagPop - Populate the database
from the contents of ARCHIVE.TXT
- MagParse
- Parses either a downloaded ISFDb file or a formatted text file into FM format
- Mrg_sty
- Merges details from an FM format file into STORY.IDX & BOOKS.IDX
- Validate
- Validate contents of an (S)BOOKS file
- Valnames - Validate contents of
NAMES.CVT and PSEUD.CVT
- XValidate
- cross-validates the contents of a group of FM format files
The way in which these programs are
used in practice can be found in the section on the Indexing
Process.
Note that MagPop and MagGen
work as a pair and are both written in Visual C++ using a Microsoft Access database
as intermediary. The plan is to rewrite these as a single C program using sorted
lists when time allows. MagParse is also currently written in Visual
C++.
Support Programs
These programs are used regularly
by Phil S-P when maintaining the indexes, but have not historically been used
by anybody else and hence may have environment-dependent constraints
- Asort - Sorts a file in FM format
into consistent "book/magazine" order
- BookProgs - Visual Interface to
the main index (and bibliography) maintenance programs (this is written in
Visual C++)
- ChkCovers - Check all known cover
scans against database to identify missing/poor-quality cover scans
- Cvtlocus
- Converts UK format file to US format (while the US format is no longer used
as such; its legacy persists in many other parts of the system)
- Cvtuk - Converts US format file
into UK format (as with Cvtlocus this is
no longer need for its original purpose but comes in useful from time to time)
- Fsort - General Purpose File Sort
Program
- Genprt
- Formats an FM format file for printing (via Microsoft Word) or for posting
online (e.g. to Fictionmags)
- Splitcol - Splits the contents
of an FM format file into anthologies, collections and others and adds them
to the appropriate data file
- Tstauth
- Test Harness for (some of) the routines in Auth_rtn
"Historic" Programs
There are a handful of programs that
were used to create and maintain bibliography files. They are mainly written
in C (with a VC++ shell):
- Chron
- Generate Chronological Index
- Generate
- Generate and Format a Bibliography Source File
- Isbn_Lst
- Generate List of ISBNs
- Paginate
- Paginate a Bibliography
- Reboiler
- Reboiler a Bibliography Text File
- Renumber
- Renumber entries in bibliography
- Twoside
- Convert Bibliography to 2-sided Text File
There are also a number of other
programs which, at one time or another, have proved useful but which are probably
now obsolete:
- Align - Attempts to compare two
FM format files for the same magazine(s) when the order of individual items
within an issue might differ (e.g. one has page numbers and the other doesn't)
- Check
- Checks the order of entries in an FM format file
- ChronBk - Generates a list of
books in chronological order from one or more FM format files
- Convert - General program for
massaging/converting an FM format file: primarily useful as a reusable shell
that reads through a set of FM format files and parses each record into its
associated type for some form of special handling
- Cvtforth
- Converts an FM format "books" file into a "Forthcoming Books"
list: was used when Ian Covell or I was maintaining the Locus column
- CvtStory - Converts Story.idx
& Books.idx to reflect changes to pseudonym files: has fallen into disuse
but should probably be used more often
- Esort - Sorts the 'E' records
in an FM format file into numerical order within book/issue: used for tidying
up existing files some years back
- Extract
- Extracts one "Locus" file from another: was used when Ian Covell
or I was maintaining the Locus column
- Merge
- Merge two or more fm format books files based purely on the first letter
of the author's surname (this has been superseded by SplitCol for the main
database)
- PrintAwd - Formats the "awards"
file for printing: the awards file has not been updated for decades
- Spilt - Converts a file containing
data for multiple magazines into a separate file per magazine: used some years
back for setting up the current database
- Splitbok
- Convert an FM format books file into those to be sent to Locus and those
not to be sent: was used when Ian Covell or I was maintaining the Locus column
- PrintBox - Generates a formatted
list of which books are in which boxes: purely internal to Phil S-P (and no
longer used)
- PrintRead - Generates a reading
list from Phil S-Ps collection
- Splitfil
- Splits a Text File into Smaller Files
- Translte
- Translates Accents in a Text File
Useful Shareware
The following Shareware programs
are recommended for use with the above programs:
- Beyond Compare (BC2):
used for comparing files or folders for changes, particularly when synchronising
data file updates (or comparing different cover scans for the same magazine
issue). It can also be used for uploading the indexes to the website. Can
be downloaded from https://www.scootersoftware.com/
- CMSort: IdxGen requires the use
of a sort program to sort the various temporary files into the relevant order.
Any sort program should work OK, but the recommended one is CMSort which is
available from http://www.chmaas.handshake.de/delphi/freeware/cmsort/cmsort.htm
- Visual Studio 6.0: All the programs
are developed using Microsoft Visual Studio 6.0 (though most are written in
"vanilla" C and "should" work with any C compiler. While
the latest versions of Visual Studio are commercial software, 6.0 is available
for free download from https://winworldpc.com/product/microsoft-visual-stu/60
(and elsewhere)
Common Routines
The above programs to one degree
or another make use of the following groups of routines:
- Abbr_rtn:
routines related to ABBREV.CVT
- Auth_rtn:
routines related to the handling of authors
- File_rtn:
file-handling routines
- Gen_rtn:
general-purpose routines that have knowledge of the structure of an (S)BOOKS
file
- Lib_rtn:
general-purpose routines that DO NOT have knowledge of the structure of an
(S)BOOKS file
- Pbdt_rtn:
routines relating to the handling of publication details strings
- Pseud_rtn:
routines related to PSEUD.CVT
- PSPCommandLineInfo: C++ routines
for handling command line arguments
- PSPErrorHandling: C++ routines
for error handling
- Publ_rtn:
routines relating to the handling of publisher names
- Scan_rtn:
routines that scan (S)BOOKS files, compiling an array of data from the contents
- Titl_rtn:
routines related to the handling of titles
- Tran_rtn:
routines that translate the data from one format to another (e.g. internal
format to HTML format)