IdxGen - Generate a set of Index Files

Introduction

IdxGen generates a set of Index Files based on the contents of an Index Configuration File. As well as a (brief) discussion of each module in IdxGen, this file discusses some of the general background issues:

• Basic Index Structure
• Calculating Bookmarks and Page Breaks
• Boilerplate Files
• Sorted Files
• Data File Naming Convention
• External Sort Command File
• Configuration Data Structure
• Other Global Data
• Notes on Some (Past) Problem Areas

Basic Index Structure

While the individual indexes look quite different one from the other, there is an over-riding structure that is common to each index, although not all indexes use all parts. The core level is what might be called the Index 1 Level which contains a list of all the "things" being indexed (e.g. Artist Names, Magazine Names, Story Titles, etc.).

In most cases there is a level below this which we could call the Listings Level which lists all the items for each "thing" (e.g. issues for each Magazine, items by each author, items in each series, etc. etc.). Note that this level is not used for the Story and Book Title Indexes as these do not have an expansion. Conversely, for Magazine Issues and for Book Authors there is a further level below which we could call the Contents Level which lists the contents of each magazine issue or book.

Above the Index 1 Level there then might be up to two higher, hierarchical, levels which provide index ranges into the lower-level indexes. The intention is that these should be dynamic based upon the number of items in the Index 1 Level. As each level will contain MAXPAGESIZE entries of the next level, we can see that with a modest page size of, say, 300 lines:

• Each Index 1 Level would contain (up to) 300 items
• Each Index 2 Level would handle up to 300x300 = 90,000 items
• Each Index 3 Level would handle up to 300x300x300 = 27,000,000 items

Currently (June 2021) the largest index (the Story Title Index) has about 2,000,000 items in it so this should do for the foreseeable future. The table below summarises the situation as of June 2021. Note that one complication is that the top level of the index is always a single page with a fixed name, so a small index will only have the Index 3 Level; a medium-sized one will have Index 3 and Index 1 Levels and large indexes would have all three.

Note also that, while the formats of Listings and Contents Levels vary quite widely from index to index, the lowest Index level will always be simply a name or a continuation line and all the levels above that will be a range of names.

The creation and the structuring of the index levels is all handled by WRITE_INDEX_LINE which depends on a set of static tables as listed in the "Defined by" line above. These are accessed by the index type which is held in the curr_index_type field in the Configuration Data structure. There are also four other static tables similarly referenced:

• idxtyp_ptr - Name of the index to be used on links from the index pages
• idxpgttl_ptr - Page Title to be used on the index pages
• idxuseital - Set to PSP_TRUE if the index pages should put the names in italics rather than in bold
• idxlinecnt - This is set up during the "set up" phase to contain the number of lines in the Level 1 index (and is used to determine the number of levels).

Using WRITE_INDEX_LINE

Using the routine for normal indexes is very straightforward:

• In the SETUP module, set idxlinecnt[IDX_xxx] to the total number of lines in the index.
• When writing a line to the index, set config_ptr->formatted_name to the name as you want it to appear in the Level 1 index and config_ptr->curr_name to the name you want to appear in the higher level indexes (this might be the same or might be a simplified form)
• Call WRITE_INDEX_LINE specifying IDXLIN_NORMAL for a normal line or IDXLIN_CONT for a continuation line
• When writing the trailers to the various files, call WRITE_INDEX_LINE specifying IDXLIN_TRAIL

Note that the code generated for the Level 1 index is of the form:

<LI><A HREF="xnn.htm#Annn">formatted name</LI> (IDXLIN_NORMAL)
<LI>________: <A HREF="xnn.htm.htm#Annn">continuation text</LI> (for IDXLIN_CONT)
<LI>special text</LI> (for others)

It is up to the caller to ensure that formatted_name and continuation text contain "</A>" at the appropriate point to terminate the hyperlink - this allows just the first part of the text to be hyperlinked if so desired.

Calculating Bookmarks and Page Breaks

The indexes are linked together by hyperlinks which requires the creation of unique anchors for anything that might be the target of a link. This obviously presents a minor challenge as we need to know what the anchors in Index 1 are when creating Index 2, but also know what the anchors are in Index 2 when creating Index 1. To handle this always requires two passes through the indexes - one to set up the anchors and the other to implement them. There are various ways of doing this, but the approach IdxGen uses is to have one set of modules (SETUP_xxx_IDX) whose primary purpose is calculate the anchors and a second set (BUILD_xxx_IDX) which generates each index in turn. A key part of the first pass is also to decide where to insert a page break in the index so that pages do not get too large - depending on the index, these may or may not be in the middle of a section, as discussed below.

There are two approaches to handling the anchors in the two passes. Each requires the anchor to be stored in the first pass, but the second pass can then take one of two approaches:

1. Look up the anchor in the table, throwing a new page when the anchor says one is needed;
2. Calculate the anchors and page breaks again and periodically check they match those stored in the table.

The first of these is potentially the easiest, but the second provides a belts-and-braces sanity check - i.e., if the anchors/page numbers don't match then the code in either the SETUP or BUILD module (or both) is incorrect. For this reason the program currently adopts the second approach, although it can take quite a bit of time to track down an anomaly if the program detects the two don't match.

The anchors are either stored in an in-memory table or added to the sorted files and the page breaks vary from index to index, as follows:

• Book Author Index (SETUP_BOKAUT_IDX): This is more complex than most indexes as it has to handle both the listings level and the contents level:
  • A line is counted in the contents level for each line in the consolidated books file and a new page is thrown in the contents level if there is an 'A' record (i.e. start of a new book) and the count > MINPAGESIZE. At the moment the assumption is that an overlong page is better than breaking a book contents listing across multiple pages so no check is made on MAXPAGESIZE.
  • A line is counted at the listings level for each 'A' record and a new page is thrown if the author name has changed and the count > MINPAGESIZE, or the count > MAXPAGESIZE; in the latter case a continuation record is needed.
  • An anchor is created at the listings level for each new author in the index and is stored internally
  • An anchor is created at the contents level for each book and is stored internally in the Issue Link Table
• Book Title Index (SETUP_BOKTTL_IDX):
  • The routine ignores anything that isn't a book or is by a secondary author or where the author is the subject.
• Magazine Issues Index (SETUP_ISSUE_IDX): Like the Book Author Index this also has to handle both the listings level and the contents level:
  • A line is counted in the contents level for each line in the consolidated magazine file (except those in an Additional Entries File) and a new page is thrown in the contents level if there is an 'A' record (i.e. start of a new issue) which does not immediately follow a header record, and the count > MINPAGESIZE. Assuming the minimum and maximum page sizes are sensible there should be no need for a continuation page here so the code simply checks to see if the count > MAXPAGESIZE and outputs a diagnostic if so.
  • A line is counted at the listings level for each 'A' record and a new page is thrown if we have a new main features record and the count > MINPAGESIZE, or the count > MAXPAGESIZE; in the latter case a continuation record is needed.
  • An anchor is created at the contents level for each issue and for each features record and is stored internally in the Issue Link Table
  • Currently no anchors are created at the listing level.
• Story Title Index (SETUP_STYTTL_IDX):
  • The routine ignores all items of types hd, mg. il, fp, pt, cv or ct, as well as any subject records
  • Otherwise a line is counted whenever the author or title changes and a new page is thrown as soon as the count > MINPAGESIZE.
  • An anchor is created for every distinct author/title pair and is stored in the styttl_ptr field in the associated SCANITEM record.
• Story Author Index (SETUP_STYAUT_IDX):
  • The routine ignores all items of types hd, mg. il, fp, pt, cv or ct, as well as any secondary author records (e.g. subjects, editors, etc.)
  • Otherwise a line is counted for each input record
  • A new page is thrown when the item changes and either the author changes and the count > MINPAGESIZE, or the count > MAXPAGESIZE; in the latter case a continuation record is needed.
  • An anchor is created for each author and is stored internally in the Names Link Table
• Artist Index (SETUP_ARTIST_IDX):
  • The routine reads the sorted artist file (IdxGen.art) and counts a line for each input record.
  • A new page is thrown when either the artist changes and the count > MINPAGESIZE, or the count > MAXPAGESIZE; in the latter case a continuation record is needed.
  • An anchor is created for each author and is stored internally in the Names Link Table
• Chronological Index (SETUP_CHRON_IDX):
  • The routine creates and reads the sorted Chronological Index file (IdxGen.crn)
  • It ignores all items of types hd, mg. il, fp, pt, cv or ct, as well as any subject records, but otherwise a line is counted for each input record
  • A new page is thrown when either the author changes and the count > MINPAGESIZE, or the count > MAXPAGESIZE; in the latter case a continuation record is needed.
  • An anchor is created for each author and is stored internally in the Names Link Table
• Series Index (SETUP_SERIES_IDX):
  • The routine creates and reads the sorted Series Index file (IdxGen.ser)
  • A new page is thrown when the series or series type changes and the count > MINPAGESIZE. Assuming the minimum and maximum page sizes are sensible there should be no need for a continuation page here so the code simply checks to see if the count > MAXPAGESIZE and outputs a diagnostic if so.
  • An anchor is created for each series and is stored internally in the Series Link Table
• Biographical Notes (SETUP_BIOG_NOTES):
  • The routine reads the contents of the Names Link Table and counts 3 lines for every author which is in the current index and has some biographical notes.
  • A new page is thrown whenever the count > MINPAGESIZE
  • An anchor is created for each author and is stored internally in the Names Link Table
• Full Text Index:
  • The routine reads the Full Text file (IdxGen.ftx) and counts a line for each input record.
  • A new page is thrown whenever the magazine name changes and the count > MINPAGESIZE, or the count > MAXPAGESIZE; in the latter case a continuation record is needed.
  • An anchor is created for each magazine name but is only used internally.
  • Note that there is no SETUP module for this index - the idxlinecnt is set up as part of creating the Full Text file in BUILD_ISSUE_IDX

Boilerplate Files

The use of boilerplate files was first used in the programs used for generating the GCP Website and provide a means for having standard, potentially fairly complex, HTML page layouts that can be easily changed without requiring programmatic changes. The principle is very straightforward - the file is a standard HTML file that can be created with any HTML editor and which contains a number of special flags of the form <!x>, <!x+> and <!x-> where x is some special code known to the program (often just a single letter).

A line that starts with <!x> indicates that the whole line should be replaced with whatever x represents to the program while <!x+> and <!x-> delimit an entire section that should be omitted under certain conditions (e.g. the "Previous Page" link on the first page).

There are three boilerplate files used by IdxGen:

• index_toc.htm - The Table of Contents
• index_hdr.htm - The Page Header
• index_trl.htm - The Page Trailer

A consistent set of substitution variables are used across the three files, even though no single file uses them all:

• 1 - The link to the Introduction Text (if any)
• 3 - The link to the Missing Issues Page (if any)
• 4 - The link to the Titles Not Included Page (if any)
• 5 - The link to the In the Next Update Page (if any)
• A - A subfolder string ("a/") to be inserted if subfolders are being used.
• B - A folder redirection string ("../") to be inserted if subfolders are being used.
• C - The names of the creator(s) of this index
  
• D - The current date in the form MMM DD, CCYY
• E - The names of the editor(s) of this index
• F - The background colour for this page (e.g. FFFFE0)
• H - The link to and name of the top page for this Index (if required)
• I - The offical name of the Index
• N - The filename for the next page on this level (if any)
• P - The filename for the previous page on this level (if any)
• T - The page title to be used on the current page
• U - The location to insert any additional index-specific lines in the Table of Contents
• X - The link to the Index by Publisher (if any)
• Y - The current year in the form CCYY
• Z - The links to the Book Indexes (only generated if any books were found in the index)

Sorted Files

While much of the data needed by the program resides in structures in memory it has proved impossible to hold everything in memory as discussed below. As such, a number of sorted data files are created and used by the program as follows:

• IdxGen.tm1 - This file is created by SETUP_FILES to contain all the records from the book files, prefixed by a special sort header.
• IdxGen.tm2 - This is a version of IdxGen.tm1 sorted by SETUP_FILES into the order needed for the Book Author Index.
• IdxGen.bks - This is identical to IdxGen.tm2 but has the sort header stripped off (also by SETUP_FILES) and is a primary input to SETUP_BOKAUT_IDX
• IdxGen.mgs - This file is created by READ_MAG_FILES (called from SETUP_FILES) and contains all the records from the magazine files, in the order required by the index, with additional fields added to the 'A' records specifying any cover scans, full text and/or about links. Note that the latter two occupy the same field as about links are only used on features records and full text links on non-features record. It is a primary input to SETUP_ISSUE_IDX
• IdxGen.tmp - This file is created by SETUP_BOKAUT_IDX & SETUP_ISSUE_IDX via the SCAN_RECORD routine.
• IdxGen.tm3 - This is a version of IdxGen.tmp sorted by SETUP_STYTTL_IDX into the order needed for the Story/Book Title Indexes.
• IdxGen.tm4 - This file is created by SETUP_STYTTL_IDX as a copy of IdxGen.tm3 but with Story Title Index anchors added for each Story Title.
• IdxGen.tm5 - This is a version of IdxGen.tm4 sorted by SETUP_STYAUT_IDX into the order needed for the Story Author Index.
• IdxGen.tm6 - This is a version of IdxGen.tm5 with additional records added for the first appearance of reprinted items.
• IdxGen.Aut - This is a version of IdxGen.tm6 sorted by SETUP_STYAUT_IDX into the order needed for the Story Author Index and is the primary input to BUILD_STYAUT_IDX.
• IdxGen.tm7 - This file is created by SETUP_STYAUT_IDX as a copy of IdxGen.Aut but with Story Author Index anchors added for each Story Title and is in Story Title Index order.
• IdxGen.Ttl - This is a version of IdxGen.tm7 sorted by SETUP_STYAUT_IDX into the order needed for the Story Title Index and is the primary input to BUILD_STYTTL_IDX.
• IdxGen.tm8 - This file is created by SETUP_CHRON_IDX and is a version of IdxGen.ttl containing only those records needed for the Chronological Index, rewritten in Chronological Index order.
• IdxGen.Crn - This is a version of IdxGen.tm8 sorted by SETUP_CHRON_IDX into the order needed for the Chronological Index and is the primary input to the main part of BUILD_CHRON_IDX.
• IdxGen.tm9 - This file is created by SETUP_SERIES_IDX and is a version of IdxGen.ttl containing only those records needed for the Series Index, rewritten to ensure all identical items are grouped together.
• IdxGen.Ser - This is a version of IdxGen.tm9 sorted by SETUP_SERIES_IDX into the order needed for the Series Index and is the primary input to the main part of BUILD_SERIES_IDX.
• IdxGen.ftx - This file is created by BUILD_ISSUE_IDX and contains details of all the Full-Text Links for this index; it is read by BUILD_FULLTEXT_IDX to create the Full Text Index.

Data File Naming Convention

The program supports the input of a mixture of magazine data files, book data files and additional reference files. At the moment, at least, these files need to be distinct (i.e. you can't have a single file with both magazine and book data, other than where the book is being listed as part of a magazine) and need to follow a strict naming convention:

• magazine data files should be named either xxxxx.mag or mags.xxx where xxx/xxxxx is the (case-insensitive) magazine abbreviation in ABBREV.CVT. The translation of this abbreviation is used to determine the order in which magazines will be displayed for those indexes where the Index Configuration File indicates the magazines should be sorted by name.
• additional reference files should be named 00xxx.mag (where, by convention, the xxx is the mnemonic of the index, although the program doesn't check this). These will be used to generate entries in the Story Author and Story Title indexes, but nowhere else.
• all other input files are assumed to be book data files.

External Sort Command File

The data in the indexes is too large for in-memory sorts with the 32-bit compiler, and a brief experiment with the 64-bit compiler resulted in 8 hours for a single full sort of the data (as opposed to 5 minutes for an external sort program) so, at various points, IdxGen calls an external command file (specified in the Index Configuration File) to sort a pair of files. As the sort order varies from instance to instance the command file is also passed a parameter indicating the type of sort that is required. Thus the formatted command that is executed might be:

   sortfil BOOKDATA IdxGen.tm1 IdxGen.Bks

where IdxGen.Tm1 is the input file and IdxGen.Bks is the output file. The possible command parameters are:

• BOOKDATA: Sort the file created by READ_BOOK_FILE - this is a straight sort into increasing order

The current tool used for sorting is CMSort where the key switches are:

• /V=$1F indicates treat the file as a CSV file separated by %x1f characters
• /SV=n,1,0 means sort the nth field (1st field =1) from the first character to the end of the field

So, for instance, a sort key could be :

    /V=$1F /SV=1,1,0 /SV=2,1,0 /SV=3,1,0 /SV=4,1,0 /SV=5,1,0 /SV=6,1,0 /SV=7,1,0 /SV=11,1,0 /SV=8,1,0 /SV=9,1,0 /SV=10,1,0

However, experimentation shows that sorting with a key like this takes 4-5 times longer than a straight sort so, instead, the files are created in a different order depending on the way we want to sort them as described in the documentation of SETUP_SCANITEM

Configuration Data Structure

To allow the code to be broken into smaller sections without huge parameter lists, the bulk of the data that the program uses is held in a structure called config_data which is passed as an argument to most routines. It contains the following groups of fields:

Settings from the Index Configuration File:

	char	idxnam[128];			/* Name of Index */
	char	editor[128];			/* Name of Editor(s) */
	char	idxdir[128];			/* Folder to generate indexes into */
	char	boiler[128];			/* Folder containing boilerplates */
	char	sortfile[128];			/* Name of the file containing the sort commands */
	char	ctrlfile[128];			/* Name of the file defining the files to include in the index */
	char	ablkfile[128];			/* Name of the file containing the about links for the index */
	char	intrfile[128];			/* Name of the file containing the Introduction */
	char	missfile[128];			/* Name of the file containing a list of missing issues */
	char	omitfile[128];			/* Name of the file containing a list of magazines deliberately omitted from the index */
	char	nxtufile[128];			/* Name of the file containing a list of items for the next update */
	char	toctext[128];			/* Name of the file containing additional text to insert in Table of Contents */
	char	lastupdate[10];			/* Date of last update */
	int	subfolders;			/* Set to PSP_TRUE if files should be generated in sub-folders */
	int	pubindex;			/* Set to PSP_TRUE if we want a "by Publisher" index */
	int	sortnames;			/* Set to PSP_TRUE if the magazine files should be sorted by name */
	int	fullimages;			/* Set to PSP_TRUE if cover scans are to be displayed full-size */
	int	minpagesize;			/* The minimum number of lines to display on a page (default 200) */
	int	maxpagesize;			/* The maximum number of lines to display on a page (default 1000) */
	int	permlinks;			/* Set to PSP_TRUE if permanent links should be output */
	int	report_diags;			/* Set to PSP_TRUE if extended diagnostics should be output */
	char	special1[128];			/* First internal special flag */
	char	special2[128];			/* Second internal special flag */
	char	special3[128];			/* Third internal special flag */
	char	special4[128];			/* Fourth internal special flag */
	char	special5[128];			/* Fifth internal special flag */
	char	special6[128];			/* Sixth internal special flag */
	char	special7[128];			/* Seventh internal special flag */
	char	special8[128];			/* Eighth internal special flag */
	char	special9[128];			/* Ninth internal special flag */

The scandata structure used by SCAN_FILE:

	struct	scandata *scandata_ptr;		/* Pointer to scandata structure */

The lists of magazine files (in a sub-structure so that they can be sorted by magazine sort name) and the consolidated magazine file name:

	struct magfile {
		char	*magnam_ptr;		/* Pointers to Magazine Names */
		char    filnam[MAXFILENAME];	/* Magazine File Name */
	};
	struct  magfile magfiles[MAXMAGFILES];	/* Magazine File & Sort Names */
	int	magfile_cnt;			/* Count of Magazine Files */
	char	magfilnam[128];			/* Magazine File Name */

The name of the consolidated book file name and flag to indicate that we have some books:

	int	got_books;			/* Flag to say we have some books */
	char	bookfilnam[128];		/* Book File Name */

A number of fields related to the index type currently being built:

	char	curr_index_type;		/* Current index type */
	FILE	*topfil_ptr;			/* Top-Level Index Output File Pointer */
	FILE	*midfil_ptr;			/* Middle-Level Index Output File Pointer */
	int	midpage_count;			/* Page and line counts for the middle-level index */
	int	midline_count;
	int	lstpage_count;			/* Page, line & anchor counts for the listings */
	int	lstline_count;
	int	lstanchor_count;
	char	first_name[1024];		/* First name for top-level index */
	char	last_name[1024];		/* Last name for top-level index */
	char	curr_name[1024];		/* Current name */
	char	formatted_name[4096];		/* formatted version of same for index headings (can be huge for house names) */

And some miscellaneous (useful) data:

	char	uplink[4];			/* Set to "../" if we have subfolders; to "" otherwise */
	char	toplink[6];			/* Set to "../a/" if we have subfolders; to "" otherwise */
	char	tmpdir[256];			/* Folder to use for temporary files */
	FILE	*namfil_ptr;			/* Names Link File */
	FILE	*dmpfil_ptr;			/* Diagnostic Dump File (if needed) */
	FILE	*csvfil_ptr;			/* CSV file for Names Link Database (if needed) */

Note that, as we store key information related to the current index type in the structure, it is critical that the index types are processed sequentially, not concurrently (see BUILD_FULLTEXT_IDX below).

Other Global Data

We also need to expose some of the data globally so that we can sort them efficiently via qsort. This includes:

The Issue Link Table containing the list of magazine issue IDs and book IDs and the associated links and a list of subscripts into that array:

static	char	*isslink_det_ptr[MAXISSUES];	/* Issue Details Table */
static	char	*isslink_exp_ptr[MAXISSUES];	/* Expanded Details Table (also used for cover scan links in READ_MAG_FILES) */
static	char	*isslink_txt_ptr[MAXISSUES];	/* Full Text/About Link Table (in READ_MAG_FILES) */
						/* Book Title Link (elsewhere) */
static	char	*isslink_pre_ptr[MAXISSUES];	/* Issue Link Page/Anchor Prefix Table */
static	int	isslink_pageno[MAXISSUES];
static	int	isslink_anchor[MAXISSUES];	/* Issue Link Page/Anchor Table */
static	char	isslink_edition[MAXISSUES];	/* and edition */
static	int	isslink_idx[MAXISSUES];		/* and Indexes into Table(s) */
static	int	isslink_cnt;			/* Count of Issue Links */

The Names Link Table containing the list of authors in the Story Author, Book Author, Artist, Chronological & Biographical Notes Indexes:

static	char	*nameslink_nrmaut_ptr [MAXNAMES]; /* Normalised author names */
static	char	*nameslink_auth_ptr [MAXNAMES];	/* Standard author names */
static	char	nameslink_namtyp[MAXNAMES];	/* Name Type Table: Bitmap indicating which indexes the name appears in: */
						/* 1=Story Author; 2=Artist; 4=Book Author */
static	int	nameslink_pseudsub[MAXNAMES];	/* Index into PSEUD.CVT */
static	int	nameslink_stypag[MAXNAMES];
static	int	nameslink_styanc[MAXNAMES];
static	int	nameslink_stylin[MAXNAMES];	/* Story Author Index Page/Anchor/Line Table */
static	int	nameslink_artpag[MAXNAMES];
static	int	nameslink_artanc[MAXNAMES];	/* Artist Index Page/Anchor Table */
static	int	nameslink_bokpag[MAXNAMES];
static	int	nameslink_bokanc[MAXNAMES];	/* Book Author Index page/anchor table */
static	int	nameslink_crnpag[MAXNAMES];
static	int	nameslink_crnanc[MAXNAMES];	/* Chronological Index Page/Anchor Table */
static	int	nameslink_biopag[MAXNAMES];
static	int	nameslink_bioanc[MAXNAMES];
static	int	nameslink_biotyp[MAXNAMES];	/* Biographical Notes page/anchor/type table */
static	int	nameslink_idx[MAXNAMES];	/* and indexes into table */
static	int	nameslink_cnt;			/* and count of Links */

Similar information for the book titles in the Book Title Index:

static	int	bokttllink_recnum[MAXBOOKS];	/* Record number in Books File */
static	int	bokttllink_pageno[MAXBOOKS];
static	int	bokttllink_anchor[MAXBOOKS];	/* Book Title page/anchor table */
static	char	*bokttllink_pubdet[MAXBOOKS];	/* Abbreviated publication details */
static	int	bokttllink_idx[MAXBOOKS];	/* and indexes into table */
static	int	bokttllink_cnt;			/* and count of Links */

Similar information for the series names in the Series Index:

static	char	*serieslink_nam_ptr [MAXSERIES]; /* Series Names Table */
static	int	serieslink_pageno[MAXSERIES];
static	int	serieslink_anchor[MAXSERIES];	/* Series Page/Anchor Table */
static	int	serieslink_idx[MAXSERIES];	/* and indexes into table */
static	int	serieslink_cnt;			/* and count of Links */

The array of column headers that should be suppressed when encountered:

static	char	*colhdr_arr[MAXCOLHDRS]; 	/* Array of column headers to be suppressed */
static	int	colhdr_cnt;			/* and count thereof */

Key information (mainly static) related to the different index types:

static	char*	topfilnam_ptr;			/* File names for the top-level index for each index type */
static	char*	idxlv2pre_ptr[];		/* Prefix letter(s) for the level 2 index for each index type */
static	char*	idxlv1pre_ptr[];		/* Prefix letter(s) for the level 1 index for each index type */
static	char*	lstingspre_ptr[];		/* Prefix letter(s) for the listings level for each index type ("" if none) */
static	char*	contentspre_ptr[];		/* Prefix letter(s) for the contents level for each index type ("" if none) */
static	char*	idxtyp_ptr[];			/* Index Type Name to be used when linking to top-level index for each index type */
static	char*	idxpgttl_ptr[];			/* Page Title to be used in Index Headers for each index type */
static	int	idxemphchr[];			/* Type of emphasis to be used in index levels: 'B' = Bold; 'I' = Italics; ' ' = None */
static	int	idxlinecnt[];			/* Number of lines in the bottom-level index */
static	int	idxsinglvl[];			/* Set to true if this index is wholly contained in the top-level index page */
static	char*	idxbgcolor_ptr[];		/* RGB colour value to set as background colour for this index */

Lastly, some odd counts for the Statistics File and for Terminal Diagnostics:

static	int	author_cnt=0;			/* Number of authors in the index */
static	int	artist_cnt=0;			/* Number of artists in the index */
static	int	magtitles_cnt=0;		/* Number of magazine titles in the index */
static	int	magissues_cnt=0;		/* Number of magazine issues in the index */
static	int	books_cnt=0;			/* Number of books in the index */
static	int	fiction_cnt=0;			/* Number of fiction items in the index */
static	int	poems_cnt=0;			/* Number of poems and plays in the index */
static	int	nonfiction_cnt=0;		/* Number of non-fiction items in the index */
static	int	covers_cnt=0;			/* Number of cover images in the index */
static	int	pages_cnt=0;			/* Number of pages in the index */
static	int	ftlinks_cnt=0;			/* Number of full-text links in the index */
static	int	biog_cnt=0;			/* Number of biographical notes in the index */

static	int	nonfatal_diags=0;		/* Number of non-fatal diagnostics (check idxgen.prt) */
static	int	max_html_files=0;		/* Maximum number of HTML files for any index type (compared to MAXHTMLFILES) */
static	int	multgrp_max=0;			/* Maximum number of different filnam/byline groups (compared to MAX_MULTGRP) */
static	int	multitm_max=0;			/* Maximum number of occurrences for a single item (compared to MAX_MULTITM) */
static	int	multrep_max=0;			/* Maximum number of entries with reprint author/title details for an item (compared to MAX_MULTREP) */
static	int	multxtr_max=0;			/* Maximum number of groups with distinct bylines/original titles (compared to MAX_MULTXTR) */

static	int	do_diag=PSP_FALSE;		/* Diagnostic flag; this can be used anywhere to create a line on which a breakpoint */
						/* can be set when a particular condition occurs */

Program Modules

The program consists of the following modules:

• The main program
• PARSE_CONFIG - Read and Parse the Index Configuration File
• SETUP_FILES - Set up the consolidated Magazine and Book Data Files
• SETUP_BOKAUT_IDX - Set up all the anchors for the Book Author Index
• SETUP_BOKTTL_IDX - Set up all the anchors for the Book Title Index
• SETUP_ISSUE_IDX - Read all the files and set up the Issue Index
• WRITE_LINKS_FILE - Create the files for links from the GCP Website
• SETUP_STYTTL_IDX - Set up all the anchors for the Story Title Index
• SETUP_STYAUT_IDX - Set up all the anchors for the Story Author Index
• SETUP_ARTIST_IDX - Set up all the anchors for the Artist Index
• SETUP_CHRON_IDX - Set up all the anchors for the Chronological Index
• SETUP_SERIES_IDX - Set up all the anchors for the Series Index
• SETUP_BIOG_NOTES - Set up all the anchors for the Biographical Notes
• WRITE_TABLE_OF_CONTENTS - Write the Table of Contents File
• BUILD_BOKAUT_IDX - Build the Book Author Index
• BUILD_BOKTTL_IDX - Build the Book Title Index
• BUILD_ISSUE_IDX - Build the Magazine Issue Index
• BUILD_FULLTEXT_IDX - Build the Full-Text Index
• BUILD_STYAUT_IDX - Build the Story Author Index
• BUILD_STYTTL_IDX - Build the Story Title Index
• BUILD_ARTIST_IDX - Build the Artist Index
• BUILD_CHRON_IDX - Build the Chronological Index
• BUILD_SERIES_IDX - Build the Series Index
• BUILD_BIOG_NOTES - Build the Biographical Notes
• WRITE_STATISTICS - Write the Statistics Page

which communicate via the Configuration Data structure and the Other Global Data. There are also a number of significant support routines:

• CREATE_NAME_LINK - Create a new nameslink entry
• CREATE_NEW_PAGE - Create a new page for the specified file
• FLUSH_DREC - Read all following 'D' records and first 'E' record (if any).
• FORMAT_AUTHOR - Format an Author Name as heading (with scanitem as input)
• FORMAT_AUTHOR2 - Format an Author Name as heading (with AUTH structure as input)
• FORMAT_AUTHTYPE - Format an Author Type as subheading
• FORMAT_BOOK_DETAILS - Format book details into summary form
• FORMAT_FULLTEXT_LINK - Format a Full Text link field
• FORMAT_ISSUE_LINK - Formats a set of publication details and adds a link to the associated issue/book if in the current index.
• FORMAT_MAGPUB - Format magazine issue details
• FORMAT_NAMES - Format a names field.
• FORMAT_NOTES - Format a notes field
• FORMAT_PUBDATE - Format Pub. Info. date field
• FORMAT_SERIES - Format series field
• FORMAT_URL - Format a URL specified in an FM data file
• GET_ANCHOR - Get the HTML to link to a specific anchor
• GET_ANCHOR2 - Get the HTML to link to a name index anchor
• GET_NAME_INDEX - Get index of specified name (if possible)
• GET_NAME_LINK - Get link to specified name (if possible)
• GET_PUBDET_LINK - Get offset of link in Issues Table for set of publication details.
• GET_SERIES_LINK - Get link to specified series (if possible)
• READ_BOOK_FILE - Read a specified book data file and output the contents to the consolidate book data file with sort header.
• READ_MAG_FILES - Read all the magazine data files (in order) and output the contents to the consolidate magazine data file with cover scan, full-text and about links added.
• REPORT_DIAGS - Report Extended Diagnostics if required
  
• SPLIT_TITLE - Split title field into constituent parts
• TIDY_DNOTES - Tidy up contents of any 'D' Notes
• WRITE_INDEX_LINE - Write a line to the intermediate index(es)
• WRITE_PAGE_HEADER - Write a standard header to a file.
• WRITE_PAGE_TRAILER - Write a standard trailer to a file.
• WRITE_PUB_INFO - Format and output all "Pub Info" records
• WRITE_STORY_ITEM - Write a single story item group.

Note that several of the BUILD_xxx routines revolve around the use of Boilerplate Files and the whole program relies on a sequence of Sorted Files. Note also that the program relies on a naming convention for the data files as discussed here and an external sort command file.

Currently IDXGEN only supports UK 7-bit format input files - it wouldn't be rocket science to extend this to handle US format files as well but it isn't a priority at present.

The main program

The main program:

• Gets the name of the Index Configuration File
• Allocates the scandata structure
• Opens the Index Configuration File (and diagnostics files) and calls PARSE_CONFIG to read and parse it
• Opens the Temporary File (IdxGen.tmp in the indicated folder) and, if required, the diagnostic dump file (IdxGen.dmp in the same folder)
• Calls SETUP_FILES to consolidate any specified magazine and/or book data file into a individual files (IdxGen.mgs & IdxGen.bks), doing any necessary sorting and adding cover scan/full text links for magazine issues.
• If we had any books it then:
• Calls SETUP_BOKAUT_IDX to read the book data file (IdxGen.bks) and store all the data in IdxGen.tmp, as well as to set up all the anchors for the Book Author Index
• Calls SETUP_BOKTTL_IDX to set up all the anchors for the Book Title Index
• Calls SETUP_ISSUE_IDX to read the magazine data file (IdxGen.mgs) and store all the data in IdxGen.tmp, as well as to set up all the anchors for the Magazine Issue Index
• Calls WRITE_LINKS_FILE to clean up the Issue Link Table and to write the Links File(s)
• Calls SETUP_STYTTL_IDX to read the data from IdxGen.tmp, set up all the anchors for the Story Title Index and output the data including the story title anchors in IdxGen.tm4
• Calls SETUP_STYAUT_IDX to sort IdxGen.tm4 into IdxGen.Aut and read it to set up all the anchors for the Story Author Index; it also outputs the data, including the story author anchors, in IdxGen.ttl; and the data, including artist information, in IdxGen.tm6
• Calls SETUP_ARTIST_IDX to sort IdxGen.tm6 into IdxGen.Art, and to set up all the anchors for the Artist Index
• Calls SETUP_CHRON_IDX to read IdxGen.ttl and extract the records needed for the Chronological Index into IdxGen.Crn, and to set up all the anchors for the Chronological Index
• Calls SETUP_SERIES_IDX to read IdxGen.ttl and extract the records needed for the Series Index into IdxGen.Ser, and to set up all the anchors for the Series Index
• Calls SETUP_BIOG_NOTES to set up the Biographical Notes Index
• Calls WRITE_TABLE_OF_CONTENTS to write the Table of Contents File
• If we had any books it then:
  • Calls BUILD_BOKAUT_IDX to build the Book Author Index
  • Calls BUILD_BOKTTL_IDX to build the Book Title Index
• Calls BUILD_ISSUE_IDX to build the Magazine Issue and store all the full text links in IdxGen.ftx
• Calls BUILD_FULLTEXT_IDX to read IdxGen.ftx and build the Full Text Index
• Calls BUILD_STYAUT_IDX to build the Story Author Index
• Calls BUILD_STYTTL_IDX to build the Story Title Index
• Calls BUILD_ARTIST_IDX to build the Artist Index
• Calls BUILD_CHRON_IDX to build the Chronological Index
• Calls BUILD_SERIES_IDX to build the Series Index
• Calls BUILD_BIOG_NOTES to build the Biographical Notes Index
• Calls WRITE_STATISTICS to build the Statistics Page
• Closes any open files, outputs usage summaries and elapsed time, and exits

PARSE_CONFIG - Read and Parse the Index Configuration File

This module simply reads the Index Configuration File, checking that all essential fields have been specified and defaulting any unspecified, optional, fields.

SETUP_FILES - Set up the list of Files

This module reads the list of files specified for the index and handles these differently depending on which type of file they are:

• Each magazine data file is added to the magfiles section of the Configuration Data structure, where the magazine name pointer is set up by calling GET_ABB_NAME - note that this is a pointer to an internal table so there is no need to allocate memory for it.
• Each additional reference file is also added to the magfiles section of the Configuration Data structure but the magazine name pointer points to a fixed string "~~~~", both to distinguish it and to sort it to the end if the magazine names are sorted.
• For each book data file, the routine first checks to see if we already have some books (got_books set) and, if not, opens the consolidated books data file (IdxGen.tm1 in the temporary folder) and calls READ_BOOK_FILE to read the data from the books file and add it to the consolidated books data file with the appropriate sort header.

When all the files have been read it checks to see if this index wants the magazine names to be sorted and, if so, calls qsort to sort the magfiles array and READ_MAG_FILES to read all the magazine files and create the consildated magazine data file (with added cover scan, full text and about link information).

It then checks to see if we had some books and, if so, calls the sort routine to sort it into order (IdxGen.tm2 in the temporary folder). It then opens the sorted file and copies the contents to the real consolidated book data file (IdxGen.bks in the temporary folder) stripping off the sort header.

SETUP_BOKAUT_IDX - Set up all the anchors for the Book Author Index

This module is only called if some books have been detected and, if so, reads all the records in the consolidated book data file (IdxGen.bks) and:

• Passes each records through to the SCAN_RECORD routine to add to/create the raw SCANITEM file.
• Sets up the structure of the Book Author Index, storing links to the book IDs in the Issue Link Table
• Stores links to the author entries in the Book Author Indexes in the internal bokautlink author structures and sorts the table

Note that:

• It is essential that the code that counts lines and calculates when to throw a new page matches that in BUILD_BOKAUT_IDX so that anchors are created where and when they should be (the latter routine does sanity checks to ensure this is the case). This routine has to do this for both the listings and contents levels as follows:
  • A new page is thrown at the contents level if we have passed the minimum page size and we have an 'A' record. Note that we should never need a continuation page at this level as we want each set of book contents on a single page, but we log a diagnostic (once per page) if we pass the maximum page size.
  • A new page is thrown at the listing level if we have passed the minimum page size and the author name has changed, or we have passed the maximum page size. Note that the latter will require a continuation page.
• We need an anchor for all books so we can link the Story Author and Story Title entries back to them so, if the book doesn't have a Book ID, we generate a fake one of the form Fnnnnn#mm where nnnnn is the current page number (padded to 5 digits) and mm is the current anchor number. In these cases we also need to create a fake "name" for the book we are linking back to that is similar to those translated via ABBREV.CVT (via FORMAT_BOOK_DETAILS)

SETUP_BOKTTL_IDX - Set up all the anchors for the Book Title Index

This routine ...

SETUP_ISSUE_IDX - Read all the files and set up the Issue Index

This module reads all the all the records in the consolidated book data file (IdxGen.mgs) and:

• Passes each records through to the SCAN_RECORD routine to add to the raw SCANITEM file (IdxGen.tmp).
• Sets up the structure of the Magazine Issue Index, storing links to the magazine issues and feature records in the Issue Link Table
• Stores links to the author entries in the Book Author Indexes in the internal author structures
• Sorts the Issue Link Table before exiting
• Adds special entries to the SCANITEM file for any 23/25/26 records in PSEUD.CVT so that we can add entries for them when needed.

Note that:

• It is essential that the code that counts lines and calculates when to throw a new page matches that in BUILD_ISSUE_IDX so that anchors are created where and when they should be (the latter routine does sanity checks to ensure this is the case). This routine has to do this for both the listings and contents levels as follows:
  • A new page is thrown at the contents level if we have passed the minimum page size and we have an 'A' record. Note that we should never need a continuation page at this level as we want each set of issue contents on a single page, but we log a diagnostic (once per page) if we pass the maximum page size.
  • A new page is thrown at the listing level if we have passed the minimum page size and we have a new main features record, or we have passed the maximum page size. Note that the latter will require a continuation page.
• We set up special anchors at the contents level for feature records so that we can handle cross-references. These start with a '%' followed by the magazine name so that they can't be confused with other entries.
• Anchors are created at the contents levels for all 'A' records including features records (the latter to allow links from the GCP website). As there can be multiple features records with the same magazine ID this routine adds ^^nmmm to the abbreviation where 'n' is 1 for a main feature record and 2 for a sub-header and mmm is the current record number to handle multiple sub-headers with the same abbreviation. After the array is sorted this means that the first entry for an abbreviation is either the first sub-header record for that abbreviation or the main features record if there were no sub-headers. The first suffix is then stripped off by the WRITE_LINKS_FILE routine.
• Although we need to create anchors for magazine features records at the listing level these are only ever used to link one level of an index to the next so there is no need to store them in the internal array (which avoids the problem that they might be duplicated).

WRITE_LINKS_FILE - Create the files for links from the GCP Website

This routine creates the Magazine Links File from the feature record entries in the Issue Link Table added by SETUP_ISSUE_IDX. It simply creates the top level file and then loops through the Issue Link Table looking for entries that start with four spaces and have an embedded ^^. As discussed above, there may be multiple entries for a given abbreviation, distinguished by a suffix after ^^ so the routine just strips off the suffix from the first such and writes an entry to the approproate Links File for it.

SETUP_STYTTL_IDX - Set up the Title Index

This module sorts the raw SCANITEM file (IdxGen.tmp) into title-order (as IdxGen.tm3). It then reads through the file setting up the structure of the Story Title Index, rewriting the file (as IdxGen.tm4) with the relevant links to the Story Title Index embedded.

SETUP_STYAUT_IDX - Set up the Story Author Index

This module sorts the title-order SCANITEM file into author-order. It then reads through the file setting up the structures of the Story Author and Artist Indexes, storing links for each author/artist in the internal author structures.

SETUP_ARTIST_IDX - Set up all the anchors for the Artist Index

This module ...

SETUP_CHRON_IDX - Set up the Chronological Index

This module sorts the title-order SCANITEM file into chronological-order. It then reads through the file setting up the structures of the Chronological Index, storing links for each author in the internal author structures.

SETUP_SERIES_IDX - Set up the Series Index

This module first needs to read through the file generated by SETUP_STYAUT_IDX (IdxGen.Ttl), which was in Story Title Index order and rewrite the entries that belong to a series (as IdxGen.tm8) in the order needed by the series index. This is complicated by a number of factors.

Firstly, if an item in a series also has a series prefix it will appear in the input file twice (once with the prefix and once without) but we only want a single entry in the series index so we ignore any titles that contain a "|" divider. A similar thing happens with articles about a series that quote the author as a subject which will have entries both for the author of the article and the subject so we want explicitly to ignore the latter.

Secondly, if the item in question is part of a serial, it will appear in the input file for each part of the serial, and the convention is that we only list the first (known) part with ", etc." appended if there are multiple parts (currently this also applies to multiple stories with the same name, as per v1, but we might want to consider changing this). We use the val_level field in the scanitem structure to indicate this - it is set to '0' if it is not part of a serial and to '1' otherwise. However, we won't know until we read the "next" record whether the flag needs to be set for the "current" record so the routine has to play around with two records at the same time.

Thirdly, if the item in question is by multiple authors, it will appear in the input file for both authors. In simple cases we just pass both through to the sorted file and then only list the first instance of each item. A complication occurs if the item is by a house name or joint pseudonym where the author(s) is known. In this instance the item will appear in the input file for the house name/joint pseudonym as well as the real author(s). For reasons that aren't currently clear the former does not contain the information needed to create a proper entry (listing house name/pseudonym as well as real author(s)) which is a problem if it sorts before the other entries. To address this the code checks to see if the current item is for a pseudonym (nonempty secnam_ptr) and the next is for a real name (empty secnam_ptr) then the current one is ignored in favour of the next. (This problem was noted for Petra Christian items authored by Christopher Priest on his own).

Fourthly, we want the items sorted in order of original publication date. If the original publication is in the file then it will appear first, but in some cases all we have in the file is a reprint so we need to sort out the original publication date, which might involve decoding a book abbreviation to see if there is a more precise date in ABBREV.CVT. In all cases it attempts to create a numeric version of the date so that we can sort by it.

Any item may belong to multiple series simultaneously so the routine next needs to split the series field into individual series. Then, for each series:

• It remembers any other series in the ednote_ptr field so we can add them later
• It works out the author name(s) to be used when sorting by author (the default): this has a few complications:
  • we want the name to be normalised but we can't use nrmaut_ptr as that is the normalised form of the byline, whereas series are grouped under the real name(s) so we need to re-nomalise the real names into the nrmaut_ptr field
  • we have to handle "Anon." by hand to sort it to the end of the list
  • we use CHK_SERIES to see if any of the authors in the current item are listed in SERIES.IDX as priority authors, prefixing the priority code to the normalised author string

processes the internal series records array, setting up the structure of the Series Index, and storing a pointer to each series entry in the internal series structures.

SETUP_BIOG_NOTES - Set up all the anchors for the Biographical Notes

This module ...

WRITE_TABLE_OF_CONTENTS - Write the Table of Contents File

BUILD_BOKAUT_IDX - Build the Book Author Index

This module ...

BUILD_BOKTTL_IDX - Build the Book Title Index

This module ...

BUILD_ISSUE_IDX - Build the Magazine Issue and Book Author Indexes

This module reads all the files defined in the (sorted) magazine file array followed by the consolidated book data file and creates the Magazine Issue and Book Author Indexes. This gets potentially complex as we are building four levels of index simultaneously:

• The top-level index (a1.htm for Books and a3.htm for Magazines)
• The middle-level index (pnnnnn.htm for both)
• The bottom-level index (bnnnnn.htm for both)
• The issue/book contents listings (tnnnnn.htm for both)

There are controlled by parallel sets of variables as follows:

• xxxfil_ptr - the file pointer for the associated file at the specified level (where xxx = top, mid, bot or iss as appropriate)
• xxxpage_count - the current page number at the specified level (mid, bot & iss only as the top-level index is always a single page)
• xxxline_count - the current line number at the specified level (mid, bot & iss only as the top-level index is always a single page)
• xxxanchor_cnt - the current anchor number for the current page at the specified level (bot & iss only as anchors aren't used in the other two)

For simplicity (?) the code is potentially executed twice - once for magazines and then again for books - on the assumption that the two passes have more in common than they have in differences (though it is debatable). For each pass, the code first creates (and writes a header to) the top-level file but, for simplicity, all the other files are created as needed based on the associated line_count and the values specified in the Configuration Data structure for minimum and maximum page sizes. Put simply:

• For the issue/contents listings we count a line for every record read from the input data files and throw a new page when we encounter an 'A' record if we have exceeded the minimum page size or unilaterally if we have exceed the maximum page size.
• For magazines, a single line is counted in the bottom-level index for each 'A' record and we throw a new page when we have a group header and have passed the minimum page size or unilaterally if we have exceeded the maximum page size.
• For books, we have the unusual situation that books without contents will only be output to the bottom-level index while for books with contents only the A & D records will be output to the bottom-level index, so we count a line for each 'A' or 'D' record and throw a new page when we have a new author and have passed the minimum page size or unilaterally if we have exceeded the maximum page size.
• For magazines, a single line is counted in the middle-level index for each 'A' record which is a features record and, again, we throw a new page when we have a group header and have passed the minimum page size or unilaterally if we have exceeded the maximum page size.
• For books, a single line is counted in the middle-level index for each new author, plus one for any unilateral page throws in the bottom-level index (in which case we need to add a continuation record to the middle-level index) and we throw a new page when we have a new author and have passed the minimum page size or unilaterally if we have exceeded the maximum page size.
• In each case, the top-level index simply has a line representing the first and last entries in each page of the middle-level index.

Note that, in the above, books edited by an author are treated as being the same as those written by the author.

One particular issue that arises with both sets of files is the handling of 'D' records for two reasons:

• In the UK format the 'D' records might contain information that is listed with the 'A' records (e.g. cover artist; series name)
• As discussed above, for book data files, whether or not the 'A' and 'D' records are output to the book contents listings depends on whether or not there are any 'E' records.

To handle this, whenever we encounter an 'A' record we call FLUSH_DREC to read all following 'D' records and the first (real) 'E' record if there is one.

BUILD_FULLTEXT_IDX - Build the Full-Text Index

This module ...

BUILD_STYAUT_IDX - Build the Story Author Index

This module reads the author-order SCANITEM data file and creates the Story Author Index.

BUILD_STYTTL_IDX - Build the Story Title Index

This module reads the title-order SCANITEM data file (IdxGen.ttl) and creates the Story Title Index. This is somewhat different to most of the build routines as the bottom level of the index is actually the Level 1 Index rather than the Listings Level.

BUILD_ARTIST_IDX - Build the Artist Index

This module ...

BUILD_CHRON_IDX - Build the Chronological Index

This module reads the chronological-order SCANITEM data file and creates the Chronological Index.

BUILD_SERIES_IDX - Build the Series Index

This module reads the internal series record array and creates the Series Index.

BUILD_BIOG_NOTES - Build the Biographical Notes

This module ...

WRITE_STATISTICS - Write the Statistics Page

This module ...

CREATE_NAME_LINK - Create a new nameslink entry

/************************************************************************/
/*									*/
/*  CREATE_NAME_LINK - Create a new nameslink entry			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	linksub = CREATE_NAME_LINK (scanitem_ptr, prtfil_ptr);		*/
/*									*/
/*  Where:								*/
/*									*/
/*	linksub	     = Subscript into nameslink table of (main) item	*/
/*		     = -1 if an error occurred				*/
/*	scanitem_ptr = Pointer to SCANITEM structure			*/
/*    	prtfil_ptr   = Pointer to diagnostics file (may be NULL)	*/
/*									*/
/*  This routine allocates an entry in the Names Link table for the	*/
/*  name specified in the supplied scanitem.				*/
/*									*/
/************************************************************************/

This routine allocates a new entry in the Names Link Table, setting up the name from the normalised name in the scanitem passed as a parameter. It calls FIND_NAME to set up the PSEUD.CVT entry (if any) and initialises all the other fields.

CREATE_NEW_PAGE - Create a new page for the specified file

/************************************************************************/
/*									*/
/*  CREATE_NEW_PAGE - Create a new page for the specified file		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = CREATE_NEW_PAGE (config_ptr, outfil_ptr, page_count,	*/
/*				  prefix_ptr, pagttl_ptr, prtfil_ptr);	*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	   = Result of operation:				*/
/*		   = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	outfil_ptr = Pointer to file pointer for creating new page	*/
/*	page_count = Current page count					*/
/*	prefix_ptr = File/Folder prefix letter				*/
/*	pagttl_ptr = Title of page					*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

All the HTML pages in the index follow the same basic format so this routine is called whenever we want to create a new page in an index level. If we have a current page (i.e. page_count > 1) it calls WRITE_PAGE_TRAILER to write the trailer to it and closes the old file. It then opens a new one and calls WRITE_PAGE_HEADER to write the page header to it. Note that, as the routine opens a new file (and hence creates a new file handle) the file parameter passed to it is a pointer to the file handle rather than the file handle itself. Note also that this routine depends on the value of config_ptr->curr_index_type when writing the page headers and trailers to indicate the home link.

FLUSH_DREC - Read all following 'D' records and first 'E' record (if any)

/************************************************************************/
/*									*/
/*  FLUSH_DREC - Read all 'D' records and 1st 'E' record		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = FLUSH_DREC (inpfil_ptr, linecnt_ptr, note_ptr_ptr,	*/
/*			     cvartist_ptr_ptr, series_ptr_ptr,		*/
/*			     flags_ptr_ptr, nxtrec_ptr_ptr, prtfil_ptr)	*/
/*									*/
/*  Where:								*/
/*									*/
/*	status		 = Routine return status:			*/
/*			 = PSP_TRUE if everything OK			*/
/*			 = PSP_FALSE if we hit an error			*/
/*	inpfil_ptr	 = Pointer to input file			*/
/*	linecnt_ptr	 = Pointer to line count to update		*/
/*	note_ptr_ptr	 = Pointer to buffer to hold pointer to notes	*/
/*	cvartist_ptr_ptr = Pointer to buffer to hold pointer to artist	*/
/*	series_ptr_ptr	 = Pointer to buffer to hold pointer to series	*/
/*	flags_ptr_ptr	 = Pointer to buffer to hold pointer to flags	*/
/*	nxtrec_ptr_ptr	 = Pointer to buffer to hold pointer to record	*/
/*    	prtfil_ptr	 = Pointer to diagnostics file (may be NULL)	*/
/*									*/
/************************************************************************/

There are a number of places where we need to know what the contents of any following 'D' records are when we process an 'A' record. It is also important to concatenate all the text in the 'D' records into a single buffer before calling CVT_TXT_TO_HTML so that it can correctly check for mismatched commands like "Bold On". This routine is called whenever we encounter a new 'A' record to read through and parse all the following 'D' records. As it won't know when we've run out of 'D' records it also has to read the first record after the 'D' records and return that to the caller. Because of this we are also able to parse any front cover records specified on a 'cv' record (see below). Some notes to bear in mind are:

• All the information being returned is built up in local static buffers, pointers to which are returned to the caller. As such, it is important that the returned information is either processed or stored elsewhere before the routine is called again.
• As this routine is reading records from the input file it needs to update the line count to ensure that the anchors generated in BUILD_ISSUE_IDX match those stored in SETUP_ISSUE_IDX
• The flags buffer is set up from any DQE records encountered and any partial data indicators (see below).
• The series buffer is set up from any DS records encountered.
• The cover artist buffer is set up either from a DC record or a record that has "E fc.A0" in the record type field and an item type of "cv". Note that this will not handle a cover artist 'E' record that has a folowing EB note. Note that the data returned is in the form Artist~Title~Title Additional~Publication Details~ where Publication Details does not include the item type.
• Any 'E' record with a title of " Need Contents" is ignored, both because we don't want to display and because it might precede a cover artist record.
• Any DP or DH records are translated in the same way as in CVTLOCUS.
• Any DE, DN or DQx records are ignored.
• All other D records are just concatenated into the buffer. In addition if any of them starts "Incomplete Data " then "PARTIAL" is added to the flags buffer.

FORMAT_AUTHOR - Format an Author Name as heading (with scanitem as input)

/************************************************************************/
/*									*/
/*  FORMAT_AUTHOR - Format an Author Name as heading			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	outbuf_ptr = FORMAT_AUTHOR (config_ptr, scanitem_ptr,		*/
/*				    auth_flag, link_typ, got_items,	*/
/*				    prtfil_ptr)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	outbuf_ptr   = Pointer to buffer containing formatted name	*/
/*		     = NULL if an error					*/
/*		     = "" if we don't want this one			*/
/*	config_ptr   = Pointer to CONFIG_DATA structure			*/
/*	scanitem_ptr = Pointer to SCANITEM structure			*/
/*	auth_flag    = FMTAUT_NEW if author name has changed		*/
/*		     = FMTAUT_CNT if we want a continuation name	*/
/*	link_typ     = Type of link required				*/
/*		     = LINK_UNKNOWN if unknown				*/
/*		     = LINK_STYAUT if story authors			*/
/*		     = LINK_BOKAUT if book authors			*/
/*		     = LINK_ARTIST if artists				*/
/*		     = LINK_CHRON if chronological index		*/
/*		     = LINK_BIOG if biographical notes			*/
/*		     = LINK_BIOG2 if biographical notes OR external	*/
/*	got_items    = PSP_TRUE if author has items in this index	*/
/*		     = PSP_FALSE otherwise				*/
/*     	prtfil_ptr   = Pointer to diagnostics file (may be NULL)	*/
/*									*/
/*  This formats a story author into the format used for the headers	*/
/*  in the listing file and for the middle-level index.			*/
/*									*/
/*  Note that the returned string has an embedded </A> to terminate	*/
/*  the <A HREF or <A NAME that will precede it.			*/
/*									*/
/************************************************************************/

This routine ...

FORMAT_AUTHOR2 - Format an Author Name as heading (with AUTH structure as input)

/************************************************************************/
/*									*/
/*  FORMAT_AUTHOR2 - Format an Author Name as heading			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	outbuf_ptr = FORMAT_AUTHOR2 (config_ptr, auth_ptr, link_typ,	*/
/*				     prtfil_ptr)			*/
/*									*/
/*  Where:								*/
/*									*/
/*	outbuf_ptr   = Pointer to buffer containing formatted name	*/
/*		     = NULL if an error					*/
/*	config_ptr   = Pointer to CONFIG_DATA structure			*/
/*	auth_ptr     = Pointer to author name				*/
/*	link_typ     = Type of link required				*/
/*		     = LINK_UNKNOWN if unknown				*/
/*		     = LINK_STYAUT if story authors			*/
/*		     = LINK_BOKAUT if book authors			*/
/*		     = LINK_ARTIST if artists				*/
/*		     = LINK_CHRON if chronological index		*/
/*		     = LINK_BIOG if biographical notes			*/
/*		     = LINK_BIOG2 if biographical notes OR external	*/
/*      prtfil_ptr   = Pointer to diagnostics file (may be NULL)	*/
/*									*/
/*  This is a shell around FORMAT_AUTHOR that is called when we don't	*/
/*  have a scanitem to hand.						*/
/*									*/
/************************************************************************/

This routine simple sets up the Normalised Author field in a local scanitem structure from the passed parameter and then calls FORMAT_AUTHOR.

FORMAT_AUTHTYPE - Format an Author Type as subheading

/************************************************************************/
/*									*/
/*  FORMAT_AUTHTYPE - Format an Author Type as subheading		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	output_ptr = FORMAT_AUTHTYPE (auth_type)			*/
/*									*/
/*  Where:								*/
/*									*/
/*	outbuf_ptr  = Pointer to formatted buffer			*/
/*	auth_type   = Author type					*/
/*									*/
/*  This formats a story author type into the format used for the	*/
/*  subheaders in the Story Author Listings File.			*/
/*									*/
/************************************************************************/

This routine checks to see if the SCANAUT_MAYBE flag is set and, if so, starts with ", [?]"; it then appends to the buffer a suffix such as ", after." that corresponds to the author type.

FORMAT_BOOK_DETAILS - Format a set of book details

/************************************************************************/
/*									*/
/*  FORMAT_BOOK_DETAILS - Format book details				*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	FORMAT_BOOK_DETAILS (outbuf_ptr, fld_ptr, bufsiz)		*/
/*									*/
/*  Where:								*/
/*									*/
/*	outbuf_ptr = Pointer to output buffer				*/
/*	fld_ptr    = Array of field pointers				*/
/*	bufsiz	   = Size of output buffer				*/
/*									*/
/************************************************************************/

This routine formats the contents of an 'A' record into the book detail format generated by FORMAT_PUBDET for a real abbreviation and is called from SETUP_BOKAUT_IDX for books without a real ID. It generates a string along the lines of:

Twelve Tales by Grant Allen, G. Richards, 1899

where the string has been made HTML-safe, the title and publisher have had any extraneous bits stripped off and the author/editor name(s) are formatted via BUILD_AUTH.

FORMAT_FULLTEXT_LINK - Format a Full Text link field

/************************************************************************/
/*									*/
/*  FORMAT_FULLTEXT_LINK - Format a Full Text link field		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	buff_ptr = FORMAT_FULLTEXT_LINK (link_ptr)			*/
/*									*/
/*  Where:								*/
/*									*/
/*	buff_ptr = Returned pointer to formatted field			*/
/*	date_ptr = Pointer to link field				*/
/*									*/
/*  This routine formats the contents of a full text link field.	*/
/*									*/
/************************************************************************/

This routine ...

FORMAT_ISSUE_LINK - Format link to specified issue (if possible)

/************************************************************************/
/*									*/
/*  FORMAT_ISSUE_LINK - Format link to specified issue (if possible)	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	link_ptr = FORMAT_ISSUE_LINK (config_ptr, pubdet_ptr,		*/
/*				  prtfil_ptr)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	link_ptr   = Pointer to Issue Details with link			*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	pubdet_ptr = Issue details to find link for			*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

This routine calls GET_PUBDET_LINK to see if the specified issue/book is in our index and, if so, to return the page and anchor references for it. It then calls FORMAT_PUBDET to format the publication details into the format we want and returns a pointer to an internal static buffer containing the formatted publication details, surrounded (if appropriate) by an A HREF link to the associated part of the index.

FORMAT_MAGPUB - Format Magazine Issue Details

/************************************************************************/
/*									*/
/*  FORMAT_MAGPUB - Format magazine issue details			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	FORMAT_MAGPUB (pubdet_ptr, fld_ptr, bufsiz)			*/
/*									*/
/*  Where:								*/
/*									*/
/*	pubdet_ptr = Pointer to output buffer				*/
/*	fld_ptr    = Array of field pointers				*/
/*	bufsiz	   = Size of output buffer				*/
/*    	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

FORMAT_NAMES - Format a names field.

/************************************************************************/
/*									*/
/*  FORMAT_NAMES - Format names field					*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	buff_ptr = FORMAT_NAMES (config_ptr, names_ptr, link_typ,	*/
/*				 format_typ, newtxt_ptr, prtfil_ptr)	*/
/*									*/
/*  Where:								*/
/*									*/
/*	buff_ptr   = Returned pointer to formatted names		*/
/*		   = NULL if we had an error				*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	names_ptr  = Pointer to names field				*/
/*	link_typ   = Type of link required				*/
/*		   = LINK_UNKNOWN if unknown				*/
/*		   = LINK_STYAUT if story authors			*/
/*		   = LINK_BOKAUT if book authors			*/
/*		   = LINK_ARTIST if artists				*/
/*		   = LINK_CHRON if chronological index			*/
/*		   = LINK_BIOG if biographical notes			*/
/*		   = LINK_BIOG2 if biographical notes OR external flag	*/
/*	format_typ = FMTNAM_NORMAL if we want all names			*/
/*		   = FMTNAM_HOUSE if we only want names in the index	*/
/*		     and want dates added to each			*/
/*		   = FMTNAM_PSEUD if we only want names in the index	*/
/*	newtxt_ptr = Replacement text to display			*/
/*		  (this is only valid if there is only a single primary	*/
/*		   name and no secondary names)				*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

This routine formats the contents of a names field, adding links to the associated section for the name. This is a fairly complex routine that attempts to parse a string of names into the format required by the indexes. It is (seriously) complicated because the basic mechanism is common to a wide range of differing circumstances that need careful handling. For example:

• In general, all names are linked to one of the five name-based indexes (BOKAUT, STYAUT, ARTIST, CHRON & BIOG) but the latter ("About") links vary depending on whether we are linking from within the BIOG index (when all links should link locally) or from elsewhere (when external links link directly to the external site) so there is an additional type, BIOG2, to handle the latter case. There is also an UNKNOWN type which, in theory, tries each of the indexes until it finds a link - it's unclear whether or not this is useful.
• In general, names are passed through as a standard name string in "internal format", but when we are parsing lists of pseudonyms or house names they are passed through as a string of authors separated by '/' characters. Technically this is the same as internal format, but in some cases there are a LOT of names in these strings which would mean extending the limits in AUTH_RTN which seems unnecessary, so we handle those by hand instead.
• In general we want to display all specified names whether they are in our index or not, but this does not apply to pseudonyms and house names where we ONLY want the names that exist in our index. Note that there might be no such names in which case an empty string is returned. Note that pseudonyms and house names are treated in the same way EXCEPT that dates are appended to each name if FMTNAM_HOUSE is specified.
• In general the (reverted) author's name is output within each hyperlink. However, if we have multiple authors with the same surname, the surname is suppressed on all but the last name. In addition, when handling embedded hyperlinks of the form "[%xxx]" there might be a replacement text specified which is used in place of the name. This is only valid if there is a single primary name and no secondary name(s).

A general complication is that we want to handle dividers neatly so that we have ", " separating all pairs of authors except the last pair which is separated by " & ", and we need to do this for the primary authors as a group, and for each set of (local or global) authors as a group. As it isn't easy to know when we've reached the last pair of authors (for primary authors because some might be suppressed; for secondary authors because there are multiple types held in the same structures) we handle this by inserting ", " dividers everywhere and remembering in each batch where we put them; then, at the end of the batch, we check to see if we had at least one divider and, if so, replace the last one with " & ".

In addition to the above, the routine is called in two different contexts. In general we have the whole author string to be formatted, but in STYAUT the author itself has already been output as the heading, so all we have to parse are the secondary names (which could be local or global). If we have both author name and secondary name then these are identical. Thus, for example:

	Maupassant, Guy de ,(tr:Wilson, Mary W.)

is translated as:

	Guy de Maupassant; translated by Mary W. Wilson

while:

	 ,(tr:Wilson, Mary W.)

is translated as:

	translated by Mary W. Wilson

However, complications arise if one of the names is specified as "Anon." or "[Unknown Author]" and we need to distinguish between the first case with a primary name of "Anon." and the second case where we don't have the primary name. For example:

	Anon. ,(by:Wilson, Mary W.)

is translated as:

	Mary W. Wilson, uncredited.

while:

	 ,(by:Wilson, Mary W.)

is translated as:

	(by Mary W. Wilson)

This is handled by the two flags got_anon which is set in the former case and no_primary_name which is set in the latter case. Note that got_anon is only set if there are some secondary names as otherwise we just output it as [uncredited].

• If we only have 1 primary author, which is "Anon." or equivalent and it has no secondary authors AND there are some global secondary names then we suppress the primary author. This is so that constructs such as "Anon. ,(as told to:Smith, Fred" are displayed as just "as told to Fred Smith" (not if link_typ is LINK_BOKAUT).
• If we have multiple authors where all the authors have the same surname we suppress the surname on all but the final author (not if link_typ is LINK_BOKAUT).
• For all except LINK_BOKAUT, the authors are output in a single string separated by commas and an ampersand for the last pair, followed by any global secondary names; for LINK_BOKAUT the first primary author is left inverted, does not have a link added and is separated from the other authors with a special divider of ^||.
• If LINK_UNKNOWN is specified the routine tries LINK_STYAUT first followed by LINK_ARTIST (not sure if this is needed or not).
  

FORMAT_NOTES - Format a Notes Field

/************************************************************************/
/*									*/
/*  FORMAT_NOTES - Format a notes field					*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = FORMAT_NOTES (config_ptr, outbuf_ptr, notes_ptr,	*/
/*			       outbuflen, link_typ, prtfil_ptr);	*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	   = PSP_TRUE if notes formatted successfully		*/
/*		   = PSP_FALSE otherwise				*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	outbuf_ptr = Pointer to output buffer				*/
/*	notes_ptr  = Pointer to input buffer				*/
/*	outbuflen  = Size of output buffer				*/
/*	link_typ   = Type of link required				*/
/*		   = LINK_UNKNOWN if unknown				*/
/*		   = LINK_STYAUT if story authors			*/
/*		   = LINK_BOKAUT if book authors			*/
/*		   = LINK_ARTIST if artists				*/
/*		   = LINK_CHRON if chronological index			*/
/*		   = LINK_BIOG if biographical notes			*/
/*		   = LINK_BIOG2 if biographical notes OR external flag	*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/*  This routine formats the contents of a notes field, translating	*/
/*  any [@ or [% hyperlinks.						*/
/*									*/
/*  Note that this routine APPENDS to the output buffer.		*/
/*									*/
/************************************************************************/

This routine ...

FORMAT_PUBDATE - Format Pub. Info. date field

/************************************************************************/
/*									*/
/*  FORMAT_PUBDATE - Format Pub. Info. date field			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	buff_ptr = FORMAT_PUBDATE (date_ptr)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	buff_ptr = Returned pointer to formatted date			*/
/*		 = NULL if we had an error				*/
/*	date_ptr = Pointer to date field				*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/*  This routine formats the contents of a date field in the Pub. Info.	*/
/*  section. These are of the form [cc]yy/mm[/dd].			*/
/*									*/
/************************************************************************/

This routine ...

FORMAT_SERIES - Format series field

/************************************************************************/
/*									*/
/*  FORMAT_SERIES - Format series field					*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	buff_ptr = FORMAT_SERIES (config_ptr, series_ptr, include_flg,	*/
/*				   prtfil_ptr)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	buff_ptr	= Returned pointer to formatted names		*/
/*			= NULL if we had an error			*/
/*			= "" if no matching series			*/
/*	config_ptr	= Pointer to CONFIG_DATA structure		*/
/*	series_ptr	= Pointer to series field			*/
/*	include_flg 	= PSP_TRUE if we want all series		*/
/*			= PSP_FALSE if we only want series in the index	*/
/*	prtfil_ptr	= Pointer to diagnostics file (may be NULL)	*/
/*									*/
/*  This routine formats the contents of a series field, adding links	*/
/*  to the associated section for the series.				*/
/*									*/
/************************************************************************/

FORMAT_URL - Format a URL specified in an FM data file

/************************************************************************/
/*									*/
/*  FORMAT_URL - Format a URL specified in an FM data file		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	output_ptr = FORMAT_URL (url_ptr, text_flg, prtfil_ptr)		*/
/*									*/
/*  Where:								*/
/*									*/
/*	outbuf_ptr = Pointer to formatted buffer			*/
/*	url_ptr	   = Pointer to input URL				*/
/*	text_ptr   = PSP_FALSE if formatting the URL as hyperlink	*/
/*		   = PSP_TRUE if formatting the URL for display		*/
/*    	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/*  This formats a URL into the format required for use as a hypelink	*/
/*  or for display to the user.						*/
/*									*/
/************************************************************************/

This routine ...

GET_ANCHOR - Get the HTML to link to a specific anchor

/************************************************************************/
/*									*/
/*  GET_ANCHOR - Get the HTML to link to a specific anchor		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	link_ptr = GET_ANCHOR (config_ptr, prefix_ptr, page_count,	*/
/*			       anchor_count)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	link_ptr     = Pointer to HTML link				*/
/*	config_ptr   = Pointer to CONFIG_DATA structure			*/
/*	prefix_ptr   = Filename/folder prefix				*/
/*	page_count   = Page number					*/
/*	anchor_count = Anchor number					*/
/*		     = -1 if just page link wanted			*/
/*									*/
/************************************************************************/

Depending on the setting in the Index Configuration File all the files in an index might be in a single folder or in multiple subfolders. To avoid checking this all over the place (and to allow for future flexibility) all links are set up via this routine which simply constructs the relevant HTML link.

GET_ANCHOR2 - Get the HTML to link to a name index anchor

/************************************************************************/
/*									*/
/*  GET_ANCHOR2 - Get the HTML to link to a name index anchor		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	link_ptr = GET_ANCHOR (config_ptr, link_typ, page_count,	*/
/*			       anchor_count)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	link_ptr     = Pointer to HTML link				*/
/*	config_ptr   = Pointer to CONFIG_DATA structure			*/
/*	link_typ     = Type of link required				*/
/*		     = LINK_STYAUT if story authors			*/
/*		     = LINK_BOKAUT if book authors			*/
/*		     = LINK_ARTIST if artists				*/
/*		     = LINK_CHRON if chronological index		*/
/*		     = LINK_BIOG or LINK_BIOG2 if biographical notes	*/
/*	page_count   = Page number					*/
/*		     = -1 if just directory link wanted			*/
/*	anchor_count = Anchor number					*/
/*		     = -1 if just page link wanted			*/
/*									*/
/*  This is just a shell around GET_ANCHOR				*/
/*									*/
/************************************************************************/

This routine ...

GET_NAME_INDEX - Get index of specified name (if possible)

/************************************************************************/
/*									*/
/*  GET_NAME_INDEX - Get index of specified name (if possible)		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	index = GET_NAME_INDEX (name_ptr, link_typ, prtfil_ptr)		*/
/*									*/
/*  Where:								*/
/*									*/
/*	index      = Index into tables for specified name		*/
/*		   = -1 if link not found				*/
/*	name_ptr   = Name to find link for				*/
/*	link_typ   = Type of link required				*/
/*		   = LINK_UNKNOWN if unknown				*/
/*		   = LINK_STYAUT if story authors			*/
/*		   = LINK_BOKAUT if book authors			*/
/*		   = LINK_ARTIST if artists				*/
/*		   = LINK_CHRON if chronological index			*/
/*		   = LINK_BIOG if biographical notes			*/
/*		   = LINK_BIOG2 if biographical notes OR external flag	*/
/*    	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

This routine ...

GET_NAME_LINK - Get link to specified name (if possible)

/************************************************************************/
/*									*/
/*  GET_NAME_LINK - Get link to specified name (if possible)		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	GET_NAME_LINK (name_ptr, link_typ, pageno_ptr, anchor_ptr,	*/
/*		       prtfil_ptr)					*/
/*									*/
/*  Where:								*/
/*									*/
/*	name_ptr   = Name to find link for				*/
/*	link_typ   = Type of link required				*/
/*		   = LINK_UNKNOWN if unknown				*/
/*		   = LINK_STYAUT if story authors			*/
/*		   = LINK_BOKAUT if book authors			*/
/*		   = LINK_ARTIST if artists				*/
/*		   = LINK_CHRON if chronological index			*/
/*		   = LINK_BIOG if biographical notes			*/
/*		   = LINK_BIOG2 if biographical notes OR external flag	*/
/*	pageno_ptr = Pointer to int to hold page number			*/
/*		   = -1 if link not found				*/
/*		   = -2 if LINK_BIOG2 & we have external link		*/
/*	anchor_ptr = Pointer to int to hold anchor			*/
/*    	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

GET_PUBDET_LINK - Get offset of link to specified item (if possible)

/************************************************************************/
/*									*/
/*  GET_PUBDET_LINK - Find set of publication details in issue links	*/
/*			  table (if possible)				*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	index = GET_PUBDET_LINK (pubdet_ptr)				*/
/*									*/
/*  Where:								*/
/*									*/
/*	index	   = Index into issue links table			*/
/*		   = -1 if no link found				*/
/*	pubdet_ptr = Publication details to look for			*/
/*									*/
/************************************************************************/

This routine simply looks up the specified publication details is the isslinks array (first converting it to the "old" format if necessary) and returns the offset into the arrays (or -1 if the details can't be found).

GET_SERIES_LINK - Get link to specified series (if possible)

/************************************************************************/
/*									*/
/*  GET_SERIES_LINK - Get link to specified series (if possible)	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	GET_SERIES_LINK (series_ptr, pageno_ptr, anchor_ptr, prtfil_ptr)*/
/*									*/
/*  Where:								*/
/*									*/
/*	series_ptr = Series name to find link for			*/
/*	pageno_ptr = Pointer to int to hold page number			*/
/*		   = -1 if link not found				*/
/*	anchor_ptr = Pointer to int to hold anchor			*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

READ_BOOK_FILE - Read book data file into consolidated book data file

/************************************************************************/
/*									*/
/*  READ_BOOK_FILE - Read book data file & add to consolidated file	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = READ_BOOK_FILE (filnam_ptr, outfil_ptr, prtfil_ptr);	*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	   = Result of operation:				*/
/*		   = PSP_TRUE if OK; else PSP_FALSE			*/
/*	filnam_ptr = Name of book data file				*/
/*	outfil_ptr = Consolidated data file to write to			*/
/*    	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/*  This routine opens the specified book data file and copies all the	*/
/*  the contents to the output file, adding a sort prefix.		*/
/*									*/
/************************************************************************/

While each individual book data file will probably be sorted into order (thanks to ASORT) we will be combining multiple book data files so we need to sort them all into a consistent order. This routine is called for each book data file and simply copies the contents to the consolidated data file with a sort prefix for each record consisting of the output from LOCUS_COMPACT for the most recent/current 'A' record followed by a six-digit record number (to keep items in order within a book and books with the same compacted form in the same order in the data file) followed by a second '~' divider, followed by the input record.

Note that this routine:

• Ignores all records before the first "real" 'A' record
• Ignores all blank records

The main complication is introduced by the use of DQN records to suppress an entire book. Early versions of the code left it up to the later code to handle this, but that led to page-sizing problems (as well as being inelegant). The next version tried to use fgetpos & fsetpos on the output file to reset it to just before the preceding 'A' record if we hit a DQN record, but that presented possible portability problems and, more to the point, didn't seem to work for obscure reasons. Rather than try to sort out the problem the next version got rid of fgetpos and fsetpos by using an array of buffers to read the records into:

• read_first loops around until it has the first 'A' record that isn't the file header; the 'A' record is stored in recbuf[0]
• read_drecs then loops around reading all following 'D' records into recbuf[n]; note that:
  • If a "DQN" record is encountered then the suppress_book flag is set to say so;
  • Otherwise all "DQ" records are ignored as we don't want them in the index itself;
  • By definition we will always end up with the next record after the last 'D' record in the array as well;
• process_recs is reached when we encounter the first non-D- record; we know we must have an 'A' record in recbuf[0] so it calls LOCUS_COMPACT to set up the sort header
• do_next then outputs each record to the output file (as long as the suppress_book flag is not set), incrementing the record count for each and drops through to read_next which reads the next record either from the array or (if we have exhausted the array) from the file; when it find the next 'A' record it resets the relevant flags, makes sure the 'A' record is in recbuf[0] (if the 'A' record immediately followed the 'D' records for the previous book it would be at the end of the array) and branch back to read_drecs to handle the next book.

READ_MAG_FILES - Read all magazine files into a consolidated magazine data file

/************************************************************************/
/*									*/
/*  READ_MAG_FILES - Read magazine data files, cover scan file and	*/
/*		     full-text link file & create consolidated file	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = READ_MAG_FILES (config_ptr, prtfil_ptr);		*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	   = Result of operation:				*/
/*		   = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*    	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

This routine sets up sorted lists of cover scan, full text and about links and then reads through the magazine data files creating a consolidated data file with the relevant links added. Note that these are added as "extra" fields (FLD_IMGLNK & FLD_TXTLNK) at the end of the 'A' record. In an ideal world we could just read the files into memory and access them when needed (as we do for PSEUD.CVT and such-like) but I'm getting paranoid about memory usage and we only need the links when processing the 'A' records and can then free up the allocated memory. It also means we only have to deal with two input files in SETUP_ISSUE_IDX and BUILD_ISSUE_IDX rather than reading through all the magazine files each time.

Note that we check the cover scans first because there are many more of them (130,000 vs. 9,000) and, when adding the full-text links, we do a binary chop on the cover links part of what we've set up to save time (we initially did a sequential search and this module took 15 seconds or more). The about links are associated only with (main) feature records so are just added as separate entries. Note that the file containing the about links is not in a standard folder and hence is specified via the Index Configuration File.

REPORT_DIAGS - Report Extended Diagnostics (if required)

/************************************************************************/
/*									*/
/*  REPORT_DIAGS - Report Extended Diagnostics (if required)		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	REPORT_DIAGS (config_ptr, report_type);				*/
/*									*/
/*  Where:								*/
/*	config_ptr  = Pointer to CONFIG_DATA structure			*/
/*	report_type = Type of data to report:				*/
/*		    = DMP_MAGLIST for Magazine File List		*/
/*		    = DMP_CONFIG for Configuration Data			*/
/*		    = IDX_ARTIST for Artist Index			*/
/*		    = IDX_BIOG for Biographical Notes			*/
/*		    = IDX_BOKAUT for Book Author Index			*/
/*		    = IDX_CHRON for Chronological Index			*/
/*		    = IDX_ISSUE for Issue Index				*/
/*		    = IDX_SERIES for Series Index			*/
/*		    = IDX_STYAUT for Story Author Index			*/
/*		    = IDX_FULLTXT for Full Text Index			*/
/*									*/
/************************************************************************/

This routine simpy checks to see if extended diagnostics are required and, if so, dumps out the array(s) associated with the report type specified.

SPLIT_TITLE - Split title field into constituent parts

/************************************************************************/
/*									*/
/*  SPLIT_TITLE - Split title field into constituent parts		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	SPLIT_TITLE (title_ptr, titlad_ptr, itemad_ptr, colhdr_ptr_ptr,	*/
/*		     divider_ptr_ptr, title_ptr_ptr)			*/
/*									*/
/*  Where:								*/
/*									*/
/*	buff_ptr	= Returned pointer to formatted title		*/
/*	title_ptr	= Pointer to title field			*/
/*	titlad_ptr	= Pointer to title additional field		*/
/*	itemad_ptr	= Pointer to item additional field		*/
/*	colhdr_ptr_ptr	= Pointer to hold pointer to column title	*/
/*	divider_ptr_ptr = Pointer to hold pointer to divider		*/
/*	titl_ptr_ptr	= Pointer to hold pointer to item title		*/
/*									*/
/*  This routine combines the title and item additional fields with	*/
/*  the title, intelligently handling sort titles, series dividers	*/
/*  and story numbers, and returns pointers to the three elements of	*/
/*  the title - the column/series name, the string to use as a divider	*/
/*  and	the item title (including any series sequence). The first	*/
/*  two fields are returned as an empty string if not relevant.		*/
/*									*/
/************************************************************************/

This routine ...

TIDY_DNOTES - Tidy up contents of any 'D' Notes

/************************************************************************/
/*									*/
/*  TIDY_DNOTES - Tidy up contents of any 'D' Notes			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	outbuf_ptr = TIDY_DNOTES (config_ptr, inpbuf_ptr, prtfil_ptr);	*/
/*									*/
/*  Where:								*/
/*									*/
/*	outbuf_ptr = Pointer to buffer with tidied notes.		*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	inpbuf_ptr = Pointer to input buffer				*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/*  This routine checks to see if the 'D' notes need any special	*/
/*  formatting (e.g. "--- see under xxx).				*/
/*									*/
/************************************************************************/

This routine first checks to see if the notes start with the special prefix "--- see under " and, if so, check the next character to see what sort of link is required:

• A '{' indicates a magazine (issue) link and can be specified as either publication details or text (i.e. a magazine name).
• A '<' indicates a book line which can only be specified as publication details.
• A '(' indicates an author name which must be in external format.
• Anything else is illegal.

Publication details and magazine names are converted via FORMAT_ISSUE_LINK (with a ':' prefixed in the latter case); names are converted to internal format via TRANSLATE_AUTH and then converted via FORMAT_NAMES. Magazine links are output in italics; book links in bold and author links normally.

If the notes do not start with the special prefix then the routine simply calls the shared FORMAT_NOTES routine.

WRITE_INDEX_LINE - Write a line to the intermediate index(es)

/************************************************************************/
/*									*/
/*  WRITE_INDEX_LINE - Write a line to the intermediate index(es)	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = WRITE_INDEX_LINE (config_ptr, reqtype, text_ptr,	*/
/*				   prtfil_ptr);				*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	    = Result of operation:				*/
/*		    = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr  = Pointer to CONFIG_DATA structure			*/
/*	reqtype	    = Request Type:					*/
/*		    = IDXLIN_NORMAL for normal index line		*/
/*		    = IDXLIN_CONT for continuation line			*/
/*		    = IDXLIN_SPECIAL for special index lines		*/
/*		    = IDXLIN_TRAIL for trailers				*/
/*	text_ptr    = text to use for continuation or for special	*/
/*		      (if IDXLIN_CONT or IDXLIN_SPECIAL)		*/
/*		    = text to use for continuation otherwise		*/
/*    	prtfil_ptr  = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/*  This routine outputs a line to the Level 1 Index, throwing a new	*/
/*  page and adding a line to the Level 2 Index, etc., if necessary.	*/
/*									*/
/************************************************************************/

This routine is called when a line is to be output to the intermediate (Level 1 to 3) Indexes and implements the 3-level index structure described in the Basic Index Structure - see the section on using WRITE_INDEX_LINE for an outline description of how to use the routine.

It outputs the relevant line to the Level 1 Index; if necessary it also throws a new page and adds a line to the Level 2 Index and, if also necessary, throws a new page in the Level 2 Index and adds a line to the Level 3 Index. In addition to the formal parameters, it relies on the following fields in config_ptr:

• standard items defining the index:
  • curr_index_type - current index type being processed
  • minpagesize - minimum number of lines in a page
  • subfolders - determines if files should be generated in sub-folders
  • idxdir - folder to generate indexes into
• information for formatting the Level 1 Index (not needed for IDXLIN_SPECIAL)
  • formatted_name - formatted name of current item (i.e. format to use in Level 1 Index; not needed for IDXLIN_SPECIAL)
  • lstpage_count - current page number in Listings Level
  • lstanchor_count - current anchor number in Listings Level (not needed for IDXLIN_SPECIAL)
• information needed for Level 2 & 3 Indexes
  • curr_name - name of current item

It operates in one of four modes depending on the contents of the reqtype parameter. By default this is set to IDXLIN_NORMAL indicating that a normal index line is to be output. If so:

• the routine first checks a static flag to see if this is the first time and, if so:
  • it opens the "top level" output file for this index (as defined by the static topfilnam_ptr table) and writes a standard page header to it via WRITE_PAGE_HEADER
  • it then works out how many index levels we need by looking at the number of lines in the level 1 index (idxlinecnt) and the minimum number of lines in a page (minpagesize); idxlinecnt is basically a count of each new item (e.g. author) needing an index entry plus each time a continuation page is thrown; the calculation is biassed in favour of fewer levels to try to avoid ever having a top page with a single line in it.
  • if only one index level is needed, it sets the top level output file as the level 1 index file; if two levels are needed it sets it as the level 2 index file and sets the level 1 line count to force an immediate new page; if three levels are needed it sets it as the level 3 index file and sets both level 1 and level 2 line counts to force an immediate new page.
  • it also works out whether names in the indexes should be output in bold or italics.
• the routine then counts another line in the level 1 index and checks to see if we have at least 2 levels of index and if the level 1 line count now exceeds the maximum for a page (always true first time round) and, if so:
  • increments the level 1 page count and, if this isn't the first page:
    • writes a page trailer to the current level 1 index file
    • writes a line to the level 2 index file describing the current level 1 page
  • calls CREATE_NEW_PAGE to close the current level 1 index file and open a new one
  • writes a header to the new level 1 index file
  • saves the current name (curr_name) as the first item for the next level 2 index line
  • increments the level 2 line count
  • if we have 3 levels of index and if the level 2 line count now exceeds the maximum for a page (always true first time round) repeat the whole process for the level 2/3 indexes
• saves the current name (curr_name) as the last item (so far) for the next level 2 & level 3 index lines
• formats and outputs the line for the level 1 index

If reqtype is set to IDXLIN_SPECIAL the routine continues exactly as above except that the line for the level 1 index is passed in the text_ptr parameter rather than being set up automatically (this is used for the Story and Book Title Indexes)

If reqtype is set to IDXLIN_CONT then we have a continuation line. In this case we still count a line in the level 1 index and output the continuation line, but do not throw any new pages.

If reqtype is set to IDXLIN_TRAIL then we need to clean up for this index and reset for the next index:

• the routine writes the trailers to the Level 1 Index and closes the associated file;
• if there are at least 2 levels for this index, it then writes a final line to the Level 2 Index, followed by the trailers, and closes the associated file;
• if there are 3 levels for this index, it then repeats the process for the Level 3 Index
• it then updates the global page count (pages_cnt) with the number of pages used in this index and resets the "first time" flag for next time round.

WRITE_PAGE_HEADER - Write a standard header to a file

/************************************************************************/
/*									*/
/*  WRITE_PAGE_HEADER - Write a standard header to the specified file	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = WRITE_PAGE_HEADER (config_ptr, outfil_ptr, pagttl_ptr,	*/
/*				    prvlnk_ptr, homlnk_ptr, prtfil_ptr);*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	   = Result of operation:				*/
/*		   = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	outfil_ptr = File to write header to				*/
/*	pagttl_ptr = Title of page					*/
/*	prvlnk_ptr = Previous link (if NULL, none created)		*/
/*	homlnk_ptr = Index home link (if NULL, none created)		*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

This routine simply reads through the Page Header boilerplate file (index_hdr.htm), making the necessary substitutions and outputting the result to the specified file. Note that, for performance reasons, the whole boilerplate file is stored in memory when it is first read to avoid unnecessary file I/O.

WRITE_PAGE_TRAILER - Write a standard trailer to a file

/************************************************************************/
/*									*/
/*  WRITE_PAGE_TRAILER - Write a standard trailer to the specified file	*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = WRITE_PAGE_TRAILER (config_ptr, outfil_ptr,		*/
/*				     nxtlnk_ptr, homlnk_ptr,		*/
/*				     prtfil_ptr);			*/
/*									*/
/*	Where:								*/
/*									*/
/*	status	   = Result of operation:				*/
/*		   = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr = Pointer to CONFIG_DATA structure			*/
/*	outfil_ptr = File to write header to				*/
/*	nxtlnk_ptr = Next link (if NULL, none created)			*/
/*	homlnk_ptr = Index home link (if NULL, none created)		*/
/*	prtfil_ptr = Pointer to diagnostics file (may be NULL)		*/
/*									*/
/************************************************************************/

WRITE_PUB_INFO - Format and output all "Pub Info" records

/************************************************************************/
/*									*/
/*  WRITE_PUB_INFO - Format and output all "Pub Info" records		*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = WRITE_PUB_INFO (config_ptr, inpfil_ptr, outfil_ptr,	*/
/*				 nxtrec_ptr_ptr, prtfil_ptr)		*/
/*									*/
/*  Where:								*/
/*									*/
/*	status	       = Result of operation:				*/
/*		       = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr     = Pointer to CONFIG_DATA structure		*/
/*	inpfil_ptr     = Pointer to input file				*/
/*	outfil_ptr     = File to write header to			*/
/*	nxtrec_ptr_ptr = Pointer to buffer to hold pointer to record	*/
/*	prtfil_ptr     = Pointer to diagnostics file (may be NULL)	*/
/*									*/
/************************************************************************/

This routine ...

WRITE_STORY_ITEM - Write a single story item group

/************************************************************************/
/*									*/
/*  WRITE_STORY_ITEM - Write a single story item group			*/
/*									*/
/*  Calling Format:							*/
/*									*/
/*	status = WRITE_STORY_ITEM (config_ptr, outfil_ptr, fldbuf_ptr,	*/
/*				   fldbufsiz, scanitem_ptr, newpage_flg,*/
/*				   prtfil_ptr)				*/
/*									*/
/*	Where:								*/
/*									*/
/*	status	     = Result of operation:				*/
/*		     = PSP_TRUE if OK; else PSP_FALSE			*/
/*	config_ptr   = Pointer to CONFIG_DATA structure			*/
/*	outfil_ptr   = File to write header to				*/
/*	fldbuf_ptr   = Pointer to buffer for current record		*/
/*	fldbufsize   = Size of buffer for current record		*/
/*	scanitem_ptr = Pointer to SCANITEM structure for current record	*/
/*	newpage_flg  = PSP_TRUE if we've just thrown a new page		*/
/*		     = PSP_FALSE otherwise				*/
/*    	prtfil_ptr   = Pointer to diagnostics file (may be NULL)	*/
/*									*/
/************************************************************************/

This routine formats and outputs all records which have the same author, author type, title, coauthors, secondary names and ED notes. While this seems fairly simple, it has turned out to be fiendishly complex for a number of reasons, primarily to do with aggregation (see the discussion under Notes on Some (Past) Problem Areas for some examples). For simple, uncomplicated, items the intent is simply to display the first appearance of an item, followed by any reprint information on subsequent lines (in chronological order) as in:

• Justice Comes to Red Creek, (nv) Exciting Western January 1951
  • Exciting Western (Canada) January 1951
  • Exciting Western (UK) October 1951

In this simple case, we have entries in the sorted data for the original appearance as well as the reprints, each of which identifies the original appearance. However, there are cases where all we have is an instance defining the reprint appearance which may or may not identify the original appearance, as in (in the WFI):

• Nightmare Island, (ss) Short Stories June 25, 1932
  • All Star Western & Frontier Magazine December 1932

Clearly, in this case, we need to output the first line (even if the first appearance is unknown) as part of processing the second line, while in the previous case above we want to suppress the first line for the reprints as we have already displayed it. We handle this by checking the pubdet_ptr field in each scanitem structure to see if it is the same as the previous one.

There are even (obscure) cases where we have (vague) original appearance data without any reprint information, as in:

• Blue-Fired Cow-Killing Crazies, (ss) Western Digest 1996

in which case we need to output the first line but not the second line even though we are dealing with a reprint record (in this case magid_ptr is empty).

However, if the item is a serial (say) then we don't want to list each part separately so we try to aggregate all the instances into a single line so that, if the above were serialised over three issues, it might appear as:

• Justice Comes to Red Creek, (nv) Exciting Western Jan, Feb, Mar 1951
  • Exciting Western (Canada) Jan, Feb, Mar 1951
  • Exciting Western (UK) Oct, Dec 1951, Feb 1952

There are several points to note about this simple case:

• To save space (as, for columns, there could be hundreds of these) the dates are abbreviated from the normal form.
• Each issue for the serial (or column) needs to be hyperlinked, but the text associated with that hyperlink might contain just the month/day, the magazine name and month/day or the month/day plus year. This is handled by some clever (i.e. messy) code in FORMAT_PUBDET but also means that we need to know what year the next item is while we're processing the current one.
• In this case, for reprints, we have to ignore which issue each item is reprinted from (or it would destroy the aggregation) so we need to ignore the setting of pubdet_ptr no matter whether it is specified or not.
• We immediately have a problem with sort orders. To achieve the chronological sequencing we need to sort the items (via SETUP_SCANITEM) in order of dtpubl_ptr (after edition which is needed to ensure the first appearance comes before any reprints with the same date). To achieve the grouping in the example just above it would be ideal to sort all the items in a given magazine together as well, but that cannot precede the publication date or the basic sequence (the first example above) might be disrupted. This means that when we are aggregating reprints, if a series of items are reprinted in multiple magazines with overlapping dates they will be presented in date order, not magazine order - this has not yet been resolved.

To make life even more interesting, items appear as a single item in their original appearance but multiple items when reprinted or vice versa (note that the first of these is particularly common if the original appearance lies outside the current index), as in (in the WFI):

• The Charmed Life, (nv) All-Story Weekly September 22, 1917
  • Best Stories Oct, Nov 1927

or:

• Trouble Trail, (n.) Western Story Magazine Aug 28, Sep 4, Sep 11, Sep 18, Sep 25, Oct 2 1926 (as by George Owen Baxter)
  • Triple Western December 1950 (as by Max Brand)
  • Triple Western (Canada) December 1950 (as by Max Brand)

This latter highlights yet another problem with aggregation, albeit one new to the v2 indexes, - i.e. bylines. Although this example is fairly straightforward, when we are talking about a long column, it is possible that the column will appear under multiple bylines (the most common example known being when an editor uses their initials for some instances). In a perfect world we would list these separately, but that would mean sorting by byline_ptr before dtpubl_ptr which would disrupt the "normal entries". We might be able to "put aside" any with a different byline while we process the current set or come up with an alternative (e.g. append the byline to only those issues that differ, which is fine as long as there is isn't an overall byline to display as well) - all this has been left as an exercise for later: for now we just (try to) ignore such differences.

To try to manage the above, the routine has a bewildering number of flags it sets and checks:

• need_header: If true means that need to output the header describing the item (and any co-authors, secondary names, etc. etc.). In most cases this only happens once but we might have the same item appearing (as first appearances) in multiple magazines or books in which case we can't aggregate them as the magazine/book has changed, nor list them as subsidiary as this implies reprints, so we need output the header again.
• need_maghdr: If true this means that the magazine name needs to be output. This is always true if need_header is true, but is also true if we're starting a new batch of reprint information.

The first part of the routine simply sets up the item header that we need whether we have one item or a thousand:

• It sets up the formatted Story Title and Column Headers. Note that we suppress the latter when doing subjects as we're only really interested in the item title in such cases.
• It sets up the Story Title Link
• It outputs the HTML for the column header if we want it (i.e. not defined in SERIES.CVT with a code of 'Z') and haven't already done it
• It then sets up the Item Header with something (not clear)
• If we have some series information it adds that (note that we skip this for secondary authors)
• We then set up any bylines, co-authors and/or secondary authors. Note that, if we're not doing a subject the byline is output before the co-authors. Initially for subjects we ignored the co-authors but this didn't work too well for book reviews (particularly when different versions of the same review cited different co-authors so that the listings were not aggregated but there was no visible reason why) so co-authors are output for book reviews, but in this case before the byline (i.e. before the author of the review). Co-authors don't look so good for other subject entries (e.g. when an article is about multiple authors) so the co-authors are still supressed in that case.
• We then set up the subject for the item (if any)

Note that this potentially causes problems with book reviews (see the discussion under Notes on Some (Past) Problem Areas below) as we might want to change the byline in the itme header within a single set of results. We bodge this by saving the offsets in the itemheader before and after we add the byline so that we can reset it when needed.

As we read them in we organise them into groups using a set of interlinked structures:

• Group Array (group_str): each instance of this identifies all occurrences under a single byline in a given file and contains:
  • The file abbreviation (scanitem_ptr->filabb_ptr)
  • The edition (scanitem_ptr->edition)
  • The book_type (scanitem_ptr->book_type)
  • An array of subscripts into the Item Array
  • A subscript into the Extras Array if necessary
  • A subscript linking to the parent group for reprints
• Group Extras Array (group_xtr): most groups are quite simple but, in some cases, additional information is needed. As there might be hundreds of groups we try to save memory by having a separate array of these, used only when necessary. Each entry contains:
  • The byline on the item (scanitem_ptr->byline_ptr)
  • The original title if there was one (scanitem_ptr->origtad_ptr + scanitem_ptr->origttl_ptr)
  • Any seconday names on the item (scanitem_ptr->secnam_ptr)
• Item Array (item_str): we have one of these for each distinct item (as of 2024, one group needed almost 1500 instances). Each contains:
  • The original publication details (scanitem_ptr->pubdet_ptr)
  • The current publication details (scanitem_ptr->magid_ptr)
  • The current publication details in "old format" (scanitem_ptr->magid_ptr converted if it starts with '|' or copied otherwise)
  • Year of publication (first 4 digits of scanitem_ptr->dtpubl_ptr)
  • A subscript into the Item Extras array if necessary
• Item Extras Array (item_rep): As with the Groupo Extras Array this is hived off to save space and contains, where relevant:
  • The reprint title (scanitem_ptr->repttl_ptr)
  • The reprint author (scanitem_ptr->repaut_ptr)

The routine first sets up the entries for the first item (i.e. the one in scanitem_ptr when the routine was first called) - in the vast majority of cases this will be the only item in the set. It then reads the next item and checks to see if it:

• is by the same author (scanitem_ptr->auth_ptr) with the same author type (scanitem_ptr->authtype)
• has the same title (scanitem_ptr->ttad_ptr + scanitem_ptr->titl_ptr)
• has the same co-authors (scanitem_ptr->coauth_ptr)
• has the same secondary names (scanitem_ptr->secnam_ptr)
• has the same ED notes (scanitem_ptr->ednote_ptr)
• has the same item type (1st 2 characters of scanitem_ptr->pubdet_ptr)
• is an item or an item reprinted as a book (i.e. scanitem_ptr->book_type !+ '1')
• is not a piece of anonymous artwork (there are simply too many instances and it's not something anybody's likely to be interested in)

If all these are true then it aggregates the item with the previous one(s). It first checks to see if it can be added to an existing group, i.e.:

• the byline matches that on the group (or neither has a byline)
• the file abbreviation exists and is the same (the former check is because book files don't have a file abbreviation and we don't want to aggregate all items in book files together; this needs changing as it currently means a new group for each instance of an item in a book file and for some items there are over 100 of these)
• the edition is the same

If not, it create a new group. It then creates an entry in the Item Array (and possibly Item Extras Array) and links it to the matched or new group. It then goes back to read the next item and repeats the above.

Once we have all the arrays set up, we need to tidy them up and sort them into the order necessary.

First we look at each reprint group (edition=2 or book_type=3) and try to find a prior group with an item with matching publication details. Generally there will always be one such because the code in SETUP_STYAUT_IDX always tries to create one, and the presence of the edition in the sort order will sort it before the reprints (and hence the group will be set up earlier). If it finds a match then it saves the Group ID as the Parent ID.

It then runs through the groups again, looking for original groups (edition=1 or book_type=1) or an orphaned reprint (i.e. parent_sub = -1) adding each to the ordered list of groups. For each group it finds it then checks the remaining groups for which it is parent and adds them next before moving on. Thus, for a complex set-up we should end up with:

• First original group (this will be the one with the earliest publication date)
  • First group with reprints from this group
  • Second group with reprints from this group
  • etc.
• Second original group
  • First group with reprints from this group
  • etc.
• etc.

We then step through each group in turn. If the original title

Notes on Some (Past) Problem Areas

Aggregation Problems

Many of the items in the database effectively contain two different pieces of publication data – details of the current publication and of the original publication – although obviously in many case these are the same and/or the second one is missing. In many/most cases there is also a record in the database for the original publication so we don't need to worry about it, but there are many instances where this is not the case.

If the original publication was under a different byline or title then this has always been catered for in FLUSH_SORT as we would need records in different areas of the sort hierarchy. However, if this was not the case, originally the issue was posponed until WRITE_STORY_ITEM when the code checked to see if we already had the original publication and, if not, output it. This works fine for simple cases, but led to problems in aggregation for two reasons:

• For repeated items (such as columns) for which we only had reprint information, some/all of the original appearances weren't listed.
• When they were listed, it was not uncommon for the original appearances to be aggregated with the reprint appearances rather than with the other original appearances.

A brief attempt was made to fix this by creating new records in FLUSH_SORT for such records but this seemed to introduce more problems (not least a major increase in the size of the files) so a second attempt (in 2024) focussed on SETUP_STYAUT_IDX. Having sorted the data into the order required for the STYAUT (i.e. Names) index, the routine then read through the data sequentially and for each set of records for a given item checked that we had a record for the original publication and, if not, added one. This worked pretty well, but there were three immediate complications:

• The original publication details might be multi-issue details (i.e. including '+' or '-'). One possibility would be to try to convert these into the equivalent "[Part 1 of m]" in the same way as XVALIDATE does but this was deemed unnecessary as the complications only occur with aggregations and such items (probably) wouldn't be aggregated if the original publication details didn't exist. Instead the code in WRITE_STORY_ITEM discussed above was left in place to handle such cases. To distinguish between cases where we did want the original publication details listed and where we didn't a new flag (done_original) was added to the scanitem structure purely for this purpose. We also don't want to create an "original publication" record if the current item type is "ex" as we don't know what it is an extract from.
• This approach breaks a cardinal (unwritten) rule that all the files from IdxGen.tmp onward have exactly the same records (or defined subsets thereof). In particular these records are added after SETUP_STYTTL_IDX has set up all the anchors for the title index but before BUILD_STYTTL_IDX checks them, which is a recipe for disaster. To avoid this the same flag (done_original) is used to flag the new items and BUILD_STYTTL_IDX simply ignores them. This can't do any harm as we only added them for the benefit of the Names Index.
• The aggregation code attempts to aggregate all records of the same "type" that come from the same file (by checking the filabb_ptr field in the scanitem structure) – this copes with cases where the same magazine changes abbreviations but is but is fundamentally the same magazine. This is a problem when creating records for the original appearance from a reprint item as we don't have the relevant data for the original printing – it will usually be the same as the magazine abbreviation but clearly this is not always the case. To address this SETUP_ISSUE_IDX creates a lookaside list of abbreviations that don't match the associated filabb_ptr and SETUP_STYAUT_IDX then checks this when adding the new record.

Even with the above fix there was still a problem with book reviews. Initially there was an additional check in the aggregation check to see if the next item "was either not about the specified author or had the same byline on the piece about the author". This works fine when the only reviews of a given book are by the same author. However, if the same book is reviewed by multiple different authors, and those reviews are then reprinted elsewhere (very common in British Reprint Editions of SF magazines) this breaks as all the original editions are sorted before all the reprint editions so that for each review the original and the reprint are probably not adjacent and hence end up in separate groups. Both before and after the above fix this meant the same review was listed twice, separated by other reviews of the same book by other people.

This was fixed by removing the check in WRITE_STORY_ITEM, when aggregating items for a "subject" that they were all by the same byline and then handling this via the group mechanism as discussed under WRITE_STORY_ITEM.