IDXGEN consists of the following main routines:
which communicate via the Configuration Data structure and the Other Global Data. There are also a number of significant support routines:
Note that several of the BUILD_xxx routines revolve around the use of Boilerplate Files and the whole program relies on a sequence of Sorted Files. Note also that the program relies on a naming convention for the data files as discussed here and an external sort command file.
Currently IDXGEN only supports UK 7-bit format input files - it wouldn't be rocket science to extend this to handle US format files as well but it isn't a priority at present.
The main program:
This module simply reads the Index Configuration File, checking that all essential fields have been specified and defaulting any unspecified, optional, fields.
This module reads the list of files specified for the index and handles these differently depending on which type of file they are:
When all the files have been read it checks to see if this index wants the magazine names to be sorted and, if so, calls qsort to sort the magfiles array and READ_MAG_FILES to read all the magazine files and create the consildated magazine data file (with added cover scan, full text and about link information).
It then checks to see if we had some books and, if so, calls the sort routine to sort it into order (IdxGen.tm2 in the temporary folder). It then opens the sorted file and copies the contents to the real consolidated book data file (IdxGen.bks in the temporary folder) stripping off the sort header.
This module is only called if some books have been detected and, if so, reads all the records in the consolidated book data file (IdxGen.bks) and:
Note that:
This routine ...
This module reads all the all the records in the consolidated book data file (IdxGen.mgs) and:
Note that:
This routine creates the Magazine Links File from the feature record entries in the Issue Link Table added by SETUP_ISSUE_IDX. It simply creates the top level file and then loops through the Issue Link Table looking for entries that start with four spaces and have an embedded ^^. As discussed above, there may be multiple entries for a given abbreviation, distinguished by a suffix after ^^ so the routine just strips off the suffix from the first such and writes an entry to the appropriate Links File for it.
This module sorts the raw SCANITEM file (IdxGen.tmp) into title-order (as IdxGen.tm3). It then reads through the file setting up the structure of the Story Title Index, rewriting the file (as IdxGen.tm4) with the relevant links to the Story Title Index embedded.
This module sorts the title-order SCANITEM file into author-order. It then reads through the file setting up the structures of the Story Author and Artist Indexes, storing links for each author/artist in the internal author structures.
This module ...
This module sorts the title-order SCANITEM file into chronological-order. It then reads through the file setting up the structures of the Chronological Index, storing links for each author in the internal author structures.
This module first needs to read through the file generated by SETUP_STYAUT_IDX (IdxGen.Ttl), which was in Story Title Index order and rewrite the entries that belong to a series (as IdxGen.tm8) in the order needed by the series index. This is complicated by a number of factors.
Firstly, if an item in a series also has a series prefix it will appear in the input file twice (once with the prefix and once without) but we only want a single entry in the series index so we ignore any titles that contain a "|" divider. A similar thing happens with articles about a series that quote the author as a subject which will have entries both for the author of the article and the subject so we want explicitly to ignore the latter.
Secondly, if the item in question is part of a serial, it will appear in the input file for each part of the serial, and the convention is that we only list the first (known) part with ", etc." appended if there are multiple parts (currently this also applies to multiple stories with the same name, as per v1, but we might want to consider changing this). We use the val_level field in the scanitem structure to indicate this - it is set to '0' if it is not part of a serial and to '1' otherwise. However, we won't know until we read the "next" record whether the flag needs to be set for the "current" record so the routine has to play around with two records at the same time.
Thirdly, if the item in question is by multiple authors, it will appear in the input file for both authors. In simple cases we just pass both through to the sorted file and then only list the first instance of each item. A complication occurs if the item is by a house name or joint pseudonym where the author(s) is known. In this instance the item will appear in the input file for the house name/joint pseudonym as well as the real author(s). For reasons that aren't currently clear the former does not contain the information needed to create a proper entry (listing house name/pseudonym as well as real author(s)) which is a problem if it sorts before the other entries. To address this the code checks to see if the current item is for a pseudonym (nonempty secnam_ptr) and the next is for a real name (empty secnam_ptr) then the current one is ignored in favour of the next. (This problem was noted for Petra Christian items authored by Christopher Priest on his own).
Fourthly, we want the items sorted in order of original publication date. If the original publication is in the file then it will appear first, but in some cases all we have in the file is a reprint so we need to sort out the original publication date, which might involve decoding a book abbreviation to see if there is a more precise date in ABBREV.CVT. In all cases it attempts to create a numeric version of the date so that we can sort by it.
Any item may belong to multiple series simultaneously so the routine next needs to split the series field into individual series. Then, for each series:
processes the internal series records array, setting up the structure of the Series Index, and storing a pointer to each series entry in the internal series structures.
This module ...
This module ...
This module ...
This module reads all the files defined in the (sorted) magazine file array followed by the consolidated book data file and creates the Magazine Issue and Book Author Indexes. This gets potentially complex as we are building four levels of index simultaneously:
There are controlled by parallel sets of variables as follows:
For simplicity (?) the code is potentially executed twice - once for magazines and then again for books - on the assumption that the two passes have more in common than they have in differences (though it is debatable). For each pass, the code first creates (and writes a header to) the top-level file but, for simplicity, all the other files are created as needed based on the associated line_count and the values specified in the Configuration Data structure for minimum and maximum page sizes. Put simply:
Note that, in the above, books edited by an author are treated as being the same as those written by the author.
One particular issue that arises with both sets of files is the handling of 'D' records for two reasons:
To handle this, whenever we encounter an 'A' record we call FLUSH_DREC to read all following 'D' records and the first (real) 'E' record if there is one.
This module ...
This module reads the author-order SCANITEM data file and creates the Story Author Index.
This module reads the title-order SCANITEM data file (IdxGen.ttl) and creates the Story Title Index. This is somewhat different to most of the build routines as the bottom level of the index is actually the Level 1 Index rather than the Listings Level.
This module ...
This module reads the chronological-order SCANITEM data file and creates the Chronological Index.
This module reads the internal series record array and creates the Series Index.
This module ...
This module ...
/************************************************************************/
/* */
/* CREATE_NAME_LINK - Create a new nameslink entry */
/* */
/* Calling Format: */
/* */
/* linksub = CREATE_NAME_LINK (scanitem_ptr, prtfil_ptr); */
/* */
/* Where: */
/* */
/* linksub = Subscript into nameslink table of (main) item */
/* = -1 if an error occurred */
/* scanitem_ptr = Pointer to SCANITEM structure */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine allocates an entry in the Names Link table for the */
/* name specified in the supplied scanitem. */
/* */
/************************************************************************/
This routine allocates a new entry in the Names Link Table, setting up the name from the normalised name in the scanitem passed as a parameter. It calls FIND_NAME to set up the PSEUD.CVT entry (if any) and initialises all the other fields.
/************************************************************************/
/* */
/* CREATE_NEW_PAGE - Create a new page for the specified file */
/* */
/* Calling Format: */
/* */
/* status = CREATE_NEW_PAGE (config_ptr, outfil_ptr, page_count, */
/* prefix_ptr, pagttl_ptr, prtfil_ptr); */
/* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* outfil_ptr = Pointer to file pointer for creating new page */
/* page_count = Current page count */
/* prefix_ptr = File/Folder prefix letter */
/* pagttl_ptr = Title of page */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
All the HTML pages in the index follow
the same basic format so this routine is called whenever we want to create a
new page in an index level. If we have a current page (i.e. page_count >
1) it calls WRITE_PAGE_TRAILER to write the
trailer to it and closes the old file. It then opens a new one and calls WRITE_PAGE_HEADER
to write the page header to it. Note that, as the routine opens a new file (and
hence creates a new file handle) the file parameter passed to it is a pointer
to the file handle rather than the file handle itself. Note also that this routine
depends on the value of config_ptr->curr_index_type when writing the page
headers and trailers to indicate the home link.
/************************************************************************/
/* */
/* FLUSH_DREC - Read all 'D' records and 1st 'E' record */
/* */
/* Calling Format: */
/* */
/* status = FLUSH_DREC (inpfil_ptr, linecnt_ptr, note_ptr_ptr, */
/* cvartist_ptr_ptr, series_ptr_ptr, */
/* flags_ptr_ptr, nxtrec_ptr_ptr, prtfil_ptr) */
/* */
/* Where: */
/* */
/* status = Routine return status: */
/* = PSP_TRUE if everything OK */
/* = PSP_FALSE if we hit an error */
/* inpfil_ptr = Pointer to input file */
/* linecnt_ptr = Pointer to line count to update */
/* note_ptr_ptr = Pointer to buffer to hold pointer to notes */
/* cvartist_ptr_ptr = Pointer to buffer to hold pointer to artist */
/* series_ptr_ptr = Pointer to buffer to hold pointer to series */
/* flags_ptr_ptr = Pointer to buffer to hold pointer to flags */
/* nxtrec_ptr_ptr = Pointer to buffer to hold pointer to record */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
There are a number of places where we need to know what the contents of any following 'D' records are when we process an 'A' record. It is also important to concatenate all the text in the 'D' records into a single buffer before calling CVT_TXT_TO_HTML so that it can correctly check for mismatched commands like "Bold On". This routine is called whenever we encounter a new 'A' record to read through and parse all the following 'D' records. As it won't know when we've run out of 'D' records it also has to read the first record after the 'D' records and return that to the caller. Because of this we are also able to parse any front cover records specified on a 'cv' record (see below). Some notes to bear in mind are:
/************************************************************************/
/* */
/* FORMAT_AUTHOR - Format an Author Name as heading */
/* */
/* Calling Format: */
/* */
/* outbuf_ptr = FORMAT_AUTHOR (config_ptr, scanitem_ptr, */
/* auth_flag, link_typ, got_items, */
/* prtfil_ptr) */
/* */
/* Where: */
/* */
/* outbuf_ptr = Pointer to buffer containing formatted name */
/* = NULL if an error */
/* = "" if we don't want this one */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* scanitem_ptr = Pointer to SCANITEM structure */
/* auth_flag = FMTAUT_NEW if author name has changed */
/* = FMTAUT_CNT if we want a continuation name */
/* link_typ = Type of link required */
/* = LINK_UNKNOWN if unknown */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG if biographical notes */
/* = LINK_BIOG2 if biographical notes OR external */
/* got_items = PSP_TRUE if author has items in this index */
/* = PSP_FALSE otherwise */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This formats a story author into the format used for the headers */
/* in the listing file and for the middle-level index. */
/* */
/* Note that the returned string has an embedded </A> to terminate */
/* the <A HREF or <A NAME that will precede it. */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* FORMAT_AUTHOR2 - Format an Author Name as heading */
/* */
/* Calling Format: */
/* */
/* outbuf_ptr = FORMAT_AUTHOR2 (config_ptr, auth_ptr, link_typ, */
/* prtfil_ptr) */
/* */
/* Where: */
/* */
/* outbuf_ptr = Pointer to buffer containing formatted name */
/* = NULL if an error */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* auth_ptr = Pointer to author name */
/* link_typ = Type of link required */
/* = LINK_UNKNOWN if unknown */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG if biographical notes */
/* = LINK_BIOG2 if biographical notes OR external */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This is a shell around FORMAT_AUTHOR that is called when we don't */
/* have a scanitem to hand. */
/* */
/************************************************************************/
This routine simple sets up the Normalised Author field in a local scanitem structure from the passed parameter and then calls FORMAT_AUTHOR.
/************************************************************************/
/* */
/* FORMAT_AUTHTYPE - Format an Author Type as subheading */
/* */
/* Calling Format: */
/* */
/* output_ptr = FORMAT_AUTHTYPE (auth_type) */
/* */
/* Where: */
/* */
/* outbuf_ptr = Pointer to formatted buffer */
/* auth_type = Author type */
/* */
/* This formats a story author type into the format used for the */
/* subheaders in the Story Author Listings File. */
/* */
/************************************************************************/
This routine checks to see if the SCANAUT_MAYBE flag is set and, if so, starts with ", [?]"; it then appends to the buffer a suffix such as ", after." that corresponds to the author type.
/************************************************************************/
/* */
/* FORMAT_BOOK_DETAILS - Format book details */
/* */
/* Calling Format: */
/* */
/* FORMAT_BOOK_DETAILS (outbuf_ptr, fld_ptr, bufsiz) */
/* */
/* Where: */
/* */
/* outbuf_ptr = Pointer to output buffer */
/* fld_ptr = Array of field pointers */
/* bufsiz = Size of output buffer */
/* */
/************************************************************************/
This routine formats the contents of an 'A' record into the book detail format generated by FORMAT_PUBDET for a real abbreviation and is called from SETUP_BOKAUT_IDX for books without a real ID. It generates a string along the lines of:
Twelve Tales by Grant Allen, G. Richards, 1899
where the string has been made HTML-safe, the title and publisher have had any extraneous bits stripped off and the author/editor name(s) are formatted via BUILD_AUTH.
/************************************************************************/
/* */
/* FORMAT_FULLTEXT_LINK - Format a Full Text link field */
/* */
/* Calling Format: */
/* */
/* buff_ptr = FORMAT_FULLTEXT_LINK (link_ptr) */
/* */
/* Where: */
/* */
/* buff_ptr = Returned pointer to formatted field */
/* date_ptr = Pointer to link field */
/* */
/* This routine formats the contents of a full text link field. */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* FORMAT_ISSUE_LINK - Format link to specified issue (if possible) */
/* */
/* Calling Format: */
/* */
/* link_ptr = FORMAT_ISSUE_LINK (config_ptr, pubdet_ptr, */
/* prtfil_ptr) */
/* */
/* Where: */
/* */
/* link_ptr = Pointer to Issue Details with link */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* pubdet_ptr = Issue details to find link for */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
This routine calls GET_PUBDET_LINK to see if the specified issue/book is in our index and, if so, to return the page and anchor references for it. It then calls FORMAT_PUBDET to format the publication details into the format we want and returns a pointer to an internal static buffer containing the formatted publication details, surrounded (if appropriate) by an A HREF link to the associated part of the index.
/************************************************************************/
/* */
/* FORMAT_MAGPUB - Format magazine issue details */
/* */
/* Calling Format: */
/* */
/* FORMAT_MAGPUB (pubdet_ptr, fld_ptr, bufsiz) */
/* */
/* Where: */
/* */
/* pubdet_ptr = Pointer to output buffer */
/* fld_ptr = Array of field pointers */
/* bufsiz = Size of output buffer */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
/************************************************************************/
/* */
/* FORMAT_NAMES - Format names field */
/* */
/* Calling Format: */
/* */
/* buff_ptr = FORMAT_NAMES (config_ptr, names_ptr, link_typ, */
/* format_typ, newtxt_ptr, prtfil_ptr) */
/* */
/* Where: */
/* */
/* buff_ptr = Returned pointer to formatted names */
/* = NULL if we had an error */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* names_ptr = Pointer to names field */
/* link_typ = Type of link required */
/* = LINK_UNKNOWN if unknown */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG if biographical notes */
/* = LINK_BIOG2 if biographical notes OR external flag */
/* format_typ = FMTNAM_NORMAL if we want all names */
/* = FMTNAM_HOUSE if we only want names in the index */
/* and want dates added to each */
/* = FMTNAM_PSEUD if we only want names in the index */
/* newtxt_ptr = Replacement text to display */
/* (this is only valid if there is only a single primary */
/* name and no secondary names) */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
This routine formats the contents of a names field, adding links to the associated section for the name. This is a fairly complex routine that attempts to parse a string of names into the format required by the indexes. It is (seriously) complicated because the basic mechanism is common to a wide range of differing circumstances that need careful handling. For example:
A general complication is that we want to handle dividers neatly so that we have ", " separating all pairs of authors except the last pair which is separated by " & ", and we need to do this for the primary authors as a group, and for each set of (local or global) authors as a group. As it isn't easy to know when we've reached the last pair of authors (for primary authors because some might be suppressed; for secondary authors because there are multiple types held in the same structures) we handle this by inserting ", " dividers everywhere and remembering in each batch where we put them; then, at the end of the batch, we check to see if we had at least one divider and, if so, replace the last one with " & ".
In addition to the above, the routine is called in two different contexts. In general we have the whole author string to be formatted, but in STYAUT the author itself has already been output as the heading, so all we have to parse are the secondary names (which could be local or global). If we have both author name and secondary name then these are identical. Thus, for example:
Maupassant, Guy de ,(tr:Wilson, Mary W.)
is translated as:
Guy de Maupassant; translated by Mary W. Wilson
while:
,(tr:Wilson, Mary W.)
is translated as:
translated by Mary W. Wilson
However, complications arise if one of the names is specified as "Anon." or "[Unknown Author]" and we need to distinguish between the first case with a primary name of "Anon." and the second case where we don't have the primary name. For example:
Anon. ,(by:Wilson, Mary W.)
is translated as:
Mary W. Wilson, uncredited.
while:
,(by:Wilson, Mary W.)
is translated as:
(by Mary W. Wilson)
This is handled by the two flags got_anon which is set in the former case and no_primary_name which is set in the latter case. Note that got_anon is only set if there are some secondary names as otherwise we just output it as [uncredited].
/************************************************************************/
/* */
/* FORMAT_NOTES - Format a notes field */
/* */
/* Calling Format: */
/* */
/* status = FORMAT_NOTES (config_ptr, outbuf_ptr, notes_ptr, */
/* outbuflen, link_typ, prtfil_ptr); */
/* */
/* Where: */
/* */
/* status = PSP_TRUE if notes formatted successfully */
/* = PSP_FALSE otherwise */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* outbuf_ptr = Pointer to output buffer */
/* notes_ptr = Pointer to input buffer */
/* outbuflen = Size of output buffer */
/* link_typ = Type of link required */
/* = LINK_UNKNOWN if unknown */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG if biographical notes */
/* = LINK_BIOG2 if biographical notes OR external flag */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine formats the contents of a notes field, translating */
/* any [@ or [% hyperlinks. */
/* */
/* Note that this routine APPENDS to the output buffer. */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* FORMAT_PUBDATE - Format Pub. Info. date field */
/* */
/* Calling Format: */
/* */
/* buff_ptr = FORMAT_PUBDATE (date_ptr) */
/* */
/* Where: */
/* */
/* buff_ptr = Returned pointer to formatted date */
/* = NULL if we had an error */
/* date_ptr = Pointer to date field */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine formats the contents of a date field in the Pub. Info. */
/* section. These are of the form [cc]yy/mm[/dd]. */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* FORMAT_SERIES - Format series field */
/* */
/* Calling Format: */
/* */
/* buff_ptr = FORMAT_SERIES (config_ptr, series_ptr, include_flg, */
/* prtfil_ptr) */
/* */
/* Where: */
/* */
/* buff_ptr = Returned pointer to formatted names */
/* = NULL if we had an error */
/* = "" if no matching series */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* series_ptr = Pointer to series field */
/* include_flg = PSP_TRUE if we want all series */
/* = PSP_FALSE if we only want series in the index */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine formats the contents of a series field, adding links */
/* to the associated section for the series. */
/* */
/************************************************************************/
/************************************************************************/
/* */
/* FORMAT_URL - Format a URL specified in an FM data file */
/* */
/* Calling Format: */
/* */
/* output_ptr = FORMAT_URL (url_ptr, text_flg, prtfil_ptr) */
/* */
/* Where: */
/* */
/* outbuf_ptr = Pointer to formatted buffer */
/* url_ptr = Pointer to input URL */
/* text_ptr = PSP_FALSE if formatting the URL as hyperlink */
/* = PSP_TRUE if formatting the URL for display */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This formats a URL into the format required for use as a hypelink */
/* or for display to the user. */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* GET_ANCHOR - Get the HTML to link to a specific anchor */
/* */
/* Calling Format: */
/* */
/* link_ptr = GET_ANCHOR (config_ptr, prefix_ptr, page_count, */
/* anchor_count) */
/* */
/* Where: */
/* */
/* link_ptr = Pointer to HTML link */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* prefix_ptr = Filename/folder prefix */
/* page_count = Page number */
/* anchor_count = Anchor number */
/* = -1 if just page link wanted */
/* */
/************************************************************************/
Depending on the setting in the Index Configuration File all the files in an index might be in a single folder or in multiple subfolders. To avoid checking this all over the place (and to allow for future flexibility) all links are set up via this routine which simply constructs the relevant HTML link.
/************************************************************************/
/* */
/* GET_ANCHOR2 - Get the HTML to link to a name index anchor */
/* */
/* Calling Format: */
/* */
/* link_ptr = GET_ANCHOR (config_ptr, link_typ, page_count, */
/* anchor_count) */
/* */
/* Where: */
/* */
/* link_ptr = Pointer to HTML link */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* link_typ = Type of link required */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG or LINK_BIOG2 if biographical notes */
/* page_count = Page number */
/* = -1 if just directory link wanted */
/* anchor_count = Anchor number */
/* = -1 if just page link wanted */
/* */
/* This is just a shell around GET_ANCHOR */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* GET_BOKTTL_INDEX - Find index in book title links table for a */
/* particular record number. */
/* */
/* Calling Format: */
/* */
/* index = GET_BOKTTL_INDEX (recnum); */
/* */
/* Where: */
/* */
/* index = Index into book title links table */
/* = -1 if no link found */
/* recnum = Record number to look for */
/* */
/************************************************************************/
This routine simply looks up the specified record number in the Book Title Index and returns the offset into the arrays (or -1 if the details can't be found).
/************************************************************************/
/* */
/* GET_NAME_INDEX - Get index of specified name (if possible) */
/* */
/* Calling Format: */
/* */
/* index = GET_NAME_INDEX (name_ptr, link_typ, prtfil_ptr) */
/* */
/* Where: */
/* */
/* index = Index into tables for specified name */
/* = -1 if link not found */
/* name_ptr = Name to find link for */
/* link_typ = Type of link required */
/* = LINK_UNKNOWN if unknown */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG if biographical notes */
/* = LINK_BIOG2 if biographical notes OR external flag */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* GET_NAME_LINK - Get link to specified name (if possible) */
/* */
/* Calling Format: */
/* */
/* GET_NAME_LINK (name_ptr, link_typ, pageno_ptr, anchor_ptr, */
/* prtfil_ptr) */
/* */
/* Where: */
/* */
/* name_ptr = Name to find link for */
/* link_typ = Type of link required */
/* = LINK_UNKNOWN if unknown */
/* = LINK_STYAUT if story authors */
/* = LINK_BOKAUT if book authors */
/* = LINK_ARTIST if artists */
/* = LINK_CHRON if chronological index */
/* = LINK_BIOG if biographical notes */
/* = LINK_BIOG2 if biographical notes OR external flag */
/* pageno_ptr = Pointer to int to hold page number */
/* = -1 if link not found */
/* = -2 if LINK_BIOG2 & we have external link */
/* anchor_ptr = Pointer to int to hold anchor */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
/************************************************************************/
/* */
/* GET_PUBDET_INDEX - Find set of publication details in issue links */
/* table (if possible) */
/* */
/* Calling Format: */
/* */
/* index = GET_PUBDET_INDEX (pubdet_ptr) */
/* */
/* Where: */
/* */
/* index = Index into issue links table */
/* = -1 if no link found */
/* pubdet_ptr = Publication details to look for */
/* */
/************************************************************************/
This routine simply looks up the specified publication details in the Issue Link Table (first converting it to the "old" format if necessary) and returns the offset into the arrays (or -1 if the details can't be found).
/************************************************************************/
/* */
/* GET_SERIES_LINK - Get link to specified series (if possible) */
/* */
/* Calling Format: */
/* */
/* GET_SERIES_LINK (series_ptr, pageno_ptr, anchor_ptr, prtfil_ptr)*/
/* */
/* Where: */
/* */
/* series_ptr = Series name to find link for */
/* pageno_ptr = Pointer to int to hold page number */
/* = -1 if link not found */
/* anchor_ptr = Pointer to int to hold anchor */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
/************************************************************************/
/* */
/* READ_BOOK_FILE - Read book data file & add to consolidated file */
/* */
/* Calling Format: */
/* */
/* status = READ_BOOK_FILE (filnam_ptr, outfil_ptr, prtfil_ptr); */
/* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* filnam_ptr = Name of book data file */
/* outfil_ptr = Consolidated data file to write to */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine opens the specified book data file and copies all the */
/* the contents to the output file, adding a sort prefix. */
/* */
/************************************************************************/
While each individual book data file will probably be sorted into order (thanks to ASORT) we will be combining multiple book data files so we need to sort them all into a consistent order. This routine is called for each book data file and simply copies the contents to the consolidated data file with a sort prefix for each record consisting of the output from LOCUS_COMPACT for the most recent/current 'A' record followed by a six-digit record number (to keep items in order within a book and books with the same compacted form in the same order in the data file) followed by a second '~' divider, followed by the input record.
Note that this routine:
The main complication is introduced by the use of DQN records to suppress an entire book. Early versions of the code left it up to the later code to handle this, but that led to page-sizing problems (as well as being inelegant). The next version tried to use fgetpos & fsetpos on the output file to reset it to just before the preceding 'A' record if we hit a DQN record, but that presented possible portability problems and, more to the point, didn't seem to work for obscure reasons. Rather than try to sort out the problem the next version got rid of fgetpos and fsetpos by using an array of buffers to read the records into:
/************************************************************************/
/* */
/* READ_MAG_FILES - Read magazine data files, cover scan file and */
/* full-text link file & create consolidated file */
/* */
/* Calling Format: */
/* */
/* status = READ_MAG_FILES (config_ptr, prtfil_ptr); */
/* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
This routine sets up sorted lists of cover scan, full text and about links and then reads through the magazine data files creating a consolidated data file with the relevant links added. Note that these are added as "extra" fields (FLD_IMGLNK & FLD_TXTLNK) at the end of the 'A' record. In an ideal world we could just read the files into memory and access them when needed (as we do for PSEUD.CVT and such-like) but I'm getting paranoid about memory usage and we only need the links when processing the 'A' records and can then free up the allocated memory. It also means we only have to deal with two input files in SETUP_ISSUE_IDX and BUILD_ISSUE_IDX rather than reading through all the magazine files each time.
Note that we check the cover scans first because there are many more of them (130,000 vs. 9,000) and, when adding the full-text links, we do a binary chop on the cover links part of what we've set up to save time (we initially did a sequential search and this module took 15 seconds or more). The about links are associated only with (main) feature records so are just added as separate entries. Note that the file containing the about links is not in a standard folder and hence is specified via the Index Configuration File.
/************************************************************************/
/* */
/* REPORT_DIAGS - Report Extended Diagnostics (if required) */
/* */
/* Calling Format: */
/* */
/* REPORT_DIAGS (config_ptr, report_type); */
/* */
/* Where: */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* report_type = Type of data to report: */
/* = DMP_MAGLIST for Magazine File List */
/* = DMP_CONFIG for Configuration Data */
/* = IDX_ARTIST for Artist Index */
/* = IDX_BIOG for Biographical Notes */
/* = IDX_BOKAUT for Book Author Index */
/* = IDX_CHRON for Chronological Index */
/* = IDX_ISSUE for Issue Index */
/* = IDX_SERIES for Series Index */
/* = IDX_STYAUT for Story Author Index */
/* = IDX_FULLTXT for Full Text Index */
/* */
/************************************************************************/
This routine simpy checks to see if extended diagnostics are required and, if so, dumps out the array(s) associated with the report type specified.
/************************************************************************/
/* */
/* SPLIT_TITLE - Split title field into constituent parts */
/* */
/* Calling Format: */
/* */
/* SPLIT_TITLE (title_ptr, titlad_ptr, itemad_ptr, colhdr_ptr_ptr, */
/* divider_ptr_ptr, title_ptr_ptr) */
/* */
/* Where: */
/* */
/* buff_ptr = Returned pointer to formatted title */
/* title_ptr = Pointer to title field */
/* titlad_ptr = Pointer to title additional field */
/* itemad_ptr = Pointer to item additional field */
/* colhdr_ptr_ptr = Pointer to hold pointer to column title */
/* divider_ptr_ptr = Pointer to hold pointer to divider */
/* titl_ptr_ptr = Pointer to hold pointer to item title */
/* */
/* This routine combines the title and item additional fields with */
/* the title, intelligently handling sort titles, series dividers */
/* and story numbers, and returns pointers to the three elements of */
/* the title - the column/series name, the string to use as a divider */
/* and the item title (including any series sequence). The first */
/* two fields are returned as an empty string if not relevant. */
/* */
/************************************************************************/
This routine ...
/************************************************************************/
/* */
/* TIDY_DNOTES - Tidy up contents of any 'D' Notes */
/* */
/* Calling Format: */
/* */
/* outbuf_ptr = TIDY_DNOTES (config_ptr, inpbuf_ptr, prtfil_ptr); */
/* */
/* Where: */
/* */
/* outbuf_ptr = Pointer to buffer with tidied notes. */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* inpbuf_ptr = Pointer to input buffer */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine checks to see if the 'D' notes need any special */
/* formatting (e.g. "--- see under xxx). */
/* */
/************************************************************************/
This routine first checks to see if the notes start with the special prefix "--- see under " and, if so, check the next character to see what sort of link is required:
Publication details and magazine names are converted via FORMAT_ISSUE_LINK (with a ':' prefixed in the latter case); names are converted to internal format via TRANSLATE_AUTH and then converted via FORMAT_NAMES. Magazine links are output in italics; book links in bold and author links normally.
If the notes do not start with the special prefix then the routine simply calls the shared FORMAT_NOTES routine.
/************************************************************************/
/* */
/* WRITE_INDEX_LINE - Write a line to the intermediate index(es) */
/* */
/* Calling Format: */
/* */
/* status = WRITE_INDEX_LINE (config_ptr, reqtype, text_ptr, */ /* prtfil_ptr); */ /* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* reqtype = Request Type: */
/* = IDXLIN_NORMAL for normal index line */
/* = IDXLIN_CONT for continuation line */
/* = IDXLIN_SPECIAL for special index lines */
/* = IDXLIN_TRAIL for trailers */
/* text_ptr = text to use for continuation or for special */
/* (if IDXLIN_CONT or IDXLIN_SPECIAL) */
/* = text to use for continuation otherwise */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/* This routine outputs a line to the Level 1 Index, throwing a new */
/* page and adding a line to the Level 2 Index, etc., if necessary. */
/* */
/************************************************************************/
This routine is called when a line is to be output to the intermediate (Level 1 to 3) Indexes and implements the 3-level index structure described in the Basic Index Structure - see the section on using WRITE_INDEX_LINE for an outline description of how to use the routine.
It outputs the relevant line to the Level 1 Index; if necessary it also throws a new page and adds a line to the Level 2 Index and, if also necessary, throws a new page in the Level 2 Index and adds a line to the Level 3 Index. In addition to the formal parameters, it relies on the following fields in config_ptr:
It operates in one of four modes depending on the contents of the reqtype parameter. By default this is set to IDXLIN_NORMAL indicating that a normal index line is to be output. If so:
If reqtype is set to IDXLIN_SPECIAL the routine continues exactly as above except that the line for the level 1 index is passed in the text_ptr parameter rather than being set up automatically (this is used for the Story and Book Title Indexes)
If reqtype is set to IDXLIN_CONT then we have a continuation line. In this case we still count a line in the level 1 index and output the continuation line, but do not throw any new pages.
If reqtype is set to IDXLIN_TRAIL then we need to clean up for this index and reset for the next index:
/************************************************************************/
/* */
/* WRITE_PAGE_HEADER - Write a standard header to the specified file */
/* */
/* Calling Format: */
/* */
/* status = WRITE_PAGE_HEADER (config_ptr, outfil_ptr, pagttl_ptr, */
/* prvlnk_ptr, homlnk_ptr, prtfil_ptr);*/
/* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* outfil_ptr = File to write header to */
/* pagttl_ptr = Title of page */
/* prvlnk_ptr = Previous link (if NULL, none created) */
/* homlnk_ptr = Index home link (if NULL, none created) */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
This routine simply reads through the Page Header boilerplate file (index_hdr.htm), making the necessary substitutions and outputting the result to the specified file. Note that, for performance reasons, the whole boilerplate file is stored in memory when it is first read to avoid unnecessary file I/O.
/************************************************************************/This routine simply reads through the Page Trailer boilerplate file (index_trl.htm), making the necessary substitutions and outputting the result to the specified file. Note that, for performance reasons, the whole boilerplate file is stored in memory when it is first read to avoid unnecessary file I/O.
/* */
/* WRITE_PAGE_TRAILER - Write a standard trailer to the specified file */
/* */
/* Calling Format: */
/* */
/* status = WRITE_PAGE_TRAILER (config_ptr, outfil_ptr, */
/* nxtlnk_ptr, homlnk_ptr, */
/* prtfil_ptr); */
/* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* outfil_ptr = File to write header to */
/* nxtlnk_ptr = Next link (if NULL, none created) */
/* homlnk_ptr = Index home link (if NULL, none created) */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
/************************************************************************/
/* */
/* WRITE_PUB_INFO - Format and output all "Pub Info" records */
/* */
/* Calling Format: */
/* */
/* status = WRITE_PUB_INFO (config_ptr, inpfil_ptr, outfil_ptr, */
/* nxtrec_ptr_ptr, prtfil_ptr) */
/* */
/* Where: */
/* */
/* status = Result of operation: */
/* = PSP_TRUE if OK; else PSP_FALSE */
/* config_ptr = Pointer to CONFIG_DATA structure */
/* inpfil_ptr = Pointer to input file */
/* outfil_ptr = File to write header to */
/* nxtrec_ptr_ptr = Pointer to buffer to hold pointer to record */
/* prtfil_ptr = Pointer to diagnostics file (may be NULL) */
/* */
/************************************************************************/
This routine ...
/************************************************************************/ /* */ /* WRITE_STORY_ITEM - Write a single story item group */ /* */ /* Calling Format: */ /* */ /* status = WRITE_STORY_ITEM (config_ptr, outfil_ptr, fldbuf_ptr, */ /* fldbufsiz, scanitem_ptr, newpage_flg,*/ /* prtfil_ptr) */ /* */ /* Where: */ /* */ /* status = Result of operation: */ /* = PSP_TRUE if OK; else PSP_FALSE */ /* config_ptr = Pointer to CONFIG_DATA structure */ /* outfil_ptr = File to write header to */ /* fldbuf_ptr = Pointer to buffer for current record */ /* fldbufsize = Size of buffer for current record */ /* scanitem_ptr = Pointer to SCANITEM structure for current record */ /* newpage_flg = PSP_TRUE if we've just thrown a new page */ /* = PSP_FALSE otherwise */ /* prtfil_ptr = Pointer to diagnostics file (may be NULL) */ /* */ /************************************************************************/
This routine formats and outputs all records which have the same author, author type, title, coauthors, secondary names and ED notes. While this seems fairly simple, it has turned out to be fiendishly complex for a number of reasons, primarily to do with aggregation. For clarity, the principles behind aggregation (and some of the past problems encountered) are discussed in the main IDXGEN documentation. It should really be part of BUILD_STYAUT_IDX (which is the only routine that calls it) but has been split out to try to make the code more manageable. Note that this routine reads most of the records while BUILD_STYAUT_IDX handles all the page throws and it is critical to ensure that the two, between them, match the code in SETUP_STYAUT_IDX as far as bookmarks and page numbers are concerned.
One complication this approach leads to is that WRITE_STORY_ITEM needs some knowledge of what the previous item looked like so that it can decide what fields do/do not need to be output, so there are a handful of pieces of static data that retained across invocations:
The routine first initialises some useful fields:
It then sets up the story and column titles, and outputs the latter if needed:
It then sets up the Item Header (itmhdrbuf) that we need whether we have one item or a thousand:
Note that the above approach for bylines potentially causes problems with book reviews (see the discussion under Notes on Some (Past) Problem Areas) as we might want to change the byline in the item header within a single set of results. We bodge this by saving the offsets in the itemheader before and after we add the byline so that we can reset it when needed.
We're now ready to try to consolidate all the items that we wish to display as a group. The specific check at the moment is for items which:
Note that this is (and should be) checking the same fields as were used to set up the Item Header except for:
As discussed in the section on aggregation we can't handle these linearly so we need to read them all and store the details in a series of interlinked structures that we can then sort, scan and manipulate as required. These are split into separate structures to try to minimise memory usage – some groups have large numbers of simple items (there are over 1200 instances of John North's "Where to Go and How to Get There" column) while other have a small number of complex items (e.g. where the title or byline varies from instance to instance and we don't want the memory overhead of allowing for the latter for all instances of the former.
At the top level we have the Group Array (group_str). Each instance of this identifies all occurrences under a single byline in a given file because we want to aggregate items in the same (magazine file) under the same byline (e.g. a repeating column or a serial). Each item in this array contains:
Most groups are quite simple but, in some cases, additional information is needed. As there might be hundreds of groups we try to save memory by having a separate array of these, the Group Extras Array (group_xtr), which is used only when necessary. Each entry contains:
Note that the contents of this array are used in three different ways so we need to be careful with them:
Below these lies the Item Array (item_str): we have one of these for each distinct item (as of 2024, one group needed almost 1500 instances). Each contains:
Item Extras Array (item_rep): As with the Group Extras Array this is hived off to save space and contains, where relevant:
The routine first sets up the entries for the first item (i.e. the one in scanitem_ptr when the routine was first called) – in the vast majority of cases this will be the only item in the set. It then reads the next item and checks to see if it is the same (as discussed above). If so then it aggregates the item with the previous one(s). It first checks to see if it can be added to an existing group, i.e.:
If not, it creates a new group. It then creates an entry in the Item Array (and possibly Item Extras Array) and links it to the matched or new group. It then goes back to read the next item and repeats the above.
Once we have all the arrays set up, we need to tidy them up and sort them into the order necessary.
First we look at each reprint group (edition=2 or book_type=3) and try to find a prior group with an item with matching publication details. Generally there will always be one such because the code in SETUP_STYAUT_IDX always tries to create one, and the presence of the edition in the sort order will sort it before the reprints (and hence the group will be set up earlier). If it finds a match then it saves the Group ID as the Parent ID.
It then runs through the groups again, looking for original groups (edition=1 or book_type=1) or an orphaned reprint (i.e. parent_sub = -1) adding each to the ordered list of groups. For each group it finds it then checks the remaining groups for which it is parent and adds them next before moving on. Thus, for a complex set-up we should end up with:
We now need to work out how to display the information we have. In the simplest case of a standalone, original, item (i.e. 1 group, 1 item) we have a single line of the form:
<itmhdrbuf>publication details<subjbuf>, e.g. * The House of the Four Winds, (br) Interzone #254, September 2014 [Ref. Mercedes R. Lackey]
A slightly more complicated case is when we have a single item that has been reprinted, and where both appearances are in the index, in which case we have:
<itmhdrbuf>original publication details<subjbuf>
reprint publication details, e.g.
* Cornel Wilde, (bg) Ranch Romances 1st November 1952 [Ref. Cornel Wilde]
• Ranch Romances (UK) #4, 1953
Note that, in this case we would have two groups each with a single item. In general this will be the case whether or not the original item is in the database because of the fake originals we create in SETUP_STYAUT_IDX as discussed here. However, there are occasions when we do not create fake originals (e.g. when the original title was different) and we have to handle these cases as well, as discussed below.
The real "fun" with aggregation, though, comes when we have multiple instances of the same item (e.g. columns or similar) which may, in turn, be reprinted so we end up with something like:
* Ranch Flicker Talk, (mr) Ranch Romances 1st Mar, 2nd Mar 1952 • Ranch Romances (UK) #1, #2 1952
where the magazine name is only output once in each instance, the year is only output when it changes and the dates are abbreviated, all in an attempt to save space. Note that the whole of "Ranch Romances 1st Mar" is hyperlinked to the first issue and "2nd Mar" to the second issue, and so on. The year is displayed in bold to help it stand out from the surrounding text. This formatting is all handled by special format types passed through to FORMAT_ISSUE_LINK and the code has to work out which format to use based on the surrounding items, as discussed below.
All of this is further complicated if the reprints appear under different titles or bylines from the original and/or if the items first appeared under a pseudonym and are now being displayed in the entry for the real author. These are handled by the Item Extras Array **More Detail Needed**
Partly for historical reasons, the rest of the code in this section falls into two halves:
If the item is original then it just does the first part; if it is a reprint for which we have done the original then it just does the second bit; if it is a reprint for which we have not done the original then it does both (with a bit of fancy footwork in the middle). This can be determined for items by looking at the parent_sub and edition flags in the group array:
Books are currently handled differently.
need_header determines whether or not the item header in itmhdrbuf needs to be output. This is initially set to PSP_TRUE and reset to PSP_FALSE when we have output the header. Within a group it is also reset to PSP_TRUE if we have a group that represents a first edition or an orphaned reprint - e.g. if we have an unconnected set of items that share the same author/title/type etc. (such as a column that appears in multiple magazines) we want to output the item header before each distinct group of items. (Anon.~About the Author~bg is the most extreme example).
need_maghdr determine whether or not we need the magazine name displayed. This is always true for the first item in a batch, but is suppressed if we have multiple issues from the same magazine (see the discussion of aggregation above).
doing_issues is primarily used to flag when we have output some issues and hence need to close off the <LI> construct. It is set to PSP_TRUE when we output the item header (as that must be followed by issues of some kind). It is then checked at the end of each group and, if set, the code outputs the </LI> and additional trailer material (see below) and sets the flag back to PSP_FALSE. It is also involved in handling cases we we want to output both original and reprint information for a single item as we flush the original details and set the flag to PSP_FALSE and then use that to spot that we need to start a new list for the reprint details.
doing_indent is used to track if we have an indented list of reprints. There might be multiple lines of these so we need to know both that we're already started the indent so that we don't keep doing it and that we've done one at all so that we can switch it off at the end of the group.
The code loops through each group and first sets up bylinebuf to contain any original title and/or byline (the latter if we haven't already done it - clarify). If also checks to see if this is a first appearance (edition=="1") or an orphan group, either of which indicates the start of a new "batch", and, if so, sets the need_header flag. In all cases it also sets the need_maghdr flag (we always need the magazine name displayed at the start of a new group).
It then loops through each item in the group.
It first checks to see if there is another item following (setting itm2_sub to point to it if so) and, if so, if it is in the same group as the current one (setting same_group accordingly). It also increments the count for the different types of item if this is an original appearance or a reprint for which we haven't done the original (we need to do it somewhere so might as well do it here).
It then sorts out how much we want to display of the next set of details. By default we set the format type to FPDT_IDXGEN (i.e. the normal display) but:
Lastly, if the need_header flag is set, the routine outputs the contents of itmhdrbuf, possibly adjusting it to change the byline (expand) and reset the need_header flag. It also sets the doing_issues flag.
As mentioned above, this is basically done if the item is an original appearance or a reprint for which the done_original flag isn't set. Specifically it checks that:
It then formats the link in pubdet (bit of a bodge here that needs clarifying) and adds it to issuebuf followed by the year if the do_year flag is set.
If we are outputting both the original and reprint details for a single item then we need to flush the set we just set up and set up the groundwork for the next bit. At the moment we check the doing_issues flag which will be set if we've done the original printing and we check we're the first item in the group. The former is done rather than checking for parent_sub = -1 and edition != 1 to cope with items reprinted as books. We need to check we're the first item in the group as we might have multiple items (e.g. parts of a serial) all reprinting from the same original.
All the code does here is:
This is done whenever the group is a reprint group (i.e. edition != 1) and the item has some reprint information (magid non-empty).
If is the first of this batch of reprints we need to output the indent (<UL>) to start them. In addition, if we're not already doing some issues then we need to output the <LI> to start the issue list; in this case we also need to change the format type to FPDT_IDXGN1 if it was FPDT_IDXGN2 to ensure the magazine name is output at the start of the list.
We then check to see if this reprint has a different author or title (clarify) and, if so, set them up in bylinebuf and output the issue details.
When we have finished processing all the items in a group we just need to do some housekeeping. First, if we were doing some issues (as is likely) we need to output the subject(s), byline(s) and ED notes (if we haven't already done them). We also need to close off the issue list (unless we're doing a column - why?) and, if we have indented some issues, close off the indentation.