1. Introduction

The document covers updating from FS 10.2 to the latest commit on its branch. The latest commit is experimental. We make every effort to keep the latest commit usable, but any commit that is not tagged for release is not intended for operations. The intended use of this document is to collect update information, as it becomes available, before the next release.

Caution
 It is expected that those using the latest commit are experts and will manage the differences for updating to the next official release themselves. No specific documentation will be provided. What commit was last updated to will determine what needs to be done. It might work to verify that all the steps in the latest version of this document, appropriately adjusted and renamed as the update from the old release to the new release, were completed. However, this may not be complete. Let the user beware.

This document is up to date with the bb5818dd028f5ffd46d732890da0ea2f62dfade1 (bb5818dd) commit. Due to small increments, such as updating this document, that may not be the literal last commit, but the differences should be small unless you happen to pull between a significant change and this document being updated.

2. Upgrading from 10.2 to the latest commit

You must have already upgraded to 10.2.0 according to the FS 10.2 Update Notes document before installing this update.

2.1. Fetch the latest commit

There are two alternatives:

  1. If you are using FSL9, FSL10, or another system that supports connections to github and you are using git (as is recommended) then as prog execute:

    cd /usr2/fs-git
git fetch
git checkout -q bb5818dd

  2. If you are using FSL8, or otherwise unable to use git:

    Please follow the steps, through the step that includes the option to set the /usr2/fs link, in the Installing from an archive subsection in the Release Model document. Please note that:

    • For FSL8, you will need the TIP in the wget step.

    • For the tag in the wget and tar commands, use the short SHA for the commit. For example, for commit under discussion here, bb5818dd028f5ffd46d732890da0ea2f62dfade1, use bb5818dd.

    • After extracting the archive, change the name of the output directory to use the latest feature release and the short SHA of the commit, like fs-10.2.0-33-gbb5818dd (note the g before the short SHA), which will then be used in the following cd command. For example (Tab completion is your friend for the long SHA):

      cd /usr2
mv fs-bb5818dd028f5ffd46d732890da0ea2f62dfade1 fs-10.2.0-33-gbb5818dd
cd /usr2/fs-10.2.0-33-gbb5818dd

    • You will end the “installing from an archive” with the step setting the link for /usr2/fs by running make install.

2.2. Make the FS

As prog:

cd /usr2/fs
make clean rmdoto rmexe all >& /dev/null
make -s

No output from the last command indicates a successful make.

2.3. Update your station code.

This step is not needed at this time.

Except if noted otherwise, these steps should all be performed as prog.

2.4. Reboot

This step is not needed at this time.

Important
 This will make sure shared memory is reallocated.

2.5. Additional steps

Except if noted otherwise, these steps should all be performed as oper.

  1. Optionally, update the comments in your .rxg files to correctly show the correct maximum number of Tcal table entries allowed.

    A script, /usr2/fs/misc/rxgfix4, has been provided for this. If the original version of the affected comments have been preserved, the will be updated to the new form. Only the comments immediately before the active Tcal entries will be updated.

    1. Execute:

      cd /usr2/control/rxg_files
/usr2/fs/misc/rxgfix4 *.rxg

      If the script stops because there are existing .bak files, you can delete them, if it is safe to do so, by adding the -d option (see /usr2/fs/misc/rxgfix4 -h for the details) to the command before the files to be updated. If it is not safe to delete them, you could, for example, rename them to end in .bak2 with rename 's/.bak$/.bak2/' *.rxg.bak first. The rename, command, by default, will not overwrite existing files, but you might want to check that your new ending is not already in use for some .rxg backup files to avoid possibly mixing different “generations” of backups.

      Note
      		 If you don’t have the (perl based) rename command, you can install it on FSLx systems as root with apt-get install rename.

3. Changes since 10.2.0

There are separate subsections with summaries of the changes in the FS and drudg.

Clickable links such as #36 connect to specific issues reported at https://github.com/nvi-inc/fs/issues.

A complete history of changes can be found using the git log command from within the FS git working directory, usually /usr2/fs-git.

Each change is listed as a numbered summary typically followed by a “Details” toggle, like:

Details

Details are shown here.

that can be clicked on to toggle showing (or not showing) a collapsible box with the details. In this way, you can view the summary as a list and only reveal the details of items that interest you. The collapsible box may be omitted if providing further details didn’t seem warranted. The collapsible boxes for the details may also have nested collapsible boxes inside them if there are many sub-items.

3.1. FS changes

  1. Update comments in .rxg files for the current maximum number of Tcal table entries (closing #211).

    Details

    The comments in the example .rxg files (/usr2/fs/st.default/control/rxg_files/*.rxg) had not kept up with the expansion of the size of the Tcal table, first from 400 entries to 600 and then 1200. The files have now been updated.

    A script, /usr2/fs/misc/rxgfix4, has been provided for optional use to update the working .rxg files at station. If the original version of the comments, with 400, have been preserved, they will be updated to the new form. Only the comments immediately before the active Tcal entries will be updated. Lines that list 100 and 600 as the maximum will also be updated. The former was apparently the size in some preliminary versions before the first official release ((9.6.9, September 2003, commit 7c26ea900dee19b01958e5c4ad846b89d64638c5) that supported .rxg files. The latter was never provided as an example, but is covered just in case. See /usr2/fs/misc/rxgfix4 -h for the details.

  2. Improve recovery for loss of an open log file.

    Details

    1. Handle the open log file being replaced by a file that is renamed to the open log file’s name.

      Previously automatic lost log file recovery worked if either the log file open in the FS have been deleted (rm) or renamed (mv). It did not cover the case of a different file being renamed to have the name of the open file. That latter most case is now detected by comparing the inodes of the open log file and the file with the name of the open log. If they don’t agree they aren’t the same file. In that case, the open log file will be recovered to a file with _recovered appended to the original name. If a file with that name already exists, the log will instead be recovered to a file with:

      _recovered.<XXXXXX>

      appended at the original name, where <XXXXXX> is a random string, chosen the kernel, that makes the filename unique, e.g., 3fDhIa.

      There is a consequence of this feature that may not initially appear to be intuitive. It occurs if the user was re-opening the original file (with log=…​ or schedule=…​) when the inode mismatch was discovered. In this case, new log entires will be appended to the file with the original name (regardless of what was already in it), not the recovered log file. The recovery process doesn’t know what the new (if there is one) log file’s name is, but will print a message warning about this possibility, which can occur whenever the recovery file’s name includes recovered. In retrospect, this may not seem so unintuitive after all.

      Warning

      The case of an open log file being overwritten by copying onto it (cp) is not handled. So it is not possible to automatically recover the log in that case. It should still be possible to recover, most, if not all, of a log lost in this way using the technique described in /usr2/misc/logrecovery.txt

      Two possible approaches to prevent overwriting by copying (cp) onto an open log are described in the Details collapsible box below. Neither have been implemented at this time.

      Details

      Two possible solutions to this are:

      1. Use chattr to make the open log file append only.

        The approach would be for ddout to use chattr to set attribute +a on the log file after opening to make it append only. When the log is closed, it would be changed to -a to allow the file to be used normally thereafter. It is probably necessary to include O_APPEND in the flags for open() calls in ddout in case for some reason -a was not applied (probably a crash) when a log was closed. To employ this method, it is necessary to give chattr the capability to set the append attribute (as root):

        setcap cap_linux_immutable=eip /usr/bin/chattr

        There are two minor drawbacks to this approach:

        1. If an existing log file is owned by a different user (perhaps prog), the attributes can’t be changed.

        2. If the attribute is not changed back to -a (perhaps due to a crash), the file becomes very difficult to work with (no renaming/moving, editing, compressing, etc.) until that is corrected.

      2. Use mandatory file locking to prevent overwriting of the open log file.

        In this case, the /usr2 partition must be mounted with the mand option. ddout would create new logs with a mode including S_ISGID. It should set the mode of an existing file that it opens to include S_ISGID Then it would need to lock the entire file (.l_whence = SEEK_SET; .l_start = 0, .l_len = 0) for reading (.l_type = RDLCK). When the log is closed, ddout would clear its setgid bit.

        There are four minor drawbacks to this approach:

        1. If an existing log file is owned by a different user (perhaps prog), the setgid bit can’t be set/cleared.

        2. If the setgid bit is not cleared (perhaps due to a crash), it will follow the file, including being compressed or renamed, until that is corrected.

        3. Mandatory file locking is not considered reliable, but the example cases where there may be problems do not seem relevant to this use.

        4. Although not deprecated, it seems as though mandatory file locking may be headed to extinction.

    2. Make recovery more robust.

      Previously if errors were detected while determining if the log file existed, recovery would be aborted. The code has been restructured to not give up for this case or for a problem determining the inode values. It now aborts only if an unrecoverable error occurs in the process of recopying the log contents. Making the recovery as robust as possible is prudent since this is the only chance to accomplish this.

    3. Update /usr2/misc/logrecovery.txt note.

      The note has been updated for more modern systems and to include more specific steps.

    4. Make miscellaneous improvements:

      • Simplify recovery logic

      • Show recovery activity even if a percentage can’t be shown

      • Make error reports more consistent

      • Report partial recovery even if an error occurred

  3. Improve overall Makefile (closing #212).

    Details

    In FSL11, the make install command would fail if the source directory was not owned by root. While this situation would not normally occur, it could if the make install was executed a second time after an initial installation. This might happen, for example, if make install were used to reset the /usr2/fs link. The failure occurred because of a confluence of two issues: (i) git was unnecessarily used to determine the version of the FS during for the install target and (ii) the version of git in FSL11, and probably later versions, will throw an error if the directory is not owned by the user. The problem was resolved by removing the use of git to determine the FS version for make install.

    Thanks to Jon Quick (HartRAO) for reporting this issue.

    Two further enhancement were made:

    • make install can only be used by root and install is the only make target allowed for root.

    • When installing from an archive, if the source directory name is incorrectly formed (missing the tag information), an explanatory error will be reported and the make will be aborted.

  4. Add rdbemsg emails to experiment log as comments (closing #213).

    Details

    Including the emails in the log avoids having to enter comments from the email messages a second time to get them into the log.

    When originally written, the program that became rdbemsg did not run on FS computers and could not easily add comments to the log. Due to on oversight when the program was ported to FS computers, adding the email message to the log, as msg does, was not added. This has been corrected for both the python2 and python3 versions of rdbemsg.

    Thanks to the staff at KPGO for reporting this issue.

  5. Restore -b startup option (closing #216).

    Details

    The removal of the -b option in FS 10.2.0 was a bit too hasty. The long form of the option was --background, which suggested that it was the inverse of --foreground. This was not the case. With the server enabled, it started the FS without a client. This might be convenient to use for starting the FS from a script or otherwise without the log being displayed on the terminal. A pitfall of using it is that there is no feedback about the whether the FS started correctly or had a start-up error and aborted. As a result, this option must be used with caution.

    To make it clearer that it does not start the client, the short option was changed to -C (not client) and the long option to --no-client. The help output now includes a warning about the lack of feedback for startup errors.

    Thanks to Christian Kristukat (AGGO) for reporting the loss of functionality when the -b option was removed. Thanks to Beppe Maccaferri (Medicina) for pointing out the possible pitfall.

  6. Comment out all lines in example mdlpo.ctl and parpo.ctl files (closing #214).

    Details

    This prevents these example files being used accidentally with values that are incorrect for a new installation. Instead, the pointing analysis programs and/or antcn (while reading the default mdlpo.ctl) will fail until the files have been setup with reasonable values for the antenna. While this may make the first time use for a new antenna a little painful, it is better than having incorrect values being used.

  7. Prevent root from running the FS.

    Details

    The FS is a user application. It is risky, and a potential security issue, to run it as root. In particular, the sy=…​ command, if misused either accidentally or on purpose, could cause serious damage. To avoid this, the FS startup program, fs, the server, fsserver, and the client program, fsclient, will refuse to run if the user is root. This is not strictly necessary for the client, but it was included for consistency. Off-line utilities, including pfmed, can still be run by root, but this is not recommended and should be avoided. It may lead to problems. However, it would not be inherently dangerous.

  8. Remove bad characters from ASCII files.

    Details

    A few ASCII files had non-ASCII characters in them.

    • There were UTF-8 characters in help/core3h_mode.j__ and misc/logrecover.txt. They were removed. They did not display properly on non-UTF-8 systems, principally FSL9 and earlier.

    • The extended ASCII character for angstroms (Å) was incorrectly specified in the help/lvdt.* files. It was replaced with the word.

  9. Cleanup 10.2 web documents

    Miscellaneous wording and typographic corrections were made.

3.2. drudg changes

There are no drudg changes yet.