1. Introduction

These instructions provide a complete method for system installation and some tuning. They are not the only method for accomplishing these goals, but have been well tested. Experts can of course use their own means, but the farther they deviate from this model, the less support we will be able to provide.

The standard configuration uses a RAID1 system with removable disks. Normally, two disks would be in use at a given time. A third disk is used as a back-up and rotated into use periodically. More disks can be used for further redundancy. You can of course provide your own back-up method and can install the system to a single disk if you do not want to use the software RAID.

If you are using the RAID configuration, you may wish to review the Recommended practices sub-section of the RAID notes for FSL10 document before installing. However, all of the practices listed there can implemented after the installation steps below are complete.

Tip
Removable disks should be used with a carrier/receiver system that can tolerate a large number of insertions; “bare” disks should not be inserted repetitively. Two receivers would normally be mounted in the computer chassis. Each disk would be in its own carrier. We can provide a recommendation for a carrier/receiver system if you need one.

Please note that for each step in this guide, we recommend you carefully read all the included caveats and notes as the material is not always logically sequential, i.e., instructions may proceed explanations that impact what you actually type.

Table 1. FSL distributions
FS Linux Release Name Debian Version Linux kernel Year

1

(Slackware)

1.2.<x>

1994

2

bo

1.3.1

2.0.29

1997

3

hamm

2.0

2.0.34

1998

slink

2.1

2.0.36

1999

4

potato

2.2

2.2.18

2000

5

woody

3.0

2.2.20/2.4.18

2002

6

sarge

3.1

2.4.27

2005

7

etch

4.0

2.6.18

2007

8

lenny

5.0

2.6.26

2009

squeeze

6.0

2.6.32

9

wheezy

7.0

3.2.0

2014

jessie

8.0

3.16.0

10

stretch

9.0

4.9.0

2020

buster

10.0

4.19.0

The FSL10 documents follow the FS font conventions, which can be found at: https://nvi-inc.github.io/fs/misc/font_conventions.html.

2. Choosing architecture and creating installation media

As of Field System version 10.0, both i386 and amd64 architectures are supported natively. The amd64 architecture is preferred and should be used if possible (it should be unless the processor is very old, from about 2010 or older). However, some work may be required to port your station code from a 32-bit to a 64-bit OS. Some automatic tools have been developed for this, and can be provided upon request. Usually the i386 architecture will work on any processor, but requires use of the Legacy (or BIOS) boot mode in most cases. The amd64 installation media will fail to boot on a system that is 32-bit only.

To install Debian 9, you can either use a DVD or USB drive. The latter is faster, and also easier if you wish to use UEFI. Directions for creating your installation media can be found online.

Note
Don’t be confused by the amd64 name, this architecture supports both AMD and Intel manufactured x86-64 processors. This includes CPU lines such as Ryzen, Epyc, Core, and Xeon. The naming scheme dates back to when Intel had a competing and incompatible 64-bit architecture ia64.

You can install from a DVD drive, USB device, or over the network. Any revision of 9.<x> installer from 9.7 onwards should work fine, but older revisions will want to download many security patches from the network which may already be included in the latest revision. Revisions before 9.7 had a serious security flaw in the package installer. Note also that installing from DVDs as described here is recommended mainly for sites with little to poor Internet connectivity (even then, use of a single DVD may suffice) and the equivalent use of a Debian GNU/Linux 9.<x> “Stretch” - Official i386/amd64 netinst CD would suffice for installation at most sites with good connectivity. Official images for the installer can be found at: https://cdimage.debian.org/cdimage/archive/ or alternatively, should your hardware require non-free firmware, unofficial images for the installer that also include all available non-free firmware can be found at: https://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/archive/

The details of creating your installation media can be found in the Debian installation guide available from: https://www.debian.org/releases/stretch/installmanual

3. Motherboard setup

Note

Some hardware may require special procedures. For cases we know about, instructions are provided in top-level sub-directories of the repository, which is visible at https://github.com/nvi-inc/fsl10. In addition to special instructions, there may be needed software/drivers in the sub-directories. The following table lists the cases that are currently covered. Some of these solutions may be useful for other hardware with the same issues. If you have the listed hardware or issue, we recommend reading the instructions before beginning the installation.

Table 2. Special installation instructions available
Directory Hardware Issue

X11SCA-F

Supermicro X11SCA-F motherboard

Newer driver required for Intel 1000e NIC

Modern motherboards offer two forms of booting: native UEFI or BIOS emulation (“Legacy”). UEFI is the preferred approach. Either mode of boot is supported by this installation guide, and you will be given alternatives when the instructions differ.

Decide which boot mode you want to use and select it through the motherboard setup menu (typically by pressing DEL during POST).

Also make sure that the motherboard time is set to the current Universal time, i.e., UTC, and the motherboard can boot from the installation media.

While you are in the motherboard menu, make sure that hot-swapping is enabled for both the primary and secondary controllers. This is necessary for disk rotation and recoverable testing.

Tip
For UEFI, some motherboards may switch to booting to the UEFI shell if they fail to find a hard disk that will boot. This might happen, for example, if you attempt to boot from a blank disk. If you become stuck booting to the UEFI shell, you may need to enter the motherboard’s setup utility boot menu and restore booting from the hard disks.

4. First Stage Installation

Warning
This guide assumes that you have two disks installed in the machine in order to set them up as a RAID pair. For the RAID to work seamlessly with a third disk later, you must make sure that the smallest disk of the three disks available is used as the first of this initial RAID pair. Use of a single disk (for a test install etc.) is also annotated below.
Tip
Installing to a single disk initially is recommended and has some advantages. It is faster and you can control when the syncing for the second disk occurs, such as when you leave for the evening. The set-up of additional disks is covered in the Post Install section (which references the RAID notes for FSL10 document). As mentioned in the warning above, you should start with the smallest disk.

4.1. Boot from the installation medium

Connect an active network cable to your lowest numbered interface (only). Usually it is on the left if there are two.

Insert/plug-in your installation media and reboot.

To boot of the installation media you may need to bring up your motherboards boot menu, which is typically accessed by pressing F11 or F12.

4.2. Set boot options and boot installer

At the Installer boot menu:

  1. Highlight Install (or Graphical install — only the installer interface differs — but this may not work on some video hardware)

    • UEFI: press e, then three times (vmlinuz), then End

      Note
      If e doesn’t work, UEFI is not available. It may be possible to enable it in the BIOS.
    • BIOS: press Tab

  2. To the end of the displayed command, add the additional options:

    locale=en_US.UTF8 netcfg/disable_dhcp=true time/zone=UTC
    Note
    Whilst typing a / (slash) it may automatically be changed (escaped) to \/ (i.e. preceded by a backslash). This is normal behaviour and harmless.
  3. Press:

    • UEFI: F10

    • BIOS: Enter

Note
You may omit the netcfg/disable_dhcp=true if you want to use DHCP to configure the network settings of this machine, though this is not advised.
Note
You can additionally use partman-partitioning/default_label=gpt if you wish to force the use of a GPT partition table on a disk that is smaller than 2 GB, but beware - some older BIOS versions cannot handle GPT formatted disks.
Note
If you do not set a locale or set locale=C, you will be prompted to select your language and your country. However some applications may have problems if a UTF8 locale is not used.

The installer will now boot.

4.3. Select a keyboard layout

Find your keyboard on the keymap list and press Enter. (The most common one is American English)

The installation media is now scanned and additional installer components loaded.

4.4. If you are presented with a dialog asking for non-free firmware files

You may need to locate the files requested (especially if they relate to your network or disk-drive interfaces) and place them on a USB stick which should be inserted at this stage. If you do have the required files select Yes, otherwise press Tab to select No then press Enter to continue. It may well be simpler just to use the unofficial installer images mentioned above that include all available non-free firmware.

4.5. If you are presented with a dialog asking which interface to use

Typically only shown if two or more network interfaces are found, which might include a virtual firewire interface in some cases. Select the interface you require (usually eno1) and press Enter.

Unless you are using DHCP (which is not advisable) you will be prompted to:

  1. Type in the required static IP address in the form xxx.xxx.xxx.xxx (where each xxx is any integer from 0 - 255 inclusive) and press Enter.

  2. Type in the required netmask in the form 255.yyy.yyy.yyy (where each yyy is typically 0, 64, 128, 192 or 255) and press Enter.

  3. Type in the required gateway IP address in the form xxx.xxx.xxx.xxx (where each xxx is any integer from 0 - 255 inclusive) and press Enter.

  4. Type in the required nameserver IP addresses, space separated, in the form xxx.xxx.xxx.xxx (where each xxx is any integer from 0 - 255 inclusive) and press Enter.

Alternatively, if you are only using the installer to initialize new disks, you may want to use Go Back and directly select Detect disks from the main menu to skip forward to Setup partitions below.

4.6. Set a hostname

Backspace over the default hostname debian and type in the name you require (if not already retrieved via DNS), then press Enter. Enter the required Internet Domain name (if not found) and press Enter.

4.7. Enter a suitable root password

Twice as prompted.

4.8. Setup first account

Enter Desktop User for the name of the new user then press Enter to accept desktop as the username and enter a (real) password twice as prompted.

4.9. Get network time

The installer now tries to set the time using NTP If this is not possible at your site due to your firewall etc., you may need to press Enter to cancel this process.

4.10. Setup partitions

Note
If you are using UEFI and the disk was previously used for BIOS, you may need to confirm forcing UEFI installation.

When prompted, select Manual

4.10.1. Setup the first disk

  1. If needed create a new partition table by:

    1. Select first disk, something like SCSI1 (0,0,0) (sda) - 4 TB ATA SATA HARDDISK, and press Enter

    2. Installer may warn: You have selected an entire device to partition…. Select Yes

  2. Select the FREE SPACE under the first device

    Note
    If some other file system, like xfs, or other old setup is displayed, you will need to delete it first. You may be able to do this by deleting individual partitions until you have a single FREE SPACE area. For more complicated layouts, it may be more expedient, and it may be necessary, to use Guided partitioning to delete the existing configuration (and temporarily create new partitions). In this case, select Guided partitioning, then select Guided - use entire disk. Then select your disk, such as listed above, do not select a RAID or your installation media device. Then select All files in one partition (recommended for new users). You may be prompted to confirm deleting RAID and/or LVM, which you must do to continue. Then you should be able to continue with step 1 above, by selecting your disk.
  3. Select Create a new partition

  4. Then for

    • UEFI: Enter 1GB in the size, then select Beginning of the disk.

    • BIOS: Enter 1MB in the size, choose Primary (rather than Logical) if asked for the partition type, then select Beginning of the disk.

  5. Then for

    • UEFI: Select Use as then select EFI System Partition (ESP)

    • BIOS: Select Use as then select Reserved BIOS boot area, or alternatively Do not use the partition if the former option is not available.

  6. Now press Done setting up the partition.

  7. Next select the FREE SPACE and Create a new partition again.

    Note
    You may see a small 1MB FREE SPACE at the start of the disk. This is fine, just be sure to choose the large FREE SPACE at the end of the disk.
  8. This time choose the whole amount of free space (the default) and choose Primary for the partition type if asked.

  9. Select Use as: physical volume for RAID, then Done setting up the partition

Note
If you physically only have one disk bay and wish to construct a FSL10 test-bed, it is possible to avoid using the software RAID layer entirely. Simply select Use as: physical volume for LVM for this partition instead and skip ahead to Setup Logical Volume Manager (LVM) below. However, please note that a single disk setup is not recommended for any operational system.

4.10.2. Setup the second disk

Repeat the process for the second disk, if present.

4.10.3. Setup RAID

  1. Select Configure software RAID, select Yes to write the changes to the disks.

  2. Select Create MD device, choose RAID1 and enter 2 as number of devices and 0 as number of spares.

  3. Select the RAID partitions we just created by pressing Space — these should be sda2 and sdb2, if you have just one disk, just pick sda2 — then press Enter to continue

  4. Select Finish.

  5. Back in partitioning, Select the space under RAID1 device #0 and press Enter

  6. Select use as then select Physical volume for LVM then Done setting up the partition

4.10.4. Setup Logical Volume Manager (LVM)

  1. Now choose Configure the Logical Volume Manager and select yes if prompted to write the changes to disk

  2. Choose Create volume group

  3. Enter a name appropriate for the machine and group, e.g., vg0, and press Enter

  4. Select the raid device md0 (or sda2 if not using RAID) by pressing Space, then press Enter to continue

  5. For each item in the following table run Create logical volume, select the your volume group and assign the corresponding label. Those marked with * are optional unless you are applying CIS hardening.

    Table 3. Logical volumes
    Mount point LV name Size

    1

    /var/log/audit

    audit *

    4 G

    2

    /boot

    boot

    1 G

    3

    /home

    home

    4 G

    4

    /var/log

    log *

    4 G

    5

    /

    root

    50 G

    6

    swap

    swap

    8 G

    7

    /tmp

    tmp

    8 G

    8

    /var

    var *

    8 G

    9

    /var/tmp

    vartmp *

    8 G

    10

    /usr2

    usr2

    remaining disk space less ~50 GB

  6. In the LVM configuration window, select Finish

  7. Then for each logical volume in the table except swap, do the following:

    1. Select the partition (e.g., #1) for each LV name (and press Enter)

    2. Select Use as and press Enter then select Ext4 journaling file system

    3. Select Mount point, press Enter, then select the appropriate mount point from the list or use Enter manually if not there.

    4. Select Done setting up this partition

  8. For the swap logical volume, select Use as then select swap area, followed by Done setting up this partition

  9. Back in the partition screen, select Finish partitioning and write changes to the disks and select Yes to write the changes. For big disks, it may take a little time to create the ext4 file systems.

The Debian base system is now installed from the installation media, which usually only takes a few minutes.

4.11. Configure the package manager

If you start from a netinst CD image, the installer now assumes you will install only from the network, and jumps straight to the Choose your country…​ part of the dialogue as detailed below.

Select the fastest Debian mirror from those available.

Tip
The new deb.debian.org mirror is a good choice for most sites as it uses DNS to find a local mirror.

Enter any necessary HTTP proxy information (usually left blank.)

If you are using DVD installer you will be prompted to scan additional DVDs. Scanning the additional DVDs (and obtaining copies of them in the first place) is entirely optional, and is only useful if you don’t have a reliable network connection to a suitable Debian mirror and hence would prefer not to download packages you could get from the DVD.

Note
If you do want to use a mirror in future, it is better not to scan any DVDs at this stage and to scan them later during Stage 2 using apt-cdrom.

For each additional DVD you wish to scan, insert it in the drive, select Yes and press Enter to perform the scan (which takes a while.)

(If you are using DVDs, and are prompted to insert another DVD, you will need to use eject /dev/cdrom from another virtual console to do this)

Select No and press Enter to continue once you are done. If prompted, insert the “Debian GNU/Linux 9.<x> Stretch - Official i386/amd64 Binary-1 DVD” back into the DVD-ROM drive and press Enter.

Warning
If you do scan additional DVDs, the following useful dialogue which allows you to select a suitable network mirror from a country-based list may be suppressed.

Select Yes and press Enter to use a network mirror (unless you have inadequate Internet access - but then you must scan all DVDs.) Choose your country from the list if available and press Enter. (If your country is not available choose the country nearest to you in a network connectivity sense.)

4.12. Do not participate in popularity-contest

When prompted to join the popularity-contest, select No and press Enter

4.13. Choose your packages

When prompted to choose packages, select SSH server by highlighting it with the arrow keys and pressing Space on it (unless you don’t want it).

Tip
If you have a small disks and are worried about space, then you can also press Space on Desktop Environment to unselect it (which may then change the dialogue presented below).

Finally press, Enter to install the standard system.

The Debian standard system is now installed from the installation media plus any updates from the network mirror and/or security.debian.org site if they can be reached.

This can take a while, up to one and a half hours or more.

4.14. Install the GRUB bootloader (BIOS boot only)

Note
With UEFI boot, you will not be presented with this option; GRUB will automatically be installed to the first ESP partition.

At Install GRUB to Master Boot Record select yes then select /dev/sda

When prompted, press Enter to install to the master boot record of the primary disk.

4.15. Remove installation media

Remove the DVD from the DVD-ROM drive (it should be auto-ejected), or unplug the USB drive and press Enter to reboot into the newly installed system.

Tip
It would generally be wise to disable booting from DVD-ROM and floppy i.e., anything other than the hard drive, in the BIOS just in case someone leaves something nasty in the machine’s removable drives by mistake.

5. Second Stage Installation

You can now boot to your new OS.

Note
If the login screen is painfully slow and your CPU does not include a GPU, you can probably fix the slowness by disabling Wayland in gdm3. However, the result may be that rebooting or shutting down will typically have an enforced 90-120 second delay (see the Fix for slow reboot/shutdown with Wayland disabled section for a possible fix). You may find the slow login screen preferable. To disable Wayland, edit /etc/gdm3/daemon.conf and uncomment the line WaylandEnable=false. Then gdm3 will need to be restarted either by rebooting or entering systemctl restart gdm3. You can restart an individual virtual console getty with systemctl restart getty@ttyN where N is the number of the virtual console.

5.1. Login as root

Tip
Previous versions of Debian ran X11 on virtual console 7. As of Debian 9, the graphical environment login is on virtual console 1. Each login there for a different user creates a session on the next unused virtual console.

Switch to Virtual Console 2, by pressing Ctrl+Alt+F2.

Enter root and press Enter, then enter the root password you set earlier.

5.2. Remove the dummy Desktop User (optional)

Unless you want another account that that is set up to use the default desktop environment, delete desktop with:

deluser --remove-home desktop
Note
If you do keep this account, you will not be able to run the FS from it unless you add this account into the additional hardware access groups such as is done for oper and prog by fsadapt.

5.3. Install GRUB to the secondary disk (if available)

  • If you installed with UEFI boot, run the command

    cp /dev/sda1 /dev/sdb1
  • If you installed with BIOS boot, install GRUB to the Master Boot Record by running: dpkg-reconfigure -plow grub-pc and after pressing Enter twice to accept the kernel command line extra arguments and default command line arguments, use the arrow keys and Space to select both /dev/sda and /dev/sdb (but not /dev/md0) and press Enter to finalise the reconfiguration. (You should then see Installation finished. No error reported appear twice in the progress messages as GRUB is re-installed to both drives.)

5.4. Setup HTTP Proxy for APT (Optional)

Should you wish to make APT use an HTTP proxy for downloads, create the new file /etc/apt/apt.conf.d/00proxies using vi containing:

ACQUIRE::http::Proxy "http://proxy.some.where:8080/";

to use a proxy proxy.some.where at port 8080 for example.

5.5. Edit /etc/apt/sources.list

Using your favourite text editor, eg vi, and comment out all cdrom entries (unless you don’t have a decent Internet connection and need to use DVDs, whereupon the dialogue presented below may differ) and check you have the equivalent of the following entries towards the top of the file, adding in contrib and/or non-free as needed:

deb http://deb.debian.org/debian/ stretch main contrib non-free
deb-src http://deb.debian.org/debian/ stretch main contrib non-free

and likewise the equivalent of the following entries towards the bottom of the file, again adding in contrib and/or non-free as needed:

deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
deb-src http://deb.debian.org/debian/ stretch-updates main contrib non-free

(where you can use any suitable mirror instead of deb.debian.org)

Also add contrib and/or non-free to the lines referring to the security.debian.org mirror in the middle of the file.

Warning
you MUST use stretch and NOT stable for the distribution in all these entries (but CD/DVD entries might use unstable.)

5.6. Update APT’s list of packages

Tip
Recent versions of Debian have the apt program, which gives a more user-friendly interface to the package manager than apt-get. We generally use apt-get except for applying updates.

Next tell APT to update its internal source list of packages using

apt-get update
Note
It is also possible to add additional DVDs at this stage using the apt-cdrom add command.

5.7. Download the FS Linux 10 package selections

  1. Install git and dselect

    apt-get install git dselect
  2. Update dselect's package lists

    dselect update
  3. Get the selections by downloading this repository:

    cd /root
    git clone https://github.com/nvi-inc/fsl10
    cd fsl10
  4. Feed the package selections into dpkg using the command, for amd64

    dpkg --set-selections < selections/fsl10_amd64.selections

    or, for i386

    dpkg --set-selections < selections/fsl10_i386.selections
  5. Start the additional package installation with

    apt-get dselect-upgrade

    then press Enter to confirm any updating of installed packages (where you have an Internet connection) and the installation of currently ~212 new packages (downloading ~196 MB from the Internet and/or DVDs) for amd64, somewhat more for i386 — unless you did not select the Desktop or added other tasks earlier.

Downloading commences for up to half an hour (depending on your Internet access and the exact revision of DVDs used).

Installation runs to completion.

5.8. Clean up the APT download directory

So that the update mechanism will work correctly, run

apt-get clean

6. Third Stage Installation

6.1. fsadapt

In the /root/fsl10 directory, start fsadapt with

./fsadapt

6.1.1. FS Adaptation: Modifications (Window 1)

Using the arrow keys and Space make your selections and press Enter.

  • For government computers select govt and noident.

  • If you are not using a GPIB board or USB dongle, you can deselect the GPIB option.

6.1.2. FS Adaptation: Setup (Window 2)

All of the steps in Window 2 need to be done once (even if you do not intend to use the serial ports) with the exception of sshkeys which can be used to generate new SSH keys if required. If you did not select the GPIB option in the previous page deselect the two related options on this page (but do not deselect set_perms as it is always required). Otherwise, simply press Enter with the OK selected to continue.

Note
The updates option relies on email to root being re-directed to some mailbox that will be read regularly, so make sure you set that up and test it as well. The installer sets it up to go the desktop account by default which would definitely be a problem if you have removed that!

6.1.3. GPIB driver configuration (Optional)

On the /etc/gpib.conf screen, use the up/down arrow keys to select the required GPIB controller and press Enter on OK to continue.

6.1.4. Serial port configuration

On the /etc/default/grub: serial port configuration screen up/down arrow keys to select the required RS232 serial card (or None if you don’t have one) and press Enter on OK to continue.

6.1.5. FS Adaptation: Settings (Window 3)

On Window 3 you can choose to modify the email or network settings if required. Simply press Enter on OK to continue.

6.1.6. FS Adaptation: Network Services (Window 4)

The Window 4 will show what services are enabled. Use the up/down arrows and Space to select secure and press Enter on OK. Thereafter use the up/down arrows and Space to select those services you actually need. If you need printing, you will need to select netipp (remote access to this can be blocked by configuring ufw with either not explicitly allowing or instead denying the CUPS service). Press Enter on OK to set them up and finish with fsadapt.

Note that the fsadapt script can be re-run at a later date should you need to change the adaptations.

6.2. Set Passwords

Set passwords for the oper and prog accounts with:

passwd oper
passwd prog

entering the passwords twice as prompted.

6.3. Install tools for RAID (Optional)

You can install some useful tools for working with the RAID, if you’re actually using it, with:

~/fsl10/RAID/install_tools

The rest of this document assumes the first three of these tools have been installed. The five tools are:

  • mdstat allows all users to check on the RAID status

  • refresh_secondary allows root to refresh a secondary disk that is from the same RAID and has been booted on its own

  • blank_secondary allows root to initialize a secondary disk, must be used with extreme care

  • drop_primary allows root deliberately to drop the primary disk out of the RAID for use as a backup

  • recover_raid allows root to re-add a disk that fell out of (or was removed from) the RAID back into it

Tip
More information about RAID operation can be found in the RAID notes for FSL10 document.

See also the Setup additional disk sub-section in the Post Install section below.

6.4. Download the Field System

   cd /usr2
   git clone https://github.com/nvi-inc/fs fs-git
   cd /usr2/fs-git
   git checkout -q tag

where tag is the latest available release, be it 10.0.0 or later.

Important

Although 10.0.0 is the current release at the time this was written, and should suffice for an initial installation, it may well not be the most up-to-date release when you are installing. To find more recent releases, go to:

You should probably use the most recent release ending in .0 (a so-called feature release) with no trailing -<string>, e.g., 10.0.0. However, if there is a more recent patch release (not ending .0) for the most recent feature release, you should use the most recent patch release. For example, if 10.1.0 is the most recent feature release and there are corresponding patch releases, 10.1.1 and 10.1.2, the last one, ending .2, is probably the best choice.

Note
Releases numbered before 10.0.0 are listed mostly for historical reference. They are not intended for operational use.

6.5. Run FS install script

This will set the /usr2/fs link, set /usr2/fs-git permissions, and install default copies of all the FS related directories.

make install

and enter y to confirm installation.

6.6. Make the FS

The FS must always be compiled as prog.

Warning
Make sure you log-out as root, and log-in again as prog.
cd /usr2/fs
make >& /dev/null

then

make -s

to confirm that everything compiled correctly (no news is good news).

6.7. Reboot

If you installed the RAID with two disks (and RAID tools) check its progress with:

mdstat

until the array no-longer shows a recovery in progress.

Important
Do not reboot until the recovery is complete.

The final steps are to remove any DVD from the machine and to restart the machine using reboot as root or Ctrl+Alt+Del whilst watching that everything starts up smoothly.

Your new FS machine should now be ready to be customised to your requirements by tailoring the control files in /usr2/control and adding suitable station specific software to /usr2/st. See the files in the /usr2/fs/misc directory for more information.

7. Post Install

All commands (except checking the RAID status) in this section need to be run as root.

7.1. Setup additional disk

Note
An additional disk should be at least as large as the smallest disk already in use in the RAID.
Note
You will need to have hot-swapping enabled in your motherboard’s setup menu, at least for the secondary controller (it should also be enabled for the primary).
Note
This sub-section assumes you have followed the directions in the Install tools for RAID (Optional) section above.

Ensure the RAID is synced by checking that

mdstat

shows no recovery in progress. If there is none, shut down the machine safely. If you installed with a second disk, remove it and place it on the shelf.

7.1.1. Initialize new disk

Warning
Do not initialize a disk unless you are sure there is no data on it that you need to preserve.

For the first time use of an additional disk with a new install, the disk should be initialized to make sure it has no already existing structure. This should be done even if the disk has been used in a different FS computer or a previous install on this computer.

Boot with just the primary disk installed. Use the script:

blank_secondary

The script will wait for the new disk to be turned on. Insert a new disk in the secondary slot. Turn the key to turn the disk on. There will be a prompts asking if wish to proceed. If it is a new disk or you are sure it safe to erase this disk, it is safe to answer y. If you are unsure about this or otherwise need to abort answer n.

7.1.2. Refresh secondary disk

Warning
You can refresh a disk if it has been erased or has previously been used in this RAID and is older than the current primary. If it is newer than the current primary (maybe from a failed FS upgrade that needs to be abandoned) or comes from a different RAID (i.e., system) or has a different structure (i.e., was previously used for something else), it will have to be erased first. The script should detect these conditions and stop with an appropriate message. In that case, consider carefully if it is safe to erase the disk (probably not). If you determine it is safe, follow the instructions for Initialize new disk.

Boot with only the primary disk installed. The new secondary disk must be keyed off or removed. The script will refuse to run if there is a second disk already turned on. This will ensure that no other disk is installed and mistaken for the disk to be refreshed.

Note
With the RAID now missing a disk, you may see ~20 of the volume group not found error messages, then the machine will boot. These error messages only appear like this the first time a disk from the RAID is booted without its partner.

Once booted, login as root.

Run the script:

refresh_secondary

When the script says it is waiting for the second disk, key it on.

Once you reach the message that it is recovering, you can resume using the computer as usual. You can stop the updating of the recovery message with Ctrl+C as described in the output. You can also safely reboot at this point, if it is needed.

If later you want to check the progress of the status of the RAID re-sync, you can use:

mdstat

When the syncing is complete, you can repeat the process of the previous sub-section and this sub-section if you have a third disk that needs to be set-up.

7.2. Consider additional customizations

Please refer to the appendix Additional Setup Items below for customizations that your system may need or that you may find useful.

Appendix A: Additional Setup Items

This appendix covers several customizations that may be helpful depending on the requirements for a system. It serves as a reference for how to make these changes, but can also be helpful as a checklist when setting up a new system. All actions in this section require root permissions.

A.1. Additional security and CIS Benchmarks

For stations that wish to conform to the additional security recommendations of the Center for Internet Security (CIS), move on to the CIS hardening FSL10 document.

A.1.1. Alternate hardening

If you don’t want the complete CIS hardening, which creates some inconveniences and is only required in certain environments, you may still be interested in applying a subset of the remediations. You can pick and choose those from the CIS hardening FSL10 document and its script.

A useful minimum set of features to apply would be to install ufw and block everything except ssh and further restrict ssh access with TCP Wrappers.

ufw set-up

To install and configure ufw to only allow ssh for incoming conections, use the commands:

apt-get -y install ufw
ufw allow OpenSSH
ufw --force enable

Addition set-up for ufw is covered below in the More firewall rules sub-section.

TCP Wrappers set-up

A base set-up for TCP Wrappers is

/etc/hosts.deny
ALL:ALL
/etc/hosts.allow
sshd:ALL

It is recommend that you further restrict sshd by using specific hosts and/or sub-domains instead of ALL. Please use man hosts_access for more information about configuring TCP Wrappers

A.2. Customize root’s .bashrc file

There are a few changes you should consider for root's .bashrc file.

  1. If you have applied the CIS remediations, you should consider uncommenting the line that sets the umask to 022. The remediations set it to 027 in /etc/profile, which may cause problems with routinely created files, including some in this section covering optional changes.

  2. Uncomment the the alias commands that add the -i option to the commands cp, mv, and rm as the default. This can help avoid some careless errors.

  3. Add the command set -o noclobber to avoid accidently overwriting existing files with I/O redirection. Other options to consider setting are physical and ignoreeof.

A.3. Network configuration changes

This sub-section requires using nm-connection-editor on a graphic display (nmtui may be an option on a text terminal, but it has not been fully verified). You will probably need to be root or desktop to do this. When you run this program and select a connection, e.g., Wired connection 1 under Ethernet, the Edit button should become active. If it stays greyed out, you don’t have sufficent permission. All the sub-sections below assume you are in program and have sufficent permision,

A.3.1. Make the connection always appear on the same interface regardless of the MAC address.

This is useful both to make the connection appear on only one interface and/or make it the same interface if the computer (or NIC) is changed.

  1. Select your connection snd click Edit.

  2. Select the Ethernet tab.

  3. Change the Device field to just list the name of the interface (typically eno1) by removing the MAC address in parentheses.

  4. You may want to also set the IPv6 Settings to use Method: Ignore.

  5. Click Save.

  6. Click Close.

A.3.2. Disable the second Ethernet port

This may be useful if your second port has a IPMI interface and the kernel detected a connection there and it is interferring with the normal or the IPMI connection.

  1. If there is no Wired connection 2, click Add. Otherwise select that connection, click Edit, and skip to step 4. It may be benign to Delete any other connections except Wired connection 1.

  2. Make sure Ethernet is selected in the drop down box and click Create…​.

  3. Change the Connection name: to Wired connection 2.

  4. Select the Ethernet tab.

  5. Make sure the Device field just lists the second ethernet device (typically eno2) with no MAC address in parentheses.

  6. Select the IPv4 Settings tab.

  7. For Method select Disabled.

  8. Select the IPv6 Settings tab.

  9. For Method select Ignore.

  10. Click Save.

  11. Click Close.

A.3.3. Update IP address, hostname, FQDN, and other network information

This is useful if the computer is physically moved to a different site or its network information needs to be be updated for a different reason. This is typically not needed if you use DHCP, which may still require some of the changes in step 6 (please let us know if you gain experience).

  1. Select your connection and click Edit.

  2. Select the IPv4 Settings (or IPv6 Settings if you are using IPv6) tab.

  3. Adjust your Manual Method configuration: Addresses, DNS Servers (comma separated), and Search domains.

  4. Click Save.

  5. Click Close.

  6. Modify other system files

    Update the information as appropriate. The system may have initially been installed with the default hostname debian and no domain name.

    /etc/hostname

    Change your hostname

    /etc/hosts

    Update your IP address, FQDN (canonical name), and alias (typically the hostname, but multiple aliases/nicknames are allowed).

    /etc/networks

    Use your local subnet (class A, B, or C) for the localnet line.

    /etc/mailname

    Use fully qualified node name.

    Note

    If your system doesn’t have a FQDN or you don’t want to show it in e-mail messages, you may be able to use a fake one. A FQDN may be necessary to allow messages to be sent successfully to some remote hosts and mailman mail lists. A possible strategy for this is to append .net to the node name you use in this file and the next. The node name in these two files can be different than the official hostname. However, these two mail related files should be consistent. You might consider fs1-<xx>.net (or fs2-<xx>.net), where <xx> is your station two letter code (lower case).

    /etc/exim4/update-exim4.conf.conf

    Look for hostnames=, use fully qualified domain name.

    Then execute:

    update-exim4.conf

    When finished, reboot.

A.4. Disable Desktop User

If you do not need the functionality available in the Desktop environment, you can disable the desktop account. You can re-enable the account later if you need it. To disable it, execute:

usermod -L desktop

You can undo this by using the -U option instead.

To prevent connecting with ssh using a key, create (or add desktop to an existing) DenyUsers line in /etc/ssh/sshd_config:

DenyUsers desktop

And restart sshd with:

systemctl restart sshd

You can undo the ssh block be removing the line (if it only has desktop) or removing desktop from the line and then restarting sshd.

A.5. Remove ModemManager package

If you use serial ports, it is strongly advised that you remove the ModemManager package to avoid conflicts over access to the ports. Execute this command:

apt-get purge modemmanger

A.6. Remove Anacron package

If you enabled the weekly update job in fsadapt (it is strongly recommended), we recommend that you also remove the anacron package so that the job will run at a fixed time every week, even if the system is turned off for some periods of time. Execute this command:

apt-get purge anacron

A.7. More firewall rules

The following tersely summarizes some ufw settings that may be useful:

#SSH
ufw allow OpenSSH
#NTP
ufw allow ntp
#remote access to metserver (or gromet) on port 50001
ufw allow 50001
#anywhere from subnet
ufw allow from 192.168.4.0/24
#RDBE multicast to addresses from subnet
ufw allow in proto udp to 239.0.2.0/24 from 192.168.4.0/24
#? RDBE multicast to group from subnet ?
#ufw allow in proto igmp to 239.0.2.0/24 from 192.168.4.0/24

A.8. Configure e-mail

The configuration described here (Internet site or mail sent by smarthost in the exim4 configuration, no incoming mail, reply-to-filter, and modified user names), provides good support of the FS msg and rdbemsg utilities.

  1. As root, enter:

    dpkg-reconfigure exim4-config

    to change the set-up. Typically you should select internet site, use your host name in place of debian when it occurs, and otherwise select defaults at all the other prompts. (The only other recommended choices are local delivery only or mail sent by smarthost; received via SMTP or fetchmail.) If you want to receive incoming mail, you will also need to enable SMTP connections in Window 4 of fsadapt (and if you are using a firewall, you will need to enable such connections for it). We recommend that you NOT receive incoming mail on this computer.

  2. If you follow the recommendation not to receive incoming mail and your system is not set-up for local delivery only, you should set the Reply-To address for outgoing messages to a real e-mail account at your institution that is read regularly. You can do this by (all as root):

    1. Create a file with contents (four lines):

      /etc/exim4/reply-to-filter
      # Exim filter          << THIS LINE REQUIRED
      
      headers remove "Reply-To"
      headers add "Reply-To: email@address"

      Where email@address is the e-mail address you want replies to be addressed to. If you want more than one, separate them with commas.

    2. In /etc/exim4/exim4.conf.template, at the beginning of the file add (two lines):

      #set reply to
      system_filter = /etc/exim4/reply-to-filter
    3. Then execute

      update-exim4.conf
      systemctl restart exim4
  3. You should change your /etc/aliases so root and prog e-mail goes to oper.

    • change root: desktop to root: oper

    • add prog: oper

    • add desktop: oper

    This is recommended as a “catch all” since the oper account is presumably under regular use and any messages sent there are likely to be noticed. This is particularly important for system error messages since they should be delivered to a mail box on the system in case there is a network problem that might prevent them from being delivered off system. You can however add additional off machine delivery of these messages to whatever addressees you wish and we recommend this as well. These should include an e-mail account at your institution that is read regularly (maybe the same address as the Reply-To address you may have set above would be a good choice). To do this, create a .forward file in oper's home directory. The permissions should be -rw-r—​r--. The contents should be similar to (left justified):

    \oper
    user@node.domain

    where user@node.domain is the off machine addressee you want the messages to go to. You can add additional lines for additional addressees. The backslash (\) before oper prevents the mail system from getting into an infinite loop re-checking oper's .forward file.

  4. If you have made the above changes to forward messages to another an e-mail account on another machine, you should customize the User Name (not login name, the User Name is the fifth field) of root, prog, oper, and desktop in /etc/passwd to identify the source of the message. For root and prog, it is recommended to append a string like at node (it is probably best to avoid FQDNs), where node is this machine, e.g., for atri you might change the 5th field for root from

    root

    to

    root at atri

    For oper, you might instead prepend your site name to the accounts for clearer reading in ops e-mail messages, e.g., for oper on atri at GSFC, we changed the 5th field for oper to:

    GSFC VLBI Operator

    and for completeness, for prog and desktop we use:

    GSFC VLBI Programmer
    GSFC Desktop User

    These changes will help the recipient (possibly you) determine which system generated this message since it may not be obvious given the modified return address.

  5. To give oper an indication at login that there is mail to read, add either (to get a count of messages):

    test ! -f /var/mail/oper || from -c

    or (to see the senders and subjects):

    test ! -f /var/mail/oper || from

    to end of oper's .profile file (if using bash as the login shell) or .login file (tcsh).

  6. Lastly, check the default mailbox directory /var/mail/ for accounts that may have messages that arrived before the e-mail system was fully configured. Be sure to resolve any system messages that may have been received. You can check to see what accounts have mail with:

    ls /var/mail

    which will list each user account mail file that exists. Check and clear each user’s mailbox (where user in the line below is the account name) that has received mail (as root):

    mail -f /var/mail/user

    If there are messages in the desktop user’s mailbox that you want to preserve and oper's mailbox is empty or non-existent, you could consider renaming desktop's mailbox to be oper's. If you do so, be sure to change the owner of the file to be oper.

A.9. Generate FQDN in HELO for outgoing mail

If mail from your system is being rejected by some servers because exim4 is not providing a Fully Qualified Domain Name (FQDN), in its HELO message, the following solution should fix the problem.

Add the following line to the beginning of /etc/exim4/exim4.conf.template:

MAIN_HARDCODE_PRIMARY_HOSTNAME=ETC_MAILNAME

Then execute:

update-exim4.conf
systemctl restart exim4

A.10. Set X display resolution at boot

If your display sometimes starts with the wrong resolution, you may be able to configure a better resolution. The following is a description of something that worked for at least one system. The details of your system may require some changes (beyond the resolution and output name).

First you need to determine the correct resolution and output name. You may be able to do this with xrandr. If the screen currently has the correct resolution, you can just execute:

xrandr

The output might look like:

Screen 0: minimum 320 x 200, current 1920 x 1200, maximum 1920 x 2048
VGA-1 connected primary 1920x1200+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
   1024x768      60.00
   800x600       60.32    56.25
   640x480       59.94
  1920x1200 (0x42) 154.000MHz +HSync -VSync
        h: width  1920 start 1968 end 2000 total 2080 skew    0 clock  74.04KHz
        v: height 1200 start 1203 end 1209 total 1235           clock  59.95Hz

Where the current screen resolution is 1920x1200 and the output name is VGA-1.

You can then generate the needed Modeline by executing:

cvt 1920 1200

Which might generate output:

# 1920x1200 59.88 Hz (CVT 2.30MA) hsync: 74.56 kHz; pclk: 193.25 MHz
 Modeline "1920x1200_60.00"  193.25  1920 2056 2256 2592  1200 1203 1209 1245 -hsync +vsync

As a test, you can make a script (use an appropriate name), that will enable that resolution. Use the output name (VGA-1 in this example) and the tokens following Modeline from above. There are three lines after the #!/bin/bash line.

~/display_1920x1200
#!/bin/bash
xrandr --newmode "1920x1200_60.00"  193.25  1920 2056 2256 2592  1200 1203 1209 1245 -hsync +vsync
xrandr --addmode VGA-1 1920x1200_60.00
xrandr --output VGA-1 --mode "1920x1200_60.00"

Be sure to chmod u+x the file before executing.

If that is successful, you can use output name (VGA-1 in this example) and Modeline from above to make a file (you may need to create the directory first):

/etc/X11/xorg.conf.d/10-monitor.conf
Section "Monitor"
Identifier     "VGA-1"
Option         "Enable" "true"
Modeline "1920x1200_60.00"  193.25  1920 2056 2256 2592  1200 1203 1209 1245 -hsync +vsync
EndSection

Section "Screen"
Identifier     "Screen0"
Device         "Device0"
Monitor        "VGA-1"
DefaultDepth    24
#Option         "TwinView" "0"
SubSection "Display"
    Depth          24
    Modes          "1920x1200_60.00"
EndSubSection
EndSection

You should chmod the permissions for directory with o+rx and the file with o+r, if those are not already set.

You could then try restarting the display (after closing all windows) with:

systemctl restart gdm3

or rebooting.

A.11. Fix for slow reboot/shutdown with Wayland disabled

TODO: Still slow for some cases (exact ones still not clear)

If you have chosen to disable Wayland for gdm3 and have a problem with slow reboots/shutdowns, the following may help.

  1. Copy the file /lib/systemd/system/gdm3.service into /etc/systemd/system/.

  2. Comment out the KillMode line (which changes it to control-group, the default).

  3. Add a line TimeoutStopSec=1 to the [Service] section.

  4. Execute:

    systemctl daemon-reload

    or reboot.

A.12. Use KeepAlive to prevent VLAN firewall inactivity time-out

If there is a VLAN firewall in use on the local network, it may be necessary to use KeepAlive for TCP connections to prevent inactivity time-outs for network connections from the FS to the VLBI equipment when no activity is occurring with the system. For some devices, having the time-out break the connection may cause an issue with the number of connections available.

To use KeepAlive to prevent the inactivity time-outs, first install the package libkeepalive0:

apt-get install libkeepalive0

Then add the follow lines for oper (and prog):

~/.profile
export KEEPCNT=20
export KEEPIDLE=180
export KEEPINTVL=60

Then add the following alias for oper (and prog):

~/.bash_aliases
alias fs='LD_PRELOAD=libkeepalive.so fs'

You will need to terminate the FS, log out, and log back in to activate these changes.

Note
If you run the FS from a script, you will need to include the setting of LD_PRELOAD explicitly in the script since scripts do not pick up aliases.

A similar alias can used to allow other individual applications to avoid the inactivity time-outs. (A better solution is available for ssh, discussed below.) It is also possible to put export LD_PRELOAD=libkeepalive.so in ~/.profile to enable it for all applications, but this may generate some error messages (in the case of xterm at least, the error is apparently benign).

If you need to have a persistent ssh connection, add the follow for oper (and prog):

~/.ssh/config file:
Host *
    ServerAliveInterval 200
    ServerAliveCountMax 2

This can be set selectively per remote system. The interval of 200 seconds is chosen to be less than the 300 seconds that some (possibly security hardened) servers may use.

If not already set correctly, set the ~/.ssh/config file’s permissions and ownership for oper (analogously for prog) with:

chmod 644 ~oper/.ssh/config
chown oper.rtx ~oper/.ssh/config

A.13. Remove login banners for commands run by ssh on remote systems

If you use ssh as oper (and maybe prog), to run commands on other systems as part of FS operations, you may get login banners mixed in with the output. You can suppress the banners by adding the following for oper (and analogously for prog):

~/.ssh/config file:
Host *
    LogLevel ERROR

This will allow errors to be displayed while suppressing the login banners of remote systems. This can be set selectively per remote system.

Please check the end of the Use KeepAlive to prevent VLAN firewall inactivity time-out section for setting the ownership and permissions on ~/.ssh/config.

A.14. Printer setup

  1. Make sure your printer is connected, to the computer or the network, as appropriate.

    Tip
    Newer computers usually do not have a parallel port (IEEE 1284). If not, and your printer requires a parallel connection, you should be able to obtain a USB/Parallel converter for less than US$20.
  2. Login in to the X-display or remotely using an X-capable display.

  3. Start firefox

  4. Enter URL: localhost:631

  5. Select Add printers and classes.

    You may be prompted to enter credentials. If your account is a member of the lpadmin group, you can use your own credentials; if not, those of the root account or another account that is a member of lpadmin will be required.

  6. Add your printers.

    Connected printers may be automatically offered to be added. You may also be able to find printers using the Find Printer function. If CUPS offers you the wrong type of printer to be automatically added or it is unclear what driver to select for a printer, you may be able to get some useful information to help with manually installing your printer by searching the Internet for the string cups and your printer model.

    Some printers will work with an AppSocket/HP JetDirect connection of the form socket://hostname.

  7. Be sure to select a printer as the default (usually by selecting Printers at the top of the page, then select the printer to be set as the default, then from the Administration drop down: Set As Server Default).

  8. Quit firefox

A.15. NTP configuration

For good performance with NTP, please follow the recommendations in /usr2/fs/misc/ntp.txt.

Additionally, to make the ntpq -c pe output more readable for local devices, you can adjust the contents of /etc/hosts. The local devices should be listed in the file, but use a nickname (15 characters or less) that is meaningful locally in place of the canonical name (the first name after the IP address). The canonical name can be listed after the nickname.

A.16. Add raid-events scripts

If your system is using a RAID configuration, you may want to install the raid-events script. The script provides email notifications of when Rebuilds (and array checks) start and end. For full details on the script and installation instructions, please see the raid-events sub-section in the Script descriptions section of the RAID Notes for FSL 10 document.

A.17. Add refresh_spare_usr2

If you are using two systems, an operational and a spare, you may want to install the refresh_spare_usr2 script. The script can be used to backup the /usr2 partition on the operational system to the spare system. For full details on the script and installation instructions, please see the refresh_spare_usr2 sub-section in the Script descriptions section of the RAID Notes for FSL 10 document.

Appendix B: Managing Security Updates

It is strongly recommended that you use the weekly cron update download (the “weekly cron job”) as configured according to the Window 2 sub-section in the fsadapt section above. This will keep you informed of the available updates on a weekly basis.

It is also recommended that you remove anacron as described in the Remove Anacron package section below. This will cause the updates to always be downloaded at what should be innocuous time, early Sunday morning (but this can be adjusted if need be).

Note
An optional method for identifying available updates without using the weekly cron job is described below in the section Manually checking for updates.

B.1. Installing updates (Upgrading)

Tip
It is recommended that a disk rotation be performed before any update is installed. This will make recovery much easier if a problem with the update is discovered. Please see the FSL10 Raid document section Recoverable testing for a streamlined method to manage testing of updates.

If updates are needed, the weekly cron job will send a message to root (or whoever e-mail to root is aliased to, typically oper) with instructions on how to install the updates. You can choose a convenient time, when not in (or about to start) operations, to install the updates and test the system.

Important
The weekly cron job message will include instructions for handling a kernel update if one is available. See the Kernel updates sub-section below for additional considerations for kernel updates.

The commands for installing the updates given by the message are (note the use of apt instead of apt-get):

apt upgrade

Enter y to confirm as needed. Then

apt clean

If the weekly cron job was installed according to the fsadapt section above (for Window 2), the first of these commands (with upgrade) will show if any NEWS items are included in the update. If there are, they will be displayed by a paging program at the beginning of the upgrade and you will be given an extra chance to abort before installing.

Note
NEWS items are, rarely occurring, announcements that may indicate additional steps are needed beyond the standard installation process. If any NEWS items are displayed, you should consider whether these will effect your system and how to handle them before installing. The first command above (with upgrade) will also cause e-mails to be sent to root with the NEWS information.

B.2. Kernel updates

Warning
Kernel updates require extra care and testing. If you are using a RAID, you should consider using the Recoverable testing procedure to give more, and easier, options for recovery in case there is a problem. That procedure contains special instructions for kernel update testing.

If there is a kernel update available, the weekly cron job output will include a warning at the end with additional instructions depending on which type is available. There are two types of kernel updates:

  1. ABI updates, e.g., from 4.9.0-11-amd64 to 4.9.0-12-amd64 (with 11 and 12 being the ABI versions), which change the kernel ABI (Application Binary Interface). The warning for this case is:

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    NB: The Linux kernel image is one of the packages due to be upgraded.
    NB: (The kernal ABI has changed as per the linux-latest source package above
    NB:  so all out-of-tree modules WILL NEED TO BE REBUILT after you REBOOT.)
    NB: Please allow _extra time_ for TESTING after the upgrade.
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  2. Non-ABI updates, which update the kernel, but do not change the ABI. The warning for this case is:

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    NB: The Linux kernel image is one of the packages due to be upgraded.
    NB: (Upgrading will OVERWRITE the running kernel and require you to REBOOT!)
    NB: Please allow _extra time_ for TESTING after the upgrade.
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Be sure to allow time to follow the instructions when planning to install these updates. As described in the ABI update warning, you will need to rebuild any out-of-tree modules after rebooting for that case. This is discussed in the Updating out-of-tree modules sub-section below.

Caution
In extreme circumstances, an ABI (but not a non-ABI) kernel update can be deferred to a later date when more extensive testing can be performed by using apt-get in place of apt in the instructions for installing the update. This works because an ABI update involves new packages. The apt-get command will install the updates for existing packages, but it will not install the new packages. While this method can be used to install the other updates, it is not recommended since there are presumably security patches needed for the kernel and they are not being installed in this case.

B.2.1. Updating out-of-tree modules

When a ABI update is installed, it will be necessary to update any, so-called, out-of-tree modules that use the kernel ABI. This must be done after rebooting with the new kernel installed.

For a normal FSL10 installations, unless you have installed other out-of-tree modules, the only module that needs to be rebuilt is the GPIB driver (if it is installed). You will need to recompile it (usually using fsadapt, Window 2, config_gpib only) after the initial reboot and then (to keep these instructions simple) reboot again.

If you have installed other out-of-tree modules (e.g., you use a special driver for some of your NICs), you will need to update them appropriately after the initial reboot and then (to keep these instructions simple) reboot again.

B.3. Recovery from a failed update

If an update fails, e.g., an updated kernel fails to boot or another problem is discovered, you can recover as described in FSL10 RAID document Recoverable testing section, if you were following that method, or from a shelf disk according to the FSL10 RAID document Recover from a shelf disk section if not and you have a good shelf disk.

B.3.1. Additional recovery option for a failed ABI kernel update

For a ABI update that has failed, it is also possible to try to use the previous kernel on the current system. For a single boot, use the Advanced option in the grub menu at boot and then select the previous kernel. You can change back permanently to the previous kernel by purging the new kernel and its headers. To do this, use:

dpkg -l|grep linux-image
dpkg -l|grep linux-headers

to determine the ABI version to be removed. For example, for the first command above, you may get:

linux-image-4.9.0-11-amd64
linux-image-4.9.0-12-amd64

The package with 12 would be the later version that should be purged:

apt-get purge linux-image-4.9.0-12-amd64

Likewise with the linux-headers. For example, for the 12 ABI version, there will be two packages you should purge:

linux-headers-4.9.0-12-amd64
linux-headers-4.9.0-12-common

B.4. Manually checking for updates

If you do not use the weekly cron job to check for updates, or if you want to make sure you have the very latest updates when you install them, you can run the distributed copy of the weekly update script manually to check for updates:

/root/fsl10/etc_cron.weekly_apt-show-upgradeable

If there is no output, there are no updates to install.

If there is output, there are updates to install. You can install them by following the installation procedure in sub-section Installing updates (Upgrading) above, except you will use the instructions from the output of the script above instead of from the weekly cron job (the outputs should be equivalent for the same set of updates). Additionally, please read the following NOTE.

Note
If the weekly cron job has not been installed, you may not get a display of NEWS items and a chance to abort when you install the updates. You can use the method below with the --which=news parameter to check for NEWS before installing an update.

Any NEWS items will be included in the script output along with the packages to be updated. If you would like to see any NEWS items more distinctly after the previous command and before installing the updates, you can run the script again using the --which=news option:

/root/fsl10/etc_cron.weekly_apt-show-upgradeable --which=news

If there are updates available and no NEWS items, you will only get the installation instructions.

You can use this second form of running the script to check for updates initially, if you do not need to review which updates are available (you will still get warnings about kernel updates). As usual, you will see no output at all if there are no updates available.

B.5. End of security updates

When support for stretch ends, currently expected in June 2022, there will be no more security updates. At that time, the existing packages will be migrated to the Debian archive site. This will be visible in the output from the weekly cron job script as errors that the packages files can’t be found. Two steps are needed at that time:

  1. If you have been using the weekly cron job, it should be deleted:

    rm /etc/cron.weekly/apt-show-upgradeable

    (you may need to answer y to confirm)

  2. Change the /etc/apt/sources.list file to point to the archive site. Although there will be no more security updates, this will enable downloading of additional packages if they are needed. The new lines that should replace the corresponding existing lines are:

    deb http://archive.debian.org/debian/ stretch main contrib non-free
    deb http://archive.debian.org/debian-security stretch/updates main contrib non-free

    And if you are using deb-src lines:

    deb-src http://archive.debian.org/debian/ stretch main contrib non-free
    deb-src http://archive.debian.org/debian-security stretch/updates main contrib non-free

    Otherwise the deb-src lines can be commented out (with a leading #). Any other deb or deb-src lines relating to updates, proposed-updates etc. should likewise be commented out.

    In addition, if you want to install packages from more recent distributions that have been backported to stretch you can add:

    deb http://archive.debian.org/debian-backports stretch-backports main contrib non-free

    However, the “backports” are not normally needed.

    Lastly, update the index files:

    apt-get update

    This may generate an error about a Release file having expired, but that is benign.

Note
Now that stretch is no longer supported, it is strongly advised that you move your FS machine behind a firewall or upgrade it to a more recent FS Linux release.

Appendix C: Rescue mode

Rescue mode is useful for repairing some problems that prevent booting and/or logging in.

Note
If your computer’s setup utility is locked with a password, you may need that password to select booting from your installation media.
Note
You should provide suitable values for your system when a specific value is required. Values that agree with the FSL10 install described in this document (or reasonable defaults) are shown in parentheses.
  1. Boot from installation media

  2. Select Advanced options …​

  3. Select …​ Rescue mode

    Note

    You could instead add parameters to the boot line (by entering e for UEFI or Tab for BIOS on the …​ Rescue mode line instead), following the directions in the Set boot options and boot installer section above. This is not necessary nor usually helpful, but if you use this approach the most useful parameters are probably netcfg/disable_dhcp=true and/or time/zone=UTC. Use of added parameters will change the dialogue below.

  4. Select Language (English)

  5. Select Location (United States)

  6. Select Keymap (American English)

  7. Network configuration

    If no network is currently available (or you know that you do not need it for the rescue), simply press Enter when DHCP autoconfiguration starts and press Enter again for the resulting Network autoconfiguration failed message. Thereafter select Do not configure the network at this time and enter in the machine’s hostname when prompted before continuing below.

    If the DHCP autoconfiguration succeeds before you can stop it, you may as well confirm the hostname and domainname and continue with the network anyway, since you never know when it might prove useful. (However, if you want to make sure you don’t use the network, you can select Go Back and press Enter for the resulting Network autoconfiguration failed message. Thereafter select Do not configure the network at this time and enter in the machine’s hostname when prompted before continuing below.)

    Otherwise if the DHCP autoconfiguration fails and you want to use the network, press Enter for the resulting Network autoconfiguration failed message. You can then select the appropriate option, most likely Configure network manually and give appropriate responses to the prompts, ultimately continuing below.

  8. Select time zone (Eastern)

    Note
    The selected time zone will have no effect on the timestamps stored on the disk for any changes you may make, but will affect the displayed times you see.
  9. Select Assemble RAID array

  10. Press Space on Automatic

    Press Enter to continue

  11. Select your root file system (/dev/vg0/root)

  12. Select Yes to mount separate /boot partition (/boot), unless it is corrupt

  13. Select Yes to mount separate /boot/efi partition (/boot/efi), unless it is corrupt

  14. Select Execute a shell in /dev/vg0/root (or whatever your root file system is)

  15. Select Continue to enter rescue mode

  16. Use whatever commands are needed for your repair

    Note

    If you need to use the network, DNS does not appear to work by default in recovery mode. Use of explicit IP addresses does work. If you need to use DNS, you can make it functional by deleting the symbolic link /etc/resolv.conf and creating it as a normal file with the nameserver information you want, e.g.:

    rm /etc/resolv.conf
    cat >>/etc/resolv.conf <<EOF
    nameserver 8.8.8.8
    EOF
  17. Use the exit command to exit when done

  18. Select Reboot the system

  19. “Bob’s your uncle” (i.e., you are done!)