Introduction

These notes detail adding extra security features to Field System Linux 10 as advised by Center for Internet Security (CIS). With the exception of the partition configuration, all actions are to be performed post-installation (see the FSL10 Installation document). All tests that failed, topics that need to be discussed, and issues with the benchmark remediations are provided in separate sections. A final section covers additional steps that are needed for FS operations.

This document is based on the results for the “CIS Debian Linux 9 Benchmark v1.0.0 - Level 2 - Server”.

Partition Configuration

During installation, be sure to create the logical volumes marked optional during the partition section.

Post-installation changes

All commands need to be run as root.

Scripted changes

As many changes as possible are implemented by the remediate script. The script is intended to be run after the “Third Stage Installation” steps in the FSL10 instructions, before any further changes have been made to the system (however initializing and adding other disks to the RAID can intervene).

cd /root/fsl10
script ../remediate.txt
./remediate
exit
Important
This script should not be run more than once on a system.
Tip
The use of the script command causes the output to be recorded in the specified file. This can be very helpful for understanding what went wrong if the script fails. The script itself uses the -x option to echo the commands as they are executed to make it easy to match the output with the commands being executed.

Manual changes

This section covers manual operations that are required after running the script.

  1. Reboot and execute:

    aa-enforce /etc/apparmor.d/*

All the CIS remediations that can applied at this point have been completed. The next section describes some other policies that can be considered.

Other policies

This section describes other policies that may be desirable. Some are beyond the CIS benchmark.

/etc/hosts.deny

Add:

ALL:ALL

/etc/hosts.allow

Add:

sshd:ALL

It is recommended that you further restrict sshd to specific hosts and/or sub-domains.

Password policies

You may wish to set minimum password length to 12.

You may wish to set password reuse remember to 24.

Set remote log host and/or retention to a minimum of one year

You may wish to configure a remote log host and/or set log retention period to at least one year.

To set a remote log host, edit the /etc/rsyslog.conf and /etc/rsyslog.d/*.conf files and add the following line (where loghost.example.com is the name of your central log host).

*.* @@loghost.example.com

Run the following command to reload the rsyslogd configuration:

kill -HUP rsyslogd

Set the retention period of system logs by editing /etc/logrotate.d/rsyslog. This should be configured to store logs for at least a minimum of a year (366 for daily 53 for weekly)..

Bootloader password

You may wish to create an encrypted password with grub-mkpasswd-pbkdf2:

grub-mkpasswd-pbkdf2
Enter password: <password>
Reenter password: <password>
Your PBKDF2 is <encrypted-password>

Add the following into /etc/grub.d/00_header or a custom /etc/grub.d configuration file:

cat <<EOF
set superusers="<username>"
password_pbkdf2 <username> <encrypted-password>
EOF

If there is a requirement to be able to boot/reboot without entering the password, edit /etc/grub.d/10_linux and add --unrestricted to the line CLASS=

Important
It is strongly recommended that booting without a password be permitted. Otherwise, if a reboot is required to continue operations it will not be possible unless some one with the password is available. If they aren’t available, this could lead to loss of VLBI data and can be a safety issue.

Example:

CLASS="--class gnu-linux --class gnu --class os --unrestricted"

Run the following commands to update the grub2 configuration and reset the grub.cfg permissions:

update-grub
chmod go-rwx /boot/grub/grub.cfg

CIS Exceptions

This section addresses the tests that failed in the CIS benchmark after all the remediations in this document were applied.

1.4.2 Ensure bootloader password is set

Must be set later by administrators.

2.2.2 Ensure X Window System is not installed

X Window system is required for FS use.

2.2.4 Ensure CUPS is not enabled

CUPS is required for operations.

2.2.11 Ensure IMAP and POP3 server is not enabled

Exim4 is required as MTA; it never accepts incoming remote connections (blocked at firewall).

3.5 Firewall Configuration

The firewall is configured with ufw instead of iptables and ip6tables. This causes four to six issues depending on the details of the installation, but ufw provides the same security as the recommended remediations. The configuration is set to by default deny for incoming connections, enable incoming SSH connections, and sets logging for all connections. Setup:

apt-get -y install ufw
ufw allow OpenSSH
ufw logging on
ufw --force enable

4.2.1.4 Ensure rsyslog is configured to send logs to a remote log host

A remote log server must be configured by system administrators later.

4.2.4 Ensure permissions on all logfiles are configured

All files except /var/log/wtmp have the requested permissions. That file changes on a reboot to rw-rw-r--, owned by root.wtmp. No sensitive information is stored in the file and making it readable for others allows commands like last and who to work.

5.2.6 Ensure SSH X11 forwarding is disabled

We require ssh X11 forwarding for use with FS for remote operations and testing.

5.4.2 Ensure system accounts are non-login

System accounts (i.e., accounts with UID less than 1000) oper and prog are needed for compatibility with the wider VLBI community and are only used as service accounts from AUID accounts. They require a valid shell, but direct local login and ssh login (including with keys) are disabled for these two accounts. Note that oper and prog have no password expiration (they has no passwords) or inactivity time-out, like other system accounts.

CIS issues that need to be addressed

This section lists further topics related to the benchmark that should be discussed.

2.2.1.2 Ensure ntp is configured

Need FS NTP configuration. That is more secure than the benchmark since it uses ignore by default.

2.3.4 Ensure telnet client is not installed

Would prefer to keep the telnet client, it is useful for debugging ASCII device protocol devices, which we have. The security weakness is telnetd, which is not installed, nor does the benchmark test for it.

4.1.1.2 Ensure system is disabled when audit logs are full

This may not be appropriate for an operational system.

5.2.13 Ensure only strong ciphers are used

What ciphers should we use?

5.2.14 Ensure only strong MAC algorithms are used

What MAC algorithms should we use?

5.2.15 Ensure only strong Key Exchange algorithms are used

What Key Exchange algorithms should we use?

5.2.16 Ensure SSH Idle Timeout Interval is configured

Five minutes is too short.

5.3.1 Ensure password creation requirements are configured

Should we use the NASA 12 character minimum?

5.4.1.4 Ensure inactive password lock is 30 days or less

This is too short for developers/troubleshooters

CIS Remediation problems

This section details problems with the recommended remediations.

2.1.2 Ensure openbsd-inetd is not installed

Remediation solves problem, but does not make the test pass. To do the latter required purge.

2.2.1.2 Ensure ntp is configured

Remediation makes it less secure. A default policy of ignore is better.

2.3.4 Ensure telnet client is not installed

The remediation does not make the test pass, that required purge.

3.2.4 Ensure suspicious packets are logged

The remediation lines added in /etc/sysctl.d/* are not respected at boot (unlike all others). To overcome this, the following lines are used in a new systemd service CISfix at boot.

sysctl -w net.ipv4.conf.all.log_martians=1
sysctl -w net.ipv4.conf.default.log_martians=1
sysctl -w net.ipv4.route.flush=1

4.1 Configure System Accounting (auditd)

Many of the remedations are described in terms of the contents of /etc/audit/audit.rules. However, the contents of that file are auto-generated at boot from the files in /etc/audit/rules.d, which is where these remediations must go.

4.1.6 Ensure events that modify the system’s network environment are collected

64-bit remediation had b64 and b32 rules concatenated on one line.

4.1.17 Ensure kernel module loading and unloading is collected

64-bit remediation was missing b32 rule.

Additional set-up

After the CIS hardening is completed, some additional set-up is needed.

Fix-ups

There are three issues that need to be corrected after the CIS hardening.

  1. The AIDE package used in CIS hardening adds a line to /etc/crontab to run aide.wrapper. Unfortunately at this time, the line inserted is incorrectly formatted, missing the “user” field, for Debian systems. This causes the cron daemon to ignore the file, thereby breaking basic cron functionality. This can be fixed by adding root as the user in a new field after the fifth field on the aide.wrapper line and then running:

    systemctl restart cron
  2. The CIS hardening configuration of PAM causes uses of sudo that require a password to increment the failure count when a correct password is accepted, leading to eventual lock-out of the user using sudo when the deny lock-out failure count is reached. To make sure the tally2 failure count is reset after a successful sudo that required a password, add to:

    /etc/pam.d/common-account
    account required pam_tally2.so
    Note
    To reset a locked-out user after CIS hardening, use /sbin/pam_tally2 -u user --reset where user is the user account. Leave off the --reset to see what the current tally is.
  3. Using the noexec option for /tmp causes a problem for the package management system. The dpkg-preconfigure program places and executes scripts on /tmp as part of package installation. The noexec option prevents the execution of the scripts. To work around this issue, you can exeucte:

    cd /root/fsl10/
    ./root_tmp

    The root_tmp script will do three things:

    1. Create a one time service at boot to clean the /root/tmp directory

    2. Set dpkg-preconfigure to use /root/tmp for temporary files

    3. Create an initial /root/tmp directory

    There may be other issues with using the noexec option for /tmp, but we don’t have any specifics at this time.

  4. Sometimes the firewall (ufw) does not work properly after rebooting. This has been noticed for remote access to gromet for met. data on port 50001. There are no other known issues. An apparent fix for this is to disable and re-enable the firewall. If you have this problem and the same solution works, a one-time service at start-up can be created to perform this action:

    cd /root/fsl10
    ./create_ufw_re-enable

    The new service will run at the next reboot. It is configured to run after ufw has been started.

Enabling user promotion to oper/prog and root

The model of FSL assumes oper and prog accounts will be used for operations and programming respectively. However, some organizations may have security and auditing restrictions that mean operators must log-in using their own account (possibly named with their Agency User ID, AUID). As the Field System currently operates, users will then need to switch to the oper or prog account after login. Likewise, if a user is allowed to elevate to root, they will need to do so after log-in to their own account. This sub-section covers how to enable this capability. The next sub-section Adding AUID accounts covers how to add an AUID account. The method described here, and in the next sub-section, uses dhorsley as an example user.

For oper and prog, we suggest creating two groups that can sudo to the accounts.

run visudo then add at end:

%operators      ALL=(oper) NOPASSWD: ALL
%programmers    ALL=(prog) NOPASSWD: ALL
%programmers    ALL=(oper) NOPASSWD: ALL

To allow operators to use refresh_secondary, shutdown, and reboot, add (respectively):

%operators      ALL=(ALL) /usr/local/sbin/refresh_secondary
%operators      ALL=(ALL) /sbin/shutdown
%operators      ALL=(ALL) /sbin/reboot

To use these commands the operators will need to enter (respectively) from their AUID accounts:

sudo refresh_secondary
sudo shutdown
sudo reboot

A password will be required. Trailing options can be used with the commands, as appropriate.

If the user can elevate to root, also add:

dhorsley       ALL=(root) ALL

Create the groups if they don’t exist:

addgroup operators
addgroup programmers

If they don’t already, make sure oper and prog have usable shells:

chsh -s /bin/bash oper
chsh -s /bin/bash prog

If the accounts haven’t been disabled for login already, do so:

usermod -L oper
usermod -L prog

To prevent connecting with ssh using a key, create (or add oper and prog to an existing) DenyUsers line in /etc/ssh/sshd_config:

Note
If you used the CIS remediate script, you should comment out the line: DenyGroup rtx as well.
DenyUsers oper prog

And restart sshd with:

systemctl restart sshd

Authorized users can then switch to oper with (similarly for prog and root):

sudo -i -u oper

No password will be required (except for root).

To ensure X authorization works do the following (this example is for user oper and works analogously for prog and root, but see the note at the end of step (1) about root's Xresources:

  1. Add this to the following file:

    ~/.profile
    if ! [ -z "$XCOOKIE" ]; then
       xauth add $XCOOKIE
    fi
    if echo $DISPLAY |grep -q localhost; then
    #   ssh from remote host with X display
        xrdb -merge ~/.Xresources
    else
        if ! [ -z $DISPLAY ]; then
          if xhost|grep -q 'SI:localuser:oper'; then
    #       local X display
            xrdb -merge ~/.Xresources
          fi
        else
    #     text terminal, do nothing
          :
        fi
    fi

    This will also set the Xresources to those of oper. (For root only the first clause would be used since Xresources would not be set.)

  2. Create the following file

    /usr/local/bin/oper_account
    #!/bin/bash
    set -e
    if echo $DISPLAY |grep -q localhost; then
     sudo -u oper XCOOKIE="$(xauth list $DISPLAY)" -i
    else
     if ! xhost|grep -q 'SI:localuser:oper'; then
       xhost +SI:localuser:oper >/dev/null
     fi
     sudo -u oper -i
    fi
  3. Execute:

    chmod a+rx /usr/local/bin/oper_account

The three numbered steps above can be executed for oper, prog, and root with:

~/fsl10/AUID/install_AUID

Adding AUID accounts

This sub-section describes how to add AUID accounts to be used with the ability to promote to oper, prog, and root as described in the previous sub-section. As in the previous sub-section, the following method uses dhorsley as an example user.

  1. Add any needed user accounts as appropriate:

    adduser dhorsley --home /usr2/dhorsley
    Important
    If you are configuring a spare computer, you will need to make sure the UID and GID for each user with a home directory on /usr2 is the same on both computers for the system-to-system backup of /usr2 to work properly.
    Note

    For normal operations, AUID users' home directories should be on /usr2. However, for some maintenance accounts, it may make sense to have the home directory some where else, typically on /home. In that case, use this command instead:

    adduser dhorsley

    The final step of this section will need to be adjusted accordingly.

    The UID and GID of the account should still be kept in agreement between the two computers.

  2. Add each user to these groups as appropriate, e.g.:

    Note
    This step assumes that the operators and programmers groups have been created as described in the previous sub-section Enabling user promotion to oper/prog and root.
    adduser dhorsley operators

    and/or:

    adduser dhorsley programmers
  3. If the user should be able to manage printers, use:

    adduser dhorsley lpadmin
  4. If the account will be used by an operator and/or programmer, the X11 environment needs to be set-up. The following command will move an existing /usr2/dhorsley to /usr2/dhorsley.FSCOPY and create a new /usr2/dhorsley with a useful skeleton for use with the FS (you will be prompted for the account name):

    /usr2/fs/misc/auid_update
    Note

    If the user’s home directory is not on /usr2, but is for example on /home, the following commands should be used instead:

    cd /home
    mv dhorsley dhorsley.FSCOPY
    cd /usr2/fs/st.default/auid
    find . -print|cpio -pmdu /home/dhorsley
    chown -R dhorsley.dhorsley /home/dhorsley

Adjust number of login failures before lock-out

The number of login failures before lock-out can cause a problem if it is set too low. The main issue is for an operator working at odd hours, alone, at a remote location, who is dealng with multiple issue, which might include: power failures, equipment problems, and logistical issues. It can be a chaotic situation. Typing long and complicated passwors in the heat of battle, particularly if they vary between machines, can be error-prone. Being locked-out will make the situation more difficult and increase the amount of data that will be lost.

If you find the number of login failures before lock-out too small, you can increase it by increasing the value of the deny parameter (5 in the example below, other typical parameters are omitted and should not be changed) in:

/etc/pam.d/common-auth
auth required pam_tally2.so deny=5

Small integer values (20 or less) should not be a signficant risk with long and complicated passwords and a lock-out of several minutes.

Setting hostname alias

These steps set a more user friendly alias for the computers of the form fs1-<xx> and fs2-<xx> where <xx> is the station’s two letter code. This provides a compact alias for local usage, even for sites with more than one system, and makes the system identifiable for remote users in a systematic way. These steps should be executed for both the operational and spare computers.

  1. Edit /etc/hosts and add the new aliases to the appropriate lines.

    If you have two computers, add the aliases for both to the file on each computer.

  2. Create a file /etc/hostname_alias that contains the new alias.

    1. Execute

      cd /etc
      cp hostname hostname_alias
    2. Edit the new file and change the contents to the new alias.

  3. Change the system’s mailname

    Note
    To allow mail to mailman mail lists to work, you may need to make a use a fake FQDN name, perhaps by appending .net to your alias, for use in /etc/mailname and /etc/exim4/update-exim4.conf.conf. The two files should be consistent.
    1. Edit the file /etc/mailname and change its contents to the new name, without a domain name unless that is required by remote mail hosts or mail lists. If so, Generate FQDN in HELO for outgoing mail in the FSL10 Installation document may also be helpful.

    2. Edit /etc/exim4/update-exim4.conf.conf, change the value of hostnames= to the new alias

    3. Execute

      update-exim4.conf
      systemctl restart exim4
  4. For the spare computer only:

    1. Update /usr/local/sbin/refresh_spare_usr2 to use the new alias of the operational computer in the ssh line.

    2. You will need to update the new alias for the operational computer to be recognized as a known host to the root account on the spare computer. You can do that by using ssh to spare@operational where operational is new alias for the operational computer. The command will give you guidance for which lines need to be deleted in /root/.ssh/known_hosts. After deleting those lines, reconnect using the same ssh command and answer yes to confirm connecting. Finally, logout of the spare account on operational computer.

  5. Use the new alias in the user prompts and xterm titles for oper, prog, and all non-system-administrator AUID accounts. In the .bashrc file for each user to be changed:

    1. Before the if block that sets PS1 add:

      hostalias_file=/etc/hostname_alias
      if [[ -f "$hostalias_file" ]]; then
          hostalias=$(cat $hostalias_file)
      else
          hostalias=$(hostname)
      fi
    2. In the two statements setting PS1 in the if block, change the use of \h to $hostalias.

    3. In the statement setting PS1 in the case block that sets the xterm window title, change the use of \h to $hostalias.

Installing refresh_spare_usr2 with CIS hardening

Note
This subsection follows the FS manual font conventions.
Tip
Read the refresh_spare_usr2 section of the RAID Notes for FSL10 document for important information on the refresh_spare_usr2 script.

This subsection describes the steps needed to enable use of the refresh_spare_usr2 script with CIS hardening. All steps must be performed as root on the specified system. You should read all of each step or sub-step before following it.

  1. On the operational system:

    1. Create spare account. Execute:

      addgroup spare --gid 2000
      adduser spare --uid 2000 --gid 2000
      Note
      The user’s home directory is on /home (by default), not /usr2.
      Note
      The UID and GID values of 2000 are chosen to make it easier to keep the values in sync for users on both systems. In other words, it is not necessary to worry about jumping over a low value on the spare system when values are assigned sequentially, as is the default. If you think you might have more than 1000 users or groups, you might want to increase the values for the spare account and group.
  2. On the spare system.

    1. Install refresh_spare_usr2. Execute:

      cd /usr/local/sbin
      cp -a /root/fsl10/RAID/refresh_spare_usr2 refresh_spare_usr2
      chown root.root refresh_spare_usr2
      chmod a+r,u+wx,go-wx refresh_spare_usr2
    2. Customize refresh_spare_usr2, following the directions in the comments in the script:

      1. Comment-out the lines (add leading #s):

        echo "This script must be customized before use.  See script for details."
        exit 1
      2. Change the operational in the line:

        remote_node=operational

        to the alias (preferred), FQDN, or IP address of your operational system.

      3. Uncomment the line for CIS hardened systems. The commented out form is:

        #remote_user=spare
    3. Create and copy a key for root. Execute:

      ssh-keygen
      ssh-copy-id spare@operational

      where operational is the alias, name, or IP of your operational system.

      Note
      If root already has a key, you only need the second command above to copy it to the spare account:
      Tip
      It is recommended to not set a passphrase.
    4. Enable running the script with sudo. Use visudo to add:

      %operators         ALL=(ALL) /usr/local/sbin/refresh_spare_usr2
      Note
      It could be setup for a specific user (but not oper or prog in a CIS hardened system) by replacing %operators with the user account name.
  3. On the operational system:

    1. Install the rrsync script. Execute:

      gunzip -c /usr/share/doc/rsync/scripts/rrsync.gz >/usr/local/bin/rrsync
      ln -s /usr/local/bin/rrsync /usr/bin/rrsync
      chmod u+x,go-x /usr/local/bin/rrsync
    2. Set the spare account to only allow a forced command with ssh by replacing the ssh-rsa at the start of the first (and only) line of ~spare/.ssh/authorized_keys line with:

      command="sudo --preserve-env rrsync -ro /usr2" ssh-rsa

      Tip
      If your spare system is registered with DNS, you can provide some additional security by adding from="node"  (note the trailing space) at the start of the line, where node is the FQDN or IP address of the spare system. It may be necessary to provide the FQDN, IP address, and/or alias of the spare system in a comma separated list in place of node to get reliable operation.
    3. Enable the spare account to run rrsync with sudo without a password and passing environment variables. Use visudo to add:

      spare          ALL=(ALL) NOPASSWD:SETENV: /usr/bin/rrsync
    4. Lock-out the spare account from normal login (but it must have a shell). This will disable password login, but not ssh login with keys, for this account. Execute:

      usermod -L spare
    5. Disable password aging and inactivity time-out for the spare account. Execute:

      chage -I -1 -M 99999 spare