Hardening RedHat Linux with Bastille

Securely installing a bastion host

By Seán Boran

This article presents a concise step-by-step approach to securely installing RedHat Linux for use in a firewall DMZ, or other sensitive environment, using Bastille. Linux has progressed rapidly and can be configured to be as secure as, if not better, than commercial UNIX.

The focus in this article is on RedHat 6.2 on SPARC + x86, RedHat7 on x86 and Mandrake 7.0 on x86.

DRAFT: Includes bastille 1.1.1

We welcome your feedback on this article.

Table of Contents:

1. Preparation
2. Initial OS installation
3. Install SSH
4. Bastille: Introduction, Running, Checking, Problems.
5. Installing tools & sysadmin software
6. Patches and Logging
7. Integrity Checking and backup
8. Install, test, harden applications.
9. Going Live

Regular maintenance
Additional Notes
References
Changes to this article

1. Preparation

• Keep things simple: it is expected that only one or two services will run on a host. Use several machines, rather than one superserver that does everything. It's easier to isolate applications, harden, troubleshoot and upgrade hw/sw. Be minimalist, only run what is absolutely necessary.
• Hardware: If you have Sun/SPARC, consider installation via the serial port console, get rid of the keyboard, screen and framebuffer. i.e. avoid using X11 and get to know the command line.
• Downloading securely: Hopefully the installation is on an isolated or non-routed private network, in this case you can ftp to the new box (as root), or ftp from the new box (via the console) to a server on this net. If you don't have an isolated network (not advisable), then change the root password to something easy, download the files and change the password again, shutdown the network interface immediately. This reduces the window of opportunity for potential attackers.
• Know exactly what the system is supposed to do, what it's hardware configuration will be etc. hardening is generic and may break certain applications. e.g. those needing RPC.
• It's important to understand how the applications work (how they use ports, devices, files), to judge what hardening is possible and to assess the risk posed.

2. Initial OS installation

On x86 hardware, screen, keyboard and mouse are needed. Boot from CDROM or the boot floppy and choose install.

On SPARC hardware, the entire installation can be done without screen or keyboard (also called a "headless server"). Connect the serial console, switch on, halt to the OK prompt by sending a Stop-A (~#, ~%b, or F5 depending on whether you use tip, cu or a vt100 terminal), then start the installation procedure:  boot cdrom - install.

The RedHat install is pretty buggy and options differ between releases:

• Ideally the "custom install" would be used to install only necessary modules, but it's buggy, e.g. on v6.1 the ftp server is not installed (for SPARC, fix this with rpm -i /mnt/cdrom/RedHat/RPMS/wu-ftpd-2.6.0-1.sparc.rpm).
• Gnome is always installed, even if you choose KDE and exclude GNOME. Fix: use startx to start GNOME and kdm to start KDE.
• On all v6.2 installation methods, the International keyboard maps are not set correctly for the commandline or GUI.

The Mandrake 7.0 install does not suffer from these shortcomings.

So, choose a server or custom install, set hostname, IP parameters, timezone, etc. Don't enable any naming services like NIS or NFS. Choose manual disk partitioning:

• Consider a separate, large /var filesystem for syslog/web/news/proxy servers or firewall filters.
• Servers containing lots of data (web, ftp) should use a separate disk for their data.
• If you don't wish to mount partitions read-only and there are no logs or application data, then consider putting the whole boot disk under root.
• Suggestion for a 2GB disk: 1500MB / (root+usr), 200MB swap,  300MB /var.
• Suggestion for a 1GB disk:  700MB / (root+usr), 100MB swap,  200MB /var.
• Suggestion for a log server with 8GB: 100MB / root, 300MB swap,  800MB /usr, 6.9GB /var.
• See also [9] for a more detailed discussion of disk partitioning.

Set a strong password (At least 8 chars with numbers, letters and punctuation) for root. Create an additional test user, as you won't be able to login over the network as root.

The "init level" should be set to 3 (command line login), rather than 5 (graphical login). If a GUI is needed, it can always be started manually with startx.

To login via the 'serial port A' on x86 Hardware, which is useful for troubleshooting, installations and getting to know the command line (it is not necessary for headless SPARCs which do this automatically). Add the following to /etc/inittab.
con:23:respawn:/sbin/getty ttyS0 VC

To allow root to login via this serial port, add ttyS0 to /etc/securetty,
echo "ttyS0" >> /etc/securetty

3. Install SSH

Install SSH, the secure Shell for login access (Bastille can do this, but you need Internet access, I had problems with SPARC, and prefer to know exactly what options are used to compile SSH). SSH is already included in some distributions, such as Redhat7.

There are two key implementations for Linux 'ssh1' and 'OpenSSH', here we use ssh1 as an example. OpenSSH is more interesting in some ways, but ssh1 also supports securid (which is useful to me). See also [7] for a detailed discussion of SSH and it's various implementations.

Either download sources or RPMs (see sites listed under [7]):

1) Sources: 
zcat ssh-1.2.30.tar.gz | tar xf - 
cd ssh-1.2.30; ./configure --prefix=/usr --without-none --without-rsh --without-idea
make && make install

2) RPMs (SPARC example shown):
rpm -i ssh-1.2.30-7i.sparc.rpm        ssh-clients-1.2.27-7i.sparc.rpm
rpm -i ssh-extras-1.2.30-7i.sparc.rpm ssh-server-1.2.27-7i.sparc.rpm

Copy a startup file (example sshd) to /etc/rc.d/init.d/sshd and setup links, unless it was done as part of the previous step.
chkconfig --add sshd

Configure an appropriate /etc/sshd_config file (see also [7]), so that access is restricted to named hosts with known public keys (/etc/ssh_known_hosts) and rhosts authentication is disabled. Avoid trusts. Only allow specific users and hosts to access SSH.

Deny daemon accounts access, for example:
DenyUsers daemon bin sync adm lp shutdown halt mail news uucp nobody operator sympa, squid, postgres, gopher, postfix, xfs.

4. Bastille

Introduction

Bastille is set of open source scripts designed to harden a virgin Red Hat 6.0 or 6.1 installation (6.2 support is planned soon). The first release was in December 1999 and significant progress has been made. V1.1 (released June 2000) was used here. What does Bastille do? Unneeded daemons are stopped, logging enabled/improved, SUID and file permissions tightened, account security improved and even a chroot environment is provided for DNS servers.
Note that Bastille does not run on other Linux variants such as SuSE (which have their own mechanisms and different startup files).

An automated and interactive interface is available. The text based menus provided by the interactive install are very useful for explaining the different options involved and generate a configuration file (tui-generated-raw-config) which is used in the next step by the Bastille back end to do the actual hardening. The configuration file can be edited or copied to other machines to speed up hardening.

Running Bastille

1. Download Bastille and extract into /root.
2. Shutdown the network interface during this next phase, just in case (the interface name may vary):

ifconfig eth0 down

3. Run the Bastille interactive script:
   cd /root/Bastille; ./InteractiveBastille.pl

   Note: On SPARC the Interactive script won't work:
   Can't load './Curses.so' for module Curses: ./Curses.so: ELF file data encoding not big-endian at /usr/lib/perl5/5.00503/sparc-linux/DynaLoader.pm line 169.
   Fix: Download Curses-1.02.tar.gz from CPAN and install:
   perl Makefile.PL; make && make install
   Then change to the Bastille dir & remove the i386 Curses library:
   cd /root/run-Bastille; mv Curses.* /tmp;
   And run InteractiveBastille.pl again.

   InteractiveBastille.pl runs through hardening setup on a step-by-step basis, asking the user what should or should not be tightened down. Unneeded daemons are stopped, logging enabled, SUID and file permissions tightened, account security improved and even a chroot environment is provided for DNS servers.
   Default answers except for the following were used (see tui-generated-raw-config):

• File permissions: set more restrictive permissions on the administration utilities
• SUID: disable all tools except ping and traceroute. 
• 2nd UID 0 account: no.
• Disable compiler (only root access)
• Specify remote logging host
• Daemons: disable Sendmail, run Sendmail via cron,  disable Apache, enable CGI, disable indices, disable printing, disable anonymous ftp download.
• RPC can be disabled, even if you need to run KDE.
  
  Review the settings you chose, by viewing config in an editor such as view.
4. Start the actual hardening process:
   ./BackEnd.pl < config > screen.log

Bug: the first time BackEnd fails with the following message:
# ./BackEnd.pl < config > screen.log
/bin/cp: /etc/banners: omitting directory

Run it again and it works fine!

Checking the results

1. Review the log of the hardening process: /root/Bastille/screen.log and /root/bastille-action-log
2. Reboot.
3. Login as root and check the process list, it should be something like:

   tests# ps -ef
   UID PID PPID C STIME TTY TIME CMD
   root 1 0 10 17:07 ? 00:00:03 init [3]
   root 2 1 0 17:07 ? 00:00:00 [kflushd]
   root 3 1 0 17:07 ? 00:00:00 [kupdate]
   root 4 1 0 17:07 ? 00:00:00 [kpiod]
   root 5 1 0 17:07 ? 00:00:00 [kswapd]
   root 6 1 0 17:08 ? 00:00:00 [mdrecoveryd]
   root 276 1 1 17:08 ? 00:00:00 syslogd -m 0 -a /home/dns/dev/lo
   root 286 1 0 17:08 ? 00:00:00 klogd
   root 301 1 0 17:08 ? 00:00:00 crond
   root 310 1 3 17:08 ? 00:00:00 /usr/sbin/sshd
   root 369 1 0 17:08 ttyS0 00:00:00 login -- root
   root 370 1 0 17:08 tty1 00:00:00 /sbin/mingetty tty1
   root 371 1 0 17:08 tty2 00:00:00 /sbin/mingetty tty2
   root 372 1 0 17:08 tty3 00:00:00 /sbin/mingetty tty3
   root 373 1 0 17:08 tty4 00:00:00 /sbin/mingetty tty4
   root 374 1 0 17:08 tty5 00:00:00 /sbin/mingetty tty5
   root 375 1 0 17:08 tty6 00:00:00 /sbin/mingetty tty6
   root 378 369 1 17:08 ttyS0 00:00:00 -bash
   root 390 378 0 17:08 ttyS0 00:00:00 ps -ef

4. Check the network connections, for example:

# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
raw 0 0 *:icmp *:* 7
raw 0 0 *:tcp *:* 7
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 3 [ ] DGRAM 239 /dev/log
unix 0 [ ] STREAM CONNECTED 108 @0000000f
unix 0 [ ] DGRAM 241 /home/dns/dev/log
unix 0 [ ] DGRAM 353
unix 0 [ ] DGRAM 282
unix 0 [ ] DGRAM 254

Bastille Problems/limitations

The Bastille Linux hardening script is a community consensus project: it attempts to integrate existing "best practices" documents and the shared knowledge of many administrators.
The Bastille team are to be congratulated on their good work, but a few problems have been noticed. It has been indicated that the following problems should be addressed in future releases, together with support for RH6.2.

• It must run on a virgin system, it cannot be undone or re-done.
• Bastille won't allow you to quit with Ctrl-C and the window size must be big enough to show the menus correctly.
• The default menu point after "MiscellaneousDaemons.pm Module 10 of 16" is "back" instead of next. Fix: manually select "next".
• Insufficient hardening:
  • Unneeded accounts like uucp, games, lp, operator, xfs are not removed. Nor are their shells set to /dev/null or /bin/false or /bin/nosuchshell.
  • The xfs (X font server) is still running. While it doesn't have major security weaknesses, some could be discovered, some day (Bugtraq are actively discussing xfs at the moment).
    Fix: disable it and reboot:  chkconfig --del xfs
    Important: You will no longer be able to use KDE or X11 on the console, but if it's just occasionally needed, manually start the font server before you need it:
    /etc/rc.d/init.d/xfs start
  • More system accounts could be added to /etc/ftpusers, such as: sympa, squid, postgres, gopher, postfix, xfs.
  • If ssh is already installed, change sshd_config (DenyUser) so that login from the same accounts as /etc/ftpusers is forbidden.
• Change the Root description (GECOS) from "root"  to "root MACHINENAME" in /etc/passwd (useful for large installations).
• There is no chroot user ftp, which would be useful to isolate user's from each other and the system files (wu-ftp does chroot for anonymous ftp). A chroot for apache would also be useful.

5. Installing tools & sysadmin software

At this stage standard tools/utilities are going to be installed. These tools should already have been compiled and tested extensively on another machine. They are typically transferred as tar files, by CD or FTP.  Some of the following settings can also be configured via linuxconf.

• Environment: /.cshrc /.profile /.bashrc /etc/profile /etc/bashrc: set aliases, variables (such as VISUAL, EDITOR and PATH don't include ".") for your favourite shell. Set umask to 077, or 027.
• Disk mounting: To reduce the risk of trojan horses and unauthorised modifications, in /etc/vfstab, mount /var and other data disks with "nosuid".
• Configure /etc/hosts with a list of critical machines (which you don't want resolved via DNS).
• DNS client (avoid if not needed): add domain name & DNS servers to /etc/resolv.conf. Add a DNS entry for "hosts" in /etc/nsswitch.conf (and remove nis and nisplus entries).
• Keyboard security: If your hosts are in secured rooms, it might be desirable to disable certain key functions such as the following. On the other hand this can be an inconvenience if the hosts are already physically secure.
  • On SPARC: enable/disable the STOP-A keyboard sequence to jump to the eprom prompt in /etc/sysconfig/init.
  • To disable hotkey interactive startup, set PROMPT=no in /etc/sysconfig/init.
  • On x86: To allow ctrl-alt-delete to shutdown the system, an entry in /etc/inittab like the following is used:
    # Trap CTRL-ALT-DELETE
    ca::ctrlaltdel:/sbin/shutdown -t3 -r now
    Comment it out and reboot or "killall -HUP init" to activate the change.
• Use default routes: add the IP address of the router to /etc/sysconfig/network.
• In /etc/inetd.conf, all services have been disabled: reopen very specific services only if absolutely needed, by adapting /etc/hosts.allow and /etc/hosts.deny.
• Email: If hosts are not supposed to send email outside the subnet, don't configure the mailhost alias. Delete /usr/lib/sendmail if you don't need any kind of email. Otherwise:
  • edit /etc/mail/aliases (at least point root to a real address)
  • set mailhost in /etc/hosts and add a hostname.YOURDOMAIN.COM alias for this machine.
  • in /etc/mail/sendmail.cf set the following to ensure all outgoing email is channeled over mailhost:
       Dj$w.YOURDOMAIN.COM.               [only needed if sendmail complains]
       DSmailhost                                     [smart relay]
       DRmailhost                                     [send emails with no host/domain here]
       DHmailhost                                     [skip this if local delivery is allowed]
       O FallbackMXhost=mailhost
  • Send a test email to check the config:
       mail -v -s test_email root </dev/null
• If a sensitive host is to be administered by several people, consider using a tool such as sudo. See also [9].
• If user accounts will be allowed on the system, consider restricting access to:
  • cron: via /etc/cron.allow and cron.deny (see also [9]).
  • at: via /etc/al.allow and at.deny
  • ftp: Disallowed users are listed in /etc/ftpusers (Bastille puts some system accounts in to start with)
  • ssh: /etc/sshd_config (look for AllowUsers DenyUsers AllowHosts DenyHosts entries)
  • General inetd services: /etc/hosts.allow and hosts.deny
  • File system groups: /etc/groups and use file and directly permissions accordingly.

6. Patches and Logging

Patches

Patch management is still very basic in Linux, there is no unique numbering, or dedicated tools for automated checking of patch levels (like Solaris for example), and no "patch bundles" to enable you to quickly get you system unto a current level.

Redhat 6.1, patches can be found at www.redhat.com/errata. Security advisories, bug fixes and package updates are on offer. Mandrake updates are at www.linux-mandrake.com/en/fupdates.php3, but Mandrake also offers a nice Patch checking and installation tool under KDE, that is worth trying out.
Only select patches that are relevant to tools/applications actually in use on your system.
Once you have downloaded the relevant patches apply them:

rpm - Uvh filename.rpm

There are also specific kernel patches to add security features such as strong crypto to the kernel, see [9].

Configure logging:

Syslog logging: The additional syslog configuration setup by Bastille is useful, but only on hosts with local logging or loghosts (syslog servers).

Syslog "clients": It is suggested that most hosts log to central loghost, with an /etc/syslog.conf such as the following:

# syslog.conf for clients
#
# send all messages to "loghost":
*.* @loghost
*.emerg *
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# Log all logins to /var/log/loginlog
auth.*;user.*;daemon.none /var/log/loginlog

# Log additional data to the Alt-F7 and Alt-F8 screens (TTY 7 and 8)
*.info;mail.none;authpriv.none /dev/tty7
authpriv.* /dev/tty7
*.warn;*.err /dev/tty7
kern.* /dev/tty7
mail.* /dev/tty8
###########

Syslog "loghost"

Give the loghost a whopping great /var disk for logs.
Use the default syslog.conf, with the additions from Bastille.
Set up log pruning as described below.

Root cron entries:

• Cron is much better on Linux that Unix. /etc/crontab contains a list of standard scripts that are run (as root) hourly, daily, weekly and monthly.
• logrotate is called daily and is configured in /etc/logrotate.conf. It is used to prune and archive system and application logs. Applications are supposed to an an appropriate script to /etc/logrotate.d, by default there are entries for syslog, ftp, apache, cron and linuxconf.  This is a really nice feature of Linux, much better than commercial Unix.
  It is suggested that the following be changed from their defaults in /etc/logrotate.conf for better log control in a production environment:

  rotate 30       [keep 30 weeks of logs - make sure you have disk space!]
  compress      [compress old logs]
  
  And set 'rotate 24' in the /var/log/wtmp and /var/log/lastlog sections, so that records or logins for the last two years are kept.

• Set date once a day with a reliable source using rdate (consider also NTP, it's more accurate, but complex, uses bandwidth and is an additional security worry):

## Synchronise the time:
0,15,30,45 * * * * /usr/bin/rdate timehost1 >/dev/null 2>&1

# process the mail queue hourly during the week:
0 * * * 1-5 /usr/lib/sendmail -q

File permissions: tighten more permissions, and restrict certain tools to root or disable:

chmod u-s /usr/sbin/sendmail       #Not for mailgateways or multi-user hosts
chmod 400 /.shosts
chmod 444 /etc/sshd_config /etc/ssh_known_hosts

Documentation:

Document configuration changes in a text file such as /etc/mods, update after each change, with date, author, files affected, description.
cat > /etc/mods <<EOF
19.12.00  sb  New install of RH6.1+Bastille & tools according to hardening guidelines
EOF

Banners.

Change login banners to warn users about unauthorised access - you'll need this if you want to prosecute intruders. Edit /etc/issue (for pre-login) and /etc/motd (for post-login). Bastille does not change /etc/issue, one might might want to remove indications about the Operating System type and add a banner (especially when the console is not in a secured room).

Reboot.

Test - Do SSH and the standard tools work? Check log entries, check console messages. Does the system behave as expected?

7.Integrity Checking and backup

At this stage, we need to install a file integrity checker that uses secure hashing algorithms, initialise it's database and run regular checks to monitor for changes. If possible keep the master database on another machine or offline or on write-once media. Even if you can't run regular checks, take a master copy and store in on a floppy, it will be a great help in detecting what has changed, should the system be penetrated at a later stage.

What options do we have for integrity checking?

• The RPM commands also be used to report changes to rpm installed files "poor man's tripwire":
  for file in $(rpm -qa); do rpm -V $file; done > rpm_changes
  It prints out a list, indicating whether Size,  MD5 hash, Links, Time (mtime), Device, User, Group, Mode (permissions) changed:
  S.5....T c /etc/host.conf
  S.5....T c /etc/hosts.allow
  S.5....T c /etc/motd
  S.5....T c /etc/securetty
  S.5....T c /etc/services
  S.5....T c /etc/localtime
  S.5....T c /etc/nsswitch.conf
  .M...... /usr/sbin/rpcinfo
  Unfortunately, the list contained 156 entries after hardening a RH6.1 SPARC system! You can store a copy off line (e.g. floppy) and use it for regular comparison.
• Tripwire [5]: There are both free and commercial versions. Red Hat x86 is the only Linux for which the Commercial version is officially available (an included in RH7)
  Linux is officially supported on RedHat 5.2 and 6.0. Other distributed versions of Linux are not officially supported, but basic functionality has been verified on RedHat 6.1, and various distributions of Debian, Caldera, Open Linux, and SuSE systems using Linux kernel 2.0.36 or higher.
  • The free version can be tricky to get working correctly and has a few bugs. Source code is provided. It crashes on very large disks (for me).
  • The commercial version is a bit pricey (for non Linux users), reports are too verbose (you may need filter scripts), more configuration examples should be provided. It is more stable than the free version, also runs on UNIX and NT and offers enhanced security by cryptographic signing of policy and configuration files. Support (even when paid for) is not great.
  • Neither version supports the use of regular expressions when defining policy rules. e.g. you can't specify a rule for "/home/*/www/cgi-bin" files, that would work even when new directories are added under /home.
  • Apparently, Tripwire will be released as OpenSource for Linux near then end of 2000. That would be useful.
• PGP can also be used, by signing files to be protected (creating lots of signature files), then writing a script to check the validity of signatures. This will not catch permission, link, inode or modify date changes though.
• MD5 signatures could also be used in a similar way, but the list of MD5 signatures should not be stored on the system being monitored, unless it is PGP signed or encrypted.
• mtree is the tool used on OpenBSD, perhaps it will be ported to Linux ?
• Aide is a new GPL tripwire replacement [8] that looks interesting, I've not had a chance to test it so far.

An example using the free Tripwire Version 1.2:

• An rpm is available for x86, the source rpm has to be compiled for SPARC. Download [5] and install the rpms.
• The tripwire binary is in /usr/sbin, configuration in /etc/tw.config, and it is checked each day from cron via /etc/cron.daily/tripwire.verify. Databases are kept in /var/spool/tripwire. See also the man pages: tripwire(8), twconvert(8), tw.config(5).
• After installation, the "initial state" of the system needs to be saved:
  /usr/sbin/tripwire -initialise
  This will create a new database in /var/spool/tripwire.
• To tell Tripwire that (change to) a file or entire directory tree is OK:
  /usr/sbin/tripwire -update [path]
• The checking can be run each day from cron, or manually via:
  /etc/cron.daily/tripwire.verify
• Run tripwire with the "-i 2" option to increase speed (it disabled one checking algorithm, snerfu, but SHA1 and MDS 5 are still used).
• For increased security and automated checking of several systems from one trusted host, copy tripwire & it's database and run it remotely at regular intervals using SSH. Delete the tripwire database on the target after checking. This makes it difficult for an attacker to know that tripwire is being used to check the system. In addition, immediately update the tripwire database, so that only differences are reported by successive runs. See the sample script for doing exactly that trip_linux.sh. If this method is used, then local tripwire checking can be disabled:
  mv /etc/cron.daily/tripwire.verify /etc/cron.daily/.tripwire.verify
• Regularly copy the configuration and database to floppy disk or write-one media, in a crisis it will be very useful.

8. Install, test, harden applications

Depending on the function of the server, applications such as ftpd, BIND, proxies, etc. are installed at this point.

Hardening of specific applications like ftp, DNS, Email and also general application tips are discussed in a separate document [16].

9. Going Live

Preparing to go live

Consider installing a script to check that important daemons are running. Install monitor_processes.pl and add a root cron entry:
## Check that important processes are running during office hours:
## [If you run 7x24, modify accordingly]
0,30 8-19 * * 1-5 /secure/monitor_processes.pl inetd sshd httpd

If data partitions had to be mounted read-write during the application install/testing, consider mounting them read-only now.

Reinitialise tripwire (or equivalent integrity checker).

Backup the system, again to two tapes, one offsite.

Run a network scan on the system, to ensure that only expected services are visible. A commercial tool such as ISS or a free one like Nessus, nmap or Satan should do the job. Print out the results and archive.

If possible, have additional people do the final testing, just in case something was forgotten. Test in detail - What works? What is forbidden? Check console/log entries. Does the system behave as expected? Watch the logs very frequently during the first few days of production.

Going Live

Test in detail. Check log entries. Does the system behave as expected?

Have applications been tested in detail, by different people with different points of view, from different access points on the network?

Regular maintenance

The following activities should take place hourly, daily, weekly or monthly, depending on how critical the system is:

• Check the status of patches, update as needed. Be very wary of kernel patches (test on a non-production machine). Only install patches for software actually used. Since most daemons are disabled on bastion hosts and users don't have accounts, few patches are typically needed.
• Check all logs for errors and unusual activity: /var/log/* and application logs.
• Write scripts to report if critical daemons die, or if important systems cannot be pinged.
• Run tripwire (or equivalent integrity checker).
• Be regularly informed of new vulnerabilities and security issues, either by subscribing directly to CERT, CIAC and the vendor security lists and/or, subscribing to SecurityPortal's weekly letter or the SANS weekly/monthly letters.

Additional Notes

This article has been very specific, in the interest of making it practical. However, each security administrator has his own methods and each site has different requirements.

• Red hat provides a facility for automatic installs. Create a text file with specifications for the install, and point the Red Hat installer at it and go. See also www.redhat.com/mirrors/LDP/HOWTO/KickStart-HOWTO.html .
• Mandrake: Mandrake is a Red Hat compatible Linux distribution. It is interesting because:
  • It's installer is much better. A nice GUI that works for simple to expert users. It can also install crypto tools, if you have a (proxied) ftp connection to the Internet.
  • The root password must be at least 8 chars long.
  • It includes the MSEC package (Mandrake Security) which allows you to specify a security level for the system. MSEC is described in chapter 7 of [2]. However, MSEC has few funnies:

    To change security levels, run /etc/security/msec/init.sh X, where X is the security level 0-5. The default is 3. When converting to level 4 and rebooting, the system asked for a runlevel number and on entering '3' it just sat there after an error saying no runlevel files found. All other console ttys were disabled.

  • MSEC: init.sh can also be run with the "custom" argument, whereby a serious of questions are asked as to what security items you want switched on or off (much like Bastille). An example of the questions and answers given is in msec.txt. Note that it will kill root logon via the serial line, if you need this, re-add ttyS0 to /etc/securetty.
  • V7.0 was tested for this article, only on x86.
  • Hardware: Sun SuperSPARCs are not supported, but an UltraSPARC version is currently in beta. Likewise support for Compaq Alpha processor is also in Beta.
  • If a package seems to be missing, those available can be listed in /mnt/cdrom/Mandrake/RPMS
    and manually install with rpm -i.
  • A few nice GUI tools are provided, drakxconf in a front end to each. Examples are:
    drakxservices list active services and allow them to be started/stopped
    adduserdrake does what it says.
    diskdrake is an excellent disk partitioning GUI.
    draksec allows selection of Low/Medium/high security levels (it uses msec as the backend).
  • I would recommend Mandrake over RedHat at the moment, unless you need the commercial support etc. of RedHat.
  • See also [1] and [2].
• The command-line tool chkconfig can be used to enable/disable services. To list what services are switched on or off:
  chkconfig --list
  To (for example) switch on the Apache web server:
  chkconfig httpd on
  To stop everything except SSH on a virgin RedHat7 server:

  chkconfig httpd off
  chkconfig apmd off
  chkconfig atd off
  chkconfig xfs off
  chkconfig pcmcia off
  chkconfig lpd off
  chkconfig nfs off
  chkconfig gpm off
  chkconfig linuxconf off
  chkconfig identd off
  chkconfig portmap off

  chkconfig sendmail off

  chkconfig xinetd off

   

• Versioning: Use RCS or CVS to manage versions of configuration files. This is especially important when several administrators manage a host, changes are frequent, etc. Versioning should also be used with Jumpstart scripts & config files. An alternative is to make a copy of each config file BEFORE making changes. Append a date to the file, e.g. /etc/syslog.conf.19991023.
• Time Synchronisation with xntpd: If you think you might have to produce evidence in court, use NTP to synchronise time, not rdate. Prosecutions have failed because, on busy servers, a few seconds difference could be used to insist that there is a doubt about the identity of a session. If using NTP, either get a (cheap) radio clock (if you are near an appropriate transmitter - e.g. the Stuttgart transmitter in Germany), or setup a dedicated bastion host as a 2nd stratum NTP server, which uses 3 1st stratum sources. Setup the firewall filter to allow only NTP from the bastion to the three 1st stratums and allow Intranet hosts to query the bastion.
  NTP can also be setup for higher security by configuring DES+MD5 authentication keys on server and client. See www.eecis.udel.edu/~ntp and [9].
• Logging:
  • In addition to centralised syslogs, you might like to keep an additional local copy - in case the syslog server goes down or is subjected to a denial of service attack. Make sure /var is a separate filesystem if you use local logging, to avoid root being filled and the system stopping.
  • There are several improved syslog daemons:
    syslog-ng: www.balabit.hu/products/syslog-ng (tcp connections, content filtering, encryption, authentication)
    secure syslog: www.core-sdi.com/english/slogging/ssyslog.html
    Nsyslogd: coombs.anu.edu.au/~avalon/nsyslog.html (tcp connections & SSL)
• Intrusion detection:
  • See also Integrity Checking section.
  • Regular logfile analysis can be implemented with customer scripts or tools such as logcheck and swatch [6].
  • I have written a perl script monitor_socket.pl that listens to a list of sockets and notifies by email and syslog if a connection is received. Like klaxon above, but does not run from inetd. Originally written to detect Sybase and Satan connection attempts.
  • Snort is a great tool for network-based Intrusion Detection, but does have some limitations.
    
• It's a good idea to keep previous versions of configuration files, to allow rollback and have an audit trail. This is called versioning.  It is especially important when several administrators manage a host, changes are frequent, etc. Versioning can also be used with Jumpstart scripts & config files.  Some sysadmins use RCS (included in Yassp), some use CVS, some copy the configuration file to file.DATE before hand, some do nothing :).
  
  Francisco Mancardi [fman@uyr.com.ar] has written a simple script that can be used to save a copy of a file or directory before doing modifications.
  Saveit is a little tool to make a backup of config files before you change them. It saves a copy under /Backup.d/DATE/ and logs "who saved what file" in /Backup.d/log-DATE. The existing directory structure is preserved under the backup directory.
  It's a simple, but useful version control tool for text files. e.g.

  etc/[9]% saveit vfstab
  copying file vfstab ===> /Backup.d/20000707/etc/vfstab

  etc/[61]% saveit vfstab
  copying file vfstab ===> /Backup.d/20000707/etc/vfstab.13:37

  etc/[10]% ls -l /Backup.d/20000707/etc/vfs*
  -rw-r--r-- 1 root sys 386 Mar 14 08:31 /Backup.d/20000707/etc/vfstab
  -rw-r--r-- 1 root sys 386 Mar 14 08:31 /Backup.d/20000707/etc/vfstab.13:37
  
  ./ ../ vfstab
  etc/[11]% tail /Backup.d/log-20000707
  ----------------------------------
  Backup base directory /Backup.d
  Backup requested by root
  Date (dd/mm/aaaa) 07-07-2000
  Time 11:36
  /etc/vfstab
  --------------------------------------------------------------
  Backup base directory /Backup.d
  Backup requested by root
  Date (dd/mm/aaaa) 07-07-2000
  Time 13:37
  /etc/vfstab

  The advantage of this tool are: simplicity and no clogging up the current directory with old versions of files, rcs directories etc. The script can be downloaded [3].

References

Other links:

• Crypto Archive: www.cryptoarchive.net
  Search engine: www.cryptoarchive.net/cgi-bin/file-search.cgi
• Security Portal Research Centre
• Free Tools
  Network weaknesses scanners: Nessus, Satan, Nmap.
  Intrusion Detection: snort
• SPARC Linux tools
  ftp://sunsite.unc.edu/pub/linux/distributions/redhat/contrib/libc6/sparc
• Interesting new tools: Subdomain

Changes to this article

25.Apr.'00  Improve tripwire, ssh, references, Mandrake. Spelling. Feedback from Kurt+Jay.
29.Apr.'00  Initial Publication
28.Jun.'00  Add links to new reports [10], Correct links[9] and snort.
09.Aug.'00 New: saveit, Update: Bastille 1.1, Links.
19.Dec.'00 Update for RH7

Last Update: 11 December, 2001

Seán Boran is an IT security consultant based in Switzerland and the author of the online IT Security Cookbook.

By default after 20 reboot, the filesystems are checked (fsck). If you have several filesystems, it's better to stagger the checking so that all filesystems are not checked at the same time, avoiding a slow "21 reboot".

Each filesystem has a mount count which can be interrogated as follows:
dumpe2fs /dev/hda1 | grep 'Mount count'

Set this to a staggered number between 1 and 19, for example:
tunefs -C 6 /dev/hda1
tunefs -C 12 /dev/hda4
tunefs -C 18 /dev/hda7

Warning: umount filesystems before running tunefs.

The number of reboots between check can also be changed:
tunefs -c 10 [check every 10 reboots]
tunefs -i 5 [check every 5 days]