By Seán Boran (sean at boran.com) for SecurityPortal
July 06, 2001 - The Solaris Security Digest came to life in May 2000, and each week we have tried to include a useful tip. This document collects those tips in FAQ style for the first six months of 2001.
The previous 6 months, May-Dec 2000 are also available:
http://securityportal.com/articles/solaristips20001220.html
Just browse the table of contents to find items that interest you.
1. "ph" script:
Francisco Mancardi from U&R Consultores [fman@uyr.com.ar] is contributing a script called ph (Put Header), to create a standard header for various types of files (configuration files, readme files, cc, c++, scripts..) with a certain standard fields (customer name, hostname, full pathname, who is adding the header).
The idea is to create a standard headed containing important information for new files, for better documentation. As he says himself: "OK, maybe I have an obsession with the documentation, but I think is very useful." :)
It can be downloaded from www.boran.com/security/sp/solaris/ph.1.1.tar
2. "chk_disk" script
This is a script of my own, that I run from cron to report (via email) if any local filesystems have reached 97% or more.
It can be downloaded from www.boran.com/security/sp/solaris/chk_disk
3. The loopback file system (lofs) can be used to make a directory outside a chroot environment available within the chroot area. Lofs can also be used instead of soft links.
4. As regards headless PCs, a product exists called "PC Weasel" that allows a "GUI console" can be managed remotely via a serial null modem cable, network terminal server, etc. Apparently it's a favourite with Linux administrators. Interesting indeed. The Middle Digital Inc. website ( http://www.realweasel.com/ ) says:
"It's an 8-bit ISA board that emulates the original IBM MDA (Monochrome Display Adapter) character-based video board and the PC keyboard. Plugged into an 8-bit or 16-bit ISA slot, it takes the characters written by your CPU into its "video" memory and pumps them out its onboard RS-232 port. Characters input by you into the RS-232 port are converted into keyboard scan codes and presented to the motherboard's keyboard connector.
The pricing for our board is US $250 plus shipping (per). We only sell direct, and accept Visa, Mastercard and company PO's. Update Oct.2002: There's now a PCI board and it's $350USD
The PC Weasel distinguishes itself even further by being an open-source product. Every purchaser receives a source license for the Weasel's onboard microcontroller code. If you don't like some aspect of the board's behaviour as shipped by us, you're free to modify it using a gcc-based toolchain. The code store is flash memory that can be written without special equipment, and there's a second serial port provided for debugging."
5. If you are compiling ISC's Bind 8.2.3 or 9.1,
make sure GNUs make is installed first, otherwise installation into temporary trees will
not work, e.g.
make install DESTDIR=/tmp
6. Disksuite: An example on mirroring using Disksuite For Solaris Sparc by Robert
Banniza
http://www.rootprompt.net/disksuite_mirror.html
7. fmthard: If you wish to partition a second disk identically to a first, the fmthard tool together with prtvtoc is much faster than manually using format. For example if the master is target 3 and the second disk is target 1, and we wish to give it the disk label 'mirror', then:
/usr/sbin/prtvtoc /dev/rdsk/c0t3d0s2 | /usr/sbin/fmthard -n mirror -s - /dev/rdsk/c0t1d0s2
Garry J. Garrett writes in with another useful tip:
It is useful to know when a system shutdown or is started, especially when you have many hosts to manage and logs are not automatically monitored, or you don't have active SNMP monitoring.
Garry puts a startup file in /etc/rc3.d that kicks out e-mail when a system is stopped or booted. The message for the shutdown is different than the startup message, so if a startup happens without a shutdown, it probably crashed and it will have to be checked out.
It is installed in /etc/init.d/bootmail and make links /etc/rc3.d/S99bootmail and /etc/rc0.d/K00bootmail. Instead of "root" you can make an alias (put it in /etc/mail/aliases and run "newaliases") and e-mail it to a list of folks who care, which may include more than just the SysAdmins (besides, not everyone likes the idea of forwarding root's e-mail off of the box - you can send it to root *and* to an e-mail address that goes off of the box). Obviously, sendmail (the client, not the server) must be working properly to send e-mail off of the box (this usually boils down to defining "mailhost" say in /etc/hosts or DNS, etc.).
#!/bin/sh ##########
# bootmail
# # Send mail to SysAdmins upon reboot so that they are aware should:
# - someone else reboot the machine
# - the machine crashes
# etc.
# 27-Jan-1999 Garry J. Garrettcase "$1" in
'start' | 'boot' | 'reboot') /bin/echo "`/bin/uname -n` rebooted `date`" \ | /bin/mailx -s "`/bin/uname -n` rebooted" root ;;
'stop' | 'shutdown' | 'down') /bin/echo "`/bin/uname -n` going down `date`" \ | /bin/mailx -s "`/bin/uname -n` going down" root ;;
*) /bin/echo "Usage: /etc/init.d/bootmail { start | stop }" ;; esac
You can also download it from www.boran.com/security/sp/solaris/bootmail
Another reader tip this week!
SymbEl, by Richard Petit is a tool that helps spot performance problems and is useful for diagnostics. SymbEl comes with several terminal based and GUI monitoring tools (uses TCL/TK). It's not officially "supported" by Sun, but they let you download it "as is". A few quotes:
SymbEL (known as SE) is an interpreted language that provides an extensive toolkit for building performance tools and utilities. If you are fed up with the limitations of vmstat, iostat and sar, then this is the tool for you. We provide trivial scripts that are improved versions of the basic utilities and build on them to provide powerful rule-based performance monitors and viewers. The extensions package includes a Motif-based GUI library and the rules package implements Adrian's favourite performance rules.
July 18th 2000, a patch is now available to make the SE3.1preFCS packages install and run on Solaris 8 with some limitations.
The three packages provided are:
RICHPse, The SymbEL Interpreter
RICHPsex, The SE eXtensions Package
ANCrules, Adrian's Rules & Tools
Unfortunately I've not had a chance to test drive it, but it sounds interesting. There seems to be little development activity currently.
See also http://www.sun.com/sun-on-net/performance/se3
How do you recognise a compromised system? The appearance of kernel kits for hiding penetrations make life much more difficult. Lets examine one kernel trojan, and consider some countermeasures.
Solaris Integrated Trojan Facility 0.2
Plasmoid/THC
http://www.infowar.co.uk/thc
http://securityfocus.com/templates/tools.html?id=1006This a publicly released Solaris Loadable Kernel Module backdoor from The Hacker's Choice.
The 'sitf0.2" module features:
- File, directory and process hiding.
The default hidden expression is 'blah', when tested with files/directories it didn't always work if several entries in the current directory contained 'blah'. Interestingly, 'find' didn't catch the same entries as 'ls'. All processes containing 'blah' were perfectly hidden.- Converting magic uid to root uid (default is uid 1001).
Tests worked perfectly. a user with uid 1001 has complete root access.- Execution redirecting (default: /usr/openwin/bin/xview/xcalc is executed instead of /bin/who).
Tests worked perfectly: xcalc was executed instead of who.- Promiscuous flag hiding: I don't see how this should work since "ifconfig -a" doesn't show the Promiscuous flag anyway?
- A switch to toggle file content and directory hiding (default 'touch mykey'). Tests: this didn't work for me, I always got the message 'touch: mykey cannot create'.
The 'anm' module is really nasty: "This is probably the most stupid module I ever programmed, instead of faking syscalls or installing backdoors, this module just corrupts a system, making it slightly unusable by randomly generating different system errors." I didn't try testing this module.
Analysis:
On a Solaris 2.7 server, the following 89 (!) standard kernel modules were loaded. This shows how difficult it is to recognise trojans.
# modinfo
Id Loadaddr Size Info Rev Module Name
6 10104000 4577 1 1 specfs (filesystem for specfs)
8 10109774 2de8 1 1 TS (time sharing sched class)
9 1010bf1c 4f0 - 1 TS_DPTBL (Time sharing dispatch table)
10 1010bf70 27818 2 1 ufs (filesystem for ufs)
11 10130224 ec4c 226 1 rpcmod (RPC syscall)
11 10130224 ec4c 1 1 rpcmod (rpc interface str mod)
12 1013d920 28d74 0 1 ip (IP Streams module)
12 1013d920 28d74 3 1 ip (IP Streams device)
13 1015fe90 15e0 1 1 rootnex (sun4u root nexus)
14 1016105c 1ec 57 1 options (options driver)
15 10161180 79c 62 1 dma (Direct Memory Access driver)
16 101616e0 75cf 59 1 sbus (SBus (sysio) nexus driver)
17 10167f18 1648 12 1 sad (Streams Administrative driver's)
18 101692a8 61f 2 1 pseudo (nexus driver for 'pseudo')
19 10169728 10e4c 32 1 sd (SCSI Disk Driver 1.300)
20 10179294 7136 - 1 scsi (SCSI Bus Utility Routines)
21 1017de84 d719 61 1 esp (ESP SCSI HBA Driver v1.264)
26 101a435c 15c3 - 1 dada ( ATA Bus Utility Routines)
27 101a53e8 886 - 1 todmostek (tod module for Mostek M48T59)
28 1018a45c 128c2 5 1 procfs (filesystem for proc)
30 101b0db4 ccec 8 1 sockfs (filesystem for sockfs)
32 1019cce8 616 11 1 clone (Clone Pseudodriver 'clone')
33 101bd0a8 17a04 2 1 tcp (TCP Streams module)
33 101bd0a8 17a04 42 1 tcp (TCP Streams device)
34 1019d0d4 1055 - 1 md5 (MD5 Message-Digest Algorithm)
35 1019e030 45e0 3 1 udp (UDP Streams module)
35 1019e030 45e0 41 1 udp (UDP Streams device)
36 101a1610 3b58 4 1 icmp (ICMP Streams module)
36 101a1610 3b58 5 1 icmp (ICMP Streams device)
37 101a5b70 51a7 5 1 arp (ARP Streams module)
37 101a5b70 51a7 44 1 arp (ARP Streams driver)
38 101a9f0c 45b7 6 1 timod (transport interface str mod)
40 101cff64 8a7f 29 1 zs (Z8530 serial driver V4.120)
41 101aece8 1800 7 1 ms (streams module for mouse)
42 101b0250 a1c 17 1 consms (Mouse Driver for Sun 'consms')
43 101d82a4 3ece 8 1 kb (streams module for keyboard)
44 101dae8c b55 16 1 conskbd (Console kbd Multiplexer driver )
45 101db684 1955 15 1 wc (Workstation multiplexer Driver )
46 101dc3ec 234f 0 1 elfexec (exec module for elf)
47 101de42c 104d 13 1 mm (memory driver)
48 101df1e8 3288 3 1 fifofs (filesystem for fifo)
49 101e1fe0 5926 9 1 ldterm (terminal line discipline)
50 101e6e90 2381 10 1 ttcompat (alt ioctl calls)
51 101e9024 14d0 26 1 ptsl (tty pseudo driver slave 'ptsl')
52 101ea15c 2053 25 1 ptc (tty pseudo driver control 'ptc')
58 101ed034 4683 105 1 tl (TPI Local Transport Driver - tl)
59 101f11b0 160a 53 1 sysmsg (System message redirection (fan)
60 101f1fdc 6d8 0 1 cn (Console redirection driver)
61 101f24b4 4c5 1 1 intpexec (exec mod for interp)
62 101a409c 2fc 42 1 pipe (pipe(2) syscall)
63 101f6494 b88e 7 1 hme (FEPS Ethernet Driver v1.114 )
65 102004d0 726a - 1 ufs_log (Logging UFS Module)
66 101f28c4 d70 12 1 fdfs (filesystem for fd)
67 101f3374 7f6 90 1 kstat (kernel statistics driver)
68 10206ec0 d8a2 11 1 tmpfs (filesystem for tmpfs)
69 101f39ac 9db 21 1 log (streams log driver)
70 1020bf08 3e12 201 1 doorfs (doors)
71 101f40ac 8c3 22 1 sy (Indirect driver for tty 'sy')
72 101f475c 875 12 1 pfmod (streams packet filter module)
73 101f4e14 1423 13 1 bufmod (streams buffer mod)
74 1020f7bc 1488 4 1 namefs (filesystem for namefs)
75 10210954 5018 91 1 vol (Volume Management Driver, 1.85)
76 1021546c b01d 36 1 fd (Floppy Driver v1.102)
77 1021f750 25c80 106 1 nfs (NFS syscall, client, and common)
77 1021f750 25c80 15 1 nfs (network filesystem)
77 1021f750 25c80 7 1 nfs (network filesystem version 2)
77 1021f750 25c80 16 1 nfs (network filesystem version 3)
78 10241a3c 92a3 - 1 rpcsec (kernel RPC security module.)
79 102486b0 1c19 - 1 tlimod (KTLI misc module)
80 1024a094 2290 53 1 semsys (System V semaphore facility)
81 101f5fc4 2d8 - 1 ipc (common ipc code)
82 1024c104 21b8 52 1 shmsys (System V shared memory)
83 1019bd54 f0f 23 1 ptm (Master streams driver 'ptm')
84 101ebe88 e53 24 1 pts (Slave Stream Pseudo Terminal dr)
85 101ad694 163b 14 1 ptem (pty hardware emulator)
86 10107f44 1934 49 1 msgsys (System V message facility)
87 101eca14 858 72 1 ksyms (kernel symbols driver)
88 1024dfbc 80d 15 1 pckt (pckt module)
89 1024e588 11b1 38 1 openeepr (OPENPROM/NVRAM Driver)A Solaris 8 system had 109 modules loaded.
The system compiled and runs on Solaris 8 too, when loaded, modinfo displayed the trojan like this:
110 fe99d59e ab3 - 1 sitf0.2 (Solaris ITF)
Detecting malevolent kernel modules:
Summary: This tool is really worrying. If any readers can add to the information/detecting measures above I like to hear from you.
Note: An interesting read which proposes lkms as security wrappers for applications is:
"Using Kernel Hypervisors to Secure Applications", by Terrence Mitchem, Raymond Lu and Richard O'Brien www.securecomputing.com/khyper
To illustrate the practicality of the kernel hypervisor concept, we prototyped three client kernel hypervisors: one for wrapping the Netscape browser, one for replicating files, and one for wrapping the Apache Web Server.
BIND, the well known, DNS server has popped up a few serious vulnerabilities (discovered by NAI) which have caused widespread concern. We present here, the original Bulletins, our analysis and point to relevant articles.
Vulnerabilities in BIND 4 and 8
http://archives.neohapsis.com/archives/bugtraq/2001-01/0472.html
The original report from NAI's COVERT labs.
CERT Advisory CA-2001-02 Multiple Vulnerabilities in BIND
http://www.cert.org/advisories/CA-2001-02.htmlThe CERT/CC has recently learned of four vulnerabilities spanning multiple versions of the Internet Software Consortium's (ISC) Berkeley Internet Name Domain (BIND) server. BIND is an implementation of the Domain Name System (DNS) that is maintained by the ISC. Because the majority of name servers in operation today run BIND, these vulnerabilities present a serious threat to the Internet infrastructure.
Domain Name System (DNS) Servers running various versions of ISC BIND (including both 4.9.x prior to 4.9.8 and 8.2.x prior to 8.2.3; 9.x is not affected) and derivatives. Because the normal operation of most services on the Internet depends on the proper operation of DNS servers, other services could be impacted if these vulnerabilities are exploited. The four vulnerabilities are:
- Vulnerability Note VU#196945 - ISC BIND 8 contains buffer overflow in transaction signature (TSIG) handling code http://www.kb.cert.org/vuls/id/196945
- Vulnerability Note VU#572183 - ISC BIND 4 contains buffer overflow in nslookupComplain()
http://www.kb.cert.org/vuls/id/572183- Vulnerability Note VU#868916 - ISC BIND 4 contains input validation error in nslookupComplain() http://www.kb.cert.org/vuls/id/868916
- Vulnerability Note VU#325431 - Queries to ISC BIND servers may disclose environment variables http://www.kb.cert.org/vuls/id/325431
The Internet Software Consortium has posted information about all four vulnerabilities at http://www.isc.org/products/BIND/bind-security.html. Upgrading to BIND version 9.1 is strongly recommended. If that is not possible for your site, upgrading at least to BIND version 8.2.3 is imperative.
Bugtraq database
2001-01-29: ISC Bind 8 Transaction Signatures Buffer Overflow Vulnerability
http://www.securityfocus.com/vdb/bottom.html?vid=23022001-01-29: ISC Bind 8 Transaction Signatures Heap Overflow Vulnerability
http://www.securityfocus.com/vdb/bottom.html?vid=23042001-01-29: ISC Bind 4 nslookupComplain() Buffer Overflow Vulnerability
http://www.securityfocus.com/vdb/bottom.html?vid=23072001-01-29: ISC Bind 4 nslookupComplain() Format String Vulnerability
http://www.securityfocus.com/vdb/bottom.html?vid=2309
BIND holes mean big trouble
Kevin Poulsen
http://www.securityfocus.com/news/144
ISC wants to limit access to BIND Security advisories to a closed group of 3rd parties, to avoid the reduce the window of exposure between an announcement being released, attackers automating attacks, vendors releasing patches and sysadmins installing the patches. http://www.securityfocus.com/frames/?content=/templates/archive.pike%3Fmid%3D159741%26threads%3D0%26end%3D2001-02-03%26fromthread%3D0%26list%3D1%26start%3D2001-01-28%26
- It is recommended to upgrade your critical Internet DNS servers soon, DNS is just too important.
- Exploit code for the TSIG weakness was announced, but then it turned out that the exploit was in fact a trojan that attacks dns1.nai.com:
http://archives.neohapsis.com/archives/bugtraq/2001-01/0517.html- Sun has not yet released patches (you're better off running ISC's bind anyway), some Linux vendors already had fixes available on Tuesday.
- How could you defend against these bugs in advance? Chroot'ing helps, as does hiding the BIND version, running as a non-root user, using a dedicate machine and monitoring logs. Some people don't believe in hiding the version number, as it is "security by obscurity", but I maintain that it at least helps against the script kiddies who are roaming the net looking for obvious targets. Defending against the pros is a different matter.
- The fix is to point upgrade 8.2.3, or the newer 9.1. ISC recommend going to 9.1.
- Upgrading to 8.2.3 is easy enough, one more library is needed to be added to the chroot jail on a Solaris 8 primary. See also the article "Hardening the BIND DNS Server", http://securityportal.com/cover/coverstory20001002.html
- Personally, I've been reluctant to go to 9.1, since it's a complete rewrite and one would expect bugs in such a scenario. BIND v9 is now several months old however, and does not suffer from these new weaknesses (which is a good sign). So this may be the time (and excuse) to tackle v9.1.
"Underground: Hacking, madness and obsession on the electronic
frontier"
By Suelette Dreyfus with Research by Julian Assange
ISBN 1 86330 595 5
http://www.underground-book.com
This book was published in 1997 in paper form. It's now been released free for download as a text file. It is a fascinating read, an excellent documentation of the hacker scene of the late 80s/early 90s. Well worth reading - in fact, save your eyes some strain and buy the paper copy. :)
Points that stand out for me are the sheer determination and genius of some hackers, and the futility of Security by Obscurity for sensitive systems - VMS systems, and X.25 financial networks have been penetrated, without the hackers having access to decent determination).
Two minor BIND tips this week, which popped up after migrating to 8.2.3:
named[9708]: tcp_send: bind(query_source): Permission denied
The problem arises because a query fails as a UDP packet (e.g. due to heave server or network load) and must be retried using TCP. The TCP connection is re-tried from a privileged source port (the one specified in the log file), which is not allowed since BIND is not running as root (or shouldn't be anyway). Hopefully ISC will fix this in the next release. In the meantime, if this happens to you alot, a patch posted by Mark Andrews of Nominum makes the TCP connection using a random non-privileged source port.
http://www.isc.org/ml-archives/bind-users/2001/02/msg01433.html
named[6251]: [ID 295310 daemon.notice] denied update from ....
It may not be an attack, or even an incorrectly configured DNS secondary, but simple a Windows200 machine. I've seen this several times, the exact reasons for Windows2000 trying to update itself into DNS are not clear.
I'd not read the CERT documents on security in a long time and was pleasantly
surprised, they are looking good and worth a visit:
http://www.cert.org/security-improvement/index.html#unix
On a more comprehensive level, the english translation of the IT Baseline Protection Manual (itbpm) is available on http://www.bsi.bund.de/english.
You need to compile some tools on a Sun, but the Sun has been already hardened. What packages need to be added to get a compilation environment?
The following Solaris packages need to be installed: SUNWbtool, SUNWsprot, SUNWtoo, SUNWhea, SUNWarc, SUNWlibm, SUNWlibms. The pkginfo tool can be used to check if these packages are installed, an error will be produced if the package is missing
% pkginfo SUNWbtool SUNWsprot SUNWtoo SUNWhea SUNWarc SUNWlibm SUNWlibms
system SUNWarc Archive Libraries
system SUNWbtool CCS tools bundled with SunOS
system SUNWhea SunOS Header Files
system SUNWlibm Sun WorkShop Bundled libm
system SUNWlibms Sun WorkShop Bundled shared libm
system SUNWsprot Solaris Bundled tools
system SUNWtoo Programming Tools
The missing packages can be added with pkgadd from the Solaris CD.
Other tips:
Further reading:
The 'script' tool can be used to log all activity in an interactive login session.
For example, to run a few commands and have them appended to /var/tmp/script.log:
% script -a /var/tmp/script.log
Script started, file is /var/tmp/script.log# pwd
/secure
# df -k
Filesystem kbytes used avail capacity Mounted on
/proc 0 0 0 0% /proc
/dev/dsk/c0t2d0s0 2054233 1261038 731569 64% /
swap 204800 8 204792 1% /tmp# exit
Script done, file is /var/tmp/script.log
Notes
/bin/script -a /var/tmp/script.$$.log; logout;
.login should only be writeable by root. This is not completely-water tight, but if in parallel you do a 'tail -f ' on the script, you can follow what the vendor is doing and watch for attempts to manipulate the script file.
When the user logs in, a message "Script started, file is /var/tmp/script.log" will appear, I've not found a way of suppressing this, but it's probably good that the user is aware of the logging.
The strip command is great for reducing the size of binary files.
As the man page for strip(1) says:
The strip command removes the symbol table, debugging information, and line number information from ELF object files. Once this stripping process has been done, no symbolic debugging access will be available for that file; therefore, this command is normally run only on production modules that have been debugged and tested.
Why is this useful?
OK, how do we use strip?
Strip is in /usr/ccs/bin, if it's not there you need to install the SUNWbtool package.
Using the BIND 9 distribution as an example, I noticed that the program binaries are very big:
% cd bin
% ls -l
total 52762
drwxr-x--- 2 boran other 512 Mar 8 16:37 ./
drwx------ 6 boran other 512 Mar 29 11:32 ../
-rwxr-x--- 1 boran other 6755108 Mar 8 16:37 dig
-rwxr-x--- 1 boran other 6758872 Mar 8 16:37 host
-rwxr-x--- 1 boran other 2489 Mar 8 16:37 isc-config.sh
-rwxr-x--- 1 boran other 6750772 Mar 8 16:37 nslookup
-rwxr-x--- 1 boran other 6689948 Mar 8 16:37 nsupdateSo lets strip these files and check the new size:
% /usr/ccs/bin/strip *
/usr/ccs/bin/strip: isc-config.sh: invalid file type% ls -al
total 10202
drwxr-x--- 2 boran other 512 Mar 8 16:37 ./
drwx------ 6 boran other 512 Mar 29 11:32 ../
-rwxr-x--- 1 boran other 1301616 Mar 29 11:35 dig
-rwxr-x--- 1 boran other 1295640 Mar 29 11:35 host
-rwxr-x--- 1 boran other 2489 Mar 8 16:37 isc-config.sh
-rwxr-x--- 1 boran other 1297936 Mar 29 11:35 nslookup
-rwxr-x--- 1 boran other 1275160 Mar 29 11:35 nsupdateStrip complains about isc-config.sh, which is fair enough, it's a shell script not an ELF binary and so it cannot be stripped. We've saved 20MB on these four files alone!
Going back to the Bind example above, by stripping all binaries and also deleting the include directory (which is not needed on a production server), the uncompressed distribution was reduced from 90MB to 25MB. Compression with gzip further reduced this to a manageable 8MB tarball.
The "pwck" can be used to check the consistency of /etc/passwd entries.
pwck scans the password file and notes any inconsistencies. The checks include validation of the number of fields, login name, user ID, group ID, and whether the login directory and the program-to-use-as-shell exist. The default password file is /etc/passwd.
Likewise grpck:
grpck verifies all entries in the group file. This verification includes a check of the number of fields, group name, group ID, whether any login names belong to more than NGROUPS_MAX groups and that all login names appear in the password file. The default group file is /etc/group.
Examples
%/usr/sbin/pwck uucp:x:5:5:uucp Admin:/usr/lib/uucp: Login directory not found nuucp:x:9:9:uucp Admin:/var/spool/uucppublic:/usr/lib/uucp/uucico Login directory not found Optional shell file not found
Comment: The uucp packages were removed from this system, hence the lack of uucp home directories..
These tools can be very useful in detecting errors on servers with large numbers of users, or constant changes.
Don't forget to check for accounts with empty passwords now and again as well:
awk -F: '{if ($2=="") print $1}' /etc/shadow
Preamble: 3 more Solaris SUID weaknesses this week. More worries, more work. We discuss how to disable a maximum of SUID files, to reduce the risk posed by these rogue SUID buffer overflows, even before vulnerabilities are announced..
Files which have the SUID bit set (an "s" where the execute bit for the owner/group is shown in 'ls' listings) allow the user executing the program to assume the identity/group of the owner of the program. This is typically used to allow normal users access to certain functions only allowed to root, for example binding to low ports, mounting a floppy disk, etc. The problem is that historically, many security weakness have been found in such programs (for example 3 this week alone!) allowing attackers with local accounts to become root by exploiting buffer over flows, race conditions etc.
What SUID files are on the system?
The find command can be used to list all SUID files:
> find / -perm -u+s -ls
or all SGID files:
> find / -perm -g+s -ls
They are also listed in the package database /var/adm/install.
How should we handle SUID files? Possible courses of action, in order of preference, are:
What SUID files need to be limited?
After appling the above commands on a Solaris 7 or 8 "user bundle" install,
the list of SUIDs left is reduced to the following:
find / -type f \( -perm -u+s -o -perm -g+s \) -ls
SUID files:
/usr/lib/pt_chmod /usr/lib/utmp_update /usr/bin/login /usr/bin/passwd /usr/bin/pfexec
/usr/bin/su /usr/sbin/ping /opt/local/bin/ssh
SGID files:
/usr/bin/mail /usr/bin/mailx
Note: You'll also find that /usr/bin/yppasswd /usr/bin/nispasswd are SUID, but they
are links to /usr/bin/passwd, so removing the SUID bits will stop normal users from
changing their local passwords!
Some auditing ideas:
When formatting disks make sure you don't overlap cylinders, or you'll end up with messages like the following in the logs:
WARNING: ufs_readir: bad dir, inumber = 332288, fs = /
What do overlapping cylinders look like?
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 884 2.00GB (885/0/0) 4194900
1 swap wm 884 - 1186 701.28MB (303/0/0) 1436220
2 backup wu 0 - 7498 16.95GB (7499/0/0) 35545260
6 var wm 1186 - 7498 14.27GB (6313/0/0) 29923620
The following is what should be used:
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 884 2.00GB (885/0/0) 4194900
1 swap wm 885 - 1185 696.65MB (301/0/0) 1426740
2 backup wu 0 - 7498 16.95GB (7499/0/0) 35545260
6 var wm 1186 - 7498 14.27GB (6313/0/0) 29923620
This kind of mistake is easy to do where you're tired/distracted .... and Solaris doesn't catch it (which seems strange).
Sun's BigAdmin has a list of useful command-line examples:
http://www.sun.com/bigadmin/shellme/index.html;$sessionid$0OHCCVQAAAY3BAMTA1LU4GQI especially like the one for listing files by size of a particular partition, e.g. /var:
du -ad /var |sort -n
Another way of doing this is to use find, e.g. list files greater that 1MB older than 7 days on the current filesystem:
find . -xdev -mtime -7 -size +1000 -ls
I've written a little script that you may find useful:
Backup key configuration files of several machines, remotely via SSH, into a compressed tarball.
Configuration files of many tools are automatically recognised. OpenBSD, Solaris and Redhat have been tested. Backups are maintained for as long as needed, manual purging of the backup directory is required.
Download
The script:
www.boran.com/security/sp/solaris/backup_configsAn example email log of the output generated by the script:
www.boran.com/security/sp/solaris/backup_configs.log
When working with Solaris 8 on PCs, IDE drives greater than 40 GB need patch 110202-01 for the full capacity to be visible (large scsi drives work fine).
The Netra X1 is a neat server that is cheap (starts at $999.- in the US), very compact (smaller than a pizza box) and includes the Lights Out Management (LOM) prom-like software for remote power down/up. A pity the fan is so noisy. Anyway, I don't want to sell you any X1s, but discuss some problems when you want o reinstall it from scratch yourself. These problems may well apply to other recent Sun hardware.
There is no CD reader and no scsi bus (only IDE), so booting from an internal or external CD or disk is not an option.
Which leaves remote booting/Jumpstart. The Jumpstart server must make Solaris 8 10/00 or later available. I used Solaris 8 01/01, which was the latest available for download last week (although Version 04/01 was "released" the week previously).
However, special drivers need to be patched onto the Install Server's Solaris directory (with the modify_install_server script) so that appropriate drivers are made available to the X1. This is strange, given that the X1 was release with Solaris 6 10/00 and one would have expected the new drivers to be rolled in Solaris 8 01/01. Perhaps they are in Solaris 8 04/01? The file containing the drivers is mis.netra-x1.259-3836-02.zip and can be downloaded from SunSolve.
There is a similar problem with Sun Blade 100s:
Sun Alert ID: 25969
Synopsis: Installing or Re-installing Solaris 8 10/00 or 01/01 on Sun Blade 100-based Platforms Requires CD0 or Patched Netinstall
http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=salert%2F25969
I had to setup a automated Solaris install (Jumpstart) server this week and though I pass on some links and tips on the subject. Jumpstart is a very useful feature that saves time and can be used to ensure consistent, completely, high-quality installation. Although it's relatively simply, it can be difficult to debug and run satisfactorily.
We won't run through setting up Jumpstart, there are several resources already available, for example:
Hands-off Jumpstart using "sysidcfg" with no Nameservice
http://sunsolve.sun.com/pub-cgi/show.pl?target=content/doc16484
How to copy a directory from server to client via a Jumpstart finish script http://sunsolve.sun.com/pub-cgi/show.pl?target=content/doc23323
Solaris 8 Installation Collection >> Solaris 8 Advanced
Installation Guide
http://docs.sun.com/ab2/coll.214.7/SPARCINSTALL/@Ab2TocView?Ab2Lang=C&Ab2Enc=iso-8859-1&DwebQuery=jumpstart&oqt=jumpstart
Adding a Kernel Patch to a Jumpstart Installation Boot Image
http://sunsolve.sun.com/pub-cgi/show.pl?target=content/doc21571
jumpstart: How to install two or more different Solaris install images?
http://sunsolve.sun.com/pub-cgi/show.pl?target=content/doc19282
Solaris Jumpstart Automated Installation
http://www.amorin.org/professional/jumpstart.php
Configuring Solaris Jumpstart Without a Name Service
http://www.cuug.ab.ca/~leblancj/labs/jumpstart.html
The Jumpstart Rules and Class Files
http://www.bu.edu/dsgsupport/sun/jumpstart/rules.html
Creating Finish Scripts
http://sgi-sw.rz.ruhr-uni-bochum.de/advanced_install_documentation/files/c0702.htm
Using Jumpstart to install/upgrade systems over the network
http://www.netwiz.net/~varmav/tips-tools/jumpstart/
Troubleshooting tips - during the initial booting from the network;
Rarp and bootparam daemons can be run in debug mode on the boot server:
/usr/sbin/in.rarpd -d -a
/usr/sbin/rpc.bootparamd -dIf you make an changes to /etc/bootparams, kill the daemon and restart:
/usr/sbin/rpc.bootparamd
in.tftpd doesn't have any debug option unfortunately, but you can enable inetd logging by adding '-t' to it's startup line and watching for connections
/usr/sbin/inetd -st
snoop can be used to show relevant network traffic:
snoop rarp or port tftp
To record exactly what happens during a "hands-off" install, and to help with troubleshooting (for example when the interactive installation programs starts unexpectedly), I attach the serial console to another Sun, and use Mindterm SSH to connect to the Sun (and use tip to access the console). Why Mindterm? Well, it has a "capture to file" option which can create a nice log of all activity in the Mindterm window (i.e. in this the case the output from the Jumpstart Installation).
After a Jumpstart installation, the logs in /var/sadm/system/logs should be checked, especially the "finish.log"
The install image contains the default patches on the Solaris CD. It makes senses to update this to the latest "Security & Recommended" bundle regularly, so that newly installed machines are up to date. For example if the install image is Solaris 8 in the directory /space/sparc_8:
# move existing patches
cd /space/sparc_8/Solaris_8/Patches;
mkdir .oldpatches.`date +%yy%mm%dd`
mv * .oldpatches.`date +%yy%mm%dd`
# get latest recommended bundle
cd /space/sparc_8/Solaris_8/Patches;
ftp sunsolve.sun.com
# extract bundle and clean up
unzip -q 8_Recommended.zip
mv 8_Recommended/* .
rmdir 8_Recommended
rm 8_Recommended.zip CLUSTER_README copyright install_cluster patch_order
Some example Jumpstart files that you may find useful:
www.boran.com/security/sp/solaris/any_machine
www.boran.com/security/sp/solaris/finish
www.boran.com/security/sp/solaris/rules
www.boran.com/security/sp/solaris/sysidcfg
Sun has just announced a very interesting service:
"The Solaris Fingerprint Database (sfpDB) is a free SunSolve Online service that enables users to verify the integrity of files distributed with the Solaris Operating Environment. Examples of these files include the /bin/su executable file, Solaris patches, and unbundled products such as Sun Forte Developer Tools. The list of checksums, generated for a system, must be updated after the system is modified by patch installation and software installations. The issue with these tools has always been verifying that the files used to generate the baseline checksums are correct and current."
...... "Our goal is to provide a comprehensive collection of digital fingerprint
for Solaris software. To this end, the Solaris Fingerprint Database is updated daily, and
it now contains close to 1 million digital fingerprints for files used in the Solaris
Operating Environment, Solaris patches, and unbundled products.
Limitations: Currently, foreign language versions of the Solaris Operating Environment and
many encryption products are not supported. If you would like to suggest a product to be
added to sfpDB, please send email to fingerprints@sun.com."
http://www.sun.com/blueprints/0501/Fingerprint.html
http://www.sun.com/blueprints/0501/Fingerprint.pd
Alex Noordergraaf, Lou Ordorica
I like to use tripwire to verify file integrity on critical Solaris systems, but tripwire does have it's limits. For instance if you've never run tripwire on a system, there is obviously little tripwire can do to detect changes. The Sun method however, can detect changes on a system that has never had an integrity checking tool installed! Nice.
To try it out, run md5 on a Solaris binary of your choice, and paste it into the appropriate field on http://sunsolve.sun.com/pub-cgi/fileFingerprints.pl. If you paste more than one entry, make sure they are cleanly separate by newlines. For example, pasting:
MD5(listdgrp)= ee06584223290be70843fe0b807ef1f2
MD5(listusers)= 8e055aebc9825e743374334dd29290d4
results in:
ee06584223290be70843fe0b807ef1f2 - (listdgrp) - 1 match(es)
canonical-path: /usr/bin/listdgrp
package: SUNWcsu
version: 11.8.0,REV=2000.01.08.18.12
architecture: sparc
source: Solaris 8/SPARC
8e055aebc9825e743374334dd29290d4 - (listusers) - 1 match(es)
canonical-path: /usr/bin/listusers
package: SUNWcsu
version: 11.8.0,REV=2000.01.08.18.12
architecture: sparc
source: Solaris 8/SPARC
Accompanying tools: Sun provide a few tools to help with automating the fingerprint checking, which can be downloaded from http://www.sun.com/blueprints/tools/fingerprint_license.html .
"The Solaris Fingerprint Database Companion (sfpC) is a tool designed to automate the process of querying the Solaris Fingerprint Database (sfpDB). sfpC is used to process MD5 file signatures and present the collected database output information in human readable form. The tool eliminates the need for the manual task of cut and pasting MD5 output onto an HTML form. In addition, the tool performs the necessary checks to enable files of arbitrary size to be processed using multiple queries if necessary." An example of usage:
% find / -type f -perm -2000 -o -perm -4000 \
-exec /opt/md5/md5-sparc {} \;> md5.list
% spfC.pl md5.list
"SideKick is a (bourne shell) tool developed to automate the collection of MD5 file signatures. SideKick can be used to collect signatures for files known to be replaced by "rootkits", files with Set-UID or Set-GID permissions in addition to several other collection methods. SideKick can optionally be used with sfpC to automated the collection and processing of MD5 file signatures. SideKick can also be used in a standalone capacity for distributed signature collection."
A new notes
Recent experience has taught me once again that system and network hardening is of limited use if users don't behave in a security conscious fashion. This often repeated, but valid adage of using good passwords is as valid today as it was 20 years ago.
A useful article on the subject to re-read is:
Choosing Secure Passwords, by Benjamin D. Thomas, 07/12/2000
http://www.linuxsecurity.com/tips/tip-6.html
A few tips I would add to those listed in the above article:
I recently had to scan a few networks to check for the existance of Sun servers which had been compromised by the above worm. More details on the worm are available on CERT. The approach used and scripts are documented here, as they may be of use to readers auditing their systems.
#!/bin/sh # /root/sean/nmap_port # Sean Boran, 16.Jun.01 port="600"; user="root"; ignore="^Port|filtered|^$|seconds to scan|Initiating" f=$0.$$; cmd="/usr/local/bin/nmap -P0 -sS -p $port"; for target in 176.17.16.0/24 176.17.17.0/24; do date > $f echo "$cmd $target" >> $f 2>&1 $cmd $target |egrep -v "$ignore" >> $f 2>&1 date >> $f mailx -s "Port $port scan $target" $user <$f # empty file for next scan: #cat $f mv $f $f.done done
Checking 193.5.227.50
RPC program 100232 is running!
program 100232 version 10 ready and waiting
#!/bin/sh # sadmin_scan # Sean Boran, 7.Jun.01 result1=$0.$$ for network in 176.17.16 176.17.17; do host=1; while [ "$host" -lt "255" ] ; do echo "Checking $network.$host" rpcinfo -u "$network.$host" 100232 > $result1 2>&1 if [ $? = 0 ] ; then echo "RPC program 100232 is running! " cat $result1 fi host=`expr $host \+ 1;` done done
A copy of the above scripts can be found on: www.boran.com/security/sp/solaris.
An interesting document has been published, which explains how to test your security, how to measure the testing done by your external consultants and finally, understand the approaches an attacker would use. Good read. The description below is quoted from the website. It's open source too, so you can join and contribute your experience too..
The Open-Source Security Testing Methodology Manual
http://www.ideahamster.org/
This manual is to set forth a standard for Internet security testing. Disregarding the credentials of many a security tester and focusing on the how, I present a solution to a problem which exists currently. Regardless of firm size, finance capital, and vendor backing, any network or security expert who meets the outline requirements in this manual is said to have completed a successful security snapshot. Not to say one cannot perform a test faster, more in depth, or of a different flavour. No, the tester following the methodology herein is said to have followed the standard model and therefore if nothing else, has been thorough.
......
I feel it is valid to be able to ask companies if they meet a certain standard. I would be thrilled if they went above the standard. I would also know that the standard is what they charge a certain price for and that I am not just getting a port scan to 10,000 ports and a check of 4,800 vulnerabilities. Especially since most of which only apply to a certain OS or application. I'd like to see vulnerability scanners break down that number by OS and application. I know if I go into Bugtraq (the only true vulnerability checking is research on BT) that I will be able to find all the known vulnerabilities by OS and application. If the scanner checks for 50 Redhat holes in a certain flavour and 5 Microsoft NT holes and I'm an NT shop; I think I may try a different scanner.
So following an open-source, standardised methodology that anyone and everyone can open and dissect and add to and complain about is the most valuable contribution we can make to Internet security. And if you need to know why you should recognise it and admit it exists whether or not you follow it to the letter is because you, your colleagues, and your fellow professionals have helped design it and write it. Supporting an open-source methodology is not a problem of making you equal with all the other security testers-- it's matter of showing you are just as good as all the other security testers. The rest is about firm size, finance capital, and vendor backing.
Jass v0.3 has been released and the associated documents updated. Jass is Sun's answer to Yassp and Titan, and appears to have evolved into a capable, interesting, Solaris hardening tool.
"The Solaris Security Toolkit is a tool designed to assist in creation and deployment of secured Solaris Operating Environment systems. The Toolkit is comprised of a set of scripts and directories implementing the recommendations made in the Sun BluePrints OnLine program.
These scripts can be executed on Solaris systems through the JumpStart technology or directly from the command line. The Toolkit includes scripts to harden, patch, and minimise Solaris Operating Environment systems. Sun does not support the Toolkit."http://www.sun.com/blueprints
http://www.sun.com/blueprints/tools/
http://www.sun.com/blueprints/0601/jass_release_notes-v03.pdf
http://www.sun.com/blueprints/0601/jass_internals-v03.pdf
http://www.sun.com/blueprints/0601/jass_quick_start-v03.pdf
http://www.sun.com/blueprints/0601/jass_conf_install-v03.pdf
Summary of changes since version 0.2 (November 2000):
The license is still quite tight: you can use freely for you own private or corporate, but cannot distribute or publish derivative works.
I hope to do some tests over the coming week and report back in the next Tip of the Week.
JASS stands for JumpStart Architecture and Security Scripts (Toolkit). The primary goal behind the development of this Toolkit was to simplify and automate the process of securing Solaris systems through JumpStart or in a standalone mode. It implements the recommendations Sun's BluePrints security articles
Jass v0.3 was released last week and we gave you a description of what it's all about in 'Tip of the Week', this time we take it for a test-drive and present you with the results.
We start off with an example of running Jass on a new Solaris 8 workstation installed with a 'user bundle'. An example log of the output is [1]
First we install Jass:
pkgadd SUNWjass-0.3.pkg
Then we run the default Jass hardening for standalone use:
/opt/SUNWjass/jass-execute -d secure.driver
On rebooting it was surprising to find many daemons still running:
Although inetd is running, no services are available in inetd.conf.
BSM auditing is enabled. I have mixed feelings about this due to the patches need to make it work, logs it generates, and the problems with root crontabs. It does however create the 'root.au' file. See also [3].
Sendmail is left running is Queue mode (will deliver but not accept remote emails), which is fine.
The following daemons are left running because they're not considered risky. Personally, I prefer to stop every daemon that is not strictly necessary:
root 230 1 0 10:18:21 ? 0:00 /usr/sbin/nscd
root 240 1 0 10:18:22 ? 0:00 /usr/lib/utmpd
root 43 1 0 10:17:59 ? 0:00 /usr/lib/devfsadm/devfseventd
root 45 1 0 10:18:00 ? 0:00 /usr/lib/devfsadm/devfsadmd
Next we try the 'undo' feature which allows us to go back to the configuration before Jass was run. It very nicely asks us which 'Jass run' we would like to undo:
/opt/SUNWjass/jass-execute -u
The undo seems to work fine, except for BSM auditing which is not cleanly removed, see also the undo log [4].
Jass can be run several times, and the undo can remove the effects of each previous run or all runs. Nice.
This time, we customise Jass a bit, for a more real-world experience and let it run:
A patch bundle is copied to /opt/SUNWjass/Patches and the script Finish/install-recommended-patches.fin was adapted so that 'install_cluster' was called with the '-nosave' option. On a new installs, I don't see the point in saving old patches and tying up tens of megabytes of space.
In addition, /opt/SUNWjass/Drivers/user.init was created with some custom settings, to tailor behaviour for this system:
# user.init
# sb, 26.Jun.01
JASS_AGING_MAXWEEKS="26"
JASS_LOGIN_RETRIES="5"
JASS_PASS_LENGTH="6"
JASS_SENDMAIL_MODE="\"\""
JASS_TMPFS_SIZE="100m"
JASS_UMASK="027"Next step was to add a new script to Finish/disable-nscd.fin which disables the nscd daemon (since it's not needed for our test system), containing:
echo "Disabling nscd startup and shutdown scripts"
echo ""
if [ "${JASS_KILL_SCRIPT_DISABLE}" = "1" ]; then
disable_rc_file ${JASS_ROOT_DIR}/etc/rcS.d K40nscd
disable_rc_file ${JASS_ROOT_DIR}/etc/rc0.d K40nscd
disable_rc_file ${JASS_ROOT_DIR}/etc/rc1.d K40nscd
fi
disable_rc_file ${JASS_ROOT_DIR}/etc/rc2.d S76nscdThen we add 'disable-nscd.fin' to the JASS_SCRIPTS section of Drivers/hardening.driver, to activate the above script..
Finally we set Jass running with the usual command:
/opt/SUNWjass/jass-execute -d secure.driver
Jass is an interesting tool, well worth checking out.
[0] 'Tip of the Week' that gave an overview of Jass - last week:
http://securityportal.com/topnews/weekly/solaris20010625.html[1] Log of Jass install and standalone configuration
www.boran.com/security/sp/solaris/jass03_install.txt[2] Log of Jass install and standalone configuration
www.boran.com/security/sp/solaris/jass03_startup.txt[3] Solaris C2/BSM security notes - Sean Boran
www.boran.com/security/sp/Solaris_bsm.html[4] Log of Jass 'undo'
www.boran.com/security/sp/solaris/jass03_undo.txt
Discussion form for Jass feedback:
http://supportforum.sun.com/cgi-bin/WebX.cgi?/security.jass.toolkit
Seán Boran (sean at boran.com) is an IT security consultant based in Switzerland and the author of the online IT Security Cookbook.
Published version: http://securityportal.com/articles/solaristips20010706.html |