Linux Operating System

Linux Administration (3.2)

In this section, you will learn about basic Linux server administration and the Linux file system.

Basic Server Administration (3.2.1)

In this topic, you will learn about Linux server configuration files, hardening Linux servers, and monitoring Linux services.

Service Configuration Files (3.2.1.1)

In Linux, services are managed using configuration files. Common options are port number, location of the hosted resources, and client authorization details. When the service starts, it looks for its configuration files, loads them into memory, and adjusts itself according to the settings in the files. Configuration file modifications often require restarting the service before the changes take effect.

Because services often require superuser privileges to run, service configuration files often require superuser privileges for editing.

Example 3-1 shows a portion of the configuration file for Nginx, a lightweight web server for Linux.

Example 3-1 Nginx Web Server Configuration File

[analyst@secOps ~]$ cat /etc/nginx/nginx.conf

#user html;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                   '$status $body_bytes_sent "$http_referer" '
    #                   '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

<output omitted>

[analyst@secOps ~]$

Example 3-2 shows the configuration file for the Network Time Protocol, NTP.

Example 3-2 NTP Configuration File

[analyst@secOps ~]$ cat ls /etc/ntp.conf
cat: ls: No such file or directory
# Please consider joining the pool:
#
#     http://www.pool.ntp.org/join.html
#
# For additional information see:
# - https://wiki.archlinux.org/index.php/Network_Time_Protocol_daemon
# - http://support.ntp.org/bin/view/Support/GettingStarted
# - the ntp.conf man page

# Associate to Arch's NTP pool
server 0.arch.pool.ntp.org
server 1.arch.pool.ntp.org
server 2.arch.pool.ntp.org
server 3.arch.pool.ntp.org

# By default, the server allows:
# - all queries from the local host
# - only time queries from remote hosts, protected by rate limiting and kod
restrict default kod limited nomodify nopeer noquery notrap
restrict 127.0.0.1
restrict ::1

# Location of drift file
driftfile /var/lib/ntp/ntp.drift
[analyst@secOps ~]$

Example 3-3 shows the configuration file for Snort, a Linux-based intrusion detection system (IDS).

Example 3-3 Snort Configuration File

[analyst@secOps ~]$ cat /etc/snort/snort.conf
#--------------------------------------------------
#   VRT Rule Packages Snort.conf
#
#   For more information visit us at:
#     http://www.snort.org                  Snort Website
#     http://vrt-blog.snort.org/    Sourcefire VRT Blog
#
#     Mailing list Contact:      snort-sigs@lists.sourceforge.net
#     False Positive reports:    fp@sourcefire.com
#     Snort bugs:                bugs@snort.org
#
#     Compatible with Snort Versions:
#     VERSIONS : 2.9.9.0
#
#     Snort build options:
#     OPTIONS : --enable-gre --enable-mpls --enable-targetbased --enable-ppm
 --enable-perfprofiling --enable-zlib --enable-active-response --enable-normalizer
 --enable-reload --enable-react --enable-flexresp3
#
#     Additional information:
#     This configuration file enables active response, to run snort in
#     test mode -T you are required to supply an interface -i <interface>
#     or test mode will fail to fully validate the configuration and
#     exit with a FATAL error
#--------------------------------------------------

###################################################
# This file contains a sample snort configuration.
# You should take the following steps to create your own custom configuration:
#
#  1) Set the network variables.
#  2) Configure the decoder
#  3) Configure the base detection engine
#  4) Configure dynamic loaded libraries
#  5) Configure preprocessors
#  6) Configure output plugins
#  7) Customize your rule set
#  8) Customize preprocessor and decoder rule set
#  9) Customize shared object rule set
###################################################

###################################################
# Step #1: Set the network variables.  For more information, see README.variables
###################################################

# Setup the network addresses you are protecting
###ipvar HOME_NET any
###ipvar HOME_NET [192.168.0.0/24,192.168.1.0/24]
ipvar HOME_NET [209.165.200.224/27]
# Set up the external network addresses. Leave as "any" in most situations
ipvar EXTERNAL_NET any

# List of DNS servers on your network
###ipvar DNS_SERVERS $HOME_NET
ipvar DNS_SERVERS 209.165.200.236

# List of SMTP servers on your network
###ipvar SMTP_SERVERS $HOME_NET
ipvar SMTP_SERVERS 209.165.200.236

# List of web servers on your network
###ipvar HTTP_SERVERS $HOME_NET
ipvar HTTP_SERVERS 209.165.200.235

# List of sql servers on your network
###ipvar SQL_SERVERS $HOME_NET
ipvar SQL_SERVERS 209.165.200.235

# List of telnet servers on your network
###ipvar TELNET_SERVERS $HOME_NET
ipvar TELNET_SERVERS 209.165.200.236

# List of ssh servers on your network
###ipvar SSH_SERVERS $HOME_NET
ipvar SSH_SERVERS 209.165.200.236

<output omitted>
[analyst@secOps ~]$

There is no rule for a configuration file format; it is the choice of the service’s developer. However, the option = value format is often used. In Example 3-3, variable ipvar is configured with several options. The first option, HOME_NET, has the value 209.165.200.224/27.

Hardening Devices (3.2.1.2)

Device hardening involves implementing proven methods of securing the device and protecting its administrative access. Some of these methods involve maintaining passwords, configuring enhanced remote login features, and implementing SSH. Defining administrative roles in terms of access is another important aspect of securing infrastructure devices because not all information technology personnel should have the same level of access to the infrastructure devices.

Depending on the Linux distribution, many services are enabled by default. Some of these features are enabled for historical reasons, but are no longer required. Stopping such services and ensuring they do not automatically start at boot time is another device hardening technique.

OS updates are also extremely important to maintaining a hardened device. New vulnerabilities are discovered every day. OS developers create and issue fixes and patches regularly. An up-to-date computer is less likely to be compromised.

The following list provides a few basic recommended steps for device hardening:

• Ensure physical security.

• Minimize installed packages.

• Disable unused services.

• Use SSH and disable the root account login over SSH.

• Keep the system updated.

• Disable USB auto-detection.

• Enforce strong passwords.

• Force periodic password changes.

• Keep users from reusing old passwords.

• Review logs regularly.

Many other steps exist and are often service- or application-dependent.

Monitoring Service Logs (3.2.1.3)

Log files are the records that a computer stores to keep track of important events. Kernel, services, and applications events are all recorded in log files. It is very important for an administrator to periodically review the logs of a computer to keep it healthy. By monitoring Linux log files, an administrator gains a clear picture of the computer’s performance, security status, and any underlying issues. Log file analysis allows an administrator to guard against upcoming issues before they occur.

In Linux, log files can be categorized as

• Application logs

• Event logs

• Service logs

• System logs

Some logs contain information about daemons that are running in the Linux system. A daemon is a background process that runs without the need for user interaction. For example, the System Security Services Daemon (SSSD) manages remote access and authentication for single sign-on capabilities. The following are a few popular Linux log files and their functions:

• /var/log/messages: This directory contains generic computer activity logs. It is mainly used to store informational and noncritical system messages. In Debian-based computers, the /var/log/syslog directory serves the same purpose.

• /var/log/auth.log: This file stores all authentication-related events in Debian and Ubuntu computers. Anything involving the user authorization mechanism can be found in this file.

• /var/log/secure: This directory is used by Red Hat and CentOS computers instead of /var/log/auth.log. It also tracks sudo logins, SSH logins, and other errors logged by SSSD.

• /var/log/boot.log: This file stores boot-related information and messages logged during the computer startup process.

• /var/log/dmesg: This directory contains kernel ring buffer messages. Information related to hardware devices and their drivers is recorded here. It is very important because, due to their low-level nature, logging systems such as syslog are not running when these events take place and, therefore, are often unavailable to the administrator in real time.

• /var/log/kern.log: This file contains information logged by the kernel.

• /var/log/cron: Cron is a service used to schedule automated tasks in Linux and this directory stores its events. Whenever a scheduled task (also called a cron job) runs, all its relevant information including execution status and error messages are stored here.

• /var/log/mysqld.log or /var/log/mysql.log: This is the MySQL log file. All debug, failure, and success messages related to the mysqld process and mysqld_safe daemon are logged here. Red Hat, CentOS, and Fedora store MySQL logs under /var/log/mysqld.log, while Debian and Ubuntu maintain the log in the /var/log/mysql.log file.

Example 3-4 shows a portion of the /var/log/syslog log file. Each line represents a logged event. The timestamps at the beginning of the lines mark the moment the event took place.

Example 3-4 Output of /var/log/syslog

[analyst@secOps]$ cat /var/log/syslog
Nov 15 09:17:13 secOps kernel: [    0.000000] Linux version 4.10.10-1-ARCH
  (builduser@tobias) (gcc versi
on 6.3.1 20170306 (GCC) ) #1 SMP PREEMPT Wed Apr 12 19:10:48 CEST 2017
Nov 15 09:17:13 secOps kernel: [    0.000000] ------------[ cut here ]------------
Nov 15 09:17:13 secOps kernel: [    0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/
  kernel/fpu/xstate.c:595
 fpu__init_system_xstate+0x465/0x7b2
Nov 15 09:17:13 secOps kernel: [    0.000000] XSAVE consistency problem, dumping
  leaves
Nov 15 09:17:13 secOps kernel: [    0.000000] Modules linked in:
Nov 15 09:17:13 secOps kernel: [    0.000000] CPU: 0 PID: 0 Comm: swapper Not
  tainted 4.10.10-1-ARCH #1
Nov 15 09:17:13 secOps kernel: [    0.000000] Call Trace:
Nov 15 09:17:13 secOps kernel: [    0.000000]  dump_stack+0x58/0x74
Nov 15 09:17:13 secOps kernel: [    0.000000]  __warn+0xea/0x110
Nov 15 09:17:13 secOps kernel: [    0.000000]  ? fpu__init_system_xstate+0x465/0x7b2
Nov 15 09:17:13 secOps kernel: [    0.000000]  warn_slowpath_fmt+0x46/0x60
Nov 15 09:17:13 secOps kernel: [    0.000000]  fpu__init_system_xstate+0x465/0x7b2
Nov 15 09:17:13 secOps kernel: [    0.000000]  fpu__init_system+0x18c/0x1b1
Nov 15 09:17:13 secOps kernel: [    0.000000]  early_cpu_init+0x110/0x113
Nov 15 09:17:13 secOps kernel: [    0.000000]  setup_arch+0xe4/0xbb6
Nov 15 09:17:13 secOps kernel: [    0.000000]  start_kernel+0x8f/0x3ce
Nov 15 09:17:13 secOps kernel: [    0.000000]  i386_start_kernel+0x91/0x95
Nov 15 09:17:13 secOps kernel: [    0.000000]  startup_32_smp+0x16b/0x16d
Nov 15 09:17:13 secOps kernel: [    0.000000] ---[ end trace c61a827435bb526d ]---
<output omitted>
[analyst@secOps]$

The Linux File System (3.2.2)

In this topic, you will learn about Linux file system types, Linux roles and file permissions, and creating hard and symbolic links.

The File System Types in Linux (3.2.2.1)

There are many different kinds of file systems, varying in properties of speed, flexibility, security, size, structure, logic, and more. It is up to the administrator to decide which file system type best suits the operating system and the files it will store. Below are a few file system types commonly found and supported by Linux:

• ext2 (second extended file system): ext2 was the default file system in several major Linux distributions until supplanted by ext3. Almost fully compatible with ext2, ext3 also supports journaling (see below). ext2 is still the file system of choice for flash-based storage media because its lack of a journal increases performance and minimizes the number of writes. Because flash memory devices have a limited number of write operations, minimizing write operations increases the device’s lifetime. However, contemporary Linux kernels also support ext4, an even more modern file system, with better performance and which can also operate in a journal-less mode.

• ext3 (third extended file system): ext3 is a journaled file system designed to improve the existing ext2 file system. A journal, the main feature added to ext3, is a technique used to minimize the risk of file system corruption in the event of sudden power loss. The file system keeps a log (or journal) of all the file system changes about to be made. If the computer crashes before the change is complete, the journal can be used to restore or correct any eventual issues created by the crash. The maximum file size in ext3 file systems is 32 TB.

• ext4 (fourth extended file system): Designed as a successor of ext3, ext4 was created based on a series of extensions to ext3. While the extensions improve the performance of ext3 and increase supported file sizes, Linux kernel developers were concerned about stability issues and were opposed to adding the extensions to the stable ext3. The ext3 project was split in two; one kept as ext3 and its normal development and the other, named ext4, incorporated the mentioned extensions.

• NFS (Network File System): NFS is a network-based file system, allowing file access over the network. From the user standpoint, there is no difference between accessing a file stored locally or on another computer on the network. NFS is an open standard which allows anyone to implement it.

• CDFS (Compact Disc File System): CDFS was created specifically for optical disk media.

• Swap file system: The swap file system is used by Linux when it runs out of RAM. Technically, it is a swap partition that does not have a specific file system but it is relevant to the file system discussion. When this happens, the kernel moves inactive RAM content to the swap partition on the disk. While swap partitions (also known as swap space) can be useful to Linux computers with a limited amount of memory, they should not be considered as a primary solution. A swap partition is stored on disk, which has much lower access speeds than RAM.

• HFS Plus or HFS+ (Hierarchical File System Plus): Primary file system used by Apple in its Macintosh computers. The Linux kernel includes a module for mounting HFS+ for read-write operations.

• Master boot record (MBR): Located in the first sector of a partitioned computer, the MBR stores all the information about the way in which the file system is organized. The MBR quickly hands over control to a loading function, which loads the OS.

Mounting is the term used for the process of assigning a directory to a partition. After a successful mount operation, the file system contained on the partition is accessible through the specified directory. In this context, the directory is called the mounting point for that file system. Windows users may be familiar with a similar concept: the drive letter.

Example 3-5 shows the output of the mount command issued in the Cisco CyberOPS VM.

Example 3-5 The Output of mount in the CyberOPS VM

[analyst@secOps ~]$ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=511056k,nr_inodes=127764,
  mode=755)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
  (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/
  systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,
  perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,
  cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,
  timeout=0,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=102812k,mode=700,
  uid=1000,gid=1000)
[analyst@secOps ~]$

When issued with no options, mount returns the list of file systems currently mounted in a Linux computer. While many files systems shown are out of the scope of this course, notice the root file system (highlighted). The root file system is represented by the “/” symbol and holds all files in the computer by default. It is also shown in Example 3-5 that the root file system was formatted as ext4 and occupies the first partition of the first drive (/dev/sda1).

Linux Roles and File Permissions (3.2.2.2)

In Linux, most system entities are treated as files. In order to organize the system and reinforce boundaries within the computer, Linux uses file permissions. File permissions are built into the file system structure and provide a mechanism to define permissions on every file. Every file in Linux carries its file permissions, defining the actions that the owner, the group, and others can do with the file. The possible permission rights are Read, Write, and Execute. The ls command with the -l parameter lists additional information about the file. Consider the output of the ls -l command in Example 3-6.

Example 3-6 Viewing Permissions for a Linux File

[analyst@secOps ~]$ ls -l space.txt
-rwxrw-r-- 1 analyst staff 253 May 20 12:49 space.txt
[analyst@secOps ~]$

The output provides a lot of information about the file space.txt.

The first field of the output displays the permissions associated to space.txt (-rwxrw-r--). File permissions are always displayed in the User, Group, and Other order, so you can interpret the first field as follows:

• The dash (-) means that this is a file. For directories, the first dash would be a “d” instead.

• The first set of characters is for user permissions (rwx). The user, analyst, who owns the file can Read, Write, and eXecute the file.

• The second set of characters is for group permissions (rw-). The group, staff, who owns the file can Read and Write to the file.

• The third set of characters is for any other user or group permissions (r--). Any other user or group on the computer can only Read the file.

The second field defines the number of hard links to the file (the number 1 after the permissions). A hard link creates another file with a different name linked to the same place in the file system (called an inode). This is in contrast to a symbolic link, which is discussed next.

The third and fourth field display the user (analyst) and group (staff) who own the file, respectively.

The fifth field displays the file size in bytes. The space.txt file has 253 bytes.

The sixth field displays the date and time of the last modification.

The seventh field displays the filename.

Figure 3-8 shows a breakdown of file permissions in Linux.

Figure 3-8 File Permissions

Table 3-4 shows how to interpret the octal values for permissions.

Table 3-4 Octal Values for Permissions

File permissions are a fundamental part of Linux and cannot be broken. A user has as much rights to a file as the file permissions allow. The only user that can override file permission on a Linux computer is the root user. Because the root user has the power to override file permissions, the root user can write to any file. Because everything is treated as a file, the root user has full control over a Linux computer. Root access is often required before performing maintenance and administrative tasks.

Hard Links and Symbolic Links (3.2.2.3)

A hard link is another file that points to the same location as the original file. Use the command ln to create a hard link. The first argument is the existing file and the second argument is the new file. The file space.txt is linked to space.hard.txt in Example 3-7 and the link field now shows 2.

Example 3-7 Creating Hard Links in Linux

[analyst@secOps ~]$ ln space.txt space.hard.txt
[analyst@secOps ~]$ ls -l space*
-rwxrw-r-- 2 analyst staff 253 May 20 14:41 space.hard.txt
-rwxrw-r-- 2 analyst staff 253 May 20 14:41 space.txt
[analyst@secOps ~]$ echo "Testing hard link" >> space.txt
[analyst@secOps ~]$ ls -l space*
-rwxrw-r-- 2 analyst staff 273 May 20 14:41 space.hard.txt
-rwxrw-r-- 2 analyst staff 273 May 20 14:41 space.txt
[analyst@secOps ~]$ rm space.hard.txt
[analyst@secOps ~]$ more space.txt
"Space is big. Really big. You just won't believe how vastly, hugely, mindbog-
  glingly big it is. I mean, you may think it's a long way down the road to the
  chemist, but that's just peanuts in space."
--Douglas Adams, The Hitchhiker's Guide to the Galaxy

Testing hard link
[analyst@secOps ~]$

Both files point to the same location in the file system. If you change one file, the other is changed, as well. The echo command is used to add some text to space.txt. Notice that the file size for both space.txt and space.hard.txt increased to 273 bytes. If you delete the space.hard.txt file with the rm command (remove), the space.txt file still exists, as verified with the more space.txt command.

A symbolic link, also called a symlink or soft link, is similar to a hard link in that applying changes to the symbolic link will also change the original file. As shown in Example 3-8, use the ln command option -s to create a symbolic link. Notice that adding a line of text to test.txt also adds the line to mytest.txt. However, unlike a hard link, deleting the original test.txt file means that mytest.txt is now linked to a file that no longer exists, as shown with the more mytest.txt and ls -l mytest.txt commands.

Example 3-8 Creating Symbolic Links in Linux

[analyst@secOps ~]$ echo "Hello World!" > test.txt
[analyst@secOps ~]$ ln -s test.txt mytest.txt
[analyst@secOps ~]$ echo "It's a lovely day!" >> mytest.txt
[analyst@secOps ~]$ more test.txt
Hello World!
It's a lovely day!
[analyst@secOps ~]$ more mytest.txt
Hello World!
It's a lovely day!
[analyst@secOps ~]$ rm test.txt
[analyst@secOps ~]$ more mytest.txt
more: stat of mytest.txt failed: No such file or directory
[analyst@secOps ~]$ ls -l mytest.txt
lrwxrwxrwx 1 analyst staff 8 May 20 15:15 mytest.txt -> test.txt
[analyst@secOps ~]$

Although symbolic links have a single point of failure (the underlying file), symbolic links have several benefits over hard links:

• Locating hard links is more difficult. Symbolic links show the location of the original file in the ls -l command, as shown in the last line of output in Example 3-8 (mytest.txt -> test.txt).

• Hard links are limited to the file system in which they are created. Symbolic links can link to a file in another file system.

• Hard links cannot link to a directory because the system itself uses hard links to define the hierarchy of the directory structure. However, symbolic links can link to directories.