Set Up a New Server

Table of Contents

Use this as a guide to enable proper monitoring and maintenace of any new server on the network.

Sample Home Network:

graph TD;
        Modem<-->Router;
        Router<-->MainSwitch;
        Router<-->WiFi;
        Router-->GuestWiFi;
        GuestWiFi-->Camera;
        MainSwitch<-->ServerSwitch;
        MainSwitch<-->Desktop;
        MainSwitch<-->Laptop;
        ServerSwitch-->MailServer;
        ServerSwitch-->WebServer;
        ServerSwitch-->NetworkAccessStorage;

Sometimes the Modem, Router and Main Switch are one unit, or there is no modem.

-----> Single Arrow is a limited access network (VLAN)

<----> Double Arrows is an Open Network

The key point here is that the servers are isolated on a seperate switch for performance and security reasons, using a VLAN (Virtual Local Area Network) local to the Server Switch. Server VLAN network packets between each other never leave the Server Switch. Each server has another IP address not on the VLAN for public access.

A guest WiFi service does not have access to the Main Switch because it is on it's own VLAN, so local resources are protected from that experimental 12 year old guest.

If your camera accesses a cloud service (most do) link it to the Guest WiFi, for security purposes. Any other untrusted device should also be on the Guest Wifi, like Robot Vacuum Cleaners, Car Chargers, Car, TV streaming box, VOIP (Telephone VoiceOverIP), Garage Door Opener, Door Locks, etc...

Check your Network

  • See Network Interfaces
$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.8  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::0000:1111:3333:2222  prefixlen 64  scopeid 0x20<link>
        ether 2a:53:9b:00:f9:21  txqueuelen 1000  (Ethernet)
        RX packets 342047883  bytes 378663045018 (378.6 GB)
        RX errors 0  dropped 54131  overruns 0  frame 0
        TX packets 221663343  bytes 165067861773 (165.0 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xc0b00000-c0b20000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 32619960  bytes 99080107335 (99.0 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32619960  bytes 99080107335 (99.0 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 36:27:10:52:32:96 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 192.168.0.5/24 brd 10.0.2.255 scope global dynamic noprefixroute ens3
       valid_lft 66645sec preferred_lft 66645sec
    inet6 fec0::000:ff:1111:1111/64 scope site dynamic noprefixroute 
       valid_lft 86390sec preferred_lft 14390sec
    inet6 fe80::000:ff:1111:1111/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
  • System Log is
/var/log/syslog

/var/log/messages
  • Bring Network Up:
$ sudo cat /etc/netplan/01-network-manager-all.yaml
network:
    version: 2
    renderer: networkd
    ethernets:
        enp3s0:
            dhcp4: true
$ sudo netplan try
$ sudo netplan -d apply
$ sudo systemctl restart system-networkd

Reference: https://netplan.io/

$ sudo nmcli connection up ens3

vi /etc/sysconfig/network-scripts/ifcfg-ens3
~
ONBOOT=yes
~
:wq

Reference: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-an-ethernet-connection_configuring-and-managing-networking

Update Package Repository

Debian

Always refresh the package repository before getting started.

$ sudo apt-get update

You may need to add extra repositories, just check the sources.

The four main repositories are:

  • Main - Canonical-supported free and open-source software.
  • Universe - Community-maintained free and open-source software.
  • Restricted - Proprietary drivers for devices.
  • Multiverse - Software restricted by copyright or legal issues.
  • Backports[1] - Software from older releases that should still work.
  1. Backports for old stuff you really really need. 
$ sudo add-apt-repository universe
$ sudo grep '^deb ' /etc/apt/sources.list
...
deb http://us.archive.ubuntu.com/ubuntu/ jammy universe
...
$ sudo add-apt-repository --remove universe

RedHat

Always refresh the package repository before getting started.

$ sudo dnf update
  • Now you can disable subscription-manager if you do not have a RedHat subscription.

Change: From -> enabled=1 To -> enabled=0

$ sudo vi /etc/yum/pluginconf.d/subscription-manager.conf
$ sudo yum clean all
0 files removed
  • You may need to add extra repositories, just check the sources.

The EPEL repository provides additional high-quality packages for RHEL-based distributions. EPEL is a selection of packages from Fedora, but only packages that are not in RHEL or its layered products to avoid conflicts.

The folks at Fedora have very nicely put up an automatic build and repo system and they are calling it COPR (Cool Other Package Repositories).

Reference: 

$ sudo dnf install epel-release 'dnf-command(copr)'

New User and Group to Use

Be sure to match the same user numbers across systems, because when you share files using NFS, the numbers need to match.

$ sudo adduser don --uid 1001
Adding user `don' ...
Adding new group `don' (1001) ...
Adding new user `don' (1001) with group `don' ...
Creating home directory `/home/don' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for don
Enter the new value, or press ENTER for the default
 Â Â Â  Full Name []: Don
 Â Â Â  Room Number []:
 Â Â Â  Work Phone []:
 Â Â Â  Home Phone []:
 Â Â Â  Other []:
Is the information correct? [Y/n] y
Adding new user `don' to extra groups ...
Adding user `don' to group `dialout' ...
Adding user `don' to group `i2c' ...
Adding user `don' to group `spi' ...
Adding user `don' to group `cdrom' ...
Adding user `don' to group `floppy' ...
Adding user `don' to group `audio' ...
Adding user `don' to group `video' ...
Adding user `don' to group `plugdev' ...
Adding user `don' to group `users' ...

If this user is an administrator;

Debian

Add them to group sudo. 
$ sudo usermod -aG sudo rootbk

Check your user for group '27(sudo)'.

$ id rootbk
uid=1002(rootbk) gid=1002(rootbk) 
groups=1002(rootbk),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),44(video),46(plugdev),100(users),114(i2c),993(spi)

RedHat

Add them to group wheel. 
$ sudo usermod -aG wheel rootbk

Check your user for group '10(wheel)'.

# id don
uid=1002(rootbk) gid=1002(rootbk) groups=1002(rootbk),10(wheel)
Also set the root password in case the system will not boot
$ sudo passwd root
[sudo] password for don: 
New password: 
Retype new password: 
passwd: password updated successfully

Firewall

Every machine should have a firewall enabled, especially before connecting to the internet.

Now adays there is a choice between Uncomplicated Firewall (ufw) and firewalld. Choose wisely or be hacked.

Debian

ufw - Uncomplicated Firewall 
$ sudo apt-get install ufw
$ sudo ufw allow 22/tcp
$ sudo ufw enable
$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     ALLOW IN    Anywhere                               
[ 2] 22/tcp (v6)                ALLOW IN    Anywhere (v6)           

$ sudo ufw delete 2
$ sudo ufw logging low
$ sudo ufw logging on

NOTE: Logging values are: low|medium|high. Only network blocks are logged on low.

Reference: https://launchpad.net/ufw

RedHat

firewalld 
$ sudo dnf install firewalld
$ sudo firewall-cmd --add-service=ssh
success
$ sudo firewall-cmd --list-services
cockpit dhcpv6-client ssh
$ sudo firewall-cmd --remove-service=cockpit
success
$ sudo firewall-cmd --remove-service=dhcpv6-client
success
$ sudo firewall-cmd --list-services
ssh
$ sudo firewall-cmd --runtime-to-permanent
success

Reference: https://firewalld.org/

Block bad actors, whole Autonomous System [1] groups at a time

Using IP addresses from Logwatch. Logcheck, and Logwatcher, feed them into the firewall.sh script.

Run the git clone and install the dependencies based on your operating system.

File: ~/firewall.sh

#!/bin/bash
##############################################################################
#
# File: firewall.sh
#
# Purpose: Block IP address or CIDR range using OS firewall
#
# Dependencies: 
#
#  git clone https://github.com/nitefood/asn 
#
#  * For more detail get an ~/.asn/iqs_token
#     from https://github.com/nitefood/asn#ip-reputation-api-token
#
# * **Debian 10 / Ubuntu 20.04 (or newer):**
#
#  ```
#  apt -y install curl whois bind9-host mtr-tiny jq ipcalc grepcidr nmap ncat aha
#  ```
#  * Enable ufw and allow the ports you need
#
#  # ufw allow 22
#  # ufw enable
#
#  * Delete rules not needed:
#  # ufw status numbered
#  # ufw delete <number>
#
# * **CentOS / RHEL / Rocky Linux 8:**
#
# # Install repos:
# $ sudo dnf repolist
# repo id                                repo name
# appstream                              CentOS Stream 9 - AppStream
# baseos                                 CentOS Stream 9 - BaseOS
# epel                                   Extra Packages for Enterprise Linux 9 - x86_64
# epel-next                              Extra Packages for Enterprise Linux 9 - Next - x86_64
# extras-common                          CentOS Stream 9 - Extras packages
#
# $ ls /etc/yum.repos.d
# centos-addons.repo  centos.repo  epel-next.repo  epel-next-testing.repo  epel.repo  epel-testing.repo  redhat.repo
#
# dnf install bind-utils jq whois curl nmap ipcalc grepcidr aha
#
#   If you have a list of IP addresses to block (text file, each IP on a separate line),
#     you can easily import that to your block list:
#
#    # firewall-cmd --permanent --ipset=networkblock --add-entries-from-file=/path/to/blocklist.txt
#    # firewall-cmd --reload
#
#   To view ipsets:
#    # firewall-cmd --permanent --get-ipsets
#     networkblock
#    # firewall-cmd --permanent --info-ipset=networkblock
#     networkblock
#       type: hash:net
#       options: maxelem=1000000 family=inet hashsize=4096
#       entries: 46.148.40.0/24
#
#   # firewall-cmd --add-service=smtp
#   success
#   # firewall-cmd --add-service=smtps
#   success
#   # firewall-cmd --list-services
#   cockpit dhcpv6-client smtp smtps ssh
#   # firewall-cmd --remove-service=cockpit
#   success
#   # firewall-cmd --remove-service=dhcpv6-client
#   success
#   # firewall-cmd --list-services
#   smtp smtps ssh
#   # firewall-cmd --runtime-to-permanent
#   success
#
#   # firewall-cmd --list-all
#     public (active)
#       target: default
#       icmp-block-inversion: no
#       interfaces: ens3
#       sources: 
#       services: smtp smtps ssh
#       ports: 
#       protocols: 
#       forward: no
#       masquerade: no
#       forward-ports: 
#       source-ports: 
#       icmp-blocks: 
#       rich rules: 
#
# https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-setting_and_controlling_ip_sets_using_firewalld
#
# * TO UNDO A MISTAKEN BLOCK:
#   # firewall-cmd --permanent --ipset=networkblock --remove-entry=x.x.x.x/y
#   # firewall-cmd --reload
#
# * To drop ipset
#   # firewall-cmd --permanent --delete-ipset=networkblock
#   # firewall-cmd --reload
#
# Author     Date     Description
# ---------- -------- --------------------------------------------------------
# D. Cohoon  Jan-2023 Created
# D. Cohoon  Feb-2023 Add RedHat firewalld
##############################################################################
DIR=/root
LOG=${DIR}/firewall.log
WHO=/tmp/whois.txt
CIDR=/tmp/whois.cidr
IP="${1}"
OS=$(/usr/bin/hostnamectl|/usr/bin/grep 'Operating System'|/usr/bin/cut -d: -f2|/usr/bin/awk '{print $1}')
#.............................................................................
function set_ipset() {
  FOUND_IPSET=0
  while read SET
  do
    if [ ! -z ${SET} ] && [ ${SET} == "networkblock" ]; then
      FOUND_IPSET=1
    fi
  done <<< $(sudo /usr/bin/firewall-cmd --permanent --get-ipsets) 
  #
  if [ $FOUND_IPSET -eq 0 ]; then
    # Create networkblock ipset
    sudo /usr/bin/firewall-cmd --permanent --new-ipset=networkblock --type=hash:net \
      --option=maxelem=1000000 --option=family=inet --option=hashsize=4096
    # Add new ipset to drop zone
    sudo /usr/bin/firewall-cmd --permanent --zone=drop --add-source=ipset:networkblock
    # reload
    sudo /usr/bin/firewall-cmd --reload
  fi
}
#
#.............................................................................
function run_asn() {
  ${DIR}/asn/asn -n ${IP} > ${WHO}
  /usr/bin/cat ${WHO}
  RANGE=$(/usr/bin/cat ${WHO} | /usr/bin/grep 'NET' | /usr/bin/grep '/' | /usr/bin/awk -Fm '{print $6}' | /usr/bin/cut -d" " -f1)
  /usr/bin/echo "CDR: ${RANGE}"
  /usr/bin/echo "${RANGE}" > ${CIDR}
}
#.............................................................................
#
if [ ${1} ]; then
  run_asn
else
  /usr/bin/echo "Usage: ${0} <IP Address>"
  exit 1
fi
#.............................................................................
#
  /usr/bin/grep -v deaggregate ${CIDR} > ${CIDR}.block
  while read -r IP
  do
    /usr/bin/echo "$(/usr/bin/date) - OS: ${OS}" | /usr/bin/tee -a ${LOG}
    /usr/bin/echo "Blocking: ${IP}" | /usr/bin/tee -a ${LOG}
    case ${OS} in
      AlmaLinux|CentOS) 
        /usr/bin/echo "Firewalld"
        set_ipset
        sudo /usr/bin/firewall-cmd --permanent --ipset=networkblock --add-entry=${IP}
        sudo /usr/bin/firewall-cmd --reload
        ;;
      Ubuntu|Debian) 
        /usr/bin/echo "ufw"
        sudo /usr/sbin/ufw prepend deny from ${IP} to any 2>&1 |tee -a $LOG
        ;;
    esac
  done < ${CIDR}.block
  1. https://www.arin.net/resources/guide/asn/

Set your host and domain names

File: /etc/hosts

127.0.0.1	  localhost
192.168.1.5 	www.example.com 	example.com	www

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

File: /etc/hostname

don.example.com

Rsyslog - Send Syslog Entries to Remote Syslog Host

It is a good idea to send log messages to another host in case the system crashes. You will be able to see that last gasping breath of the dying server. Also in the event of a compromised system hackers usually zero out the local syslog to cover their tracks. Now you still have any trace of the hackers on the central rsyslog host. It makes things simpler for detailed log analysis with combined logs on one system too.

Local System

Replicate log entries: Add the following to cause log entries to be in /var/log/syslog locally and be sent to a remote syslog host. If you do not have a Remote Syslog Host, skip this.

File: /etc/rsyslog.conf

Add these lines on local system.

~
# Remote logging - Aug 2020 Don
# Provides UDP forwarding
*.* @192.168.1.5:514 #this is the logging host
~

Alert: Create the following to send syslog alerts to email if the severity is high (3 or below).

File: /etc/rsyslog.d/alert.conf

Create the file if it does not exist and replace with these lines.

module(load="ommail")
template (name="mailBody"  type="string" string="Alert for %hostname%:\n\nTimestamp: %timereported%\nSeverity:  %syslogseverity-text%\nProgram:   %programname%\nMessage:  %msg%")
template (name="mailSubject" type="string" string="[%hostname%] Syslog alert for %programname%")

if $syslogseverity <= 3 and not ($msg contains 'brcmfmac') then {
   action(type="ommail" server="192.168.1.3" port="25"
          mailfrom="rsyslog@localhost"
          mailto="don@example.com"
          subject.template="mailSubject"
          template="mailBody"
          action.execonlyonceeveryinterval="3600")
}

Remote Syslog Host

Allow remote hosts to log here: Open firewall port 514/udp on remote syslog host.

$ sudo ufw allow 514/udp

File: /etc/rsyslog.conf

Add these lines to remote syslog host.

~
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
~
# Process remote logs into seperate directories, then stop. Do not duplicate into syslog
$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& stop

Restart rsyslog

$ sudo systemctl restart rsyslog

Time Control

All servers should be set up to synchronize their time over the network using Network Time Protocol (NTP). This is critical in validating security certificates. For offline systems, consider using a Real Time Clock (RTC) attached to something like BeagleBone.

timezone

Change to match your timezone.

File: /etc/timezone

$ cat /etc/timezone
America/New_York

Set timezone with timedatactl, and verify.

$ sudo timedatectl set-timezone America/New_York
$ timedatectl
               Local time: Sun 2022-10-09 18:27:11 EDT
           Universal time: Sun 2022-10-09 22:27:11 UTC
                 RTC time: n/a
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

RedHat

RedHat uses chronyd service

File: /etc/chrony.conf

server pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony

Restart to pick up new config

$ sudo systemctl restart chronyd
$ sudo systemctl status chronyd
● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2023-02-17 15:47:28 EST; 20h ago
     Docs: man:chronyd(8)
           man:chrony.conf(5)
  Process: 798 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 789 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 796 (chronyd)
    Tasks: 1 (limit: 11366)
   Memory: 2.3M
   CGroup: /system.slice/chronyd.service
           └─796 /usr/sbin/chronyd

Test

$ chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^? your-ip-name-d>             0   8     0     -     +0ns[   +0ns] +/-    0ns
...
$ timedatectl
               Local time: Wed 2023-07-12 09:03:57 EDT
           Universal time: Wed 2023-07-12 13:03:57 UTC
                 RTC time: Wed 2023-07-12 13:03:57
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
...
$ systemctl is-active chronyd.service
active
...
$ chronyc tracking
Reference ID    : 404F64C5 (ntpool1.258.ntp.org)
Stratum         : 3
Ref time (UTC)  : Wed Jul 12 12:47:34 2023
System time     : 0.000000002 seconds slow of NTP time
Last offset     : +1.114915133 seconds
RMS offset      : 1.114915133 seconds
Frequency       : 32.362 ppm slow
Residual freq   : +22.860 ppm
Skew            : 3.901 ppm
Root delay      : 0.046558209 seconds
Root dispersion : 0.050582517 seconds
Update interval : 0.0 seconds
Leap status     : Normal

Reference: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/using-chrony-to-configure-ntp_configuring-basic-system-settings

Debian

Debian uses systemd-timesyncd service.

$ sudo systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Sun 2022-07-24 12:06:36 EDT; 2 weeks 3 days ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 559 (systemd-timesyn)
   Status: "Synchronized to time server for the first time 192.155.94.72:123 (2.debian.pool.ntp.org)."
    Tasks: 2 (limit: 951)
   Memory: 1.0M
   CGroup: /system.slice/systemd-timesyncd.service
           └─559 /lib/systemd/systemd-timesyncd

Reference:

E-Mail - Client for Sending Local Mail

Identify Mail Client Host and Domain

Edit the following files:

  • /etc/hostname (add fully qualified host & domain; i.e.: www.example.com)
  • /etc/mailname (add domain; i.e.: example.com)

Mail Transport Agent (MTA) packages

Install an SMTP daemon to transfer mail to the E-Mail server.

  • Debian

Install postfix

$ sudo apt-get install postfix

Reconfigure postfix, if it does not pop up, and select sattelite system.

$ sudo dpkg-reconfigure postfix

The following assumes your host is named app and your email server is smtp.<domain>

File: /etc/postfix/main.cf

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on
# fresh installs.
compatibility_level = 3.6



# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_security_level=may

smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache


smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = app
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = app.example.com, $myhostname, app, localhost.localdomain, localhost
relayhost = smtp.example.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = loopback-only
inet_protocols = all

Check postfix systemd service.

$ sudo systemctl status postfix
● postfix.service - Postfix Mail Transport Agent
     Loaded: loaded (/lib/systemd/system/postfix.service; enabled; preset: enabled)
     Active: active (exited) since Sat 2023-07-22 09:32:00 EDT; 5h 26min ago
       Docs: man:postfix(1)
    Process: 1178 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
   Main PID: 1178 (code=exited, status=0/SUCCESS)
        CPU: 1ms

Jul 22 09:32:00 app.example.com systemd[1]: Starting postfix.service - Postfix Mail Transport Agent...
Jul 22 09:32:00 app.example.com systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
  • RedHat

Install postfix

$ sudo dnf install postfix

The following assumes your host is named app and your email server is smtp.<domain>

File: /etc/postfix/main.cf

smtpd_banner = $myhostname ESMTP $mail_name (Linux)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 2 on
# fresh installs.
compatibility_level = 2

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = app.example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = app.example.com, localhost.example.com, localhost
relayhost = smtp.example.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 192.168.1.0/32
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = loopback-only
inet_protocols = all

Check the status of postfix

$ sudo systemctl status postfix
[sudo] password for don: 
● postfix.service - Postfix Mail Transport Agent
     Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; preset: disabled)
     Active: active (running) since Wed 2023-06-07 17:43:43 EDT; 22h ago
   Main PID: 2182 (master)
      Tasks: 3 (limit: 99462)
     Memory: 8.6M
        CPU: 1.975s
     CGroup: /system.slice/postfix.service
             ├─ 2182 /usr/libexec/postfix/master -w
             ├─ 2184 qmgr -l -t unix -u
             └─15581 pickup -l -t unix -u

MAC OS send mail to local server, not system configured in mail app

Use the IP address of the local mail server, or you can edit /etc/hosts and use that name. Postfix does not run as a daemon, but is run by the SMTP process, probably fired of by a listener for port 25.

File: /etc/postfix/main.cf

% sudo vi /etc/postfix/main.cf
~
myhostname = square.example.com
~
mydomain = example.com
~
relayhost = [192.168.1.3]

Test mail

First install the command line mail interface(s). I use mail and mutt.

  • Debian
$ sudo apt-get install mailutils mutt
  • RedHat
$ sudo dnf install s-nail mutt

% mail -s "Hello internal mail"  don@example.com </dev/null
Null message body; hope that's ok

Shows up as: don@square.example.com
...
mutt -s "Hello internal mail from mutt"  don@example.com

Shows up as: don@square.local

Mutt change from address

File: ~/.muttrc 

set from="Square <don@square.example.com>"
set hostname="square.example.com"

Shows up as: don@square.example.com

  • Update root destination in aliases

File: /etc/aliases

~
# Person who should get root's mail
#root:		marc
root:		bob@example.com
~

Update aliases into database format

[don@ash ~]$ sudo newaliases
  • Create mail script to set variables

File: ~/mail.sh

#!/bin/bash
#######################################################################
#
# File: mail.sh
#
# Usage: mail.sh <File Name to Mail> <Subject>
#  Change the REPLYTO, FROM, and MAILTO variables
#  and choose RedHat or Debian
#
# Who       When        Why
# --------- ----------- -----------------------------------------------
# D. Cohoon Feb-2023    VPS host name cannot be changed, so set headers
#######################################################################
function usage () {
   /usr/bin/echo "Usage: ${0} <File Name to Mail> <Subject>"
   exit 1
}
#------------------
if [ $# -lt 2 ]; then
  usage
fi
#
if [ ! -z ${1} ] && [ ! -f ${1} ]; then
  usage 
fi
#
#------------------
HOSTNAME=$(hostname -s)
DOMAINNAME=$(hostname -d)
FILE=${1}    # First arg
shift 1
SUBJECT="${HOSTNAME}.${DOMAINNAME}:${@}" # Remainder of args
#
#------------------
export REPLYTO=root@app.example.com
FROM=root@app.example.com
#FROM="${HOSTNAME}@${DOMAINNAME}"
MAILTO=bob@example.com
#
#------------------
# Debian: install mailutils
#/usr/bin/cat ${FILE} | /usr/bin/mail -aFROM:${FROM}  -s "${SUBJECT}" ${MAILTO}
# RedHat: install s-nail
#/usr/bin/cat ${FILE} | /usr/bin/mail --from-address=${FROM} -s "${SUBJECT}" ${MAILTO}

Monit - Monitor System and Restart Processes

Monit is a small Open Source utility for managing and monitoring Unix systems. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Reference: https://mmonit.com/monit/

Installation

$ sudo apt-get install monit

$ sudo dnf install monit

Configuration

Change the mailserver to yours, and add some general monitoring.

File: /etc/monit/monitrc:

# Mail server
set mailserver www.example.com port 25   # primary mailserver
# Don 28-Dec 2021 - general monitoring
check system $HOST
  if loadavg (1min) per core > 2 for 5 cycles then alert
  if loadavg (5min) per core > 1.5 for 10 cycles then alert
  if cpu usage > 95% for 5 cycles then alert
  if memory usage > 90% then alert
  if swap usage > 50% then alert

check device root with path /
  if space usage > 90% then alert
  if inode usage > 90% then alert
  if changed fsflags then alert
  if service time > 250 milliseconds for 5 cycles then alert
  if read rate > 500 operations/s for 5 cycles then alert
  if write rate > 200 operations/s for 5 cycles then alert

check network eth0 with interface eth0
  if failed link then alert
  if changed link then alert
  if saturation > 90%  for 2 cycles then alert
  if download > 10 MB/s for 5 cycles then alert
  if total uploaded > 1 GB in last hour then alert

check host REACHABILITY with address 1.1.1.1
  if failed ping with timeout 10 seconds then alert

Process

Monitor and restart the ssh process (and others that you may need using this as a guide).

File: /etc/monit/conf.d/sshd

check process sshd with pidfile /var/run/sshd.pid
    alert root@example.com with mail-format {
           from: monit@example.com
        subject: monit alert: $SERVICE $EVENT $DATE
        message: $DESCRIPTION
    }
  start program "/etc/init.d/ssh start"
  stop program "/etc/init.d/ssh stop"

Munin - Resource History Monitor

Munin is a networked resource monitoring tool (started in 2002) that can help analyze resource trends and what just happened to kill our performance? problems. It is designed to be very plug and play.

A default installation provides a lot of graphs with almost no work. Requires Apache or nginx for graphs.

Reference: http://guide.munin-monitoring.org/en/latest/tutorial/alert.html

Munin-Architecture.png

Installation

On all nodes

package libdb-pg-perl is required for postgresql

# sudo apt-get install munin libdbd-pg-perl

package perl-DBD-Pg is required for postgresql

# sudo dnf install munin perl-DBD-Pg

On Munin-Master node, add the following list of hosts to monitor:

File: /etc/munin.munin.conf

~
# Local Host
[app.example.com]
    address 127.0.0.1
    use_node_name yes

# E-Mail host
[www.example.com]
    address 192.168.1.3
    use_node_name yes
~

On Munin-Node node, allow master into IPv6 port:

$ sudo ufw allow 4949
$ sudo ufw status | grep 4949
4949                       ALLOW       Anywhere                  
4949 (v6)                  ALLOW       Anywhere (v6)    

$ sudo firewall-cmd --permanent --zone=public --add-port=4949/tcp
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --permanent --list-ports
4949/tcp

On Munin-Node node, add the Munin-Master IP address to the following:

File: /etc/munin/munin-node.conf

~
# A list of addresses that are allowed to connect.  This must be a
# regular expression, since Net::Server does not understand CIDR-style
# network notation unless the perl module Net::CIDR is installed.  You
# may repeat the allow line as many times as you'd like

allow ^127\.0\.0\.1$
allow ^::1$
allow ^192\.168\.1\.3$
allow ^fe80::abcd:1234:0000:abcd$
~

Check your munin-node functions from command line using the network cat utility Debian -> $ sudo apt-get install ncat: RedHat -> $ sudo dnf install ncat:

Try comands:

  • list
  • nodes
  • config
  • fetch
  • version
  • quit quit
$ ncat 127.0.0.1 4949
# munin node at app.example.com
list
acpi apache_accesses apache_processes apache_volume cpu df df_inode entropy forks fw_packets http_loadtime if_em1 if_eno1 if_err_em1 if_err_eno1 if_err_tun0 if_err_wlp58s0 if_tun0 if_wlp58s0 interrupts irqstats load lpstat memory munin_stats netstat ntp_198.23.200.19 ntp_208.94.243.142 ntp_216.218.254.202 ntp_91.189.94.4 ntp_96.126.100.203 ntp_kernel_err ntp_kernel_pll_freq ntp_kernel_pll_off ntp_offset ntp_states open_files open_inodes postfix_mailqueue postfix_mailvolume postgres_autovacuum postgres_bgwriter postgres_cache_ALL postgres_cache_twotree postgres_checkpoints postgres_connections_ALL postgres_connections_db postgres_connections_twotree postgres_locks_ALL postgres_locks_twotree postgres_querylength_ALL postgres_querylength_twotree postgres_scans_twotree postgres_size_ALL postgres_size_twotree postgres_transactions_ALL postgres_transactions_twotree postgres_tuples_twotree postgres_users postgres_xlog proc_pri processes swap threads uptime users vmstat
.
fetch df
_dev_nvme0n1p2.value 34.4721606114914
_dev_shm.value 0.000540811610733636
_run.value 0.183629672785198
_run_lock.value 0.15625
_run_qemu.value 0
_dev_sda1.value 39.2182617590549
_dev_nvme0n1p1.value 1.02513530868728
_dev_sdb1.value 29.1143742265263
.
fetch cpu
user.value 3392679
nice.value 310628
system.value 792413
idle.value 49254331
iowait.value 202415
irq.value 0
softirq.value 56281
steal.value 0
guest.value 0
.

Apache monitoring requires the mod_status to be enabled and add your IP address range to the status.conf.

Enable apache module mod_status:

$ sudo a2enmod status

Check the IP addresses in the apache status configuration. Change Require ip <address> to allow other IP addresses to connect to the munin monitor.

File: /etc/apache2/mods-enabled/status.conf

~
	<Location /server-status>
		SetHandler server-status
		Require local
		Require ip 192.168.1.0/24
		#Require ip 192.0.2.0/24
	</Location>
~

Check apache plugin:

$ sudo munin-run apache_volume
volume80.value 500736

Check postgresql plugin:

$ sudo munin-run postgres_connections_miniflux
active.value 0
idle.value 1
idletransaction.value 0
unknown.value 0
waiting.value 0

Check the munin-node daemon status:

$ sudo systemctl status munin-node

Check the munin-master daemon status:

$ sudo systemctl status munin

The utility munin-node-configure is used by the Munin installation procedure to check which plugins are suitable for your node and create the links automatically. It can be called every time when a system configuration changes (services, hardware, etc) on the node and it will adjust the collection of plugins accordingly. '-shell' will display new configuration plugin links 'ln -s ...' for you.

For instance, below a new network interface (if) was discovered since the last configuration of munin. To enable the new monitoring simply execute the 'ln -s ...' commands to create soft links, so interface veth2e40fe9 will be monitored.

$ sudo munin-node-configure -shell
ln -s '/usr/share/munin/plugins/if_' '/etc/munin/plugins/if_veth2e40fe9'
ln -s '/usr/share/munin/plugins/if_err_' '/etc/munin/plugins/if_err_veth2e40fe9'

To have munin-node-configure display 'rm ...' commands for plugins with software that may no longer be installed, use the option ‘–remove-also’.

$ sudo munin-node-configure -shell -remove-also
ln -s '/usr/share/munin/plugins/if_' '/etc/munin/plugins/if_veth2e40fe9'
rm -f '/etc/munin/plugins/if_veth0049d71'
ln -s '/usr/share/munin/plugins/if_err_' '/etc/munin/plugins/if_err_veth2e40fe9'
rm -f '/etc/munin/plugins/if_err_veth0049d71'

Enabled monitors can be found in the same location

$ ls -l /etc/munin/plugins | grep apache
lrwxrwxrwx 1 root root 40 Jun 30  2018 apache_accesses -> /usr/share/munin/plugins/apache_accesses
lrwxrwxrwx 1 root root 41 Jun 30  2018 apache_processes -> /usr/share/munin/plugins/apache_processes
lrwxrwxrwx 1 root root 38 Jun 30  2018 apache_volume -> /usr/share/munin/plugins/apache_volume

Testing new plugins has an autoconf option to munin-run. Errors will be displayed, and a debug '-d' option is also available.

$ sudo munin-run postgres_connections_miniflux autoconf
yes

$ sudo munin-run -d postgres_connections_miniflux autoconf
# Running 'munin-run' via 'systemd-run' with systemd properties based on 'munin-node.service'.
# Command invocation: systemd-run --collect --pipe --quiet --wait --property EnvironmentFile=/tmp/YRfAa1dq9U --property UMask=0022 --property LimitCPU=infinity --property LimitFSIZE=infinity --property LimitDATA=infinity --property LimitSTACK=infinity --property LimitCORE=infinity --property LimitRSS=infinity --property LimitNOFILE=524288 --property LimitAS=infinity --property LimitNPROC=14150 --property LimitMEMLOCK=65536 --property LimitLOCKS=infinity --property LimitSIGPENDING=14150 --property LimitMSGQUEUE=819200 --property LimitNICE=0 --property LimitRTPRIO=0 --property LimitRTTIME=infinity --property SecureBits=0 --property 'CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf cap_checkpoint_restore' --property AmbientCapabilities= --property DynamicUser=no --property MountFlags= --property PrivateTmp=yes --property PrivateDevices=no --property ProtectClock=no --property ProtectKernelTunables=no --property ProtectKernelModules=no --property ProtectKernelLogs=no --property ProtectControlGroups=no --property PrivateNetwork=no --property PrivateUsers=no --property PrivateMounts=no --property ProtectHome=yes --property ProtectSystem=full --property NoNewPrivileges=no --property LockPersonality=no --property MemoryDenyWriteExecute=no --property RestrictRealtime=no --property RestrictSUIDSGID=no --property RestrictNamespaces=no --property ProtectProc=default --property ProtectHostname=no -- /usr/sbin/munin-run --ignore-systemd-properties -d postgres_connections_miniflux autoconf
# Processing plugin configuration from /etc/munin/plugin-conf.d/README
# Processing plugin configuration from /etc/munin/plugin-conf.d/dhcpd3
# Processing plugin configuration from /etc/munin/plugin-conf.d/munin-node
# Processing plugin configuration from /etc/munin/plugin-conf.d/spamstats
# Setting /rgid/ruid/ to /130/117/
# Setting /egid/euid/ to /130 130/117/
# Setting up environment
# Environment PGPORT = 5432
# Environment PGUSER = postgres
# About to run '/etc/munin/plugins/postgres_connections_miniflux autoconf'
yes

Alert via e-mail:

Reference: https://guide.munin-monitoring.org/en/latest/tutorial/alert.html#alerts-send-by-local-system-tools

Change your email here

File: /etc/munin/munin.conf

~
contact.email.command mail -s "Munin-notification for ${var:group} :: ${var:host}" your@email.address.here
~

Adjust disk full thresholds:

Adjust this in master /etc/munin.conf section for node

File: /etc/munin.conf

[beaglebone]
    address 192.168.1.7
    use_node_name yes
    diskstats_latency.mmcblk0.avgrdwait.warning 0:10
    diskstats_latency.mmcblk0.avgrdwait.critical -5:5
    diskstats_latency.mmcblk0.avgwdwait.warning 0:10
    diskstats_latency.mmcblk0.avgwdwait.critical -5:5
    diskstats_latency.mmcblk0.avgwait.warning 0:10
    diskstats_latency.mmcblk0.avgwait.critical -5:5

Graph Examplemunin.png

Redhat Alternative

Cockpit

Cockpit Dashboard

$ sudo systemctl start Cockpit

To log in to Cockpit, open your web browser to localhost:9090 and enter your Linux username and password.

Reference: https://www.redhat.com/sysadmin/intro-cockpit

Fail2ban - Automatic Firewall Blocking

Daemon to ban hosts that cause multiple authentication errors by monitoring system logs.

Reference: https://github.com/fail2ban/fail2ban

Install

$ sudo apt-get install fail2ban

$ sudo dnf install fail2ban

Configure

Create a jail.local file to override the defaults. Update your email and IP addresses to suit your environment. Also add or disable applications you do not run. See the reference above for example of how to do that.

action = %(action_)s This defines the action to execute when a limit is reached. By default it will only block the user.

To receive an email at each ban, set it to:

action = %(action_mw)

To receive the logs with the mail, set it to:

action = %(action_mwl)

File: /etc/fail2ban/jail.local

[DEFAULT]
# email
destemail = don@example.com
sender = root@example.com
# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action = %(action_mwl)s
# whitelist
ignoreip = 127.0.0.1 192.168.1.0/24 8.8.8.8 1.1.1.1

Secure sshd

File: /etc/fail2ban/jail.d/sshd.local

[sshd]
enabled = true
port = 22
filter = sshd
action = iptables-multiport[name=sshd, port="ssh"]
logpath = /var/log/auth.log
maxretry = 3
bantime = 1d

[sshd]
enabled = true
port = 22
filter = sshd
action = iptables-multiport[name=sshd, port="ssh"]
logpath = /var/log/secure
maxretry = 3
bantime = 1d

More filters show which daemons are available to be enabled here: /etc/fail2ban/filter.d/

Backup - Save Your Files Daily

To find the proper name of your USB stick, check the current mounts:

$ sudo lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   1.8T  0 disk  
└─sda1        8:1    0   1.8T  0 part  
sdb           8:16   1     0B  0 disk  
sdc           8:32   1     0B  0 disk  
nvme0n1     259:0    0 238.5G  0 disk  
├─nvme0n1p1 259:1    0   300M  0 part  /boot/efi
├─nvme0n1p2 259:2    0 119.2G  0 part  /
├─nvme0n1p3 259:3    0  16.9G  0 part  [SWAP]

Plug it is, then check dmesg -x immediately after plugging it in. Look for :

[201373.210797]  sdd: sdd1
[201373.211917] sd 2:0:0:0: [sdd] Attached SCSI removable disk

Then run blkid again. You can see the new entry, sdd.

$ sudo lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   1.8T  0 disk  
└─sda1        8:1    0   1.8T  0 part  
sdb           8:16   1     0B  0 disk  
sdc           8:32   1     0B  0 disk  
sdd           8:48   1  28.9G  0 disk  <----- New USB Stick
nvme0n1     259:0    0 238.5G  0 disk  
├─nvme0n1p1 259:1    0   300M  0 part  /boot/efi
├─nvme0n1p2 259:2    0 119.2G  0 part  /
├─nvme0n1p3 259:3    0  16.9G  0 part  [SWAP]

Automatic backup to USB disk

Format new USB stick.

Here are the commands to fdisk:

  • m - menu
  • p - print existing partitions
  • d - delete partition
  • n - create new partition (in this case only a primary partition is required)
  • w - write partition
  • q - quit
$ sudo fdisk /dev/sdd

Check the USB stick label and filesystem (this example has no filesystem)

$ sudo blkid /dev/sdd1
/dev/sdd1: PARTUUID="66bc7da7-1234-abcd-1234-ea4bfe7e00a7"

So create an ext4 filesystem on the new partition(1)

$ sudo mkfs.ext4 /dev/sdd1
mke2fs 1.46.2 (28-Feb-2021)
/dev/sdc1 contains a vfat file system
Proceed anyway? (y,N) y
Creating filesystem with 7566075 4k blocks and 1892352 inodes
Filesystem UUID: 8e33672c-1283-49de-98b8-6fd841372db6
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Check the new filesystem, now we see it is TYPE="ext4"

$ sudo blkid /dev/sdd1
/dev/sdc1: UUID="8e33672c-1234-49de-abcd-6fd841372db6" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="66bc7da7-c9c3-4342-8073-ea4bfe7e00a7"

Label the USB stick as 'backup' so autobackup can find and mount it, then verify that LABEL="backup"

$ sudo e2label /dev/sdd1 backup
$ sudo blkid /dev/sdd1
/dev/sdd1: LABEL="backup" UUID="8e33672c-1234-49de-abcd-6fd841372db6" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="66bc7da7-c9c3-4342-8073-ea4bfe7e00a7"

Copy the autobackup software onto this server

$ git clone https://github.com/bablokb/autobackup-service

Install the autobackup software

$ cd autobackup-service/
$ sudo ./tools/install

RedHat Changes

Lines: 21, 30, 31

File: ./tools/install

 17 check_packages() {
 18   local p
 19   for p in "$@"; do
 20     echo -en "Checking $p ... " >&2
 21     rpm -qa "$p" 2>/dev/null | grep -q "Status.*ok" || return 0
 22     echo "ok" >&2
 23   done
 24   return 1
 25 }
 26 
 27 install_packages() {
 28   if [ -n "$PACKAGES" ] && check_packages $PACKAGES; then
 29     echo -e "[INFO] installing additional packages" 2>&1
 30     dnf update
 31     dnf -y --no-upgrade install $PACKAGES
 32   fi
 33 }

Edit the autobackup configuration file, assigning LABEL=backup, and other items below:

File: /etc/autobackup.conf

~
# => File: /etc/autobackup.conf <=
# label of backup partition
LABEL=backup

# write messages to syslog
SYSLOG=1

# wait for device to appear (in seconds)
WAIT_FOR_DEVICE=2

# run a backup on every mount (i.e. multiple daily backups)
force_daily=0

# backup-levels - this must match your entries in /etc/rsnapshot.conf,
i.e.
# you must have a corresponding 'retain' or 'interval' entry.
# The autobackup-script will skip empty levels
daily="day"
weekly="week"
monthly="month"
yearly=""
"/etc/autobackup.conf" line 29 of 29 --100%--

Edit the rsnapshot configuration file, be sure to use TABS in the BACKUP POINTS / SCRIPTS section.

File: /etc/rsnapshot.conf

~
# => File /etc/rsnapshot.conf <=
###########################
# SNAPSHOT ROOT DIRECTORY #
###########################
 
# All snapshots will be stored under this root directory.
#
#snapshot_root  /var/cache/rsnapshot/
snapshot_root   /tmp/autobackup/.autobackup/
~ 
~
#########################################
#     BACKUP LEVELS / INTERVALS         #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc.         #
#########################################

retain  day     7
retain  week    4
retain  month   3
#retain year    3

###############################
### BACKUP POINTS / SCRIPTS ###
###############################
 
# LOCALHOST
# backup  /etc/           ./
# backup  /var/backups/   ./
# backup  /usr/local/     ./
# backup  /home           ./
# NOTE: Use tabs!
# LOCALHOST$
backup^I/etc/^I./$
backup^I/var/backups/^I./$
backup^I/usr/local/^I./$
backup^I/home^I^I./$    
~
> "/etc/rsnapshot.conf"

Copy autobackup script from install to your home directory

cp autobackup-service/files/usr/local/sbin/autobackup $HOME/autobackup-service/autobackup.sh

Comment out lines 58 through 61 from "<" to ">" below, to allow running in cron.

autobackup-service normally runs automatically when a USB stick with the proper label is inserted into the machine. Comment out the if statement to allow it to run by cron.

File: $HOME/autobackup-service/autobackup.sh

58,61c60,64
<   if [ "${DEVICE:5:3}" != "$udev_arg" ]; then
<     msg "info: partition with label $LABEL is not on newly plugged device $udev_arg"
<     exit 0
<   fi
---
> # Don -> do not check, as we are screduling through cron
> #  if [ "${DEVICE:5:3}" != "$udev_arg" ]; then
> #    msg "info: partition with label $LABEL is not on newly plugged device $udev_arg"
> #    exit 0
> #  fi

Schedule in /etc/cron.d (change your home directory):

File: /etc/cron.d/autobackup-daily

# This is a cron file for autobackup/rsnapshot.
# 0 */4		* * *		root	/usr/bin/rsnapshot alpha
# 30 3   	* * *		root	/usr/bin/rsnapshot beta
# 0  3   	* * 1		root    /usr/bin/rsnapshot gamma
# 30 2   	1 * *		root	/usr/bin/rsnapshot delta
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO="don@example.com"
# m h  dom mon dow user  command
55 12  *   *   *   root  /home/don/autobackup-service/autobackup.sh

Log entries will be in the syslog

$ sudo grep autobackup.sh /var/log/syslog
Jan 23 12:51:11 box autobackup.sh: info: LABEL           = backup
Jan 23 12:51:11 box autobackup.sh: info: WAIT_FOR_DEVICE = 2
Jan 23 12:51:11 box autobackup.sh: info: force_daily     = 0
Jan 23 12:51:11 box autobackup.sh: info: yearly          = 
Jan 23 12:51:11 box autobackup.sh: info: monthly         = month
Jan 23 12:51:11 box autobackup.sh: info: weekly          = week
Jan 23 12:51:11 box autobackup.sh: info: daily           = day
Jan 23 12:51:13 box autobackup.sh: info: checking: 
Jan 23 12:51:13 box autobackup.sh: info: mount-directory: /tmp/autobackup
Jan 23 12:51:13 box autobackup.sh: info: current year:  2021
Jan 23 12:51:13 box autobackup.sh: info: current month: 01
Jan 23 12:51:13 box autobackup.sh: info: current week:  03
Jan 23 12:51:13 box autobackup.sh: info: current day:   023
Jan 23 12:51:13 box autobackup.sh: info: starting backup for interval: month (last backup: 0)
Jan 23 12:51:13 box autobackup.sh: info: starting backup for interval: week (last backup: 0)
Jan 23 12:51:13 box autobackup.sh: info: starting backup for interval: day (last backup: 0)
Jan 23 12:51:15 box autobackup.sh: info: umounting /dev/sda1

Automatic Backup of PostgreSQL Database

Place a script in /etc/cron.daily and it will be run once a day, using the root account.

cron.daily

To check the times look here:

$ grep run-parts /etc/crontab
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

# cat /etc/anacrontab 
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days   delay in minutes   job-identifier   command
1	5	cron.daily		nice run-parts /etc/cron.daily
7	25	cron.weekly		nice run-parts /etc/cron.weekly
@monthly 45	cron.monthly		nice run-parts /etc/cron.monthly

So our daily runs start at 6:25am every day.

Backing up a PostrgreSQL database can be done while everything is up and running with the following script. Backups are stored in /data/backups.

File: /etc/cron.daily/backup_nextcloud

#!/bin/bash
LOGFILE=/var/log/backup_db.log
ID=$(id -un)
if [ ${ID} != "root" ]; then
  echo "Must run as root, try sudo"
  exit 1
fi
#
echo $(date) ${0} >> $LOGFILE

umask 027
export DATA=/data/backups
if cd ${DATA}; then
  # Postgres
  #/usr/bin/pg_dump -c nextcloud > $DATA/nextcloud.db.$(date +%j) </dev/null
  sudo -u postgres /usr/bin/pg_dump -c nextcloud > $DATA/nextcloud.db </dev/null
  
  date >> $LOGFILE
  sync
  sync
  sync
  sync
  savelog -c 7 nextcloud.db >>$LOGFILE 2>&1
fi

Rsync - Remote File Synchronization

Rsync is a good way to keep a daily backup as it only copies changed files to the destination. Make sure you use a seperate disk and preferrably seperate system, as rsync works great over the network.

The PostgreSQL backup above should be sent off to another system using this method. Another rsync script should be called by PostgreSQL backup to do the database network backup. Just copy this one, change the directories, and call it at the end of the database backup.

Schedule

This cron entry will run at 8:40am every day by user root.

File: /etc/cron.d/rsync

# This is a cron file for rsync to NAS 
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO="don@example.com"
# m h  dom mon dow user  command
40 8  *   *   *   root  /mnt/raid1/rsync.sh noask

Script

This script will backup the local directory, /mnt/raid1/data, to a remote system (IP Address 192.168.1.2). The files on the remote system will be at /mnt/vol09/backups. The first run will copy everything, all next runs will only copy changed files. Any files deleted on the source will also be deleted on the destination.

To schedule in cron the parameter 'noask' is used, as shown above. Otherwise there is a prompt for y/n.

The last run's history is in log file /mnt/raid1/rsync.log.

File: /mnt/raid1/rsync.sh

#!/bin/bash
DIR=/mnt/raid1
LOG=${DIR}/rsync.log
cd ${DIR}
date >${LOG}
ASK=${1}
if [ -z ${ASK} ]; then
	echo "Asking"
fi
#
if [ -z ${ASK} ]; then
  echo -n "Copy data? y/n: "
  read askme
  if [[ $askme =~ ^[Yy]$ ]]; then
    rsync -avzz --ignore-errors --progress --delete ${DIR}/data root@192.168.1.2:/mnt/vol09/backups/ |tee -a ${LOG}
  else
    echo "Sync of data skipped"
    echo ". . ."
  fi
else
  rsync -avzz --ignore-errors --progress --delete ${DIR}/data root@192.168.1.2:/mnt/vol09/backups/ |tee -a ${LOG}
fi
#
date >>${LOG}

Logwatch - Daily Alert of Logging Activity

Logwatch is a customizable, pluggable log-monitoring system. It will go through your logs for a given period of time and make a report in the areas that you wish with the detail that you wish. Logwatch is being used for Linux and many types of UNIX.

Installation

Debian:

$ sudo apt-get install logwatch

Redhat:

$ sudo dnf install logwatch

Schedule

File: /etc/cron.daily/00logwatch

#!/bin/bash

#Check if removed-but-not-purged
test -x /usr/share/logwatch/scripts/logwatch.pl || exit 0

#execute
#/usr/sbin/logwatch --output mail
/usr/sbin/logwatch --mailto don@example.com

#Note: It's possible to force the recipient in above command
#Just pass --mailto address@a.com instead of --output mail

Add services

You can add iptables summary on the daily report. It shows which IP addresses have been blocked by UFW.

$ sudo cp /usr/share/logwatch/default.conf/services/iptables.conf /etc/logwatch/conf/services/

You may need to add syslog on Ubuntu servers

File: /etc/logwatch/conf/services/iptables.conf

~
# Which logfile group...
LogFile = syslog
~

Logcheck - mails anomalies in the system logfiles to the admin

The logcheck program helps spot problems and security violations in your logfiles automatically and will send the results to you periodically in an e-mail. By default logcheck runs as an hourly cronjob just off the hour and after every reboot.

Installation

$ sudo apt-get install logcheck

$ sudo dnf install epel-release 'dnf-command(copr)'
$ sudo dnf copr enable brianjmurrell/epel-8
$ sudo dnf install logcheck
$ sudo setfacl -R -m u:logcheck:rx /var/log/secure*
$ sudo setfacl -R -m u:logcheck:rx /var/log/messages*
$ sudo dnf copr disable brianjmurrell/epel-8

Schedule

Normally the package installation will schedule a cron job for you. Check it here:

File: /etc/cron.d/logcheck

# Cron job runs at 2 minutes past every hour
# /etc/cron.d/logcheck: crontab entries for the logcheck package

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

@reboot         logcheck    if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R; fi
2 * * * *       logcheck    if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi

# EOF

Change email destination

Change SENDMAILTO variable to point to your email.

File: /etc/logcheck/logcheck.conf

~
# Controls the address mail goes to:
# *NOTE* the script does not set a default value for this variable!
# Should be set to an offsite "emailaddress@some.domain.tld"

#SENDMAILTO="logcheck"
SENDMAILTO="don@example.com"
~

Sysstat - Gather System Usage Statistics

The sysstat[1] package contains various utilities, common to many commercial Unixes, to monitor system performance and usage activity:

  • iostat reports CPU statistics and input/output statistics for block devices and partitions.
  • mpstat reports individual or combined processor related statistics.
  • pidstat reports statistics for Linux tasks (processes) : I/O, CPU, memory, etc.
  • tapestat reports statistics for tape drives connected to the system.
  • cifsiostat reports CIFS statistics.

Sysstat also contains tools you can schedule via cron or systemd to collect and historize performance and activity data:

  • sar collects, reports and saves system activity information (see below a list of metrics collected by sar).
  • sadc is the system activity data collector, used as a backend for sar.
  • sa1 collects and stores binary data in the system activity daily data file. It is a front end to sadc designed to be run from cron or systemd.
  • sa2 writes a summarized daily activity report. It is a front end to sar designed to be run from cron or systemd.
  • sadf displays data collected by sar in multiple formats (CSV, XML, JSON, etc.) and can be used for data exchange with other programs. This command can also be used to draw graphs for the various activities collected by sar using SVG (Scalable Vector Graphics) format.

Default sampling interval is 10 minutes but this can be changed of course (it can be as small as 1 second).

Redhat Cockpit uses pmlogger.service [2] from systemd. Install from Cockpit's Overview, Metrics and history.

RedHat pmstat [3]

$ pmstat
@ Mon Jun 12 10:02:01 2023
 loadavg                      memory      swap        io    system         cpu
   1 min   swpd   free   buff  cache   pi   po   bi   bo   in   cs  us  sy  id
    0.00 116224 231636   3284 13116m    0    0    0   17  348  387   0   0 100
    0.00 116224 233328   3284 13116m    0    0    0    0  339  383   0   0 100
    0.00 116224 228704   3284 13116m    0    0    0    0  333  358   0   0 100
    0.00 116224 227192   3284 13116m    0    0    0    6  493  548   0   0  99
^C
$ pmstat -a /var/log/pcp/pmlogger/bob.example.com/20230610.0.xz -t 2hour -A 1hour -z
Note: timezone set to local timezone of host "bob.example.com" from archive

@ Sat Jun 10 01:00:00 2023
 loadavg                      memory      swap        io    system         cpu
   1 min   swpd   free   buff  cache   pi   po   bi   bo   in   cs  us  sy  id
    0.08   2048  7646m   6440  6591m    0    0    0    3  198  237   0   0 100
    0.08   2048  7650m   6440  6596m    0    0    0    3  202  237   0   0 100
    0.06   2048  7643m   6440  6600m    0    0    0    3  204  236   0   0 100
    0.00   2048  7597m   6440  6624m    0    0    2   27  219  261   0   0 100
    0.09   2048  7609m   6440  6629m    0    0    0    3  215  259   0   0 100
    0.03   2048  7593m   6440  6633m    0    0    0    3  220  261   0   0 100
    0.00   2048  7585m   6440  6638m    0    0    0    4  223  263   0   0 100
    0.01      0 14402m   6740 495508    ?    ?    ?    ?    ?    ?   ?   ?   ?
    0.00      0 14268m   6740 630344    ?    ?    ?    ?    ?    ?   ?   ?   ?
    0.15      0 14272m   6740 634764    0    0    0    2  162  151   0   0 100
    0.13      0 14266m   6740 639188    0    0    0    2  164  152   0   0 100
 pmFetchGroup: End of PCP archive log

Reference: 1 https://github.com/sysstat/sysstat 2 https://cockpit-project.org/guide/latest/feature-pcp.html 3 https://pcp.readthedocs.io/en/latest/UAG/MonitoringSystemPerformance.html#the-pmstat-command

Installation

$ sudo apt-get install sysstat

$ sudo dnf install sysstat

Configuration

It should configure itself, but just in case:

$ sudo dpkg-reconfigure sysstat
Replacing config file /etc/default/sysstat with new version

$ vi /etc/sysconfig/sysstat
$ sudo systemctl enable --now sysstat

The history files are kept here:

$ ls /var/log/sysstat/
sa07

$ ls /var/log/sa/
sa07

The timer is in the systemd configuration file. OnCalendar defines the interval. In this case data is collected every ten minutes. WantedBy defines that the timer should be active when the sysstat.service is running.

Use systemctl edit sysstat-collect.timer [1] to edit this file. It will automatically create an override file in the right place [2] and enable it for you, and preserve the change over release updates.

File: /usr/lib/systemd/system/sysstat-collect.timer

# /lib/systemd/system/sysstat-collect.timer
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-12.5.2 systemd unit file:
#        Activates activity collector every 10 minutes

[Unit]
Description=Run system activity accounting tool every 10 minutes

[Timer]
OnCalendar=*:00/10

[Install]
WantedBy=sysstat.service
  1. https://www.catalyst2.com/knowledgebase/server-management/how-to-install-configure-sysstat/
  2. Systemd edit override example changing the interval from 10 minutes to 5:
# ls -lrt /etc/systemd/system/sysstat-collect.timer.d/
total 4
-rw-r--r--. 1 root root 27 Feb 19 09:38 override.conf
# more /etc/systemd/system/sysstat-collect.timer.d/override.conf 
[Timer]
OnCalendar=*:00/05

Past Statistics Report

Report on system statistics over the last few days.

File: sar.sh

#!/bin/bash
#################################
# Files are here:
# ls -l /var/log/sysstat/
#  -rw-r--r-- 1 root root 49064 Feb  9 16:35 sa09
#
# Report on some other day:
#  sar -u 2 3 -f /var/log/sysstat/sa15
#
# Output to file:
#  sar -u 2 3 -o /tmp/logfile
#################################
echo "Disk"
sar -d 2 3
echo "Network"
sar -n DEV 2 3
echo "CPU"
sar -u 2 3
sar -P ALL -u 2 3
echo "Memory"
sar -r 2 3
echo "Paging"
sar -B 2 3
echo "Swap"
sar -S 2 3
echo "Load"
sar -q 2 3

Sample Runs

Memory

$ sar -r
Linux 5.10.120-ti-arm64-r64 (app.example.com) 	11/07/2022 	_aarch64_	(2 CPU)

02:09:36 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
02:10:01 PM    872556   2467364   1037140     27.37    162024   1529680   3898804     66.24    999112   1645384       536
Average:       872556   2467364   1037140     27.37    162024   1529680   3898804     66.24    999112   1645384       536

IO

$ sar -b
Linux 5.10.120-ti-arm64-r64 (app.example.com) 	11/07/2022 	_aarch64_	(2 CPU)

02:09:36 PM       tps      rtps      wtps      dtps   bread/s   bwrtn/s   bdscd/s
02:10:01 PM      6.63      0.16      6.47      0.00      2.25    312.46      0.00
Average:         6.63      0.16      6.47      0.00      2.25    312.46      0.00

Network

$ sar -n DEV
Linux 5.10.120-ti-arm64-r64 (app.example.com) 	11/07/2022 	_aarch64_	(2 CPU)

02:09:36 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
02:10:01 PM        lo      3.94      3.94      1.66      1.66      0.00      0.00      0.00      0.00
02:10:01 PM      eth0      4.10      5.27      0.37      1.72      0.00      0.00      0.00      0.00
02:10:01 PM      usb0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01 PM      usb1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01 PM   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:           lo      3.94      3.94      1.66      1.66      0.00      0.00      0.00      0.00
Average:         eth0      4.10      5.27      0.37      1.72      0.00      0.00      0.00      0.00
Average:         usb0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         usb1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Load and Run Queue

$ sar -q
Linux 5.10.120-ti-arm64-r64 (app.example.com) 	11/07/2022 	_aarch64_	(2 CPU)

02:09:36 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
02:10:01 PM         3       531      0.74      0.61      0.51         0
Average:            3       531      0.74      0.61      0.51         0

S.M.A.R.T. Disk Monitoring

Monitor and notify disk health using smartmontools, and email any notifications.

Install software:

$ sudo apt-get install smartmontools

$ sudo dnf install smartmontools

Configure

Add Long monitoring test for Sunday (/dev/sda through /dev/sdX) and comment out DEVICESCAN:

File: /etc/smartd.conf

File: /etc/smartmontools/smartd.conf

~
# Don - 5-Nov-2021
#   -a      Default: equivalent to -H -f -t -l error -l selftest -C 197 -U 198
#   -d TYPE Set the device type: ata, scsi, marvell, removable, 3ware,N, hpt,L/M/N
#   -n MODE No check. MODE is one of: never, sleep, standby, idle
#   -s REGE Start self-test when type/date matches regular expression (see man page)
#           T/MM/DD/d/HH 
#           ^ ^  ^  ^ ^
#           | |  |  | + 24 Hour
#           | |  |  +-- Day of week, 1(monday) through 7(sunday)
#           | |  +----- Day of month, 1 ~ 31
#           | +-------- Month of year, 01 (January) to 12 (December)
#           +---------- T is the type of test that should be run, options are:
#
#                       L for long self-test
#                       S for short self-test
#                       C for conveyance test
#                       O for an Offline immediate Test
#
#   -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit
#   -m ADD  Send warning email to ADD for -H, -l error, -l selftest, and -f
#/dev/nvme0 -a -n never -W 2,30,40 -m don@example.com
# Start long tests on Sunday 9am and short
#  self-tests every night at 2am and send errors to me
#/dev/sda   -a -n never -s (L/../../7/09|S/../.././02) -W 2,30,40 -m don@example.com -M test
/dev/sda   -a -n never -s (L/../../7/09|S/../.././02) -W 2,42,50 -m don@example.com -M diminishing
#/dev/sdb   -a -n never -s (L/../../7/09|S/../.././02) -W 2,30,40 -m don@example.com
# Don - 5-Nov-2021
~
#DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
~

Restart

Restart smartd daemon to pick up configuration changes

$ sudo systemctl restart smartd

Monitoring script for testing and reporting.

Change the DEV below and see if your disks support SMART monitoring.

#!/bin/bash
DEV="/dev/sda"
# Info
sudo smartctl -i "${DEV}"
# Show
sudo smartctl -P show "${DEV}"
# turn smart on/off
#sudo smartctl -s on "${DEV}"
# Errors?
sudo smartctl -l error "${DEV}"
# Health Check
sudo smartctl -Hc "${DEV}"
# Selftest Log
sudo smartctl -l selftest "${DEV}"
# Attributes
#  Problems if...
#    Reallocated_Sector_Ct > 0
#    Current_Pending_Sector > 0
sudo smartctl -A "${DEV}"
#
#.... T E S T S ....
# -> short ... couple of minutes
# sudo smartctl -t short /dev/sda
# -> long ... one hour
# sudo smartctl -t long /dev/sda
# -> Look at test results
# sudo smartctl -a /dev/sda
# 
#.... R E P O R T ....
sudo smartctl --attributes --log=selftest   "${DEV}"
#
# - Get the temprature
sudo hddtemp /dev/sda

smartd database

The history of each smartd monitored disk is located here:

$ ls /var/lib/smartmontools/
drivedb						       smartd.Samsung_SSD_980_1TB-S64ANS0T408956M.nvme.state~  smartd.Samsung_SSD_980_1TB-S64ANS0T418940T.nvme.state~
smartd.Samsung_SSD_980_1TB-S64ANS0T408956M.nvme.state  smartd.Samsung_SSD_980_1TB-S64ANS0T418940T.nvme.state

If you replace the disks and get error reports, you can remove the files and new ones will be created

$ sudo rm /var/lib/smartmontools/smartd.Samsung_SSD_980_1TB-S64ANS0T4*

Run smartctl for each disk:

sudo smartctl -i /dev/sda
sudo smartctl -i /dev/sdb
...

Then check the history data files. New ones should show up.

$ ls /var/lib/smartmontools/
attrlog.CT1000BX500SSD1-2251E695AE97.ata.csv  smartd.CT1000BX500SSD1-2251E695AE97.ata.state   smartd.CT1000BX500SSD1-2251E695AE9E.ata.state~
attrlog.CT1000BX500SSD1-2251E695AE9E.ata.csv  smartd.CT1000BX500SSD1-2251E695AE97.ata.state~  smartd.Samsung_SSD_980_1TB-S64ANS0T408956M.nvme.state
drivedb					      smartd.CT1000BX500SSD1-2251E695AE9E.ata.state   smartd.Samsung_SSD_980_1TB-S64ANS0T418940T.nvme.state

Login Notices to Users - motd/issue/issue.net

You can customize the Message of the Day (motd), and infomation displayed when loggin in to the system.

Installation

$ sudo apt-get install cowsay fortune

$ sudo dnf install cowsay fortune-mod

Configuration

Update the message (/etc/motd) every hour.

File: /etc/cron.hourly/motd

#!/bin/bash
/usr/games/cowsay $(/usr/games/fortune) > /etc/motd

#!/bin/bash
/bin/cowsay $(/bin/fortune) > /etc/motd

Verify

File: /etc/motd

 _________________________________________
/ Your mind is the part of you that says, \
| "Why'n'tcha eat that piece of cake?"    |
| ... and then, twenty minutes later,     |
| says, "Y'know, if I were you, I         |
| wouldn't have done that!" -- Steven and |
\ Ondrea Levine                           /
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

File: /etc/issue

Look out!

File: /etc/issue.net

Looke out!

Example login

% ssh don@example         
Looke out!
 _________________________________________
/ Individuality My words are easy to      \
| understand And my actions are easy to   |
| perform Yet no other can understand or  |
| perform them. My words have meaning; my |
| actions have reason; Yet these cannot   |
| be known and I cannot be known. We are  |
| each unique, and therefore valuable;    |
| Though the sage wears coarse clothes,   |
| his heart is jade. -- Lao Tse, "Tao Te  |
\ Ching"                                  /
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
Last login: Sun Aug 21 10:13:57 2022 from 192.168.0.5

Login notification

Add the following lines to the end of the system bashrc for notification whenever any user logs into the system (with the bash shell).

File: /etc/bash.bashrc

File: /etc/bashrc

~
# Email logins - Don November 2020
echo $(who am i) ' just logged on ' $(hostname) ' ' $(date) $(who) | mail -s "Login on" don@example.com

Continue

Now that you have set up your new server, consider giving it an internet name with DNS.

Proceed in the order presented, some things are depending on prior setups.

Book Last Updated: 29-March-2024


Set up a New Server - Linux in the House - https://linux-in-the-house.org Creative Commons License