Advanced Linux System Administration

Master Professional Linux Server Management

By Garry September 7, 2025 25 min read Linux

Advanced Linux system administration goes far beyond basic command-line skills. This comprehensive guide covers professional-level techniques for monitoring, optimizing, securing, and automating Linux servers in production environments.

🔍 System Monitoring and Analysis

Real-time System Monitoring

# Comprehensive system overview
htop

# Detailed process information
ps aux --sort=-%cpu | head -20

# Real-time disk I/O
iotop

# Network connections and traffic
netstat -tuln
ss -tuln

# System resource usage
vmstat 1 5
iostat -x 1 5

# Memory usage analysis
free -h
cat /proc/meminfo

Performance Bottleneck Identification

# CPU performance analysis
sar -u 1 10

# Memory performance
sar -r 1 10

# Disk I/O performance
sar -d 1 10

# Network performance
sar -n DEV 1 10

# Load average trends
uptime
cat /proc/loadavg

# Find top CPU consuming processes
top -o %CPU

# Find top memory consuming processes
top -o %MEM

⚡ Performance Optimization

Kernel Parameter Tuning

# View current kernel parameters
sysctl -a

# Optimize network performance
echo 'net.core.rmem_max = 16777216' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 16777216' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 65536 16777216' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 16777216' >> /etc/sysctl.conf

# Optimize file system performance
echo 'vm.dirty_ratio = 10' >> /etc/sysctl.conf
echo 'vm.dirty_background_ratio = 5' >> /etc/sysctl.conf
echo 'vm.swappiness = 10' >> /etc/sysctl.conf

# Apply changes
sysctl -p

I/O Scheduling Optimization

# Check current I/O scheduler
cat /sys/block/sda/queue/scheduler

# Set I/O scheduler for SSDs
echo noop > /sys/block/sda/queue/scheduler

# Set I/O scheduler for HDDs
echo deadline > /sys/block/sda/queue/scheduler

# Make permanent in GRUB
# Add elevator=deadline to GRUB_CMDLINE_LINUX in /etc/default/grub
sudo update-grub

CPU Affinity and Process Prioritization

# Set CPU affinity for a process
taskset -c 0,1 command

# Change process priority
nice -n -10 command
renice -10 -p PID

# View CPU affinity
taskset -p PID

# Set CPU governor for performance
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

🛡️ Security Hardening

SSH Security Configuration

# Edit /etc/ssh/sshd_config
Port 2222                          # Change default port
PermitRootLogin no                  # Disable root login
PasswordAuthentication no           # Use key-based auth only
AllowUsers user1 user2              # Restrict users
MaxAuthTries 3                      # Limit auth attempts
ClientAliveInterval 300             # Auto-disconnect idle sessions
ClientAliveCountMax 2

# Restart SSH service
systemctl restart sshd

# Generate SSH key pair
ssh-keygen -t ed25519 -C "your_email@example.com"

# Copy public key to server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

Firewall Configuration

# UFW (Ubuntu/Debian)
ufw enable
ufw default deny incoming
ufw default allow outgoing
ufw allow 2222/tcp
ufw allow 80/tcp
ufw allow 443/tcp

# iptables rules
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 2222 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Save iptables rules
iptables-save > /etc/iptables/rules.v4

File System Security

# Set secure permissions
chmod 700 /root
chmod 755 /home
chmod 644 /etc/passwd
chmod 600 /etc/shadow

# Find files with excessive permissions
find / -type f -perm -4000 2>/dev/null
find / -type f -perm -2000 2>/dev/null

# Secure mount options in /etc/fstab
/dev/sda1 /boot ext4 defaults,nodev,nosuid,noexec 0 2
/dev/sda2 /tmp ext4 defaults,nodev,nosuid,noexec 0 2

# Enable audit logging
systemctl enable auditd
systemctl start auditd

📊 Log Management and Analysis

Centralized Logging with rsyslog

# Configure rsyslog server (/etc/rsyslog.conf)
$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 0.0.0.0

# Template for log file organization
$template DynamicFile,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?DynamicFile

# Configure rsyslog client
*.* @@logserver:514

# Restart rsyslog
systemctl restart rsyslog

Log Rotation and Cleanup

# Configure logrotate (/etc/logrotate.d/myapp)
/var/log/myapp/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 644 root root
    postrotate
        systemctl reload myapp
    endscript
}

# Test logrotate configuration
logrotate -d /etc/logrotate.d/myapp

# Force logrotate
logrotate -f /etc/logrotate.conf

Log Analysis with Command Line Tools

# Analyze Apache access logs
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -10

# Find failed SSH login attempts
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr

# Monitor real-time logs
tail -f /var/log/syslog | grep ERROR

# Search logs with journalctl
journalctl -u sshd -f
journalctl --since "2025-09-07 10:00:00" --until "2025-09-07 11:00:00"

# Analyze disk usage by log files
du -sh /var/log/* | sort -hr

🔄 Automation and Scripting

System Health Monitoring Script

#!/bin/bash
# system_health_check.sh

# Configuration
THRESHOLD_CPU=80
THRESHOLD_MEMORY=90
THRESHOLD_DISK=85
EMAIL="admin@example.com"

# Check CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
if (( $(echo "$CPU_USAGE > $THRESHOLD_CPU" | bc -l) )); then
    echo "HIGH CPU USAGE: $CPU_USAGE%" | mail -s "CPU Alert" $EMAIL
fi

# Check memory usage
MEMORY_USAGE=$(free | grep Mem | awk '{printf("%.2f", $3/$2 * 100.0)}')
if (( $(echo "$MEMORY_USAGE > $THRESHOLD_MEMORY" | bc -l) )); then
    echo "HIGH MEMORY USAGE: $MEMORY_USAGE%" | mail -s "Memory Alert" $EMAIL
fi

# Check disk usage
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt $THRESHOLD_DISK ]; then
    echo "HIGH DISK USAGE: $DISK_USAGE%" | mail -s "Disk Alert" $EMAIL
fi

# Log system metrics
echo "$(date): CPU=$CPU_USAGE%, MEM=$MEMORY_USAGE%, DISK=$DISK_USAGE%" >> /var/log/system_health.log

Automated Backup Script

#!/bin/bash
# backup_system.sh

BACKUP_DIR="/backup"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7

# Create backup directory
mkdir -p $BACKUP_DIR

# System configuration backup
tar -czf $BACKUP_DIR/system_config_$DATE.tar.gz \
    /etc \
    /var/www \
    /home \
    --exclude=/home/*/.*cache* \
    --exclude=/home/*/Downloads

# Database backup (if MySQL/MariaDB is installed)
if systemctl is-active --quiet mysql; then
    mysqldump --all-databases --single-transaction --lock-tables=false \
        | gzip > $BACKUP_DIR/mysql_backup_$DATE.sql.gz
fi

# Remove old backups
find $BACKUP_DIR -type f -mtime +$RETENTION_DAYS -delete

# Log backup completion
echo "$(date): Backup completed successfully" >> /var/log/backup.log

Cron Job Automation

# Edit crontab
crontab -e

# Example cron jobs
# Daily system health check at 6 AM
0 6 * * * /root/scripts/system_health_check.sh

# Weekly backup every Sunday at 2 AM
0 2 * * 0 /root/scripts/backup_system.sh

# Update package cache daily at midnight
0 0 * * * apt update

# Clean temporary files weekly
0 3 * * 0 find /tmp -type f -atime +7 -delete

# Monitor disk space every hour
0 * * * * df -h | mail -s "Disk Space Report" admin@example.com

🌐 Network Administration

Advanced Network Configuration

# Configure static IP with Netplan (Ubuntu)
# /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: no
      addresses:
        - 192.168.1.100/24
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]

# Apply network configuration
netplan apply

# Configure bonding for high availability
# /etc/netplan/01-bond.yaml
network:
  version: 2
  ethernets:
    enp0s3:
      dhcp4: no
    enp0s8:
      dhcp4: no
  bonds:
    bond0:
      interfaces: [enp0s3, enp0s8]
      addresses: [192.168.1.100/24]
      gateway4: 192.168.1.1
      parameters:
        mode: active-backup
        primary: enp0s3

Network Troubleshooting Tools

# Network connectivity testing
ping -c 4 google.com
traceroute google.com
mtr google.com

# Port scanning and testing
nmap -sS -O target_ip
nc -zv target_ip 80

# Bandwidth testing
iperf3 -s                    # Server mode
iperf3 -c server_ip          # Client mode

# DNS troubleshooting
dig google.com
nslookup google.com
host google.com

# Network interface statistics
ip -s link show
cat /proc/net/dev

💾 Storage Management

LVM (Logical Volume Management)

# Create physical volume
pvcreate /dev/sdb1

# Create volume group
vgcreate vg_data /dev/sdb1

# Create logical volume
lvcreate -L 10G -n lv_data vg_data

# Format and mount
mkfs.ext4 /dev/vg_data/lv_data
mount /dev/vg_data/lv_data /data

# Extend logical volume
lvextend -L +5G /dev/vg_data/lv_data
resize2fs /dev/vg_data/lv_data

# View LVM information
pvdisplay
vgdisplay
lvdisplay

RAID Configuration

# Create RAID 1 array
mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

# Check RAID status
cat /proc/mdstat
mdadm --detail /dev/md0

# Save RAID configuration
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

# Monitor RAID array
mdadm --monitor --daemonise /dev/md0

🎯 Conclusion

Advanced Linux system administration requires a deep understanding of system internals, security principles, and automation techniques. The skills covered in this guide form the foundation for managing enterprise-level Linux environments effectively.

Remember: Always test changes in a non-production environment first, maintain comprehensive backups, and document your configurations for future reference.

🚀 Next Steps

Set up a lab environment to practice these techniques
Learn configuration management tools like Ansible or Puppet
Explore containerization with Docker and Kubernetes
Study cloud platforms like AWS, Azure, or GCP
Implement comprehensive monitoring with Prometheus and Grafana