If you run a dedicated server, you already know the power and flexibility it gives you. Full root access, guaranteed resources, complete control over your environment, it's the gold standard for businesses that need raw performance and reliability. But that same openness is exactly what makes a dedicated server a high-value target for attackers.
Every day, thousands of brute force attacks and automated port scanning bots sweep the internet looking for misconfigured or unprotected servers. Without proper server hardening, even a freshly provisioned dedicated server can be compromised within hours of going live.
This guide walks you through proven, actionable techniques to protect your dedicated server, whether you're running a web application, a game server, or a mission-critical business platform. You'll learn not just what to do, but why each layer of protection matters.
At COLO BIRD, we believe server security isn't optional; it's the foundation of every high-performance dedicated hosting environment.
Server hardening is the process of reducing a system's attack surface by disabling unnecessary services, enforcing strict access controls, and configuring security tools that detect and block threats. On a dedicated server, you're responsible for every layer of security, from the OS to the application level.
Unlike shared hosting, where the provider manages much of the security infrastructure, a dedicated server puts the security burden on you (or your managed hosting provider). This is both a freedom and a responsibility.
The two most common attack vectors targeting dedicated servers are:
Brute force attacks: Automated scripts that try thousands of username/password combinations per minute to gain unauthorized SSH, RDP, or admin panel access
Port scanning: Reconnaissance techniques used by attackers to map open ports, identify running services, and find exploitable vulnerabilities before launching a targeted attack
Hardening your server against both of these is the first line of defense in any dedicated server security strategy.
Before applying security controls, you need to know exactly what your server is exposing to the internet. Attackers use tools like Nmap to discover this information, and so should you.
Run a Baseline Port Scan on Your Own Server
nmap -sV -sC -p- your.server.ip.address
This command reveals every open port, running service, and version number that an attacker can see. Any port or service that isn't essential to your workload is a potential entry point.
Key principle: Every open port that you don't actively use is a door you forgot to lock.
Identify Unnecessary Running Services
ss -tulnp # List all listening ports and the processes behind them
systemctl list-units --type=service --state=running
Disable any service you don't need. On a dedicated web server, for example, you likely don't need Bluetooth support, NFS, or CUPS running.
SSH (Secure Shell) is the primary management interface for most Linux dedicated servers, and the single most attacked service on any internet-facing machine. Hardening SSH should be your first priority.
2.1 Change the Default SSH Port
The default SSH port is 22, and it's bombarded by automated bots 24/7. Changing it to a non-standard high port (e.g., between 49152 - 65535) dramatically reduces automated login attempts. While not a security measure on its own, it's a simple, effective way to reduce noise.
Edit your SSH configuration:
sudo nano /etc/ssh/sshd_config
Port 47291 # Choose any non-standard high port
2.2 Disable Root Login Over SSH
Never allow direct root login over SSH. This forces attackers to guess both a username and a password, and it eliminates the risk of root being directly compromised.
PermitRootLogin no
2.3 Enforce SSH Key-Based Authentication
This is the single most effective defense against brute force attacks on SSH. Password authentication can be brute-forced. A 4096-bit RSA or Ed25519 key pair cannot.
Generate a key pair on your local machine:
ssh-keygen -t ed25519 -C "[email protected]"
Copy the public key to your server:
ssh-copy-id -p 47291 [email protected]
Then disable password authentication entirely:
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no
PubkeyAuthentication yes
Restart SSH after every config change:
sudo systemctl restart sshd
2.4 Limit SSH Access to Specific Users and IPs
Only allow SSH connections from known IP addresses or trusted users. This is especially powerful for dedicated server environments where admin access comes from a fixed office or VPN IP.
AllowUsers youruser
AllowGroups sshusers
For IP-based restriction, use your firewall (covered in Section 4).
2.5 Set SSH Timeout and Login Limits
Reduce the window an attacker has during a connection attempt:
LoginGraceTime 30
MaxAuthTries 3
MaxSessions 5
ClientAliveInterval 300
ClientAliveCountMax 2
Fail2Ban is one of the most important security tools for any dedicated server. It monitors log files in real time, detects patterns of failed login attempts, and automatically bans the offending IP addresses using your firewall.
3.1 Install Fail2Ban
# Ubuntu/Debian
sudo apt update && sudo apt install fail2ban -y
# CentOS/AlmaLinux/RHEL
sudo dnf install fail2ban -y
3.2 Configure a Local Jail
Never edit the main jail.conf directly, it gets overwritten on updates. Create a local override:
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local
Recommended base configuration for SSH protection:
[DEFAULT]
bantime = 3600 ; Ban IPs for 1 hour (increase for repeat offenders)
findtime = 600 ; Look for failures within 10 minutes
maxretry = 5 ; Ban after 5 failures
[sshd]
enabled = true
port = 47291 ; Your custom SSH port
logpath = %(sshd_log)s
maxretry = 3
bantime = 86400 ; 24-hour ban for SSH — be aggressive here
3.3 Protect Other Services with Fail2Ban
Fail2Ban isn't limited to SSH. You can also protect:
Web servers: [nginx-http-auth], [apache-auth]
Mail servers: [postfix], [dovecot]
FTP services: [vsftpd], [proftpd]
Control panels: Custom jails for cPanel, Plesk, or WHM login pages
3.4 Whitelist Your Own IPs
Always whitelist your management IPs before activating aggressive ban rules:
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 YOUR.OFFICE.IP.HERE
3.5 Monitor and Manage Fail2Ban
sudo fail2ban-client status # Overview of all active jails
sudo fail2ban-client status sshd # SSH jail details
sudo fail2ban-client set sshd unbanip BANNED.IP.ADDRESS # Unban a legitimate IP
A properly configured firewall is essential for dedicated server protection. It controls what traffic is even allowed to reach your services, stopping port scanning and unauthorized access attempts before they reach your applications.
4.1 Use UFW (Uncomplicated Firewall) on Ubuntu/Debian
UFW provides a simple interface over iptables that's perfect for dedicated server environments:
sudo apt install ufw -y
# Set default policies — deny all incoming, allow all outgoing
sudo ufw default deny incoming
sudo ufw default allow outgoing
# Allow only what you need
sudo ufw allow 47291/tcp # Your custom SSH port
sudo ufw allow 80/tcp # HTTP
sudo ufw allow 443/tcp # HTTPS
# Limit SSH connection rate (additional brute force protection)
sudo ufw limit 47291/tcp
# Enable the firewall
sudo ufw enable
sudo ufw status verbose
4.2 Use firewalld on CentOS/RHEL/AlmaLinux
sudo systemctl enable --now firewalld
# Allow services
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --permanent --add-port=47291/tcp
# Remove the default SSH port if you've changed it
sudo firewall-cmd --permanent --remove-service=ssh
sudo firewall-cmd --reload
4.3 Implement Rate Limiting to Counter Port Scanning
Port scanners work by sending connection requests to many ports rapidly. Rate limiting detects and throttles this behavior at the firewall level.
Using iptables directly for rate limiting:
# Limit new connections to 10 per minute per IP across all ports
sudo iptables -A INPUT -m state --state NEW -m recent --set --name PORT_SCAN
sudo iptables -A INPUT -m state --state NEW -m recent --update --seconds 60 --hitcount 10 --name PORT_SCAN -j DROP
4.4 Block Known Malicious IP Ranges
Use threat intelligence lists to proactively block IP ranges known to host botnets, Tor exit nodes, and scanning services:
# Install ipset for efficient large blocklists
sudo apt install ipset -y
# Example: Import and apply a known bad-IP blocklist
sudo ipset create blacklist hash:ip
sudo ipset add blacklist MALICIOUS.IP.HERE
sudo iptables -I INPUT -m set --match-set blacklist src -j DROP
Tools like FireHOL or Blocklist.de provide regularly updated IP reputation feeds that integrate well with iptables/ipset on dedicated servers.
Port knocking is a technique where all ports appear closed by default. A service only opens temporarily when a client sends a specific sequence of connection attempts to predefined ports, in the right order and within a time window. It's one of the most effective ways to hide SSH and other management ports from automated scanners.
Install and Configure Knockd
sudo apt install knockd -y
sudo nano /etc/knockd.conf
[options]
UseSyslog
[openSSH]
sequence = 7000,8000,9000 ; Knock these ports in order
seq_timeout = 10 ; Must complete within 10 seconds
command = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 47291 -j ACCEPT
tcpflags = syn
[closeSSH]
sequence = 9000,8000,7000 ; Reverse sequence to close
seq_timeout = 10
command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 47291 -j ACCEPT
tcpflags = syn
Enable knockd and set it to start on boot:
sudo nano /etc/default/knockd
# Set: START_KNOCKD=1 and KNOCKD_OPTS="-i eth0"
sudo systemctl enable --now knockd
To connect from your client machine:
knock your.server.ip 7000 8000 9000 # Open the port
ssh -p 47291 [email protected] # Connect while it's open
knock your.server.ip 9000 8000 7000 # Close it again
With port knocking active, your SSH port is completely invisible to any scanner; it won't even show as "closed." It simply won't exist.
Many of the most effective dedicated server hardening techniques happen at the kernel level. The Linux sysctl interface lets you tune networking parameters that directly defend against port scanning, IP spoofing, SYN flood attacks, and other common network-level exploits.
Edit /etc/sysctl.conf or create a dedicated file at /etc/sysctl.d/99-security.conf:
sudo nano /etc/sysctl.d/99-security.conf
Recommended kernel security parameters:
# Disable IP source routing (prevents IP spoofing)
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
# Ignore ICMP redirects (prevents routing table manipulation)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
# Enable reverse path filtering (prevents spoofed packets)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Enable SYN cookies to defend against SYN flood attacks
net.ipv4.tcp_syncookies = 1
# Increase the size of the SYN backlog
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5
# Log suspicious packets (Martians — packets with impossible source addresses)
net.ipv4.conf.all.log_martians = 1
# Ignore ICMP broadcast requests (Smurf attack protection)
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Ignore bogus error responses
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Disable IPv6 if not in use
net.ipv6.conf.all.disable_ipv6 = 1
Apply changes immediately:
sudo sysctl --system
While a firewall and Fail2Ban react to known attack patterns, an Intrusion Detection System (IDS) provides deeper visibility, analyzing traffic content, file system changes, and system behavior to catch sophisticated attacks that bypass simpler defenses.
7.1 AIDE - File Integrity Monitoring
AIDE (Advanced Intrusion Detection Environment) creates a cryptographic baseline of your file system and alerts you when files are added, modified, or deleted, a critical signal of compromise.
sudo apt install aide -y
sudo aideinit # Build the initial database
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# Schedule daily integrity checks via cron
echo "0 3 * * * root /usr/bin/aide --check" | sudo tee /etc/cron.d/aide-check
7.2 Wazuh or OSSEC - Host-Based Intrusion Detection
Wazuh (the maintained successor to OSSEC) is a comprehensive, open-source security platform that provides:
Real-time log analysis and threat correlation
File integrity monitoring
Vulnerability detection
Active response (automated blocking similar to Fail2Ban, but deeper)
Compliance monitoring (PCI DSS, HIPAA, CIS benchmarks)
For dedicated server environments running production workloads, Wazuh offers enterprise-grade protection at no license cost.
7.3 Suricata - Network-Level Intrusion Detection
Suricata analyzes network traffic against a database of known attack signatures. Unlike host-based tools, it detects threats at the network level, including port scan patterns, exploit attempts, and command-and-control traffic.
sudo apt install suricata -y
sudo suricata-update # Download latest threat intelligence rules
sudo systemctl enable --now suricata
Even with SSH keys enforced, adding two-factor authentication (2FA) creates a second verification layer that stops compromised key attacks in their tracks.
8.1 Enable Google Authenticator for SSH
sudo apt install libpam-google-authenticator -y
google-authenticator # Run as the user you want to protect
Configure PAM to require 2FA:
sudo nano /etc/pam.d/sshd
Add at the top:
auth required pam_google_authenticator.so
Update SSH config:
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
8.2 Implement the Principle of Least Privilege
Every user and process on your dedicated server should have only the permissions needed to do their job, nothing more.
Create separate system users for each application (web server, database, mail)
Never run web applications as root
Use sudo with granular permissions rather than sharing root credentials
Audit sudoers configuration regularly: sudo visudo
8.3 Lock Down /etc/passwd and Disable Unused System Accounts
# List all accounts with login shells
grep -v '/sbin/nologin\|/bin/false' /etc/passwd
# Disable accounts that don't need login access
sudo usermod -s /sbin/nologin unused_account
sudo passwd -l unused_account
You cannot defend what you cannot see. Comprehensive logging and real-time log analysis are non-negotiable components of a hardened dedicated server environment.
9.1 Centralize and Protect Log Files
# Key log files to monitor on a dedicated server
/var/log/auth.log # Authentication and authorization events (Debian/Ubuntu)
/var/log/secure # Authentication events (RHEL/CentOS)
/var/log/syslog # General system events
/var/log/fail2ban.log # Fail2Ban ban/unban activity
/var/log/nginx/access.log # Web server access
/var/log/nginx/error.log # Web server errors
Use logrotate to manage log size and enable log shipping to an external SIEM or log management service so that compromised servers can't have their evidence deleted.
9.2 Enable auditd for System Call Auditing
The Linux Audit Daemon records system calls at the kernel level, capturing file access, privilege escalation, and user actions that no application-level logging can see.
sudo apt install auditd -y
sudo systemctl enable --now auditd
# Watch for changes to the sudoers file
sudo auditctl -w /etc/sudoers -p wa -k sudoers_changes
# Watch for failed login attempts beyond what auth.log captures
sudo auditctl -w /var/log/lastlog -p wa -k login_events
9.3 Set Up Real-Time Alerting
Configure email or Slack/webhook alerts for critical security events:
Successful root login from any source
New user account creation
SSH connection from a new country or IP
File integrity violation detected by AIDE
Unusual outbound connections (potential C2 communication)
Server hardening is not a one-time event. It's an ongoing process. The security posture of a dedicated server degrades over time as new vulnerabilities are discovered, software versions age, and configurations drift.
10.1 Automate Security Updates
Unpatched software is the root cause of the majority of successful server compromises. Enable automatic security updates:
# Ubuntu/Debian
sudo apt install unattended-upgrades -y
sudo dpkg-reconfigure --priority=low unattended-upgrades
For production servers, test updates in a staging environment first, but never let security patches languish for more than 48 hours after public disclosure of a critical CVE.
10.2 Run Periodic Vulnerability Scans
Use Lynis, an open-source security auditing tool designed for Linux servers, to get a hardening score and actionable recommendations:
sudo apt install lynis -y
sudo lynis audit system
Lynis will assess everything from SSH configuration to kernel parameters, file permissions, and installed software, and generate a prioritized list of security improvements tailored to your dedicated server environment.
10.3 CIS Benchmarks for Dedicated Servers
The Center for Internet Security (CIS) publishes detailed hardening benchmarks for every major Linux distribution. Measuring your server against the CIS Benchmark for your OS gives you a structured, industry-recognized framework for security configuration.
Use this checklist to audit your dedicated server's hardening status:
SSH Hardening
Default SSH port changed from 22
Root login disabled
Password authentication disabled
SSH key-based authentication is enforced
Login attempt limits configured
2FA enabled for SSH
Firewall and Network
The default-deny firewall policy is active
Only necessary ports open
Connection rate limiting enabled
Known malicious IPs blocked
Port knocking is configured for sensitive services
Kernel sysctl security parameters applied
Brute Force Protection
Fail2Ban is installed and running
SSH jail configured with aggressive ban times
All exposed login services are protected
Admin IPs whitelisted
Intrusion Detection
AIDE or equivalent file integrity monitoring is active
auditd enabled and configured
Log centralization in place
Real-time alerting configured
System Hardening
All non-essential services disabled
Principle of least privilege enforced
Automatic security updates enabled
Lynis audit score reviewed
CIS benchmark compliance checked
A brute force attack tries every possible character combination for a password, while a dictionary attack uses a precompiled list of common passwords, phrases, and leaked credentials. Both can be effectively stopped by disabling password authentication and enforcing SSH key-based login on your dedicated server.
Run automated scans (like Lynis) weekly and review your firewall rules and access logs monthly. After any significant change to your server's software stack, run a full audit before the change goes live in production. At COLO BIRD, we recommend treating security audits as a recurring operational task, not a one-off project.
Yes, measurably so. While it's not a security control by itself (security through obscurity), moving SSH off port 22 eliminates the vast majority of automated bot traffic that scans port 22 exclusively. Combined with key-based auth and Fail2Ban, it's a worthwhile and trivially easy measure.
Leaving SSH on port 22 with root login enabled and password authentication allowed is the single most dangerous default configuration. This combination allows automated tools to brute-force their way into root access with no friction. Fix these three things first, above all else.
Port knocking works with your firewall; the firewall keeps all ports blocked by default, and the knockd daemon listens for the knock sequence and temporarily modifies firewall rules to open a specific port for the requesting IP only. It's complementary to, not a replacement for, a properly configured firewall.
Protecting a dedicated server against brute force attacks and port scanning requires multiple layers of complementary defense. No single tool or technique is sufficient on its own, but when you combine SSH hardening, Fail2Ban, a well-configured firewall, kernel-level protections, intrusion detection, and regular auditing, you create a security posture that stops the overwhelming majority of automated and opportunistic attacks.
The attackers scanning your server are mostly automated, bots running 24/7, looking for the path of least resistance. By systematically removing that path, you don't need to be impenetrable; you just need to be harder to compromise than the next server in the scan queue.
At COLO BIRD, our dedicated server infrastructure is built with security as a foundational layer, not an afterthought. Whether you're managing your own dedicated server or looking for a managed solution where hardening is handled for you, understanding these principles puts you in control of your server's security posture.