Server compromises happen faster than you think. One misconfigured Docker container exposed my Hetzner server to attackers who installed cryptocurrency mining malware and SSH key hijackers within hours. This article shares the hard lessons I learned from that experience and the security practices that could have prevented it.
I had a PostgreSQL database running in Docker with port 5432 exposed to the internet without proper authentication. The configuration seemed convenient for development, but it was a critical security flaw that attackers discovered quickly.
Attackers scanned for exposed PostgreSQL instances, gained access through the weak configuration, and leveraged this foothold to compromise the entire server. They installed sophisticated malware including cryptocurrency miners that consumed CPU resources and SSH key hijackers that maintained persistent access.
I noticed unusual CPU usage patterns and network traffic. Investigation revealed multiple malicious processes running and unauthorized SSH keys in the authorized_keys file. The server had become part of a botnet mining cryptocurrency for the attackers.
Databases should never be exposed directly to the internet. PostgreSQL, MongoDB, Redis, and MySQL are common targets for automated scanning tools. Attackers continuously scan for these services with default or weak credentials.
Many attacks succeed simply because services use default credentials or weak passwords. Attackers maintain databases of default credentials for common services and try them systematically.
Unpatched software contains known vulnerabilities that attackers exploit. Security patches exist for a reason, and delayed updates give attackers a window of opportunity.
Docker makes deployment easy but can introduce security risks if misconfigured. Exposed ports, running containers as root, and inadequate network isolation are common issues.
Use UFW or iptables to restrict inbound traffic. Only allow necessary ports and restrict access to trusted IP addresses when possible. For databases, never expose them to the public internet.
Configure your firewall before deploying services. Default-deny policies are safer than trying to remember to block ports after deployment.
Disable password authentication and use key-based authentication only. Change the default SSH port to reduce automated scanning. Disable root login and use sudo for administrative tasks instead.
Implement fail2ban to automatically block IP addresses after failed login attempts. This stops brute force attacks before they succeed.
Never expose databases directly to the internet. Use localhost binding or private networks for database access. Implement strong passwords and consider certificate-based authentication for additional security.
For remote database access, use SSH tunnels or VPNs. These add encryption and authentication layers that protect your data.
Don't bind container ports to 0.0.0.0 unless absolutely necessary. Use Docker networks to isolate containers. Run containers as non-root users whenever possible.
Review docker-compose files carefully before deployment. A single exposed port can compromise your entire infrastructure.
Monitor CPU, memory, and network usage for anomalies. Cryptocurrency miners consume significant CPU resources, making them detectable through resource monitoring.
Set up alerts for unusual resource usage patterns. Tools like htop, netdata, or Prometheus can help identify suspicious activity.
Regularly review system logs for unusual authentication attempts, privilege escalations, or unexpected service starts. Centralized logging makes this easier for multiple servers.
Failed SSH login attempts, new user creations, and sudo command usage are important events to monitor.
Use tools like AIDE or Tripwire to detect unauthorized changes to critical system files. Check SSH authorized_keys files regularly for unknown public keys.
In my case, the attackers added their own SSH keys. Regular integrity checks would have caught this immediately.
Monitor outbound connections for unusual destinations or traffic patterns. Cryptocurrency mining often involves connections to mining pools that can be identified through network analysis.
When you discover a compromise, disconnect the server from the network immediately. Take snapshots or backups before making changes. Document everything you observe for later analysis.
Identify the attack vector and all compromised components. Check for backdoors, malicious cron jobs, and unauthorized user accounts. Review all SSH keys and remove unknown ones.
Look for persistence mechanisms like modified system binaries, kernel modules, or startup scripts.
The safest approach is complete reinstallation from clean backups. Cleaning an infected system is difficult because you can never be certain all malware has been removed.
If you must clean without reinstalling, remove all malware, close vulnerabilities, change all credentials, and monitor closely for signs of reinfection.
After reinstallation, restore only data, not executables or configurations from the compromised system. Implement all security measures before reconnecting to the internet.
I completely rebuilt my Hetzner server, implementing proper firewalls, SSH hardening, and Docker network isolation before restoring services.
Separate public-facing services from internal infrastructure. Use private networks for database and internal service communication. This limits the blast radius if one component is compromised.
Grant only the minimum necessary permissions. Application users shouldn't have sudo access. Database users should only access required databases with minimal privileges.
Don't rely on a single security measure. Layer multiple defensive strategies so that if one fails, others still protect the system.
Firewalls, authentication, encryption, monitoring, and regular updates all contribute to a robust security posture.
Configure unattended-upgrades for automatic security patches. This ensures critical updates are applied promptly without manual intervention.
For production systems, test updates in staging first, but automate the deployment process to reduce delays.
Use Ansible, Puppet, or similar tools to enforce consistent security configurations across servers. This prevents configuration drift that can introduce vulnerabilities.
Implement regular vulnerability scanning with tools like OpenVAS or Nessus. Scan both internally and externally to identify weaknesses before attackers do.
Bind to localhost when external access isn't needed. Use specific IP addresses rather than 0.0.0.0 when possible. Document why each port needs to be exposed.
Never put credentials directly in docker-compose files. Use environment variable files that are excluded from version control. Rotate credentials regularly.
Use official images from trusted sources. Scan images for vulnerabilities before deployment. Keep base images updated and rebuild regularly.
Create separate Docker networks for different components. Don't put all containers on the default bridge network. Use internal networks for backend services that don't need internet access.
Conduct security audits quarterly at minimum. Review firewall rules, user accounts, SSH keys, and running services. Remove anything unnecessary.
Establish a regular update schedule. Test updates in non-production environments first. Have rollback plans ready in case updates cause issues.
Backups are useless if they don't work. Regularly test restoration procedures. Store backups securely and separately from production systems.
Document all security configurations and procedures. This helps during incident response and ensures knowledge isn't lost when team members change.
Exposing the PostgreSQL port seemed convenient for development, but the risk far outweighed the benefit. Always use SSH tunnels or VPNs for remote access instead.
Design systems assuming they will be compromised eventually. Implement monitoring and detection so you can respond quickly when it happens.
Security isn't a one-time setup. It requires ongoing maintenance, monitoring, and updates. Automated tools help, but human oversight remains essential.
The cryptocurrency mining attack taught me valuable lessons about server security. What seemed like a minor configuration issue became a complete server compromise requiring full reinstallation.
Implementing proper security measures before deployment is far easier than recovering from a breach. Use firewalls, strong authentication, minimal exposure, and continuous monitoring to protect your infrastructure.
Don't learn these lessons the hard way like I did. Take security seriously from the start, stay vigilant, and remember that attackers are constantly scanning for vulnerable systems. Your server could be next if you're not properly protected.