What to Do After Discovering Your Linux Mail Server Was Compromised

Discovering malware or unauthorized access on a Linux mail server changes the situation immediately.

You are no longer troubleshooting an ordinary service failure.

You are responding to a security incident.

Killing a suspicious process, deleting a malicious file, or removing a bad cron entry may stop what you can currently see. It does not prove that the server is clean.

The attacker may have installed another persistence mechanism, added SSH keys, stolen administrator credentials, copied TLS private keys, modified application files, created users, installed systemd services, altered mail routing, or moved to another server.

This guide explains how to contain, investigate, preserve evidence from, and recover a compromised Linux mail server.

The procedures are based partly on lessons from our compromised Zimbra server, where we encountered malicious processes, executable files in temporary and unusual directories, cron-based persistence, suspicious SSH keys, and evidence of successful SSH authentication.

The principles apply to most Linux mail servers.

Current NIST guidance treats incident response as a cycle involving detection, response, recovery, and continuous improvement. CISA also recommends quarantining compromised hosts, collecting artifacts, reimaging confirmed compromised systems, and issuing new credentials.


1. Do Not Treat Malware Removal as Incident Recovery

Suppose you discover this:

ps aux | grep suspicious-process

You kill it:

kill -9 PID

Then you delete its executable:

rm /path/to/malware

The process disappears.

That does not mean the system is trustworthy again.

An attacker who gained sufficient access may have made changes you have not discovered.

For example, they may have:

Added another cron job
Installed a systemd service
Added an SSH authorized key
Created another user
Changed sudo configuration
Modified shell startup files
Installed a kernel-level rootkit
Modified the mail application
Stolen passwords
Copied API credentials
Copied TLS private keys
Copied DKIM signing keys
Modified mailbox forwarding rules
Installed another payload under a different filename
Moved laterally to another server

This is the main mindset change required during incident response.

You are not asking:

“Did I remove the malware?”

You are asking:

“Can I prove that I understand what the attacker did, what credentials were exposed, what other systems may have been reached, and whether this operating system can still be trusted?”

If the answer is no, rebuilding is usually safer than continuing to clean the existing installation.

CISA’s guidance for confirmed Zimbra compromises specifically recommends collecting and reviewing artifacts, quarantining affected hosts, reimaging compromised hosts, and provisioning new credentials.


2. Lessons From Our Compromised Zimbra Server

Our Zimbra incident demonstrated why visible malware is only one part of a compromise.

We encountered suspicious executables and related files under locations including:

/dev/shm/
/home/SSL/

Suspicious process names included:

idle
javab

We also discovered persistence through a Zimbra user’s crontab. A malicious command was being executed every minute.

The investigation later found suspicious entries in:

/opt/zimbra/.ssh/authorized_keys

and authentication logs showed successful SSH public-key access to the Zimbra account.

This changed the incident significantly.

The problem was no longer:

Malware process -> kill malware -> problem solved

It became:

Initial compromise
       |
       +--> malicious executable
       |
       +--> cron persistence
       |
       +--> SSH persistence
       |
       +--> possible credential exposure
       |
       +--> possible lateral movement
       |
       +--> unknown additional modifications

That is the correct way to think about a compromised mail server.


3. First Priority: Contain the Server

Once you have credible evidence of compromise, limit the attacker’s ability to continue operating.

CISA recommends immediately isolating affected systems and collecting relevant logs, data, and artifacts.

Possible containment methods include disconnecting the server from the network, placing it in an isolated VLAN, restricting communication using upstream firewall rules, or blocking internet access while allowing only controlled incident-response access. CISA specifically recognizes physical disconnection, network-layer isolation, restricted VLANs, and strict firewall controls as valid isolation approaches depending on the situation.

For a mail server, however, containment requires judgment.

Immediately unplugging the server may destroy volatile information that exists only in RAM.

That information can include:

Running malware
Process arguments
Network sessions
Open files
Logged-in sessions
Temporary credentials
Encryption material
Command history still held in memory
Attacker connections

NIST SP 800-86 explains that volatile information can disappear after shutdown or reboot and recommends deciding early whether it needs to be collected.

If an attacker is actively using the machine, containment normally takes priority.

If the attacker appears inactive and you have incident-response capability available, collect important volatile information before shutting the machine down.


4. Do Not Reboot Immediately

One of the first instincts administrators have is:

reboot

Avoid doing this automatically.

A reboot destroys useful volatile evidence.

It can remove:

Running processes
Memory contents
Current network connections
Temporary files in tmpfs
Processes deleted from disk but still running
Active attacker sessions

It can also trigger persistence mechanisms that only execute at boot.

NIST notes that collecting volatile operating-system information is only possible while the affected system remains running.

There are situations where immediate shutdown is justified, particularly when continued operation creates severe business or security risk.

The key is to make the decision deliberately rather than rebooting as a troubleshooting habit.


5. Start an Incident Timeline

Write down what happened from the moment you discovered the issue.

Record timestamps in UTC where possible.

For example:

date -u

Record:

When the issue was discovered
Who discovered it
Symptoms observed
Suspicious filenames
Process IDs
User accounts involved
Source IP addresses
Destination IP addresses
Commands executed by administrators
Files deleted or moved
Firewall changes
Accounts disabled
Credentials rotated
Systems disconnected
Services stopped

Do not rely on memory.

Your own administrative commands change the compromised system.

NIST forensic guidance stresses documenting evidence handling and the actions taken during an investigation.

For serious incidents, preserve a formal chain of custody if legal action, insurance, contractual investigation, or regulatory reporting may become relevant.


6. Capture Volatile Information

Before killing suspicious processes, collect enough information to understand them.

Create an evidence destination. Ideally, use external storage or a trusted remote collection system rather than storing everything on the compromised server.

For a basic triage collection:

date -u
hostname
uname -a
uptime
who
w
last
lastb

Capture running processes:

ps auxww

A process tree can reveal parent-child relationships:

ps auxf

or:

pstree -ap

Check active network connections:

ss -plant

and UDP:

ss -panu

Check listening services:

ss -lntup

Open files can also be useful:

lsof -nP

Check deleted files that are still held open by processes:

lsof +L1

This is particularly useful when an attacker deletes a malware executable after launching it.

You may see something similar to:

malware  12345 attacker  txt  REG ... /tmp/.something (deleted)

Do not assume the malware is gone merely because its file is no longer visible.


7. Preserve Suspicious Files Before Deleting Them

If you discover something suspicious such as:

/tmp/.something
/dev/shm/program
/home/user/.hiddenfile

do not immediately run:

rm -rf suspicious-file

Preserve it first.

Never execute the suspicious file.

Record metadata:

stat suspicious-file

Check its type:

file suspicious-file

Inspect extended attributes if available:

lsattr suspicious-file

Calculate cryptographic hashes:

sha256sum suspicious-file
sha512sum suspicious-file

Copy it to protected evidence storage:

cp -a suspicious-file /mnt/incident-evidence/

Then hash the preserved copy again:

sha256sum /mnt/incident-evidence/suspicious-file

Hashes help establish that the file being analyzed later is the same file you originally collected.

If you have proper forensic capability, acquiring a disk image is preferable to performing extensive analysis directly on the affected filesystem.

NIST SP 800-86 explains that forensic analysis is safer when performed on captured evidence rather than repeatedly modifying the original system.


8. Stop Malicious Processes When Necessary

If malicious activity is active and containment cannot sufficiently restrict it, stopping the process may be necessary.

First identify it:

ps auxww | grep PROCESS

Check its executable:

readlink -f /proc/PID/exe

Inspect its command line:

tr '\0' ' ' < /proc/PID/cmdline
echo

Check its environment if required:

tr '\0' '\n' < /proc/PID/environ

Check open network connections:

lsof -Pan -p PID -i

Preserve these results before terminating the process when possible.

Then:

kill PID

If it refuses to terminate:

kill -9 PID

Be aware that killing the process only terminates that instance.

If persistence remains, it may return within seconds or minutes.

That is exactly what happened during our Zimbra investigation. Killing malicious processes did not resolve the issue because another mechanism was starting them again.


9. Search for Cron Persistence

Cron remains a common persistence mechanism because it is simple and easy to hide among legitimate scheduled tasks.

Check the root crontab:

crontab -l

Check application users:

crontab -u zimbra -l

or:

crontab -u postfix -l

Check all user crontabs where appropriate.

Also inspect:

cat /etc/crontab

and:

ls -la /etc/cron.d/
ls -la /etc/cron.hourly/
ls -la /etc/cron.daily/
ls -la /etc/cron.weekly/
ls -la /etc/cron.monthly/

Search scheduled-task configuration for suspicious paths:

grep -R "/tmp\|/dev/shm\|/var/tmp\|/home" \
    /etc/cron* \
    /var/spool/cron* \
    2>/dev/null

Do not assume a cron line is legitimate because it runs under an application account.

In our Zimbra incident, malware persistence existed directly in the Zimbra user’s crontab and ran every minute.

That explained why deleting or killing the visible malware did not permanently stop it.

Before removing a malicious cron entry, preserve:

The complete crontab
Modification timestamps
The executed command
Referenced scripts and binaries
Hashes of those files

Then remove or disable it.


10. Check systemd Persistence

Modern Linux systems provide attackers with another powerful persistence mechanism through systemd.

List services:

systemctl list-units --type=service --all

List installed service definitions:

systemctl list-unit-files --type=service

Check timers:

systemctl list-timers --all

systemctl list-timers displays systemd timer units and the services they activate.

Review system service locations:

/etc/systemd/system/
/run/systemd/system/
/usr/lib/systemd/system/
/lib/systemd/system/

Depending on the distribution, some of these directories may not exist.

Look for recently modified unit files:

find /etc/systemd/system \
     /usr/lib/systemd/system \
     /lib/systemd/system \
     -type f \
     -mtime -30 \
     -ls 2>/dev/null

Inspect suspicious units:

systemctl cat suspicious.service

Check their status:

systemctl status suspicious.service

Also investigate per-user systemd services:

~/.config/systemd/user/

Attackers do not need to name a malicious service:

malware.service

They may use innocent-looking names such as:

system-update.service
network-helper.service
cache-cleaner.service

Judge the unit by what it executes, where that executable came from, who created it, and whether it belongs to a legitimate package.


11. Check Other Persistence Locations

Cron and systemd are important, but they are not the only mechanisms.

Inspect:

/etc/rc.local
/etc/init.d/
/etc/profile
/etc/profile.d/
~/.bashrc
~/.profile
~/.bash_profile
~/.ssh/rc
/etc/ssh/sshrc

Check scheduled at jobs:

atq

Investigate new or unexpected users:

cat /etc/passwd

Look for users with UID 0:

awk -F: '$3 == 0 {print}' /etc/passwd

Normally only root should have UID 0.

Inspect sudo configuration:

visudo -c

and:

ls -la /etc/sudoers.d/

Do not limit your investigation to one persistence mechanism simply because you already found one.

Attackers can establish several independent access paths.


12. Investigate SSH Access Carefully

SSH deserves special attention after a Linux server compromise.

Check:

/root/.ssh/authorized_keys
/home/*/.ssh/authorized_keys

Application users should also be inspected.

For Zimbra:

/opt/zimbra/.ssh/authorized_keys

OpenSSH normally obtains user public keys from configured authorized-key files. The standard configuration commonly includes .ssh/authorized_keys, although AuthorizedKeysFile can be changed in sshd_config.

Search broadly:

find /root /home /opt \
    -type f \
    -name authorized_keys \
    -print 2>/dev/null

Do not look only for unknown public keys.

OpenSSH authorized-key lines can contain options before the key.

For example:

command="some-command",no-port-forwarding ssh-ed25519 AAAA...

Options such as command= can force execution of a particular command whenever that key authenticates.

This makes an unexpected authorized-key entry particularly important during an investigation. OpenSSH documents the authorized-key format and its optional key settings.

In our Zimbra investigation, suspicious entries in the Zimbra account’s authorized-key configuration became an important indicator that the compromise involved more than a single malware process.


13. Review SSH Authentication Logs

Now determine whether suspicious keys or passwords were actually used.

On Ubuntu and Debian:

grep sshd /var/log/auth.log

Search successful authentication:

grep "Accepted" /var/log/auth.log

Specifically:

grep "Accepted publickey" /var/log/auth.log

and:

grep "Accepted password" /var/log/auth.log

On systems using the journal:

journalctl -u ssh

or:

journalctl -u sshd

Search by time:

journalctl -u sshd \
    --since "2026-09-01 00:00:00" \
    --until "2026-09-02 23:59:59"

Look for:

Unexpected IP addresses
Unexpected login times
Application-account SSH access
Root login
Authentication from neighboring servers
New public keys
Repeated failures followed by success

In our incident, successful public-key authentication from an internal address was particularly important.

An internal source address does not automatically make authentication legitimate.

It can indicate that another internal machine was compromised first.


14. Investigate Lateral Movement

Once a server is confirmed compromised, ask:

What systems could this server access?

Check:

Database servers
LDAP servers
Backup servers
Monitoring servers
Storage servers
DNS servers
Web applications
Management servers
Other mail nodes
File servers
Hypervisors
Cloud services
Administrative workstations

Look at SSH client configuration:

cat ~/.ssh/config

Search for private keys:

find /root /home /opt \
    -type f \
    \( -name "id_rsa" \
       -o -name "id_ed25519" \
       -o -name "*.pem" \
       -o -name "*.key" \) \
    -ls 2>/dev/null

Do not assume those files are all malicious.

The goal is to identify credentials the compromised host could use elsewhere.

Check known hosts:

find /root /home /opt \
    -type f \
    -name known_hosts \
    -print 2>/dev/null

These files can help identify systems administrators or applications have previously connected to.

Also examine shell history where available:

cat ~/.bash_history

but remember that attackers can delete, modify, disable, or bypass shell history.

CISA recommends assuming possible lateral movement in serious compromises and investigating connected systems rather than limiting analysis to the first discovered host.


15. Rotate Credentials From a Trusted Machine

Do not change passwords while assuming the compromised server can safely handle those new credentials.

If an attacker installed credential-stealing software, your newly entered password could immediately be captured.

Perform credential rotation from a known-good administrative workstation.

Potential credentials include:

Root password
Administrator passwords
Application administrator accounts
SSH keys
Database credentials
LDAP credentials
SMTP relay credentials
Backup credentials
Object-storage credentials
Monitoring credentials
API tokens
Cloud credentials
DNS credentials
Registrar credentials
Service-account passwords
Automation credentials

For a mail system, also consider credentials stored inside application configuration files.

CISA recommends provisioning new account credentials after confirmed compromise.

Do not simply change the password of the account you know was abused.

Assume any credential accessible to the attacker may have been copied.


16. Rotate SSH Keys Where Exposure Is Possible

If an attacker obtained root privileges, assume they could read SSH private keys stored on that server.

Generate replacement keys on a trusted machine.

Remove unauthorized public keys from servers where the old key had access.

Then revoke or remove the old legitimate key where compromise cannot be ruled out.

Do not forget service accounts.

If the compromised mail server used passwordless SSH to:

Backup servers
Storage systems
Automation hosts
Monitoring machines
Other mail servers

those relationships must be reviewed.

Otherwise rebuilding the mail server may still leave the attacker with valid credentials to another system.


17. Review TLS Certificates and Private Keys

Mail servers contain valuable private keys.

These can include:

HTTPS certificates
SMTP TLS certificates
IMAP TLS certificates
LDAP certificates
Internal CA certificates
Client certificates

If an attacker had sufficient privilege to read those private keys, consider them exposed.

Inspect your certificate files and determine:

Which private keys were present
Which accounts could read them
Whether the attacker reached those accounts
Whether those keys protect public services
Whether certificates should be revoked and reissued

You can inspect a certificate with OpenSSL:

openssl x509 \
    -in certificate.crt \
    -noout \
    -subject \
    -issuer \
    -dates \
    -fingerprint \
    -sha256

If compromise of the private key is possible, generate a new key pair and reissue the certificate rather than simply reinstalling the same certificate.

OpenSSL provides mechanisms for checking relationships between certificates and private keys, but cryptographic consistency does not tell you whether a private key was stolen.


18. Do Not Forget DKIM Keys

This is particularly important for mail servers.

If your server performs DKIM signing, private DKIM keys may be stored locally.

An attacker who copies a DKIM private key could potentially sign messages that validate as originating from your domain until the associated DNS key is changed or revoked.

Locate your DKIM configuration and signing keys.

The exact locations depend on your mail software.

For example, OpenDKIM installations often keep key material under locations such as:

/etc/opendkim/
/etc/opendkim/keys/

Zimbra manages DKIM differently.

If there is reasonable evidence that DKIM private keys could have been read, create new keys and update the corresponding DNS records.

Also inspect:

SPF
DKIM
DMARC
MX records
Mail relay configuration

for unauthorized changes.


19. Review Mail Configuration for Abuse

Attackers may compromise a mail server without leaving an obvious malware process running.

Inspect mail configuration for unauthorized modifications.

Depending on your platform, check:

Aliases
Forwarders
Transport maps
Relay configuration
Mailbox rules
Catch-all addresses
Administrator accounts
Delegated administration
SMTP authentication
Virtual domains
Mail routing
Milter configuration
Content filters
Webmail plugins

Check the outgoing mail queue.

Postfix:

postqueue -p

or:

mailq

Exim:

exim -bp

Look for unusual volumes, senders, recipients, or destinations.

If the server was used for spam, your public IP address or domain reputation may also have been damaged.


20. Preserve Mail and Authentication Logs

Copy relevant logs before rotation or cleanup destroys older evidence.

Potential sources include:

/var/log/auth.log
/var/log/syslog
/var/log/messages
/var/log/secure
/var/log/mail.log
/var/log/maillog
/var/log/audit/
/var/log/journal/

For Zimbra, also examine relevant logs under:

/opt/zimbra/log/
/var/log/zimbra.log

Preserve rotated logs as well:

auth.log.1
auth.log.2.gz
mail.log.1
syslog.1

Copy them without modifying the originals unnecessarily.

For example:

cp -a /var/log/auth.log* /mnt/incident-evidence/
cp -a /var/log/mail.log* /mnt/incident-evidence/

Hash the evidence:

find /mnt/incident-evidence \
    -type f \
    -exec sha256sum {} \; \
    > /mnt/incident-evidence/SHA256SUMS.txt

Treat evidence storage as sensitive.

Logs may contain:

Usernames
IP addresses
Internal hostnames
Tokens
Email addresses
Authentication information
Infrastructure details

21. Check for Package and Application Modifications

If your distribution supports package verification, use it as another signal.

Debian and Ubuntu:

dpkg -V

RPM-based distributions:

rpm -Va

These commands can reveal package-managed files that differ from expected metadata.

Do not automatically assume every difference indicates compromise.

Configuration files are often legitimately modified.

Also inspect recently modified files:

find /etc /usr /opt \
    -xdev \
    -type f \
    -mtime -7 \
    -ls 2>/dev/null

The time range should correspond to the suspected compromise window.

Remember that timestamps are evidence, but attackers with sufficient access can alter them.


22. Identify How the Attacker Entered

Recovery is incomplete until you understand the likely entry point.

Possible causes include:

Unpatched mail application vulnerability
Exposed administrative interface
Stolen SSH key
Weak password
Credential reuse
Compromised administrator workstation
Vulnerable webmail component
Malicious plugin
Application vulnerability
Misconfigured firewall
Compromised neighboring system

This matters because rebuilding without fixing the entry point can produce:

Old server compromised
        |
        v
Server rebuilt
        |
        v
Same vulnerability remains
        |
        v
New server compromised

CISA has previously documented active exploitation of Zimbra vulnerabilities and recommended patch management, restricted management interfaces, MFA where possible, network segmentation, incident response, and reimaging compromised systems.


23. Check Neighboring Systems

Once you identify indicators of compromise, search other systems for them.

Useful indicators include:

File hashes
Filenames
IP addresses
Domains
SSH key fingerprints
Usernames
Cron commands
Process names
Systemd unit names
Destination ports
Timestamps

If one mail server connected to another machine using SSH, investigate that machine.

If an authentication originated from an internal server, investigate that source.

If credentials stored on the mail server allowed access to backups, databases, DNS, or monitoring systems, those systems become part of the investigation.

Do not wait until obvious malware appears there.


24. Protect Your Backups

A backup becomes critical during recovery.

It can also become another infection source.

Do not immediately restore the newest backup without investigation.

Determine:

When the compromise started
When the backup was created
Whether malicious files exist in the backup
Whether persistence exists in configuration backups
Whether unauthorized SSH keys are included
Whether credentials stored in the backup are still valid

CISA recommends securing backups and ensuring they remain protected during incident response.

A backup from yesterday is not a clean backup if the attacker gained access three weeks ago.


25. Cleanup or Complete Rebuild?

This is one of the most important decisions.

Cleanup may be reasonable when all of the following are true:

The compromise is narrowly understood.
The attacker never obtained privileged access.
Persistence mechanisms are fully identified.
Affected files are known.
Credential exposure is understood.
Integrity can be validated against trusted sources.
Business requirements justify cleanup.

A complete rebuild should be strongly considered when:

Root access was obtained.
An application account with extensive privileges was compromised.
Unknown SSH keys were installed.
System files may have been modified.
Multiple persistence mechanisms were discovered.
The attacker had access for an unknown period.
Credentials could have been stolen.
Evidence suggests lateral movement.
You cannot determine the full scope.
You cannot prove system integrity.

For a fully compromised server, rebuilding from a known-good source is generally safer.

CISA’s Zimbra compromise guidance explicitly recommends reimaging compromised hosts.

That is an important distinction.

You do not rebuild because:

“We found malware.”

You rebuild because:

“We can no longer establish that the operating system and application stack have maintained integrity.”


26. Why Cleaning the Existing Server May Not Restore Trust

Imagine that you discovered five attacker modifications:

1. Malware executable
2. Cron persistence
3. SSH key
4. Hidden script
5. Modified application file

You successfully remove all five.

The problem is that you cannot necessarily prove there was no sixth modification.

Or twentieth.

If the attacker had privileged access, the affected operating system itself becomes an unreliable witness.

Commands such as:

ps
ls
netstat
ss
find

normally help you investigate.

But sophisticated malware can interfere with the operating system or utilities you rely on.

NIST forensic guidance specifically warns that a deeply compromised system may return misleading information and recommends using trusted forensic tools when possible.

This is the foundation of the rebuild decision.

The concern is not simply malware.

The concern is loss of trust in system integrity.


27. Build the Replacement Server From Trusted Sources

If rebuilding, do not clone the compromised operating system.

Install from a trusted installation source.

Then:

Apply current security updates
Install only required software
Restore only necessary configuration
Restore mail data from a validated backup
Generate new credentials
Generate new SSH keys where appropriate
Generate new private keys when exposure is possible
Patch the vulnerability used for entry
Restrict management access
Review firewall rules
Enable monitoring
Enable centralized logging

Do not copy the entire old filesystem because it is convenient.

Configuration should be reviewed before being transferred.

For example, instead of copying:

cp -a /etc /new-server/

restore specific validated files.

A rebuild is also an opportunity to compare the new system against the old server and determine what should legitimately exist.


28. Restore Mail Services Carefully

Do not reconnect the rebuilt server to the internet immediately.

First test internally.

Confirm:

SMTP works
IMAP/POP works if used
Webmail works
Authentication works
LDAP works if applicable
Database services work
TLS certificates are correct
DKIM signing works
SPF is correct
DMARC is correct
Mail queues behave normally
Backups work
Monitoring works
Logging works

Check listening ports:

ss -lntup

Check processes:

ps auxf

Test outbound and inbound mail through controlled accounts.

Verify TLS.

Check that your new server is not attempting connections to any suspicious infrastructure observed during the incident.

Only restore public connectivity after validation.


29. Monitor Closely After Recovery

Recovery does not end when mail starts flowing again.

Increase monitoring after the replacement server goes online.

Watch:

Authentication failures
Successful SSH logins
New processes
Unexpected outbound connections
CPU usage
Network traffic
Mail volume
SMTP authentication
New users
Cron changes
systemd changes
Filesystem changes
Mail queues
DNS changes
Administrator logins

Also watch the neighboring systems identified during the investigation.

If the original compromise resulted from stolen administrator credentials rather than a server vulnerability, rebuilding only the server will not solve the problem.


30. Preserve the Old Server

If possible, do not immediately erase the compromised server.

Keep it:

Powered off
Disconnected from production networks
Clearly labeled
Access controlled
Documented

If storage permits, retain forensic disk images rather than depending only on the physical server.

Evidence may become useful later when:

A second compromised machine is discovered
A new indicator appears
Management requests a report
A customer asks for an explanation
Law enforcement becomes involved
An insurer requests evidence
You need to determine the initial compromise date

NIST recommends preserving relevant forensic evidence and documenting how it was collected and handled.


31. The Actionable Linux Mail Server Incident-Response Checklist

Use this checklist when a Linux mail server compromise is confirmed or strongly suspected.

  • Record the current UTC date and time.
  • Start an incident timeline.
  • Notify the appropriate technical and management personnel.
  • Do not discuss the incident through the compromised mail system if attacker access may still exist.
  • Restrict or isolate the affected host.
  • Avoid rebooting before deciding whether volatile evidence is required.
  • Capture running processes.
  • Capture active network connections.
  • Capture logged-in users and recent logins.
  • Preserve suspicious process command lines and executable paths.
  • Copy suspicious files before deleting them.
  • Calculate SHA-256 hashes of collected files.
  • Preserve authentication logs.
  • Preserve mail logs.
  • Preserve system logs.
  • Preserve application-specific logs.
  • Check root’s crontab.
  • Check mail-application user crontabs.
  • Inspect /etc/cron.* and user cron locations.
  • Check systemd services.
  • Check systemd timers.
  • Check user-level systemd services.
  • Check at jobs.
  • Inspect shell startup files.
  • Inspect /etc/sudoers and /etc/sudoers.d/.
  • Check for unexpected accounts.
  • Check for unexpected UID 0 accounts.
  • Inspect all relevant authorized_keys files.
  • Investigate SSH authentication history.
  • Identify the initial access method if possible.
  • Determine whether privileged access occurred.
  • Identify credentials accessible from the server.
  • Rotate exposed credentials from a trusted machine.
  • Replace potentially exposed SSH keys.
  • Review TLS private keys and reissue certificates where necessary.
  • Review or rotate DKIM signing keys where exposure is possible.
  • Check DNS, MX, SPF, DKIM, and DMARC records.
  • Inspect mail forwarding and transport configuration.
  • Inspect outgoing mail queues.
  • Search neighboring systems for the same indicators.
  • Investigate internal machines that authenticated to the compromised server.
  • Protect and inspect backups.
  • Determine the earliest known compromise time.
  • Patch the vulnerability or access path that enabled entry.
  • Decide whether cleanup can actually establish system integrity.
  • Rebuild if privileged compromise or uncertain scope prevents establishing trust.
  • Restore only validated data and configuration.
  • Generate new credentials on the replacement system.
  • Validate SMTP, webmail, authentication, TLS, DKIM, and DNS before production use.
  • Increase monitoring after recovery.
  • Preserve the original compromised system or forensic image.
  • Produce a final incident report.
  • Document lessons learned and update your incident-response procedures.

32. Evidence Preservation Warning

There is an unavoidable tension during incident response.

You need to stop the attacker.

You also need to preserve evidence.

Every command you run on the compromised system can change its state.

For example:

ls
find
ps
cp
kill
rm
systemctl

can alter access times, process state, logs, caches, or other information.

This does not mean administrators should refuse to touch a compromised system.

It means actions should be deliberate and documented.

For a serious compromise involving financial loss, customer information, regulatory requirements, legal proceedings, or a potentially sophisticated attacker, involve qualified digital-forensics personnel before performing extensive cleanup.

NIST SP 800-86 specifically warns that live forensic collection changes system state and that organizations must weigh the value of volatile evidence against the risks associated with collecting it.


33. What Our Zimbra Incident Ultimately Teaches

Our Zimbra experience provides a useful example.

At first, suspicious processes and executables were visible.

Those processes could be killed.

The files could be disabled or removed.

But additional investigation uncovered persistence through cron and suspicious SSH authorization.

That changed the security assessment.

The lesson is simple:

Finding malware tells you that something happened.

Removing malware tells you that one artifact is gone.

Neither tells you that the server is trustworthy.

Trust must be re-established.

Sometimes investigation can provide enough confidence to retain a system.

When privileged access, multiple persistence mechanisms, stolen credentials, or uncertain scope are involved, a clean rebuild usually provides a much stronger basis for restoring trust.

For a mail server, this is particularly important.

A mail server often contains:

Thousands of user accounts
Authentication data
Email contents
TLS private keys
DKIM keys
Administrative credentials
Internal network access
DNS dependencies
Backup credentials
Customer and business information

Treating a confirmed compromise as an ordinary malware infection can leave the attacker with access long after the visible process has disappeared.

Treat it as an incident.

Contain it.

Preserve evidence.

Determine the scope.

Rotate trust material.

Investigate connected systems.

Then rebuild when you cannot confidently prove the integrity of the existing installation.

Official References

NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
NIST SP 800-61 Rev. 3

NIST SP 800-86, Guide to Integrating Forensic Techniques into Incident Response
NIST SP 800-86

CISA, Federal Government Cybersecurity Incident and Vulnerability Response Playbooks
CISA Incident and Vulnerability Response Playbooks

CISA and MS-ISAC, Threat Actors Exploiting Multiple CVEs Against Zimbra Collaboration Suite
CISA Zimbra Security Advisory AA22-228A

OpenSSH sshd Documentation
OpenSSH sshd Manual

systemctl Documentation
Linux systemctl Manual

SEO Metadata

SEO Title:
What to Do After Your Linux Mail Server Is Compromised

SEO Description:
Learn how to respond to a compromised Linux mail server, preserve evidence, investigate SSH and persistence, rotate credentials, check connected systems, and decide when a complete rebuild is required.

Social Title:
Your Linux Mail Server Was Compromised. What Should You Do Next?

Social Description:
A practical incident-response guide for compromised Linux mail servers covering containment, evidence preservation, cron and systemd persistence, SSH investigation, credential rotation, recovery, and rebuilding.

Focus Keyword:
Linux mail server compromised

Tags:
Linux Security, Incident Response, Mail Server Security, Zimbra Security, Linux Server, Cybersecurity, Digital Forensics, SSH Security, Malware Investigation, System Administration, Zimbra, PinoyLinux

lordfrancs3

lordfrancs3

Lordfrancis3 is a member of PinoyLinux since its establishment in 2011. With a wealth of experience spanning numerous years, he possesses a profound understanding of managing and deploying intricate infrastructure. His contributions have undoubtedly played a pivotal role in shaping the community's growth and success. His expertise and dedication reflect in every aspect of the journey, as PinoyLinux continues to champion the ideals of Linux and open-source technology. LordFrancis3's extensive experience remains an invaluable asset, and his commitment inspires fellow members to reach new heights. His enduring dedication to PinoyLinux's evolution is truly commendable.

Articles: 71