Zimbra Server Compromise Indicators: Suspicious Processes, Cron Jobs, SSH Keys, and Malware Artifacts

A strange process called javab does not prove that your Zimbra server has been compromised.

Neither does a file named .khp, an unfamiliar object under /dev/shm, an SSH key you do not immediately recognize, or an unexpected cron job.

Indicators become meaningful when they connect.

During a real PinoyLinux investigation of a compromised Zimbra Collaboration server running on Ubuntu, several pieces of evidence eventually formed a consistent picture:

zimbra account
      |
      +---- suspicious processes
      |        |
      |        +---- javab
      |        +---- idle
      |
      +---- suspicious files
      |        |
      |        +---- /dev/shm
      |        +---- /home/SSL
      |        +---- /tmp
      |
      +---- /home/SSL/.khp
      |        |
      |        +---- executed by zimbra crontab
      |
      +---- SSH authorization data
      |        |
      |        +---- additional keys
      |        +---- forced-command entry
      |
      +---- authentication logs
               |
               +---- successful public-key login

The importance was not any single filename. It was the relationship among processes, files, ownership, cron execution, SSH configuration, and logs. The original PinoyLinux investigation confirmed unauthorized processes and files operating under the zimbra account, an unauthorized cron job executing /home/SSL/.khp every minute, and SSH-related activity that required further investigation.

This page organizes those indicators into one investigation reference.

It does not replace the detailed incident articles. Use it to understand what was actually found, what each indicator meant in context, and where to continue the investigation.


Start With an Important Rule: An Indicator Is Not Proof

Incident response becomes unreliable when filenames are treated as verdicts.

Consider these names:

javab
idle
ksmd
.khp
.rguard

Linux allows executables and processes to use almost any filename.

An attacker can choose a legitimate-looking name.

A legitimate administrator or application can also create a strange-looking name.

The correct question is therefore not:

Is javab malware?

The better questions are:

Which user is running it?

Where is the executable stored?

What is its parent process?

What command-line arguments are present?

What files does it open?

Does it communicate over the network?

When did it appear?

What starts it?

Does it return after termination?

Does another configuration file reference it?

During the PinoyLinux Zimbra incident, javab and idle became significant because they were associated with the zimbra account, related artifacts appeared in unusual locations, and the activity returned after the processes were terminated.

That correlation turned suspicious names into incident evidence.


Confirmed Incident Indicators at a Glance

IndicatorStatus in the PinoyLinux incidentWhy it mattered
javabConfirmed suspicious process/artifactRan under the zimbra account and was associated with suspicious files and recurring activity
idleConfirmed suspicious process/artifactAppeared with the same broader unauthorized activity
/home/SSL/.khpConfirmed suspicious artifact and persistence componentReferenced by the unauthorized zimbra cron entry
.khp_tsConfirmed related artifactFound with other suspicious files during the investigation
.rguardConfirmed related artifactFound among related suspicious artifacts
/dev/shm artifactsConfirmed incident locationContained files associated with unauthorized activity
/home/SSLConfirmed suspicious artifact locationContained .khp, javab, idle, and other related files
/tmp artifactsConfirmed related evidenceLock-related objects connected with names already being investigated
* * * * * /home/SSL/.khpConfirmed unauthorized cron persistenceExecuted the suspicious file every minute
/var/log/syslog cron recordsConfirmed execution evidenceShowed cron actually running /home/SSL/.khp as zimbra
/opt/zimbra/.ssh/authorized_keysConfirmed area requiring investigationContained multiple keys and at least one forced-command entry
Successful Accepted publickey eventConfirmed SSH authentication evidenceShowed successful authentication to zimbra from an internal IP during the incident period
ksmdSearch indicator, not clearly confirmed in the published incident evidenceIncluded in targeted filename searches, but the current articles do not establish it as a confirmed artifact

That last distinction matters.

ksmd appeared in the filename search used during the investigation:

find \
  /home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \
  -xdev \
  \( \
    -name 'javab' -o \
    -name 'idle' -o \
    -name '.khp' -o \
    -name '.khp_ts' -o \
    -name '.rguard' -o \
    -name 'ksmd' \
  \) \
  -ls 2>/dev/null

But the published PinoyLinux incident articles clearly identify javab, idle, .khp, .khp_ts, .rguard, /dev/shm, /home/SSL, and related /tmp artifacts as findings. They do not provide the same confirmation for ksmd.

So:

ksmd found in your environment
        !=
proof of this malware

Treat it as something to investigate.


1. javab: The Process That Started the Investigation

One of the clearest incident indicators was a process named:

javab

A suspicious process used a command resembling:

./javab -o <REMOTE_HOST>:<REMOTE_PORT>

The actual destination was removed from the published incident report.

The important details were:

  • it was not expected Zimbra software
  • it ran under the zimbra account
  • related files existed in suspicious locations
  • it appeared together with other unauthorized activity
  • terminating it did not permanently stop the activity

Those facts made it an incident indicator. The filename alone did not.

Safe first checks

pgrep -af 'javab'

Then inspect a specific PID:

ps -o pid,ppid,user,lstart,etime,args -p <PID>

Check what executable Linux says is running:

readlink -f /proc/<PID>/exe

Inspect the command line:

tr '\0' ' ' < /proc/<PID>/cmdline
echo

Linux exposes /proc/<PID>/exe as a symbolic link to the executed program. /proc/<PID>/cmdline exposes the process command-line data, although a process can alter what appears there, so it should be treated as one source of evidence rather than absolute truth.

For the full process investigation, read:

How to Investigate Suspicious Processes and Files on a Linux Server


2. idle: Significant Because of the Context Around It

The second recurring process name was:

idle

Again, idle by itself proves nothing.

What made it relevant was that it appeared in the same environment as:

javab
.khp
/dev/shm artifacts
/home/SSL artifacts
cron persistence

The process also returned after containment attempts, which helped push the investigation toward persistence rather than simple process termination.

Check it the same way:

pgrep -af 'idle'

Then:

ps -o pid,ppid,user,lstart,etime,args -p <PID>
readlink -f /proc/<PID>/exe

Do not use a rule such as:

process name = idle
therefore malware

Process names can be misleading.


3. /dev/shm: Suspicious in Context, Normal by Design

Several suspicious artifacts appeared under:

/dev/shm

This directory deserves careful explanation.

/dev/shm is not a malware directory.

On Linux, POSIX shared-memory objects are normally backed by a tmpfs filesystem mounted under /dev/shm. Legitimate applications use it every day.

During this incident, files under /dev/shm included names associated with the unauthorized activity, including:

javab
idle
.khp
.khp_ts
.rguard

That relationship made the location relevant.

Inspect without deleting

ls -lah /dev/shm

For an individual object:

stat /dev/shm/<FILE>
file /dev/shm/<FILE>
sha256sum /dev/shm/<FILE>

You can also inspect recently modified files:

find /dev/shm \
  -xdev \
  -type f \
  -mmin -120 \
  -ls

A result means:

investigate

It does not automatically mean:

delete

4. /home/SSL: A Legitimate-Looking Name Hiding Suspicious Artifacts

Another major incident location was:

/home/SSL

The name looks plausible.

An administrator seeing SSL may initially assume it contains certificates or TLS-related material.

During the incident, however, it contained suspicious components including:

.khp
.khp_ts
javab
idle

The broader investigation also encountered related names such as .app_args and .java_cache at different stages.

The lesson is simple:

Do not trust a directory because its name sounds legitimate.

Inspect it.

ls -lah /home/SSL

Record directory metadata:

stat /home/SSL

Then inspect individual files:

stat /home/SSL/.khp
file /home/SSL/.khp
sha256sum /home/SSL/.khp

Do not execute an unknown file merely to discover what it does.


5. .khp: The Indicator That Connected Files to Persistence

The file:

/home/SSL/.khp

became one of the most important indicators in the entire investigation.

It was not important simply because its name began with a dot.

Hidden files are normal on Linux.

Its importance came from another discovery:

* * * * * /home/SSL/.khp

That line was present in the zimbra user’s crontab.

The crontab therefore established a persistence relationship:

zimbra crontab
      |
      v
/home/SSL/.khp
      |
      v
runs every minute
      |
      v
suspicious activity returns

Linux crontabs contain scheduling fields followed by a command. Asterisks in all five time fields cause the command to be eligible to run every minute.

This was a confirmed persistence mechanism in the incident.

For the full persistence investigation, read:

Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server


6. The Unauthorized zimbra Crontab Entry

The critical cron entry was:

* * * * * /home/SSL/.khp

You can inspect the Zimbra account’s scheduled jobs with:

crontab -u zimbra -l

Do not assume that every zimbra cron entry is malicious.

Zimbra legitimately uses scheduled jobs.

The problem in this incident was a specific unauthorized entry pointing to a file that was already connected to suspicious activity. Zimbra’s own compromise-check guidance explicitly recommends checking the zimbra and root crontabs for unauthorized entries.

This distinction matters:

zimbra has a crontab
        -> normal

zimbra crontab contains an unexpected command
executing a suspicious artifact every minute
        -> investigate immediately

7. /var/log/syslog: Turning a Cron Hypothesis Into Execution Evidence

Finding a cron entry tells you what is configured.

Logs can tell you whether it actually ran.

During the PinoyLinux incident, /var/log/syslog contained records showing cron executing:

/home/SSL/.khp

as the zimbra user.

A useful search is:

grep -hE 'CRON.*zimbra' \
  /var/log/syslog \
  /var/log/syslog.1 \
  2>/dev/null

Once you know the suspicious path:

grep -R '/home/SSL/.khp' /var/log 2>/dev/null

You can also search the known process names:

grep -R -E 'javab|idle|\.khp' /var/log 2>/dev/null

Be careful with large log trees. Recursive searches can consume substantial disk I/O.

The evidence chain became:

crontab says .khp should execute
              |
              v
syslog shows .khp executing
              |
              v
suspicious processes appear

That is much stronger evidence than a filename alone.


8. .khp_ts, .rguard, and Related Files

Other confirmed related artifacts included:

.khp_ts
.rguard

These were found in the same broader group of suspicious files during the investigation.

Their names alone should not become universal detection signatures.

Instead of creating a rule like:

find .rguard -> server hacked

use them as pivots:

find \
  /home /tmp /var/tmp /dev/shm /run /opt/zimbra \
  -xdev \
  \( \
    -name '.khp' -o \
    -name '.khp_ts' -o \
    -name '.rguard' \
  \) \
  -ls 2>/dev/null

Then investigate:

  • owner
  • group
  • permissions
  • hashes
  • timestamps
  • parent directories
  • references from cron or services
  • open processes
  • network connections
  • neighboring files created at similar times

GNU find supports searching directory hierarchies using criteria such as name, type, ownership, time, and permissions.


9. What About ksmd?

ksmd deserves special treatment because it can easily become an overstated IOC.

The PinoyLinux investigation included ksmd in a targeted filename search:

-name 'ksmd'

However, the current published incident articles do not identify ksmd with the same level of confirmation as:

javab
idle
.khp
.khp_ts
.rguard

Therefore, this article does not classify ksmd as a confirmed artifact from the incident.

If you find something called ksmd, investigate:

pgrep -af 'ksmd'

If it is a process:

readlink -f /proc/<PID>/exe
ps -o pid,ppid,user,lstart,etime,args -p <PID>

If it is a file:

stat <PATH>
file <PATH>
sha256sum <PATH>

Then correlate it with the rest of the system.

The correct label is:

candidate indicator

until evidence establishes what it is.


10. Suspicious SSH Authorization Data

The investigation later expanded into:

/opt/zimbra/.ssh/authorized_keys

The file contained:

  • multiple SSH public-key entries requiring investigation
  • at least one entry using a forced command
  • additional keys whose legitimacy needed verification

The incident also contained successful public-key authentication involving the zimbra account.

OpenSSH supports AuthorizedKeysFile for defining the files containing public keys allowed to authenticate users. The default commonly includes .ssh/authorized_keys, although administrators can configure another path.

Preserve before editing

cp -a /opt/zimbra/.ssh/authorized_keys \
  /root/incident-evidence/authorized_keys.before-change

Then inspect:

nl -ba /opt/zimbra/.ssh/authorized_keys

Do not publish the complete key material.

Generate fingerprints instead:

ssh-keygen -E sha256 -lf \
  /opt/zimbra/.ssh/authorized_keys

A fingerprint is easier and safer to compare with administrator inventories.

For the detailed SSH investigation, read:

Investigating Suspicious SSH Access on a Compromised Linux Server


11. Successful SSH Authentication Was Evidence, Not Root-Cause Proof

The incident’s /var/log/auth.log contained a successful public-key authentication involving the zimbra account from an internal IP address during the broader incident period.

A sanitized event looks like:

<DATE> <HOST> sshd[<PID>]:
Accepted publickey for zimbra from <INTERNAL_IP>
port <SOURCE_PORT> ssh2

Search:

grep -E 'Accepted publickey for zimbra' \
  /var/log/auth.log

Also inspect rotated logs where available:

grep -hE 'Accepted publickey for zimbra' \
  /var/log/auth.log \
  /var/log/auth.log.1 \
  2>/dev/null

This record establishes that OpenSSH recorded successful public-key authentication.

It does not prove:

  • that SSH was the original compromise vector
  • that the internal IP belonged to an attacker
  • that every SSH key was malicious
  • that the login created .khp
  • that the same person controlled every suspicious component

The PinoyLinux investigation deliberately avoided claiming SSH as the original intrusion vector because the available evidence did not establish that conclusion.

This is the correct incident-response discipline.


Evidence, Hypothesis, Containment, Remediation, and Root Cause Are Different

These terms should not be mixed.

Evidence

Evidence is something actually observed.

Examples from this incident:

javab process exists

/home/SSL/.khp exists

zimbra crontab contains:
* * * * * /home/SSL/.khp

syslog records cron executing .khp

authorized_keys contains entries requiring investigation

auth.log records successful public-key authentication

Hypothesis

A hypothesis is an explanation that may fit the evidence.

For example:

The SSH key may have provided persistence.

or:

An attacker may have obtained control of the zimbra account.

Those require further proof.

Containment

Containment tries to stop active damage.

Examples can include:

  • isolating the host
  • terminating confirmed unauthorized processes
  • disabling confirmed persistence
  • blocking malicious destinations
  • restricting compromised accounts

Containment changes the system and can destroy evidence, so collect what you can first.

Remediation

Remediation removes the cause and restores the environment safely.

It can include:

  • patching vulnerable software
  • rotating credentials
  • replacing SSH keys
  • restoring trusted configuration
  • rebuilding systems
  • reviewing neighboring hosts

Root-cause attribution

Root cause answers:

How did the compromise begin?

The published PinoyLinux incident established that the server was compromised.

It did not conclusively establish the original attack vector or attacker identity.

That distinction should remain intact.


A Safe Evidence-Collection Workflow

When you find similar indicators on a Zimbra server, avoid turning the investigation into an immediate cleanup session.

Start by recording state.

Record running processes

ps auxwwf

Then target known indicators:

pgrep -af 'javab|idle|\.khp'

pgrep searches currently running processes and can match process attributes and command lines.

Inspect process details

ps -o pid,ppid,user,lstart,etime,args -p <PID>
readlink -f /proc/<PID>/exe
readlink -f /proc/<PID>/cwd
tr '\0' ' ' < /proc/<PID>/cmdline
echo

Record suspicious-file information

ls -lah <PATH>
stat <PATH>
file <PATH>
sha256sum <PATH>

Search targeted directories

find \
  /home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \
  -xdev \
  \( \
    -name 'javab' -o \
    -name 'idle' -o \
    -name '.khp' -o \
    -name '.khp_ts' -o \
    -name '.rguard' -o \
    -name 'ksmd' \
  \) \
  -ls 2>/dev/null

Remember that ksmd in this command is a search pivot, not a confirmed finding from the published case.

Preserve scheduled tasks

crontab -u zimbra -l \
  > /root/incident-evidence/zimbra-crontab.txt

Also inspect:

crontab -u root -l

and:

ls -lah /etc/cron.d/
ls -lah /etc/cron.hourly/
ls -lah /etc/cron.daily/

Zimbra’s own compromise checklist recommends checking newly added cron entries for both zimbra and root.

Preserve SSH authorization data

cp -a /opt/zimbra/.ssh/authorized_keys \
  /root/incident-evidence/

Then:

ssh-keygen -E sha256 -lf \
  /opt/zimbra/.ssh/authorized_keys

Review authentication activity

grep -E \
  'Accepted|Failed|Invalid user' \
  /var/log/auth.log

Narrow the time window when possible.

Review cron activity

grep -hE 'CRON.*zimbra' \
  /var/log/syslog \
  /var/log/syslog.1 \
  2>/dev/null

Check Zimbra service state

su - zimbra -c 'zmcontrol status'

Zimbra documents zmcontrol status as displaying the running or stopped state of Zimbra components.

A healthy service list does not prove the operating system is clean.

That was one of the important lessons from the incident.


Additional Checks When You Find Similar Evidence

The following are investigation recommendations.

They are not additional confirmed findings from the PinoyLinux incident.

Check recently changed Zimbra files

Zimbra’s compromise guidance recommends looking for unknown and recently modified files, including files beneath the web application directories and elsewhere under /opt/zimbra.

Example:

find /opt/zimbra/jetty/webapps/ \
  -type f \
  -newerct "-30 days" \
  -ls

Do not delete everything returned by this command.

Software updates also create new files.

Review listening ports

ss -lntup

Compare the listeners with the services you expect.

Inspect systemd persistence

systemctl list-timers --all
systemctl list-unit-files --type=service

The PinoyLinux incident confirmed cron persistence.

It did not establish a malicious systemd service or timer.

Search for other SSH keys

find /root /home /opt/zimbra \
  -type f \
  -name authorized_keys \
  -ls 2>/dev/null

Zimbra’s current compromise checklist recommends validating SSH configuration, checking which authorized-keys file is in use, and reviewing authorized keys for users.

Correlate everything by time

Compare:

SSH authentication time
file creation/change time
process start time
cron execution time
Zimbra log events
system log events
network-device logs

One event may mean little.

Five related events within minutes can tell a very different story.


Why You Should Not Immediately Delete the Indicators

Imagine you find:

/home/SSL/.khp

and immediately run:

rm -f /home/SSL/.khp

You may lose:

  • timestamps
  • ownership
  • permissions
  • hash information
  • file contents
  • metadata
  • a sample needed for later analysis

The same applies to an unknown SSH key.

Deleting it may help containment, but it also modifies the original evidence.

Where practical:

collect
        |
        v
preserve
        |
        v
hash
        |
        v
document
        |
        v
contain

For serious production incidents, evidence should ideally be preserved on trusted storage separate from the compromised host.


Zimbra’s Own Compromise Guidance Supports This Approach

Zimbra’s official compromise checklist recommends administrators inspect:

  • recently changed and unknown files
  • web application directories
  • unexpected files elsewhere on the server
  • zimbra and root cron entries
  • unknown administrator accounts
  • unknown Zimlets
  • SSH daemon configuration
  • authorized-key configuration and contents
  • open network ports

Zimbra also states that if evidence shows a server has been compromised, rebuilding the system is an appropriate response path rather than assuming removal of the visible files has restored trust.

That aligns closely with what the PinoyLinux incident demonstrated.


Cleaning the Indicators Does Not Restore Trust

Suppose you:

kill javab
kill idle
remove the bad cron entry
delete .khp
remove an unauthorized SSH key

You have removed visible components.

You have not proven that:

  • no other persistence exists
  • no additional credentials were stolen
  • no Zimbra files were modified
  • no web shell exists
  • no privileged account was added
  • no neighboring system was affected
  • no unknown binaries remain
  • no application data was accessed
  • no attacker-controlled configuration remains

A critical production mail server should therefore be treated as untrusted after a serious compromise until you have a defensible recovery plan.

Zimbra publishes rebuild procedures specifically for compromised ZCS servers.

A safer recovery approach may involve:

  1. Preserve required evidence.
  2. Determine the affected time window.
  3. Rotate passwords, SSH keys, API credentials, and other secrets that may have been exposed.
  4. Build a clean replacement host.
  5. Install a supported and fully patched Zimbra release.
  6. Restore only trusted mail data and configuration.
  7. Validate administrator accounts and SSH keys.
  8. Review firewall exposure.
  9. Monitor the replacement system closely.
  10. Retain the old system only as required for forensic investigation.

Containment stops activity. Recovery restores service. Rebuilding trust requires more.


What We Can Say About the PinoyLinux Incident

Based on the current published evidence, we can say:

The Zimbra server was compromised.

Unauthorized processes named javab and idle ran under the zimbra account.

Suspicious files existed under locations including /dev/shm, /home/SSL, and /tmp.

The file /home/SSL/.khp was connected to an unauthorized zimbra crontab entry.

That entry ran every minute.

System logs confirmed cron executing /home/SSL/.khp.

SSH authorization information under /opt/zimbra/.ssh/authorized_keys required investigation.

At least one key entry used a forced command.

Successful public-key authentication to the zimbra account was found in /var/log/auth.log during the broader incident period.

We should not claim from the published evidence that:

SSH definitely caused the compromise.

A specific Zimbra vulnerability definitely caused the compromise.

The internal SSH source IP was definitely the attacker.

Every unfamiliar SSH key was malicious.

ksmd was definitely part of the malware.

A filename alone identifies the malware family.

Removing the visible files made the server trustworthy again.

Those statements go beyond the evidence currently documented.


Related PinoyLinux Incident Articles

Full incident investigation

How We Investigated a Compromised Zimbra Mail Server on Ubuntu

Start here for the complete overview of the original compromise investigation.

Cron persistence investigation

Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server

Read this for the detailed connection between .khp, the zimbra crontab, recurring processes, and /var/log/syslog.

Process and filesystem investigation

How to Investigate Suspicious Processes and Files on a Linux Server

Read this for process inspection, /proc, /dev/shm, /tmp, /home/SSL, file metadata, hashing, and targeted filesystem searching.

SSH investigation

Investigating Suspicious SSH Access on a Compromised Linux Server

Read this for authorized_keys, forced commands, key fingerprints, Accepted publickey events, and the limits of SSH evidence.


Key Takeaways

A Zimbra compromise should not be diagnosed from a filename.

javab was significant because it ran under the affected zimbra account, appeared with related artifacts, and formed part of recurring unauthorized activity.

idle mattered for the same reason.

/dev/shm is a legitimate Linux shared-memory location. Its presence in an incident becomes meaningful only when the files inside correlate with other evidence.

/home/SSL looked legitimate by name but contained suspicious artifacts.

.khp became especially important when the zimbra user’s crontab showed:

* * * * * /home/SSL/.khp

and /var/log/syslog confirmed that cron actually executed the command.

SSH evidence added another layer. Unknown or questionable keys and a successful public-key login demanded investigation, but they did not prove that SSH was the initial compromise vector.

ksmd should not be promoted into a confirmed IOC merely because it appeared in a search expression.

The strongest incident findings come from relationships:

process
   +
file
   +
user
   +
persistence
   +
logs
   +
authentication
   +
timeline

That is the difference between finding something strange and understanding a compromise.

And once a critical Zimbra server has been seriously compromised, removing the visible malware is not the same as restoring trust.

lordfrancs3

lordfrancs3

Lordfrancis3 is a member of PinoyLinux since its establishment in 2011. With a wealth of experience spanning numerous years, he possesses a profound understanding of managing and deploying intricate infrastructure. His contributions have undoubtedly played a pivotal role in shaping the community's growth and success. His expertise and dedication reflect in every aspect of the journey, as PinoyLinux continues to champion the ideals of Linux and open-source technology. LordFrancis3's extensive experience remains an invaluable asset, and his commitment inspires fellow members to reach new heights. His enduring dedication to PinoyLinux's evolution is truly commendable.

Articles: 71