Physical Address
Metro Manila, Philippines
Physical Address
Metro Manila, Philippines
A strange process called javab does not prove that your Zimbra server has been compromised.
Neither does a file named .khp, an unfamiliar object under /dev/shm, an SSH key you do not immediately recognize, or an unexpected cron job.
Indicators become meaningful when they connect.
During a real PinoyLinux investigation of a compromised Zimbra Collaboration server running on Ubuntu, several pieces of evidence eventually formed a consistent picture:
zimbra account
|
+---- suspicious processes
| |
| +---- javab
| +---- idle
|
+---- suspicious files
| |
| +---- /dev/shm
| +---- /home/SSL
| +---- /tmp
|
+---- /home/SSL/.khp
| |
| +---- executed by zimbra crontab
|
+---- SSH authorization data
| |
| +---- additional keys
| +---- forced-command entry
|
+---- authentication logs
|
+---- successful public-key loginThe importance was not any single filename. It was the relationship among processes, files, ownership, cron execution, SSH configuration, and logs. The original PinoyLinux investigation confirmed unauthorized processes and files operating under the zimbra account, an unauthorized cron job executing /home/SSL/.khp every minute, and SSH-related activity that required further investigation.
This page organizes those indicators into one investigation reference.
It does not replace the detailed incident articles. Use it to understand what was actually found, what each indicator meant in context, and where to continue the investigation.
Incident response becomes unreliable when filenames are treated as verdicts.
Consider these names:
javab
idle
ksmd
.khp
.rguardLinux allows executables and processes to use almost any filename.
An attacker can choose a legitimate-looking name.
A legitimate administrator or application can also create a strange-looking name.
The correct question is therefore not:
Is javab malware?The better questions are:
Which user is running it?
Where is the executable stored?
What is its parent process?
What command-line arguments are present?
What files does it open?
Does it communicate over the network?
When did it appear?
What starts it?
Does it return after termination?
Does another configuration file reference it?During the PinoyLinux Zimbra incident, javab and idle became significant because they were associated with the zimbra account, related artifacts appeared in unusual locations, and the activity returned after the processes were terminated.
That correlation turned suspicious names into incident evidence.
| Indicator | Status in the PinoyLinux incident | Why it mattered |
|---|---|---|
javab | Confirmed suspicious process/artifact | Ran under the zimbra account and was associated with suspicious files and recurring activity |
idle | Confirmed suspicious process/artifact | Appeared with the same broader unauthorized activity |
/home/SSL/.khp | Confirmed suspicious artifact and persistence component | Referenced by the unauthorized zimbra cron entry |
.khp_ts | Confirmed related artifact | Found with other suspicious files during the investigation |
.rguard | Confirmed related artifact | Found among related suspicious artifacts |
/dev/shm artifacts | Confirmed incident location | Contained files associated with unauthorized activity |
/home/SSL | Confirmed suspicious artifact location | Contained .khp, javab, idle, and other related files |
/tmp artifacts | Confirmed related evidence | Lock-related objects connected with names already being investigated |
* * * * * /home/SSL/.khp | Confirmed unauthorized cron persistence | Executed the suspicious file every minute |
/var/log/syslog cron records | Confirmed execution evidence | Showed cron actually running /home/SSL/.khp as zimbra |
/opt/zimbra/.ssh/authorized_keys | Confirmed area requiring investigation | Contained multiple keys and at least one forced-command entry |
Successful Accepted publickey event | Confirmed SSH authentication evidence | Showed successful authentication to zimbra from an internal IP during the incident period |
ksmd | Search indicator, not clearly confirmed in the published incident evidence | Included in targeted filename searches, but the current articles do not establish it as a confirmed artifact |
That last distinction matters.
ksmd appeared in the filename search used during the investigation:
find \
/home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \
-xdev \
\( \
-name 'javab' -o \
-name 'idle' -o \
-name '.khp' -o \
-name '.khp_ts' -o \
-name '.rguard' -o \
-name 'ksmd' \
\) \
-ls 2>/dev/nullBut the published PinoyLinux incident articles clearly identify javab, idle, .khp, .khp_ts, .rguard, /dev/shm, /home/SSL, and related /tmp artifacts as findings. They do not provide the same confirmation for ksmd.
So:
ksmd found in your environment
!=
proof of this malwareTreat it as something to investigate.
javab: The Process That Started the InvestigationOne of the clearest incident indicators was a process named:
javabA suspicious process used a command resembling:
./javab -o <REMOTE_HOST>:<REMOTE_PORT>The actual destination was removed from the published incident report.
The important details were:
zimbra accountThose facts made it an incident indicator. The filename alone did not.
pgrep -af 'javab'Then inspect a specific PID:
ps -o pid,ppid,user,lstart,etime,args -p <PID>Check what executable Linux says is running:
readlink -f /proc/<PID>/exeInspect the command line:
tr '\0' ' ' < /proc/<PID>/cmdline
echoLinux exposes /proc/<PID>/exe as a symbolic link to the executed program. /proc/<PID>/cmdline exposes the process command-line data, although a process can alter what appears there, so it should be treated as one source of evidence rather than absolute truth.
For the full process investigation, read:
How to Investigate Suspicious Processes and Files on a Linux Server
idle: Significant Because of the Context Around ItThe second recurring process name was:
idleAgain, idle by itself proves nothing.
What made it relevant was that it appeared in the same environment as:
javab
.khp
/dev/shm artifacts
/home/SSL artifacts
cron persistenceThe process also returned after containment attempts, which helped push the investigation toward persistence rather than simple process termination.
Check it the same way:
pgrep -af 'idle'Then:
ps -o pid,ppid,user,lstart,etime,args -p <PID>
readlink -f /proc/<PID>/exeDo not use a rule such as:
process name = idle
therefore malwareProcess names can be misleading.
/dev/shm: Suspicious in Context, Normal by DesignSeveral suspicious artifacts appeared under:
/dev/shmThis directory deserves careful explanation.
/dev/shm is not a malware directory.
On Linux, POSIX shared-memory objects are normally backed by a tmpfs filesystem mounted under /dev/shm. Legitimate applications use it every day.
During this incident, files under /dev/shm included names associated with the unauthorized activity, including:
javab
idle
.khp
.khp_ts
.rguardThat relationship made the location relevant.
ls -lah /dev/shmFor an individual object:
stat /dev/shm/<FILE>
file /dev/shm/<FILE>
sha256sum /dev/shm/<FILE>You can also inspect recently modified files:
find /dev/shm \
-xdev \
-type f \
-mmin -120 \
-lsA result means:
investigateIt does not automatically mean:
delete/home/SSL: A Legitimate-Looking Name Hiding Suspicious ArtifactsAnother major incident location was:
/home/SSLThe name looks plausible.
An administrator seeing SSL may initially assume it contains certificates or TLS-related material.
During the incident, however, it contained suspicious components including:
.khp
.khp_ts
javab
idleThe broader investigation also encountered related names such as .app_args and .java_cache at different stages.
The lesson is simple:
Do not trust a directory because its name sounds legitimate.
Inspect it.
ls -lah /home/SSLRecord directory metadata:
stat /home/SSLThen inspect individual files:
stat /home/SSL/.khp
file /home/SSL/.khp
sha256sum /home/SSL/.khpDo not execute an unknown file merely to discover what it does.
.khp: The Indicator That Connected Files to PersistenceThe file:
/home/SSL/.khpbecame one of the most important indicators in the entire investigation.
It was not important simply because its name began with a dot.
Hidden files are normal on Linux.
Its importance came from another discovery:
* * * * * /home/SSL/.khpThat line was present in the zimbra user’s crontab.
The crontab therefore established a persistence relationship:
zimbra crontab
|
v
/home/SSL/.khp
|
v
runs every minute
|
v
suspicious activity returnsLinux crontabs contain scheduling fields followed by a command. Asterisks in all five time fields cause the command to be eligible to run every minute.
This was a confirmed persistence mechanism in the incident.
For the full persistence investigation, read:
Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server
zimbra Crontab EntryThe critical cron entry was:
* * * * * /home/SSL/.khpYou can inspect the Zimbra account’s scheduled jobs with:
crontab -u zimbra -lDo not assume that every zimbra cron entry is malicious.
Zimbra legitimately uses scheduled jobs.
The problem in this incident was a specific unauthorized entry pointing to a file that was already connected to suspicious activity. Zimbra’s own compromise-check guidance explicitly recommends checking the zimbra and root crontabs for unauthorized entries.
This distinction matters:
zimbra has a crontab
-> normal
zimbra crontab contains an unexpected command
executing a suspicious artifact every minute
-> investigate immediately/var/log/syslog: Turning a Cron Hypothesis Into Execution EvidenceFinding a cron entry tells you what is configured.
Logs can tell you whether it actually ran.
During the PinoyLinux incident, /var/log/syslog contained records showing cron executing:
/home/SSL/.khpas the zimbra user.
A useful search is:
grep -hE 'CRON.*zimbra' \
/var/log/syslog \
/var/log/syslog.1 \
2>/dev/nullOnce you know the suspicious path:
grep -R '/home/SSL/.khp' /var/log 2>/dev/nullYou can also search the known process names:
grep -R -E 'javab|idle|\.khp' /var/log 2>/dev/nullBe careful with large log trees. Recursive searches can consume substantial disk I/O.
The evidence chain became:
crontab says .khp should execute
|
v
syslog shows .khp executing
|
v
suspicious processes appearThat is much stronger evidence than a filename alone.
.khp_ts, .rguard, and Related FilesOther confirmed related artifacts included:
.khp_ts
.rguardThese were found in the same broader group of suspicious files during the investigation.
Their names alone should not become universal detection signatures.
Instead of creating a rule like:
find .rguard -> server hackeduse them as pivots:
find \
/home /tmp /var/tmp /dev/shm /run /opt/zimbra \
-xdev \
\( \
-name '.khp' -o \
-name '.khp_ts' -o \
-name '.rguard' \
\) \
-ls 2>/dev/nullThen investigate:
GNU find supports searching directory hierarchies using criteria such as name, type, ownership, time, and permissions.
ksmd?ksmd deserves special treatment because it can easily become an overstated IOC.
The PinoyLinux investigation included ksmd in a targeted filename search:
-name 'ksmd'However, the current published incident articles do not identify ksmd with the same level of confirmation as:
javab
idle
.khp
.khp_ts
.rguardTherefore, this article does not classify ksmd as a confirmed artifact from the incident.
If you find something called ksmd, investigate:
pgrep -af 'ksmd'If it is a process:
readlink -f /proc/<PID>/exe
ps -o pid,ppid,user,lstart,etime,args -p <PID>If it is a file:
stat <PATH>
file <PATH>
sha256sum <PATH>Then correlate it with the rest of the system.
The correct label is:
candidate indicatoruntil evidence establishes what it is.
The investigation later expanded into:
/opt/zimbra/.ssh/authorized_keysThe file contained:
The incident also contained successful public-key authentication involving the zimbra account.
OpenSSH supports AuthorizedKeysFile for defining the files containing public keys allowed to authenticate users. The default commonly includes .ssh/authorized_keys, although administrators can configure another path.
cp -a /opt/zimbra/.ssh/authorized_keys \
/root/incident-evidence/authorized_keys.before-changeThen inspect:
nl -ba /opt/zimbra/.ssh/authorized_keysDo not publish the complete key material.
Generate fingerprints instead:
ssh-keygen -E sha256 -lf \
/opt/zimbra/.ssh/authorized_keysA fingerprint is easier and safer to compare with administrator inventories.
For the detailed SSH investigation, read:
Investigating Suspicious SSH Access on a Compromised Linux Server
The incident’s /var/log/auth.log contained a successful public-key authentication involving the zimbra account from an internal IP address during the broader incident period.
A sanitized event looks like:
<DATE> <HOST> sshd[<PID>]:
Accepted publickey for zimbra from <INTERNAL_IP>
port <SOURCE_PORT> ssh2Search:
grep -E 'Accepted publickey for zimbra' \
/var/log/auth.logAlso inspect rotated logs where available:
grep -hE 'Accepted publickey for zimbra' \
/var/log/auth.log \
/var/log/auth.log.1 \
2>/dev/nullThis record establishes that OpenSSH recorded successful public-key authentication.
It does not prove:
.khpThe PinoyLinux investigation deliberately avoided claiming SSH as the original intrusion vector because the available evidence did not establish that conclusion.
This is the correct incident-response discipline.
These terms should not be mixed.
Evidence is something actually observed.
Examples from this incident:
javab process exists
/home/SSL/.khp exists
zimbra crontab contains:
* * * * * /home/SSL/.khp
syslog records cron executing .khp
authorized_keys contains entries requiring investigation
auth.log records successful public-key authenticationA hypothesis is an explanation that may fit the evidence.
For example:
The SSH key may have provided persistence.or:
An attacker may have obtained control of the zimbra account.Those require further proof.
Containment tries to stop active damage.
Examples can include:
Containment changes the system and can destroy evidence, so collect what you can first.
Remediation removes the cause and restores the environment safely.
It can include:
Root cause answers:
How did the compromise begin?The published PinoyLinux incident established that the server was compromised.
It did not conclusively establish the original attack vector or attacker identity.
That distinction should remain intact.
When you find similar indicators on a Zimbra server, avoid turning the investigation into an immediate cleanup session.
Start by recording state.
ps auxwwfThen target known indicators:
pgrep -af 'javab|idle|\.khp'pgrep searches currently running processes and can match process attributes and command lines.
ps -o pid,ppid,user,lstart,etime,args -p <PID>
readlink -f /proc/<PID>/exe
readlink -f /proc/<PID>/cwd
tr '\0' ' ' < /proc/<PID>/cmdline
echols -lah <PATH>
stat <PATH>
file <PATH>
sha256sum <PATH>find \
/home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \
-xdev \
\( \
-name 'javab' -o \
-name 'idle' -o \
-name '.khp' -o \
-name '.khp_ts' -o \
-name '.rguard' -o \
-name 'ksmd' \
\) \
-ls 2>/dev/nullRemember that ksmd in this command is a search pivot, not a confirmed finding from the published case.
crontab -u zimbra -l \
> /root/incident-evidence/zimbra-crontab.txtAlso inspect:
crontab -u root -land:
ls -lah /etc/cron.d/
ls -lah /etc/cron.hourly/
ls -lah /etc/cron.daily/Zimbra’s own compromise checklist recommends checking newly added cron entries for both zimbra and root.
cp -a /opt/zimbra/.ssh/authorized_keys \
/root/incident-evidence/Then:
ssh-keygen -E sha256 -lf \
/opt/zimbra/.ssh/authorized_keysgrep -E \
'Accepted|Failed|Invalid user' \
/var/log/auth.logNarrow the time window when possible.
grep -hE 'CRON.*zimbra' \
/var/log/syslog \
/var/log/syslog.1 \
2>/dev/nullsu - zimbra -c 'zmcontrol status'Zimbra documents zmcontrol status as displaying the running or stopped state of Zimbra components.
A healthy service list does not prove the operating system is clean.
That was one of the important lessons from the incident.
The following are investigation recommendations.
They are not additional confirmed findings from the PinoyLinux incident.
Zimbra’s compromise guidance recommends looking for unknown and recently modified files, including files beneath the web application directories and elsewhere under /opt/zimbra.
Example:
find /opt/zimbra/jetty/webapps/ \
-type f \
-newerct "-30 days" \
-lsDo not delete everything returned by this command.
Software updates also create new files.
ss -lntupCompare the listeners with the services you expect.
systemctl list-timers --all
systemctl list-unit-files --type=serviceThe PinoyLinux incident confirmed cron persistence.
It did not establish a malicious systemd service or timer.
find /root /home /opt/zimbra \
-type f \
-name authorized_keys \
-ls 2>/dev/nullZimbra’s current compromise checklist recommends validating SSH configuration, checking which authorized-keys file is in use, and reviewing authorized keys for users.
Compare:
SSH authentication time
file creation/change time
process start time
cron execution time
Zimbra log events
system log events
network-device logsOne event may mean little.
Five related events within minutes can tell a very different story.
Imagine you find:
/home/SSL/.khpand immediately run:
rm -f /home/SSL/.khpYou may lose:
The same applies to an unknown SSH key.
Deleting it may help containment, but it also modifies the original evidence.
Where practical:
collect
|
v
preserve
|
v
hash
|
v
document
|
v
containFor serious production incidents, evidence should ideally be preserved on trusted storage separate from the compromised host.
Zimbra’s official compromise checklist recommends administrators inspect:
zimbra and root cron entriesZimbra also states that if evidence shows a server has been compromised, rebuilding the system is an appropriate response path rather than assuming removal of the visible files has restored trust.
That aligns closely with what the PinoyLinux incident demonstrated.
Suppose you:
kill javab
kill idle
remove the bad cron entry
delete .khp
remove an unauthorized SSH keyYou have removed visible components.
You have not proven that:
A critical production mail server should therefore be treated as untrusted after a serious compromise until you have a defensible recovery plan.
Zimbra publishes rebuild procedures specifically for compromised ZCS servers.
A safer recovery approach may involve:
Containment stops activity. Recovery restores service. Rebuilding trust requires more.
Based on the current published evidence, we can say:
The Zimbra server was compromised.
Unauthorized processes named javab and idle ran under the zimbra account.
Suspicious files existed under locations including /dev/shm, /home/SSL, and /tmp.
The file /home/SSL/.khp was connected to an unauthorized zimbra crontab entry.
That entry ran every minute.
System logs confirmed cron executing /home/SSL/.khp.
SSH authorization information under /opt/zimbra/.ssh/authorized_keys required investigation.
At least one key entry used a forced command.
Successful public-key authentication to the zimbra account was found in /var/log/auth.log during the broader incident period.
We should not claim from the published evidence that:
SSH definitely caused the compromise.
A specific Zimbra vulnerability definitely caused the compromise.
The internal SSH source IP was definitely the attacker.
Every unfamiliar SSH key was malicious.
ksmd was definitely part of the malware.
A filename alone identifies the malware family.
Removing the visible files made the server trustworthy again.Those statements go beyond the evidence currently documented.
How We Investigated a Compromised Zimbra Mail Server on Ubuntu
Start here for the complete overview of the original compromise investigation.
Finding Malware Persistence in Linux: How a Cron Job Kept Reinfecting Our Zimbra Server
Read this for the detailed connection between .khp, the zimbra crontab, recurring processes, and /var/log/syslog.
How to Investigate Suspicious Processes and Files on a Linux Server
Read this for process inspection, /proc, /dev/shm, /tmp, /home/SSL, file metadata, hashing, and targeted filesystem searching.
Investigating Suspicious SSH Access on a Compromised Linux Server
Read this for authorized_keys, forced commands, key fingerprints, Accepted publickey events, and the limits of SSH evidence.
A Zimbra compromise should not be diagnosed from a filename.
javab was significant because it ran under the affected zimbra account, appeared with related artifacts, and formed part of recurring unauthorized activity.
idle mattered for the same reason.
/dev/shm is a legitimate Linux shared-memory location. Its presence in an incident becomes meaningful only when the files inside correlate with other evidence.
/home/SSL looked legitimate by name but contained suspicious artifacts.
.khp became especially important when the zimbra user’s crontab showed:
* * * * * /home/SSL/.khpand /var/log/syslog confirmed that cron actually executed the command.
SSH evidence added another layer. Unknown or questionable keys and a successful public-key login demanded investigation, but they did not prove that SSH was the initial compromise vector.
ksmd should not be promoted into a confirmed IOC merely because it appeared in a search expression.
The strongest incident findings come from relationships:
process
+
file
+
user
+
persistence
+
logs
+
authentication
+
timelineThat is the difference between finding something strange and understanding a compromise.
And once a critical Zimbra server has been seriously compromised, removing the visible malware is not the same as restoring trust.