A strange process name on a Linux server does not automatically mean the server has been compromised. Neither does finding an unfamiliar file under /tmp or /dev/shm.

The real question is whether the process, file, account, location, timestamps, command line, network activity, and persistence mechanisms make sense together.

We had to answer exactly that question while investigating a compromised Zimbra mail server running on Ubuntu.

During the incident, we discovered processes named:

javab
idle

We also found suspicious files with names such as:

.khp
.khp_ts
.rguard
javab
idle

Some of the artifacts appeared in locations including:

/dev/shm
/tmp
/home/SSL

The filenames alone did not prove anything. What made them significant was the relationship between the files, running processes, scheduled execution, user ownership, and system logs.

This guide uses that incident as a real-world case study and explains how Linux administrators can investigate suspicious processes and files without immediately deleting the evidence.

Customer names, domains, IP addresses, SSH keys, credentials, remote destinations, and other identifying information have been removed.

Important: If the server may be involved in a serious security incident, preserve evidence before making changes. Commands that kill processes, change permissions, or delete files can destroy information that may be useful later.

What We Actually Found on the Zimbra Server

Before discussing general investigation techniques, it is important to separate our confirmed findings from the additional checks recommended later in this guide.

During our actual incident, we confirmed that suspicious processes named javab and idle were running under the zimbra account.

One process used a command resembling:

./javab -o <REMOTE_HOST>:<REMOTE_PORT>

The actual destination has been removed.

We found suspicious artifacts under /dev/shm and /home/SSL, including files named:

javab
idle
.khp
.khp_ts
.rguard

During the continued investigation, we also encountered suspicious lock-related files under /tmp, including names associated with the same activity.

The most important discovery came later when we found this line in the zimbra user’s crontab:

* * * * * /home/SSL/.khp

The cron job executed /home/SSL/.khp every minute. System logs also showed cron executing that command as the zimbra user.

This connected the filesystem artifacts with the recurring suspicious processes.

Zimbra’s own security guidance recommends checking running processes, recently modified files, temporary directories, SSH access, and the zimbra crontab when investigating a compromised system.

Start With the Running Processes

A good starting point is determining what is currently running.

The standard Linux ps command reports a snapshot of active processes.

One of the commands we used during the incident was:

ps aux | grep -E 'javab|idle'

A broader view is:

ps auxf

The f option provides a process hierarchy, which can help identify parent and child relationships.

For more command-line detail, I usually prefer:

ps auxwwf

The additional w options prevent output from being shortened to the terminal width.

When examining the output, pay attention to fields such as:

USER
PID
%CPU
%MEM
START
TIME
COMMAND

Do not focus only on %CPU.

A malicious process may consume significant CPU, but it can also remain nearly idle while maintaining persistence or waiting for instructions.

The USER column was particularly important in our case because the suspicious processes were associated with the zimbra account.

That gave us another direction to investigate.

Use pgrep When You Already Know the Process Name

Once we identified javab and idle, repeatedly searching the entire process table was unnecessary.

We could use:

pgrep -a javab

and:

pgrep -a idle

The -a option displays the PID and command line of each matching process.

The Linux pgrep documentation notes that process-name matching has limitations. The -f option can be used when you need to match against the complete command line instead.

For example:

pgrep -af '/home/SSL'

This can be useful when the executable name changes but the suspicious path remains visible in its command line.

Another example is:

pgrep -af '\.khp'

These broader searches are additional investigation techniques. Our confirmed indicators during the incident were already known process and file names.

Do Not Trust the Process Name

A Linux process can call itself something that looks harmless.

For example:

backup
java
system-update
monitor
worker

A name is only an indicator.

Once you have the PID, determine what executable is actually running.

Suppose the suspicious PID is:

12345

You can inspect:

readlink -f /proc/12345/exe

Linux exposes /proc/<PID>/exe as a symbolic link to the pathname of the program being executed. If the executable has already been deleted from the filesystem while the process continues running, Linux can append (deleted) to the pathname.

For example, you might see:

/home/SSL/javab

That is far more useful than seeing only:

javab

in a process list.

Check the Process Command Line Through /proc

Linux exposes process information through the /proc virtual filesystem.

To inspect the command line:

tr '\0' ' ' < /proc/12345/cmdline
echo

The arguments in /proc/<PID>/cmdline are separated by null characters, which is why tr is used to replace them with spaces.

Linux documents /proc/<PID>/cmdline as containing the process command line. However, a process can modify the information exposed there, so administrators should not treat it as unquestionable forensic evidence.

This is an important limitation.

You should correlate /proc information with:

filesystem paths
logs
open files
network connections
user ownership
timestamps
scheduled tasks

A single command rarely tells the entire story.

Check the Process Working Directory

Another useful /proc entry is:

readlink -f /proc/12345/cwd

This reveals the process’s current working directory.

For example:

/home/SSL

would deserve attention if you were already investigating suspicious files under that directory.

You can also inspect the process root:

readlink -f /proc/12345/root

and its open file descriptors:

ls -l /proc/12345/fd

These are additional general investigation techniques. They were not the primary commands that originally exposed our Zimbra compromise.

Use lsof to See What the Process Has Open

The name lsof means “list open files.”

On Linux, many resources appear as files or file descriptors. lsof can therefore help identify what files, sockets, libraries, and other resources a process has open.

For a particular PID:

lsof -p 12345

You may see entries pointing to:

/home/SSL/javab
/tmp/something
/dev/shm/something
network sockets
shared libraries
log files

If you want to concentrate on network activity:

lsof -Pan -p 12345 -i

Here:

-P

prevents port numbers from being converted to service names.

-n

prevents hostname resolution.

-a

combines the selection conditions.

This can make the output faster and easier to interpret during an incident.

Depending on permissions and Linux security restrictions, you may need root privileges to inspect another user’s processes fully.

Search for Known Indicators With find

Once we knew the suspicious filenames, we searched the server for additional copies.

One of our actual searches resembled:

find \
  /home /root /tmp /var/tmp /dev/shm /run /opt/zimbra \
  -xdev \
  \( \
    -name 'javab' -o \
    -name 'idle' -o \
    -name '.khp' -o \
    -name '.khp_ts' -o \
    -name '.rguard' -o \
    -name 'ksmd' \
  \) \
  -ls 2>/dev/null

GNU find searches directory trees from the specified starting paths and evaluates the supplied expression against each object it encounters.

Several parts of this command are worth explaining.

The starting directories were:

/home
/root
/tmp
/var/tmp
/dev/shm
/run
/opt/zimbra

These were selected because they were relevant to our incident.

The expression:

\( -name 'javab' -o -name 'idle' -o -name '.khp' \)

means:

find files named javab
OR idle
OR .khp

The -ls action displays detailed information about matching files.

The option:

-xdev

prevents find from descending into directories on another filesystem from each starting point.

Finally:

2>/dev/null

hides error output such as permission-denied messages.

That can make results easier to read, but remember that it can also hide errors that may matter. Remove the redirection if you need to see everything.

Search by Time, Not Only by Filename

Attackers can rename files easily.

If you know roughly when an incident began, timestamps can reveal other files created or modified around the same period.

For example:

find /home/SSL /tmp /dev/shm \
  -xdev \
  -type f \
  -mmin -120 \
  -ls

This searches for regular files whose contents were modified within approximately the last 120 minutes.

If the incident occurred yesterday, you might instead use:

find /home/SSL /tmp /dev/shm \
  -xdev \
  -type f \
  -mtime -2 \
  -ls

These searches can generate many legitimate results.

Time-based searches should therefore be treated as a way to identify files requiring inspection, not as a malware detector.

Inspect File Ownership and Permissions

Once you find a suspicious file, start with:

ls -la /home/SSL

The -a option matters because it displays hidden names beginning with a dot.

Without it, files such as:

.khp
.khp_ts
.rguard

may not appear in the normal directory listing.

For a particular file:

ls -l /home/SSL/.khp

Typical output might resemble:

-rwxr-xr-x 1 zimbra zimbra 4780 Aug 25 21:13 /home/SSL/.khp

From this one line you can examine:

file type
permissions
owner
group
size
modification time
pathname

The Linux ls documentation describes options for showing hidden files and selecting modification, access, metadata-change, and birth timestamps where available.

Ownership is particularly important.

If a suspicious executable under an unusual directory is owned by:

zimbra:zimbra

and suspicious processes are also running as zimbra, those findings deserve to be correlated.

Ownership alone still does not prove maliciousness.

Use stat for Better File Metadata

ls -l gives a useful summary, but stat provides more detail.

For example:

stat /home/SSL/.khp

The stat command displays file or filesystem status information.

Typical information includes:

Size
Blocks
File type
Permissions
UID
GID
Access time
Modify time
Change time
Birth time, if supported

Three timestamps are particularly useful.

mtime, or modification time, normally changes when file contents change.

ctime, or status-change time, changes when metadata such as ownership or permissions changes and also when file contents change.

atime, or access time, may change when a file is read, depending on filesystem and mount settings.

Some filesystems can also expose a creation or birth timestamp.

Do not interpret these timestamps as an unquestionable attacker timeline. Filesystem settings, restoration processes, copying, administrator actions, and deliberate manipulation can affect them.

They are evidence to correlate with other evidence.

Why /dev/shm Deserves Attention

During our incident, suspicious artifacts appeared under:

/dev/shm

That does not mean /dev/shm is inherently suspicious.

On Linux, POSIX shared-memory objects are normally backed by a tmpfs filesystem mounted at /dev/shm.

Legitimate applications use /dev/shm.

You should therefore never use a rule such as:

Everything in /dev/shm is malware.

That would be wrong.

What made /dev/shm relevant in our incident was that files there matched other known suspicious indicators and were related to unauthorized activity already under investigation.

A useful inspection command is:

ls -lah /dev/shm

Then inspect individual objects with:

stat /dev/shm/<FILE>

and:

file /dev/shm/<FILE>

You can search for recently modified regular files:

find /dev/shm \
  -xdev \
  -type f \
  -mmin -120 \
  -ls

Again, a match means “investigate this file,” not “delete this file.”

Why /tmp and /var/tmp Also Deserve Attention

Temporary directories are another useful place to inspect during an incident because many applications legitimately write temporary data there.

That also means strange-looking files can be perfectly normal.

During our Zimbra investigation, /tmp became relevant because we found lock-related artifacts associated with names we were already tracking.

The correct approach is correlation.

For example, suppose you find:

/tmp/.idle_lockd

at approximately the same time that:

idle

is running under the compromised application account.

That relationship is more interesting than either item alone.

Start with:

ls -lah /tmp

For targeted searching:

find /tmp /var/tmp \
  -xdev \
  \( \
    -name '*idle*' -o \
    -name '*javab*' -o \
    -name '*.khp*' \
  \) \
  -ls 2>/dev/null

Do not recursively delete unfamiliar files from /tmp on a production server. Active applications may depend on sockets, lock files, temporary databases, or other objects stored there.

Investigating /home/SSL

Another significant location in our incident was:

/home/SSL

The directory name itself looked plausible.

A directory named SSL could easily be assumed to contain certificates or security-related files.

Instead, our investigation found suspicious files there, including:

javab
idle
.khp
.khp_ts

This is an important operational lesson.

Do not judge a directory by its name.

Check its contents:

ls -lah /home/SSL

Check ownership:

stat /home/SSL

Then inspect individual files:

stat /home/SSL/.khp
file /home/SSL/.khp

If the file is executable:

ls -l /home/SSL/.khp

Do not execute it simply to find out what it does.

Identify the File Type

The file utility can provide useful information without directly executing the suspicious program.

For example:

file /home/SSL/javab

You may discover that something is:

ELF executable
shell script
Python script
compressed data
text
shared object

File extensions on Linux are not authoritative.

A file called:

report.txt

can still contain executable binary data.

Likewise:

javab

tells you almost nothing about the real format.

Calculate a Cryptographic Hash

Before modifying or deleting a suspicious file, calculate a hash:

sha256sum /home/SSL/.khp

The result resembles:

<64-character-SHA256-value>  /home/SSL/.khp

A SHA-256 hash gives you a stable identifier for the file contents.

You can use it to:

compare copies
correlate the same file on other servers
document evidence
search internal threat-intelligence records
verify that your preserved copy matches the source

Do not upload potentially sensitive corporate files to a public malware-analysis service without considering company policy, privacy, confidentiality, and legal requirements.

Preserve Evidence Before Deleting Anything

The natural reaction when finding malware is:

rm -f suspicious-file

That may destroy information you still need.

NIST’s Guide to Integrating Forensic Techniques into Incident Response recommends properly gathering and handling evidence, maintaining records of major actions, preserving integrity, and maintaining appropriate custody of collected evidence.

For operational troubleshooting, a basic evidence directory might begin with:

mkdir -p /root/incident-evidence
chmod 700 /root/incident-evidence

Record the current time:

date -Is > /root/incident-evidence/collection-time.txt

Capture the process list:

ps auxwwf > /root/incident-evidence/processes.txt

Record metadata before copying:

stat /home/SSL/.khp \
  > /root/incident-evidence/khp-stat.txt

Calculate the source hash:

sha256sum /home/SSL/.khp \
  > /root/incident-evidence/khp-source.sha256

Then preserve a copy:

cp -a /home/SSL/.khp \
  /root/incident-evidence/

Hash the copy:

sha256sum /root/incident-evidence/.khp \
  > /root/incident-evidence/khp-copy.sha256

This is a lightweight operational approach, not a replacement for formal forensic acquisition.

Simply reading files can affect certain metadata depending on filesystem and mount behavior. If the incident may involve legal action, regulatory requirements, or a serious breach investigation, use your organization’s forensic procedures and qualified incident-response personnel.

Be Careful Before Killing the Process

A suspicious process may contain useful volatile evidence that disappears when it terminates.

Before killing it, consider capturing:

ps -fp <PID>

readlink -f /proc/<PID>/exe

readlink -f /proc/<PID>/cwd

tr '\0' ' ' < /proc/<PID>/cmdline
echo

lsof -p <PID>

If appropriate:

lsof -Pan -p <PID> -i

Once you have collected what you need and confirmed the process is unauthorized, containment may require terminating it.

For example:

kill <PID>

or:

pkill javab

The decision depends on the incident.

If a process is actively damaging data or attacking other systems, containment may take priority over collecting every possible artifact.

Check Whether the Executable Has Already Been Deleted

One useful Linux-specific clue is:

readlink /proc/<PID>/exe

You might see:

/home/SSL/javab (deleted)

Linux documents that /proc/<PID>/exe can show (deleted) if the pathname has been unlinked while the process continues to execute.

That situation deserves investigation.

It can occur legitimately during software upgrades, but it can also appear when a suspicious executable has been deleted while the process remains alive.

Again, context matters.

Look for Parent and Child Relationships

A suspicious executable does not launch itself.

Try:

ps -ef --forest

or:

ps auxf

Find the suspicious PID and look at its parent process.

You can also display specific fields:

ps -o pid,ppid,user,lstart,etime,args -p <PID>

This shows:

PID
parent PID
user
start time
elapsed time
command

If the parent is still running, investigate it too.

A recurring process whose parent changes every minute may suggest a scheduler or another launcher.

That became relevant in our Zimbra incident because killing the visible processes did not stop the activity permanently.

We later discovered the persistence mechanism:

* * * * * /home/SSL/.khp

The file investigation therefore became a persistence investigation.

Compare Process Start Time With File Timestamps

One useful technique is to compare:

process start time
file modification time
file metadata-change time
cron execution time
authentication time
log entries

For example:

ps -o pid,lstart,args -p <PID>

and:

stat /home/SSL/.khp

can be viewed together.

Suppose the file appeared at approximately 21:13 and the associated process started soon afterward.

That does not prove causation, but it gives you a timeline to investigate.

Then search the logs around the same period.

This is much more useful than looking at every log entry from an entire week.

Check Permissions Carefully

Linux permissions can provide useful clues.

For example:

-rwxrwxr-x

can be interpreted as:

owner: rwx
group: rwx
others: r-x

The x bit means a regular file can be executed when other requirements are met. Linux chmod manages these file mode bits.

If you discover an executable owned by an application account in an unexpected directory, ask:

Should this account own this file?

Should this file be executable?

Should this directory contain executables?

Who could have written this file?

Is the group writable?

When did the mode change?

Do not immediately run:

chmod -R 000 <DIRECTORY>

on a production server.

Changing permissions recursively can break legitimate services and modify evidence.

Containment During Our Incident

After identifying /home/SSL as part of the suspicious activity, one containment action we used was:

chmod -R a-x /home/SSL

This removed execute permission recursively from the contents of the suspicious directory.

We also terminated confirmed unauthorized processes.

This was an incident-specific containment decision.

It should not be copied blindly.

Before using a recursive permission command, verify that the target directory does not contain legitimate executables required by production services.

A formal forensic response may also preserve the system state before changing permissions.

What readlink Adds to the Investigation

The readlink command resolves symbolic links and can return canonical paths with -f.

This makes it particularly useful with /proc.

For example:

readlink -f /proc/<PID>/exe

answers:

What executable is this process actually using?

While:

readlink -f /proc/<PID>/cwd

answers:

From which directory is the process currently operating?

Those answers can connect something you saw in ps with something you found on disk.

What lsof Adds to the Investigation

ps tells you about the process.

find tells you about files.

lsof helps connect the two.

For example:

lsof -p <PID>

can show whether the suspicious process currently has a configuration file, log, shared library, device, socket, or deleted file open.

If the process opens:

/home/SSL/.khp

or another known suspicious artifact, that relationship becomes another part of your evidence.

The same applies to network sockets.

A process connecting to an unexpected remote destination deserves further investigation, especially when combined with other indicators.

Do Not Rely on One Indicator

A useful Linux investigation is built from correlated evidence.

In our Zimbra incident, the important relationships eventually looked roughly like this:

zimbra account
      |
      +---- suspicious process: javab
      |
      +---- suspicious process: idle
      |
      +---- suspicious files
      |       |
      |       +---- /dev/shm
      |       +---- /home/SSL
      |       +---- /tmp
      |
      +---- /home/SSL/.khp
              |
              +---- referenced by zimbra crontab
                       |
                       +---- executed every minute

One unusual file would have been weak evidence.

One unusual process name would have been weak evidence.

The combination made the situation clear.

Actual Incident Findings Versus Additional Investigation Techniques

For clarity, these were confirmed during our incident: suspicious javab and idle processes, suspicious files under /dev/shm and /home/SSL, related artifacts under /tmp, ownership connected to the zimbra account, a suspicious .khp file, recurring activity after process termination, and a cron entry executing /home/SSL/.khp every minute.

The following techniques are broader recommendations that can improve a future investigation: inspecting /proc/<PID>/cwd, collecting complete lsof output, hashing every artifact before containment, searching files by modification time, comparing process start times with filesystem metadata, and capturing formal forensic evidence according to an incident-response procedure.

Those recommended checks should not be interpreted as additional findings from our original Zimbra incident.

A Practical Investigation Workflow

When you encounter an unfamiliar process on a Linux production server, a safe initial workflow is to identify it without changing anything, capture its PID and command line, inspect /proc, inspect open files, identify the executable, record file metadata, search for related files, calculate hashes, correlate timestamps with logs, investigate persistence, preserve the important artifacts, and only then perform containment when circumstances allow.

For example:

# Identify processes
ps auxwwf
pgrep -af '<SUSPICIOUS_NAME>'

# Inspect a specific process
ps -o pid,ppid,user,lstart,etime,args -p <PID>

readlink -f /proc/<PID>/exe
readlink -f /proc/<PID>/cwd

tr '\0' ' ' < /proc/<PID>/cmdline
echo

# Inspect open resources
lsof -p <PID>

lsof -Pan -p <PID> -i

# Inspect a suspicious file
ls -l <FILE>
stat <FILE>
file <FILE>
sha256sum <FILE>

# Search for known indicators
find /tmp /var/tmp /dev/shm /home \
  -xdev \
  \( \
    -name '<INDICATOR_1>' -o \
    -name '<INDICATOR_2>' \
  \) \
  -ls 2>/dev/null

# Search recently modified files in targeted locations
find /tmp /var/tmp /dev/shm \
  -xdev \
  -type f \
  -mmin -120 \
  -ls

This is an investigation workflow, not an automated malware-removal script.

Review every result before taking action.

Lessons From the Zimbra Incident

The biggest lesson was that suspicious processes should be investigated as relationships, not isolated objects.

javab became more significant when we discovered where it was stored and which user was running it.

.khp became more significant when we discovered it under /home/SSL.

/home/SSL/.khp became critical when we found it referenced by the zimbra user’s crontab.

The recurring processes made sense once we connected them to persistence.

Temporary directories also require context. /tmp and /dev/shm contain legitimate files on healthy Linux servers. Their contents become interesting when they match other indicators, have unexpected ownership, execute as application accounts, appear at suspicious times, or connect to other evidence.

The most important operational rule is simple:

Do not delete the first suspicious file you find. Investigate what created it, who owns it, what executes it, what it opens, when it appeared, and whether something can recreate it.

That information can tell you much more than the filename itself.

References

Zimbra, Investigating and Securing Systems

Zimbra’s security investigation documentation covers suspicious processes, recently modified files, temporary directories, SSH activity, and compromise investigation procedures.

Zimbra: Investigating and Securing Systems

Linux ps(1) Manual

Official Linux manual page for inspecting active processes.

Linux ps(1) manual

Linux pgrep(1) and pkill(1) Manual

Documents process matching, full-command-line searches, and process signalling.

Linux pgrep and pkill manual

GNU Findutils

Official GNU documentation for the find utility and its search expressions.

GNU Findutils: Invoking find

Linux lsof(8) Manual

Documents lsof and the inspection of open files and process resources.

Linux lsof(8) manual

Linux /proc/<PID>/exe

Documents the /proc/<PID>/exe symbolic link and how Linux exposes the executable associated with a process.

Linux proc_pid_exe(5) manual

Linux /proc/<PID>/cmdline

Documents process command-line information and its limitations.

Linux proc_pid_cmdline(5) manual

Linux readlink(1) Manual

Documents resolving symbolic links and canonical paths.

Linux readlink(1) manual

Linux Shared Memory and /dev/shm

Documents POSIX shared memory and the normal use of the tmpfs filesystem mounted at /dev/shm.

Linux shm_overview(7) manual

Linux tmpfs(5) manual

Linux stat(1) Manual

Documents displaying file and filesystem metadata.

Linux stat(1) manual

NIST SP 800-86

NIST’s Guide to Integrating Forensic Techniques into Incident Response provides guidance for collecting, preserving, documenting, and handling digital evidence.

NIST SP 800-86: Guide to Integrating Forensic Techniques into Incident Response