Rootkit Analysis

Repository: https://github.com/x0prc/GlowRK

A Comprehensive x64 Rootkit Analysis and Detection Platform

Introduction

In an era where cyber threats are becoming increasingly sophisticated, the need for robust rootkit detection mechanisms has never been more critical. Rootkits represent some of the most insidious forms of malware, designed to conceal their presence while providing attackers with persistent access to compromised systems. We present GlowRK, a proof-of-concept project that demonstrates the working principles of rootkits and provides effective detection capabilities for x64 Windows systems.

GlowRK is an open-source rootkit analysis platform that combines multiple detection techniques including memory dump analysis, system table inspection, and file integrity verification. The project serves both as an educational tool for understanding rootkit mechanics and as a practical detection framework for security professionals.

Core Detection Mechanisms

1. Memory Dump Analysis

The foundation of GlowRK’s detection capabilities lies in its memory analysis module. We utilize the Volatility Framework, the gold standard for memory forensics, to extract meaningful artifacts from memory dumps. This approach allows us to analyze system state at the time of capture, revealing malicious activity that may not be visible through traditional file-based analysis.

Our memory analysis module supports multiple memory dump formats including crash dumps, raw memory dumps, VMware virtual machine memory files (.vmem), and hibernation files (hiberfil.sys). This flexibility ensures compatibility with various acquisition methods used in incident response scenarios.

# src/memory_analysis.py
import subprocess
import json
 
def run_volatility(command):
    """Run a Volatility command and return the output."""
    try:
        result = subprocess.run(
            command, 
            stdout=subprocess.PIPE, 
            stderr=subprocess.PIPE, 
            text=True
        )
        if result.returncode != 0:
            raise Exception(f"Error running command: {result.stderr.strip()}")
        return result.stdout
    except Exception as e:
        print(f"An error occurred: {e}")
        return None
 
def get_process_list(memory_dump_path, profile):
    """Get a list of processes from the memory dump."""
    command = [
        'python', 'vol.py', 
        '--profile', profile, 
        'pslist', '-f', memory_dump_path
    ]
    output = run_volatility(command)
    
    if output:
        process_list = []
        lines = output.splitlines()
        for line in lines[4:]:  # Skip header lines
            parts = line.split()
            if len(parts) >= 5:
                process_info = {
                    'PID': parts[0],
                    'PPID': parts[1],
                    'Name': parts[2],
                    'Offset': parts[3],
                    'Threads': parts[4]
                }
                process_list.append(process_info)
        return process_list
    return []
 
def analyze_memory_dump(memory_dump_path, profile):
    """Analyze the memory dump and return a summary."""
    processes = get_process_list(memory_dump_path, profile)
    dlls = get_dll_list(memory_dump_path, profile)
 
    analysis_summary = {
        'Process Count': len(processes),
        'Processes': processes,
        'DLL Count': len(dlls),
        'DLLs': dlls
    }
 
    return json.dumps(analysis_summary, indent=4)

The get_process_list function demonstrates how we extract running process information from memory dumps. By parsing the Volatility output, we can identify suspicious processes that may be hidden from standard system utilities—a common characteristic of rootkit infections.

2. Interrupt Descriptor Table (IDT) Detection

The Interrupt Descriptor Table is a critical data structure in x64 architecture that manages hardware interrupts and software exceptions. Rootkits frequently hook into the IDT to intercept system events and redirect them to malicious handlers. Our IDT detection module inspects each entry in the table to identify unauthorized modifications.

Each IDT entry consists of 16 bytes containing the handler address, segment selector, and access rights. By comparing the current IDT state against known-good baselines, we can detect hooks that redirect interrupt handling to rootkit code.

# src/detections/idt_detection.py
import ctypes
import struct
 
IDT_SIZE = 256  
IDT_ENTRY_SIZE = 16 
 
# Define the IDT entry structure matching x64 architecture
class IDTEntry(ctypes.Structure):
    _fields_ = [
        ("offset_low", ctypes.c_uint16),      # Bits 0-15 of handler address
        ("selector", ctypes.c_uint16),         # Code segment selector
        ("ist_index", ctypes.c_uint8),         # Interrupt Stack Table index
        ("type_attributes", ctypes.c_uint8),  # Type and privilege level
        ("offset_middle", ctypes.c_uint16),    # Bits 16-31 of handler address
        ("offset_high", ctypes.c_uint32),      # Bits 32-63 of handler address
        ("reserved", ctypes.c_uint32)          # Reserved for future use
    ]
 
def get_idtr():
    """Retrieve the IDT Register containing base address and limit."""
    idtr = ctypes.create_string_buffer(8)  
    ctypes.windll.kernel32.RtlGetInterruptDescriptorTable(ctypes.byref(idtr))
    base, limit = struct.unpack("<II", idtr)
    return base, limit
 
def read_idt(base, limit):
    """Read all IDT entries from memory."""
    idt_entries = []
    for i in range(IDT_SIZE):
        entry_offset = base + (i * IDT_ENTRY_SIZE)
        entry = IDTEntry.from_address(entry_offset)
        idt_entries.append(entry)
    return idt_entries

The IDT entry structure precisely mirrors the x64 architecture definition. By using ctypes to map this structure directly onto memory, we can examine each interrupt handler’s address and identify those pointing to unexpected memory regions.

3. System Service Descriptor Table (SSDT) Monitoring

On Windows systems, the System Service Descriptor Table (SSDT) bridges user-mode applications and kernel-mode system calls. Rootkits commonly target the SSDT to intercept and modify system call behavior, enabling them to hide files, processes, and network connections.

Our SSDT detection module reads the service table entries and compares them against known-good values. Any deviation from the baseline may indicate malicious hook installation.

# src/detections/ssdt_detection.py
import ctypes
 
# SSDT constants for x64 Windows
SSDT_SIZE = 256  
SSDT_ENTRY_SIZE = 8  
 
# SSDT entry structure definition
class SSDTEntry(ctypes.Structure):
    _fields_ = [
        ("ServiceTable", ctypes.c_void_p),    # Pointer to service function table
        ("CounterTable", ctypes.c_void_p),    # Optional counter table
        ("NumberOfServices", ctypes.c_uint32), # Number of services in table
        ("Reserved", ctypes.c_void_p)         # Reserved field
    ]
 
def get_ssdt_address():
    """Retrieve the SSDT base address."""
    # In production, this would use kernel-mode APIs
    # or driver communication to obtain the actual address
    return 0xFFFFF78000000000  # Example: typical SSDT location
 
def read_ssdt(base):
    """Read SSDT entries to detect hooks."""
    ssdt_entries = []
    for i in range(SSDT_SIZE):
        entry_offset = base + (i * SSDT_ENTRY_SIZE)
        entry = SSDTEntry.from_address(entry_offset)
        ssdt_entries.append(entry)
    return ssdt_entries

The SSDT structure reveals how system calls are dispatched. Each entry points to a function in the kernel that handles specific system operations. When a rootkit hooks the SSDT, it can intercept any system call, making this detection mechanism particularly valuable.

4. Import Address Table (IAT) Analysis

The Import Address Table contains pointers to external DLL functions that a PE (Portable Executable) file relies upon. Malware often modifies the IAT to redirect API calls to malicious wrappers, enabling techniques like API hashing and function hooking.

Our IAT detection module extracts and analyzes the import table from PE files, identifying suspicious redirections or unexpected DLL dependencies.

# src/detections/iat_detection.py
import pefile
 
def get_iat(pe):
    """Extracts the Import Address Table from a PE file."""
    iat = []
    if hasattr(pe, 'DIRECTORY_ENTRY_IMPORT'):
        for entry in pe.DIRECTORY_ENTRY_IMPORT:
            for imp in entry.imports:
                iat.append({
                    'dll': entry.dll.decode('utf-8'),
                    'function': imp.name.decode('utf-8') if imp.name else 'N/A',
                    'address': hex(imp.address)
                })
    return iat
 
def analyze_iat(pe_file_path):
    """Analyze IAT for suspicious modifications."""
    pe = pefile.PE(pe_file_path)
    iat_entries = get_iat(pe)
    
    # Check for known malicious DLLs or functions
    suspicious_patterns = {
        'dlls': ['payload.dll', 'hook.dll'],
        'functions': ['VirtualAllocEx', 'WriteProcessMemory']
    }
    
    for entry in iat_entries:
        if entry['dll'] in suspicious_patterns['dlls']:
            print(f"ALERT: Suspicious DLL detected: {entry['dll']}")
    
    return iat_entries

By examining the IAT, we can identify when legitimate applications have been modified to load malicious code. This technique is particularly effective for detecting user-mode rootkits and DLL injection attacks.

5. File Integrity Verification

Beyond runtime memory analysis, GlowRK incorporates file integrity checking using cryptographic hashes. By maintaining baseline hashes of critical system files, we can detect any unauthorized modifications that persist across reboots.

# src/detections/integrity_check.py
import hashlib
 
def calculate_sha256(file_path):
    """Calculate SHA-256 hash of a file."""
    sha256_hash = hashlib.sha256()
    with open(file_path, "rb") as f:
        for byte_block in iter(lambda: f.read(4096), b""):
            sha256_hash.update(byte_block)
    return sha256_hash.hexdigest()
 
def integrity_check(pe_file, expected_hash):
    """Verify file integrity against expected hash."""
    calculated_hash = calculate_sha256(pe_file)
 
    if calculated_hash == expected_hash:
        return {
            'status': 'pass',
            'message': 'File integrity verified'
        }
    else:
        return {
            'status': 'fail',
            'message': 'File has been modified!',
            'expected': expected_hash,
            'calculated': calculated_hash
        }

The integrity verification module uses SHA-256 hashing, which provides strong cryptographic guarantees against tampering. Organizations can establish baseline measurements during known-good states and regularly verify system files against these baselines.

Integration and Workflow

The Flask backend orchestrates all detection modules through a unified API. When a memory dump is uploaded, the system invokes each detection mechanism and aggregates the results into a comprehensive report.

# src/main.py
from flask import Flask, request, jsonify
from analysis.memory_analysis import analyze_memory_dump
from analysis.detection import (
    detect_idt_modifications,
    detect_ssdt_modifications,
    detect_iat_modifications,
    check_file_integrity
)
 
app = Flask(__name__)
 
@app.route('/upload', methods=['POST'])
def upload_file():
    if 'memoryDump' not in request.files:
        return jsonify({"error": "No file part"}), 400
 
    file = request.files['memoryDump']
    file_path = os.path.join(UPLOAD_FOLDER, file.filename)
    file.save(file_path)
    
    # Execute comprehensive analysis
    results = analyze_memory_dump(file_path)
    
    return jsonify(results), 200
 
def analyze_memory_dump(dump_path):
    """Orchestrate all detection modules."""
    memory_data = subprocess.run(
        ['volatility', '-f', dump_path, 
         '--profile=Win7SP1x64', 'pslist'],
        capture_output=True,
        text=True
    ).stdout
 
    results = {
        "memory_analysis": memory_data,
        "idt_check": detect_idt_modifications(memory_data),
        "ssdt_check": detect_ssdt_modifications(memory_data),
        "iat_check": detect_iat_modifications(memory_data),
        "integrity_check": check_file_integrity(dump_path)
    }
 
    return results

Memory Dump Formats Supported

GlowRK supports diverse memory acquisition formats, ensuring compatibility with various forensic tools and scenarios:

Format	Description	Use Case
Crash Dumps	Windows crash dump files	Blue screen analysis
Raw Dumps	Raw memory captures	Generic forensic analysis
.vmem	VMware virtual machine memory	VM forensics
hiberfil.sys	Hibernate file	Sleep/hibernate state analysis

Conclusion

GlowRK represents a comprehensive approach to rootkit detection, combining multiple techniques that address both kernel-mode and user-mode threats. By leveraging established frameworks like Volatility and implementing specialized detection for critical system structures, we provide security professionals with a powerful tool for identifying sophisticated malware.