Real-Time Alerting for Wireshark

Repository: https://github.com/x0prc/RTA-Plugin

Overview

A powerful Wireshark extension built in Lua that brings automated threat detection and alerting capabilities to network analysis. While Wireshark excels at packet analysis, it lacks built-in alerting mechanisms for suspicious activities. This plugin bridges that gap, providing real-time notifications for potential security threats.

---

The Problem: Manual Monitoring Limitations

Network administrators and security professionals often spend hours manually monitoring Wireshark captures, looking for anomalies and suspicious patterns. This manual approach has several drawbacks:

• Time-consuming: Requires constant attention to capture windows
• Error-prone: Human observers can miss subtle patterns
• Delayed response: Threats may go unnoticed until it’s too late
• Scalability issues: Impossible to monitor high-volume traffic manually

RTA-Plugin solves these challenges by automating the detection process and providing instant alerts through email notifications and log files.

---

Deep Dive: Code Architecture

1. Protocol Definition and Field Setup

Every Wireshark dissector begins with protocol definition. The RTA-Plugin creates a custom protocol object and defines the fields it will populate:

-- Import the SMTP library for email notifications
local smtp = require("socket.smtp")
 
-- Create a new protocol object for our anomaly detector
local anomaly_detector_proto = Proto("RTA", "Anomaly Detector Protocol")
 
-- Define protocol fields that will appear in the Wireshark packet tree
local fields = anomaly_detector_proto.fields
fields.src_ip = ProtoField.ipv4("anomaly_detector.src_ip", "Source IP")
fields.dst_ip = ProtoField.ipv4("anomaly_detector.dst_ip", "Destination IP")
fields.src_port = ProtoField.uint16("anomaly_detector.src_port", "Source Port")
fields.dst_port = ProtoField.uint16("anomaly_detector.dst_port", "Destination Port")

Key Points:

• Proto() creates a new protocol dissector with a name and description
• ProtoField definitions specify the data types (IPv4 addresses, 16-bit integers for ports)
• These fields will be displayed in Wireshark’s packet details pane

2. SMTP Configuration for Email Alerts

The plugin uses SMTP for real-time email notifications. Configuration is centralized for easy modification:

-- SMTP server configuration for email alerts
-- IMPORTANT: Update these values with your actual SMTP credentials
local smtp_config = {
    server = "smtp.example.com",    -- Replace with your SMTP server
    port = 587,                     -- SMTP port (587 for TLS, 465 for SSL)
    user = "your-email@example.com", -- Your email address
    password = "your-password",      -- Your email password
    from = "<your-email@example.com>", -- Sender email
    to = "<recipient-email@example.com>", -- Recipient email
}

Security Note: Store credentials securely and consider using environment variables or separate configuration files in production environments.

3. Detection Thresholds and Data Structures

The plugin maintains several data structures for tracking network behavior:

-- Detection thresholds - adjust based on your network baseline
tráfico local traffic_threshold = 100  -- Packets per source IP before triggering alert
local error_threshold = 10     -- Error count threshold
 
-- Data structure to track traffic patterns per source IP
-- Structure: { src_ip = { total_packets, errors, destinations = { dst_ip = { total_packets, ports = { port = { count } } } } } }
local traffic_patterns = {}
 
-- Blacklist of known malicious IP addresses
local known_malicious_ips = {
    ["192.168.1.100"] = true,
    ["10.10.10.10"] = true
}
 
-- Common/allowed ports - traffic on other ports will trigger alerts
local common_ports = {
    [80] = true,    -- HTTP
    [443] = true,   -- HTTPS
    [22] = true,    -- SSH
    [53] = true     -- DNS
}

Design Insight: Using Lua tables as hash maps provides O(1) lookup time for checking malicious IPs and common ports, making the plugin efficient even with large datasets.

4. Email Alert Function

The send_email_alert function handles notification delivery:

function send_email_alert(subject, message)
    -- Construct the email message with headers and body
    local msg = {
        headers = {
            to = smtp_config.to,
            subject = subject
        },
        body = message
    }
    
    -- Send email using luasocket's SMTP module
    local r, e = smtp.send {
        from = smtp_config.from,
        rcpt = smtp_config.to,
        source = smtp.message(msg),
        user = smtp_config.user,
        password = smtp_config.password,
        server = smtp_config.server,
        port = smtp_config.port,
        authentication = "login",  -- Authentication method
        ssl = false                -- Set to true for SSL/TLS
    }
 
    -- Error handling
    if not r then
        print("Failed to send email: " .. e)
    else
        print("Alert email sent successfully!")
    end
end

Implementation Details:

• Uses the luasocket library’s SMTP module
• Supports authentication and SSL/TLS encryption
• Provides console feedback for debugging

5. Persistent Logging

In addition to email alerts, the plugin maintains a local log file for audit trails:

-- Function to log threats to a file for persistent record keeping
function log_threat(threat_message)
    -- Open log file in append mode (creates if doesn't exist)
    local log_file = io.open("RTA_log.txt", "a")
    if log_file then
        -- Write threat message with newline
        log_file:write(threat_message .. "\n")
        log_file:close()
    else
        print("Failed to open log file.")
    end
end

Best Practice: The append mode (“a”) ensures new threats are added without overwriting existing logs, creating a comprehensive audit trail.

6. Traffic Pattern Tracking

The core data collection function maintains statistics on network traffic:

function update_traffic_patterns(src_ip, dst_ip, port)
    -- Initialize tracking structure for new source IPs
    if traffic_patterns[src_ip] == nil then
        traffic_patterns[src_ip] = { 
            total_packets = 0, 
            errors = 0, 
            destinations = {} 
        }
    end
    
    -- Increment total packet count for source IP
    traffic_patterns[src_ip].total_packets = traffic_patterns[src_ip].total_packets + 1
    
    -- Initialize destination tracking if needed
    if traffic_patterns[src_ip].destinations[dst_ip] == nil then
        traffic_patterns[src_ip].destinations[dst_ip] = { 
            total_packets = 0, 
            ports = {} 
        }
    end
    
    -- Increment destination-specific packet count
    traffic_patterns[src_ip].destinations[dst_ip].total_packets = traffic_patterns[src_ip].destinations[dst_ip].total_packets + 1
    
    -- Initialize port tracking if needed
    if traffic_patterns[src_ip].destinations[dst_ip].ports[port] == nil then
        traffic_patterns[src_ip].destinations[dst_ip].ports[port] = { count = 0 }
    end
    
    -- Increment port-specific count
    traffic_patterns[src_ip].destinations[dst_ip].ports[port].count = traffic_patterns[src_ip].destinations[dst_ip].ports[port].count + 1
end

Data Structure Design:

• Hierarchical structure: Source IP → Destination IP → Port
• Enables multi-dimensional analysis (who is talking to whom, on what ports)
• Memory-efficient using Lua’s table semantics

7. Detection Functions

Traffic Spike Detection

function detect_traffic_spikes(traffic_patterns)
    -- Iterate through all tracked source IPs
    for src_ip, data in pairs(traffic_patterns) do
        -- Check if packet count exceeds threshold
        if data.total_packets > traffic_threshold then
            local message = "ALERT: Traffic spike detected from source IP " .. src_ip
            print(message)
            send_email_alert("Traffic Spike Detected", message)
            log_threat(message)
        end
    end
end

This function scans through all tracked traffic patterns and flags any source IP that exceeds the defined threshold (100 packets). This is useful for detecting:

• DDoS attacks
• Port scanning activities
• Misconfigured applications flooding the network

Uncommon Port Detection

function detect_uncommon_ports(traffic_patterns)
    -- Iterate through all source IPs
    for src_ip, data in pairs(traffic_patterns) do
        -- Iterate through all destinations for this source
        for dst_ip, dst_data in pairs(data.destinations) do
            -- Check each port used
            for port, port_data in pairs(dst_data.ports) do
                -- If port is not in our common_ports whitelist, alert
                if not common_ports[port] then
                    local message = "ALERT: Uncommon port detected! Src IP: " .. src_ip .. " Dst IP: " .. dst_ip .. " Port: " .. port
                    print(message)
                    send_email_alert("Uncommon Port Detected", message)
                    log_threat(message)
                end
            end
        end
    end
end

This multi-level nested loop provides fine-grained visibility into port usage patterns. It’s effective for detecting:

• Backdoors or unauthorized services
• Exfiltration attempts over unusual ports
• Protocol tunneling

Malformed Packet Detection

function detect_malformed_packets(pinfo)
    -- Check if Wireshark detected errors in the packet
    if pinfo.err ~= nil then
        local message = "ALERT: Malformed packet detected! Frame: " .. pinfo.number .. " Reason: " .. pinfo.err
        print(message)
        send_email_alert("Malformed Packet Detected", message)
        log_threat(message)
    end
end

Leveraging Wireshark’s built-in error detection, this function catches:

• Protocol violations
• Truncated packets
• Checksum failures
• Maliciously crafted packets designed to exploit parser vulnerabilities

High Error Rate Detection

function detect_high_error_rates(traffic_patterns)
    for src_ip, data in pairs(traffic_patterns) do
        -- Check if error count exceeds threshold
        if data.errors and data.errors > error_threshold then
            local message = "ALERT: High error rate detected from source IP " .. src_ip .. " with " .. data.errors .. " errors."
            print(message)
            send_email_alert("High Error Rate Detected", message)
            log_threat(message)
        end
    end
end

This function tracks cumulative errors per source IP, helping identify:

• Faulty network equipment
• Misconfigured hosts
• Deliberate fuzzing attacks

Known Malicious Signature Detection

-- Function to detect known malicious signatures
function detect_known_malicious_signatures(traffic_patterns)
    for src_ip, data in pairs(traffic_patterns) do
        -- Check if source IP is in our blacklist
        if known_malicious_ips[src_ip] then
            local message = "ALERT: Known malicious IP detected! Src IP: " .. src_ip
            print(message)
            send_email_alert("Malicious IP Detected", message)
            log_threat(message)
        end
    end
end

Using a simple hash lookup for O(1) performance, this function provides immediate alerts for traffic from known bad actors. The blacklist can be populated from:

• Threat intelligence feeds
• Previous incident responses
• Internal security policies

8. The Dissector Function: Integration Point

The dissector function is Wireshark’s entry point - it’s called for every packet:

-- Main dissector function - Wireshark calls this for each packet
function anomaly_detector_proto.dissector(buffer, pinfo, tree)
    -- Skip non-IP packets
    if pinfo.ip == nil then return end
 
    -- Extract packet information from Wireshark's pinfo object
    local src_ip = tostring(pinfo.src)
    local dst_ip = tostring(pinfo.dst)
    local src_port = pinfo.src_port
    local dst_port = pinfo.dst_port
 
    -- Add our protocol to the packet tree (visible in Wireshark UI)
    local subtree = tree:add(anomaly_detector_proto, buffer(), "Anomaly Detector Protocol")
    subtree:add(fields.src_ip, pinfo.src)
    subtree:add(fields.dst_ip, pinfo.dst)
    subtree:add(fields.src_port, src_port)
    subtree:add(fields.dst_port, dst_port)
 
    -- Update traffic tracking with current packet data
    update_traffic_patterns(src_ip, dst_ip, dst_port)
 
    -- Run all detection algorithms
    detect_traffic_spikes(traffic_patterns)
    detect_uncommon_ports(traffic_patterns)
    detect_malformed_packets(pinfo)
    detect_high_error_rates(traffic_patterns)
    detect_known_malicious_signatures(traffic_patterns)
 
    -- Set protocol column in Wireshark's packet list
    pinfo.cols.protocol = "RTA"
end

Parameters:

• buffer: The packet data (not used directly here, but required by Wireshark)
• pinfo: Packet information object containing metadata (IPs, ports, errors, etc.)
• tree: The protocol tree where we add our analysis results

9. Protocol Registration

Finally, we register the dissector with Wireshark:

-- Register the dissector for IP protocol
local ip_table = DissectorTable.get("ip.proto")
ip_table:add(0, anomaly_detector_proto)  -- 0 applies to all IP traffic

The value 0 tells Wireshark to apply this dissector to all IP protocol traffic, ensuring comprehensive coverage.

---

Testing the Plugin

Simulate a DDoS Attack

# Use hping3 to generate high-volume traffic
sudo hping3 -S -p 80 --flood 192.168.1.1

Port Scan Detection

# Use nmap for port scanning
nmap -p 1-65535 192.168.1.1

Malformed Packet Testing

# Use scapy to create malformed packets
# Save as test_malformed.py and run with sudo
from scapy.all import *
packet = IP(dst="192.168.1.1")/TCP(flags="FSRPAU")  # Invalid flag combination
send(packet)

Verify Alerts

Check three locations for alerts:

1. Wireshark Console: Alert messages printed to stdout
2. Email Inbox: SMTP notifications
3. RTA_log.txt: Persistent log file in the working directory

---

Performance Considerations

Memory Usage

The traffic_patterns table grows with unique source IPs, destinations, and ports. In high-volume environments:

• Problem: Unbounded growth could exhaust memory
• Solution: Implement periodic cleanup or use a circular buffer

-- Example: Add cleanup to prevent memory leaks
local MAX_ENTRIES = 10000
if table_size(traffic_patterns) > MAX_ENTRIES then
    -- Remove oldest entries or reset patterns
    traffic_patterns = {}
end

Detection Frequency

Currently, detection functions run on every packet. For performance:

• Batch processing: Run detection every N packets instead of every packet
• Sampling: Process only a percentage of packets in high-volume scenarios
• Time-based: Run detection on intervals (e.g., every 5 seconds)

SMTP Rate Limiting

Email servers often have rate limits. The current implementation may trigger multiple emails for the same threat:

-- Add debouncing to prevent email flooding
local last_alert_time = {}
local ALERT_COOLDOWN = 300  -- 5 minutes
 
function send_email_alert(subject, message)
    local key = subject .. message
    local now = os.time()
    if last_alert_time[key] and (now - last_alert_time[key]) < ALERT_COOLDOWN then
        return  -- Skip duplicate alert
    end
    last_alert_time[key] = now
    -- ... rest of email code
end

---

Conclusion

RTA-Plugin demonstrates how Wireshark’s extensible architecture can be leveraged to build powerful network security tools. By combining:

• Wireshark’s deep packet inspection capabilities
• Lua’s lightweight scripting for logic implementation
• SMTP notifications for real-time alerting
• Persistent logging for compliance and forensics

…we create a comprehensive solution for automated threat detection.

The modular design makes it easy to:

• Add new detection algorithms
• Customize thresholds for your environment
• Integrate with existing security infrastructure
• Extend functionality for specific use cases

Whether you’re protecting a small network or monitoring enterprise traffic, RTA-Plugin provides a foundation for automated security monitoring that scales with your needs.

---

Resources

• Wireshark Lua API Reference: https://www.wireshark.org/docs/wsdg_html_chunked/wslua.html
• LuaSocket Documentation: http://w3.impa.br/~diego/software/luasocket/