protocol-reverse-engineering

Protocol Reverse Engineering

Comprehensive techniques for capturing, analyzing, and documenting network protocols for security research, interoperability, and debugging.

Authorization and Scope

Capture, intercept, fuzz, or replay traffic only on networks, systems, devices, and accounts the user owns or is explicitly authorized in writing to test. Intercepting the traffic of other people can carry legal and consent obligations. Confirm the target host, port, and scope with the user before any step that sends or intercepts traffic.

Live capture, transparent proxying, and sending raw packets need elevated privileges or network capabilities. Grant only the minimum capability needed, prefer a dedicated virtual machine or isolated network namespace, and revert any firewall redirection rules after transparent proxying.

Traffic Capture

Apply a capture filter scoped to the authorized target host and port, as in the filtered examples below, and capture only on networks you are authorized to monitor. Write captures to a dedicated output directory readable only by the owner, using timestamped file names so existing files stay intact. The ring buffer example can use about a gigabyte, so check free space and size limits first, and clean up capture files after analysis.

Wireshark Capture

# Capture on specific interface
wireshark -i eth0 -k

# Capture with filter
wireshark -i eth0 -k -f "port 443"

# Capture to file
tshark -i eth0 -w capture.pcap

# Ring buffer capture (rotate files)
tshark -i eth0 -b filesize:100000 -b files:10 -w capture.pcap

tcpdump Capture

# Basic capture
tcpdump -i eth0 -w capture.pcap

# With filter
tcpdump -i eth0 port 8080 -w capture.pcap

# Capture specific bytes
tcpdump -i eth0 -s 0 -w capture.pcap  # Full packet

# Real-time display
tcpdump -i eth0 -X port 80

Man-in-the-Middle Capture

# mitmproxy for HTTP/HTTPS
# Explicit proxy mode for clients you control
mitmproxy -p 8080

# SSL/TLS interception
# Transparent mode: use only on your own devices and test accounts with
# documented authorization. It needs firewall redirection rules, which
# must be reverted afterward.
# The insecure option turns off checks of upstream server certificates.
# Use it only in an isolated lab with known self signed upstream servers,
# and install the mitmproxy certificate authority on the test client for client trust.
mitmproxy --mode transparent --ssl-insecure

# Dump to file
mitmdump -w traffic.mitm

# Burp Suite
# Configure browser proxy to 127.0.0.1:8080

Protocol Analysis

Analyze captures from untrusted sources as an unprivileged user inside a sandbox, container, or virtual machine with current tools, and run Wireshark and tshark analysis as a regular user.

Wireshark Analysis

# Display filters
tcp.port == 8080
http.request.method == "POST"
ip.addr == 192.168.1.1
tcp.flags.syn == 1 && tcp.flags.ack == 0
frame contains "password"

# Following streams
Right-click > Follow > TCP Stream
Right-click > Follow > HTTP Stream

# Export objects
File > Export Objects > HTTP

# Decryption
Edit > Preferences > Protocols > TLS
  - (Pre)-Master-Secret log filename
  - RSA keys list

tshark Analysis

# Extract specific fields
tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.port

# Statistics
tshark -r capture.pcap -q -z conv,tcp
tshark -r capture.pcap -q -z endpoints,ip

# Filter and extract
tshark -r capture.pcap -Y "http" -T json > http_traffic.json

# Protocol hierarchy
tshark -r capture.pcap -q -z io,phs

Scapy for Custom Analysis

from scapy.all import *

# Read pcap
packets = rdpcap("capture.pcap")

# Analyze packets
for pkt in packets:
    if pkt.haslayer(TCP):
        print(f"Src: {pkt[IP].src}:{pkt[TCP].sport}")
        print(f"Dst: {pkt[IP].dst}:{pkt[TCP].dport}")
        if pkt.haslayer(Raw):
            print(f"Data: {pkt[Raw].load[:50]}")

# Filter packets
http_packets = [p for p in packets if p.haslayer(TCP)
                and (p[TCP].sport == 80 or p[TCP].dport == 80)]

# Create custom packets
pkt = IP(dst="target")/TCP(dport=80)/Raw(load="GET / HTTP/1.1\r\n")

Sending a crafted packet needs raw socket privileges and authorization. Do it only as part of Active Testing against a lab host the user has confirmed.

Protocol Identification

Common Protocol Signatures

HTTP        - "HTTP/1." or "GET " or "POST " at start
TLS/SSL     - 0x16 0x03 (record layer)
DNS         - UDP port 53, specific header format
SMB         - 0xFF 0x53 0x4D 0x42 ("SMB" signature)
SSH         - "SSH-2.0" banner
FTP         - "220 " response, "USER " command
SMTP        - "220 " banner, "EHLO" command
MySQL       - 0x00 length prefix, protocol version
PostgreSQL  - 0x00 0x00 0x00 startup length
Redis       - "*" RESP array prefix
MongoDB     - BSON documents with specific header

Protocol Header Patterns

+--------+--------+--------+--------+
|  Magic number / Signature         |
+--------+--------+--------+--------+
|  Version       |  Flags          |
+--------+--------+--------+--------+
|  Length        |  Message Type   |
+--------+--------+--------+--------+
|  Sequence Number / Session ID     |
+--------+--------+--------+--------+
|  Payload...                       |
+--------+--------+--------+--------+

Before binary protocol or encryption analysis, read parsers, structure identification, and TLS techniques.

Before documenting a protocol or writing a dissector, read the specification template and Lua example.

Active Testing

Fuzzing with Boofuzz

Fuzz only an isolated lab host the user owns or is authorized to test. Replace the placeholder target with that lab host only after the user confirms the host and scope, and run crash or restart monitoring only in the lab. Get the user's confirmation of the source and version of scapy, boofuzz, mitmproxy, and Wireshark before installing them, and prefer official project sources at reviewed, pinned versions.

from boofuzz import *

def main():
    session = Session(
        target=Target(
            connection=TCPSocketConnection("target", 8888)
        )
    )

    # Define protocol structure
    s_initialize("HELLO")
    s_static(b"\x50\x52\x4f\x54")  # Magic
    s_word(1, name="version")       # Version
    s_word(0x01, name="type")       # Type (HELLO)
    s_size("payload", length=4)     # Length field
    s_block_start("payload")
    s_dword(0x12345678, name="client_id")
    s_word(0, name="flags")
    s_block_end()

    session.connect(s_get("HELLO"))
    session.fuzz()

if __name__ == "__main__":
    main()

Replay and Modification

Replay only captures of your own test traffic, and only toward an authorized isolated lab host. Change each packet destination to that lab host, remove or redact authentication and session data from payloads, and get user confirmation before transmitting. Prefer saving modified packets to a file for offline review.

from scapy.all import *

# Replay captured traffic
packets = rdpcap("capture.pcap")
for pkt in packets:
    if pkt.haslayer(TCP) and pkt[TCP].dport == 8888:
        send(pkt)

# Modify and replay
for pkt in packets:
    if pkt.haslayer(Raw):
        # Modify payload
        original = pkt[Raw].load
        modified = original.replace(b"client", b"CLIENT")
        pkt[Raw].load = modified
        # Recalculate checksums
        del pkt[IP].chksum
        del pkt[TCP].chksum
        send(pkt)

Best Practices

Analysis Workflow

1. Capture traffic: Multiple sessions, different scenarios
2. Identify boundaries: Message start/end markers
3. Map structure: Fixed header, variable payload
4. Identify fields: Compare multiple samples
5. Document format: Create specification
6. Validate understanding: Implement parser/generator
7. Test edge cases: Fuzzing, boundary conditions

Sensitive Data Handling

• Use test credentials only.
• Keep captures, proxy dumps, key logs, private keys, and exported objects in a private directory readable only by the owner, outside shared or tracked locations, and encrypted where possible.
• Refer to secrets found in traffic by type or location only, and keep their values out of agent output.
• Delete or redact these artifacts after analysis.

Common Patterns to Look For

• Magic numbers/signatures at message start
• Version fields for compatibility
• Length fields (often before variable data)
• Type/opcode fields for message identification
• Sequence numbers for ordering
• Checksums/CRCs for integrity
• Timestamps for timing
• Session/connection identifiers