• Stars
    star
    240
  • Rank 168,229 (Top 4 %)
  • Language
    Python
  • License
    MIT License
  • Created about 6 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A no-nonsense c-like structure parsing library for Python

Something new coming soon

dissect.cstruct

Structure parsing in Python made easy. With cstruct, you can write C-like structures and use them to parse binary data, either as file-like objects or bytestrings.

Parsing binary data with cstruct feels familiar and easy. No need to learn a new syntax or the quirks of a new parsing library before you can start parsing data. The syntax isn't strict C but it's compatible with most common structure definitions. You can often use structure definitions from open-source C projects and use them out of the box with little to no changes. Need to parse an EXT4 super block? Just copy the structure definition from the Linux kernel source code. Need to parse some custom file format? Write up a simple structure and immediately start parsing data, tweaking the structure as you go.

By design cstruct is incredibly simple. No complex syntax, filters, pre- or postprocessing steps. Just structure parsing.

Installation

pip install dissect.cstruct

Usage

All you need to do is instantiate a new cstruct instance and load some structure definitions in there. After that you can start using them from your Python code.

from dissect import cstruct

# Default endianness is LE, but can be configured using a kwarg or setting the 'endian' attribute
# e.g. cstruct.cstruct(endian='>') or cparser.endian = '>'
cparser = cstruct.cstruct()
cparser.load("""
#define SOME_CONSTANT   5

enum Example : uint16 {
    A, B = 0x5, C
};

struct some_struct {
    uint8   field_1;
    char    field_2[SOME_CONSTANT];
    char    field_3[field_1 & 1 * 5];  // Some random expression to calculate array length
    Example field_4[2];
};
""")

data = b'\x01helloworld\x00\x00\x06\x00'
result = cparser.some_struct(data)  # Also accepts file-like objects
assert result.field_1 == 0x01
assert result.field_2 == b'hello'
assert result.field_3 == b'world'
assert result.field_4 == [cparser.Example.A, cparser.Example.C]

assert cparser.Example.A == 0
assert cparser.Example.C == 6
assert cparser.Example(5) == cparser.Example.B

assert result.dumps() == data

# You can also instantiate structures from Python by using kwargs
# Note that array sizes are not enforced
instance = cparser.some_struct(field_1=5, field_2='lorem', field_3='ipsum', field_4=[cparser.Example.B, cparser.Example.A])
assert instance.dumps() == b'\x05loremipsum\x05\x00\x00\x00'

By default, all structures are compiled into classes that provide optimised performance. You can disable this by passing a compiled=False keyword argument to the .load() call. You can also inspect the resulting source code by accessing the source attribute of the structure: print(cparser.some_struct.source).

More examples can be found in the examples directory.

Features

Structure parsing

Write simple C-like structures and use them to parse binary data, as can be seen in the examples.

Type parsing

Aside from loading structure definitions, any of the supported types can be used individually for parsing data. For example, the following is all supported:

from dissect import cstruct
cs = cstruct.cstruct()
# Default endianness is LE, but can be configured using a kwarg or setting the attribute
# e.g. cstruct.cstruct(endian='>') or cs.endian = '>'
assert cs.uint32(b'\x05\x00\x00\x00') == 5
assert cs.uint24[2](b'\x01\x00\x00\x02\x00\x00') == [1, 2]  # You can also parse arrays using list indexing
assert cs.char[None](b'hello world!\x00') == b'hello world!'  # A list index of None means null terminated

Unions and nested structures

Unions and nested structures are support, both anonymous and named.

cdef = """
struct test_union {
    char magic[4];
    union {
        struct {
            uint32 a;
            uint32 b;
        } a;
        struct {
            char   b[8];
        } b;
    } c;
};

struct test_anonymous {
    char magic[4];
    struct {
        uint32 a;
        uint32 b;
    };
    struct {
        char   c[8];
    };
};
"""
c = cstruct.cstruct()
c.load(cdef)

assert len(c.test_union) == 12

a = c.test_union(b'ohaideadbeef')
assert a.magic == b'ohai'
assert a.c.a.a == 0x64616564
assert a.c.a.b == 0x66656562
assert a.c.b.b == b'deadbeef'

assert a.dumps() == b'ohaideadbeef'

b = c.test_anonymous(b'ohai\x39\x05\x00\x00\x28\x23\x00\x00deadbeef')
assert b.magic == b'ohai'
assert b.a == 1337
assert b.b == 9000
assert b.c == b'deadbeef'

Parse bit fields

Bit fields are supported as part of structures. They are properly aligned to their boundaries.

bitdef = """
struct test {
    uint16  a:1;
    uint16  b:1;  # Read 2 bits from an uint16
    uint32  c;    # The next field is properly aligned
    uint16  d:2;
    uint16  e:3;
};
"""
bitfields = cstruct.cstruct()
bitfields.load(bitdef)

d = b'\x03\x00\xff\x00\x00\x00\x1f\x00'
a = bitfields.test(d)

assert a.a == 0b1
assert a.b == 0b1
assert a.c == 0xff
assert a.d == 0b11
assert a.e == 0b111
assert a.dumps() == d

Enums

The API to access enum members and their values is similar to that of the native Enum type in Python 3. Functionally, it's best comparable to the IntEnum type.

Custom types

You can implement your own types by subclassing BaseType or RawType, and adding them to your cstruct instance with addtype(name, type)

Custom definition parsers

Don't like the C-like definition syntax? Write your own syntax parser!

More Repositories

1

dissect

Dissect is a digital forensics & incident response framework and toolset that allows you to quickly access and analyse forensic artefacts from various disk and file formats, developed by Fox-IT (part of NCC Group).
779
star
2

aclpwn.py

Active Directory ACL exploitation with BloodHound
Python
632
star
3

log4j-finder

Find vulnerable Log4j2 versions on disk and also inside Java Archive Files (Log4Shell CVE-2021-44228, CVE-2021-45046, CVE-2021-45105)
Python
435
star
4

cve-2019-1040-scanner

Python
276
star
5

quantuminsert

Quantum Insert
HTML
211
star
6

LDAPFragger

C#
181
star
7

mkYARA

Generating YARA rules based on binary code
Python
179
star
8

linux-luks-tpm-boot

A guide for setting up LUKS boot with a key from TPM in Linux
Shell
175
star
9

Invoke-CredentialPhisher

PowerShell
170
star
10

bloodhound-import

Python based BloodHound data importer
Python
141
star
11

dissect.cobaltstrike

Python library for dissecting and parsing Cobalt Strike related data such as Beacon payloads and Malleable C2 Profiles
Python
139
star
12

danderspritz-evtx

Parse evtx files and detect use of the DanderSpritz eventlogedit module
Python
139
star
13

cryptophp

CryptoPHP Indicators of Compromise
Python
128
star
14

cobaltstrike-extraneous-space

Historical list of {Cobalt Strike,NanoHTTPD} servers
125
star
15

cobaltstrike-beacon-data

Open Dataset of Cobalt Strike Beacon metadata (2018-2022)
Jupyter Notebook
114
star
16

OpenSSH-Session-Key-Recovery

Project containing several tools/ scripts to recover the OpenSSH session keys used to encrypt/ decrypt SSH traffic.
Python
72
star
17

acquire

acquire is a tool to quickly gather forensic artifacts from disk images or a live system into a lightweight container.
Python
57
star
18

OpenSSH-Network-Parser

Project to decrypt and parse SSH traffic
Python
54
star
19

bro-scripts

Bro-IDS scripts
Bro
51
star
20

cisco-ios-xe-implant-detection

Cisco IOS XE implant scanning & detection (CVE-2023-20198, CVE-2023-20273)
Python
38
star
21

dissect.cstruct

A Dissect module implementing a parser for C-like structures.
Python
30
star
22

operation-wocao

Operation Wocao - Indicators of Compromise
YARA
30
star
23

dissect.target

The Dissect module tying all other Dissect modules together. It provides a programming API and command line tools which allow easy access to various data sources inside disk images or file collections (a.k.a. targets).
Python
27
star
24

dll-hijacking-poc

A quick POC on how to embed a meterpreter in Firefox via DLL hijacking
C
17
star
25

citrix-netscaler-triage

Dissect triage script for Citrix NetScaler devices
Python
17
star
26

Decrypt-TFSSecretVariables

PowerShell
15
star
27

ponmocup

Ponmocup Indicators of Compromise
13
star
28

signed-phishing-email

Python
11
star
29

pcap-broker

PCAP-over-IP server written in Golang
Go
11
star
30

mofang

Mofang Indicators of Compromise
10
star
31

spookyssl-pcaps

SpookySSL PCAPS and Network Coverage
10
star
32

dissect.esedb

A Dissect module implementing a parser for Microsofts Extensible Storage Engine Database (ESEDB), used for example in Active Directory, Exchange and Windows Update.
Python
10
star
33

dissect-docs

Dissect documentation project
7
star
34

log4shell-pcaps

Log4Shell PCAPS and Network Coverage
Java
7
star
35

dissect.evidence

A Dissect module implementing a parsers for various forensic evidence file containers, currently: AD1, ASDF and EWF.
Python
7
star
36

dissect.ntfs

A Dissect module implementing a parser for the NTFS file system, used by the Windows operating system.
Python
7
star
37

dissect.eventlog

A Dissect module implementing parsers for the Windows EVT, EVTX and WEVT log file formats.
Python
6
star
38

saitama-server

Server-side implementation for the Saitama implant, useful for detection engineering purposes
Python
6
star
39

flow.record

Recordization library
Python
6
star
40

django-auth-policy

Django Authentication Policy
Python
6
star
41

psixbot

PsiXBot Indicators of Compromise
5
star
42

reasm

Extract parts of the malware and re-compile it on linux for decrypting stuff using same malware algorithms.
Python
5
star
43

aws-lambda-kinesis-windowseventlog

AWS lambda to transform the json from AWS kinesis agent to useful json documents for elasticsearch
Python
5
star
44

dissect.cim

A Dissect module implementing a parser for the Windows Common Information Model (CIM) database, used in the Windows operating system.
Python
5
star
45

dissect.hypervisor

A Dissect module implementing parsers for various hypervisor disk, backup and configuration files.
Python
4
star
46

Decrypt-OrchestratorSecretVariables

PowerShell
4
star
47

dissect.sql

A Dissect module implementing a parsers for the SQLite database file format, commonly used by applications to store configuration data.
Python
4
star
48

blister-research

Scripts, YARA and IOCs from our research on the Blister malware 🩹
Python
3
star
49

dissect.clfs

A Dissect module implementing a parser for the CLFS (Common Log File System) file system of Windows.
Python
3
star
50

dissect.volume

A Dissect module implementing a parser for different disk volume and partition systems, for example LVM2, GPT and MBR.
Python
3
star
51

dissect.ole

A Dissect module implementing a parser for the Object Linking & Embedding (OLE) format, commonly used by document editors on Windows operating systems.
Python
3
star
52

dissect.vmfs

Dissect module implementing a parser for the VMFS file system, used by VMware virtualization software.
Python
3
star
53

dissect.regf

A Dissect module implementing a parser for Windows registry file format, used to store application and OS configuration on Windows operating systems.
Python
3
star
54

Invoke-BadPwdCountSprayer

2
star
55

dissect.fat

A Dissect module implementing parsers for the FAT and exFAT file systems, commonly used on flash memory based storage devices and UEFI partitions.
Python
2
star
56

dissect.shellitem

A Dissect module implementing a parser for the Shellitem structures, commonly used by Microsoft Windows.
Python
2
star
57

dissect.util

A Dissect module implementing various utility functions for the other Dissect modules.
Python
2
star
58

dissect.ffs

A Dissect module implementing a parser for the FFS file system, commonly used by BSD operating systems.
Python
2
star
59

dissect.xfs

A Dissect module implementing a parser for the XFS file system, commonly used by RedHat Linux distributions.
Python
2
star
60

dissect-workflow-templates

Workflow templates for the dissect projects
2
star
61

dissect_legacy

Namespace and collection package for all dissect projects
Python
2
star
62

dissect.extfs

A Dissect module implementing a parser for the ExtFS file system, the native filesystem for Linux operating systems.
Python
1
star
63

dissect.btrfs

A Dissect module implementing a parser for the btrfs file system.
Python
1
star
64

dissect.etl

A Dissect module implementing a parser for Event Trace Log (ETL) files, used by the Windows operating system to log kernel events.
Python
1
star