• Stars
    star
    132
  • Rank 274,205 (Top 6 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated almost 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Event Trace Log file parser in pure Python

etl-parser

Event Trace Log file reader in pure Python

etl-parser is a pure Python 3 parser library for ETL Windows log files. ETL is the default format for ETW as well as the default format for the Kernel logger.

etl-parser has no system dependencies, and will work well on both Windows and Linux.

Since this format is not documented, we merged information from the blog of Geoff Chappel and reverse engineering activities conducted by Airbus CERT team.

What is ETL and why is it a pain to work with? Consider ETL as a container, like AVI is for video files. Reading ETL is similarly frustrating as reading an AVI file without the right codec.

etl-parser tries to solve this problem by including parsers for the following well known log formats:

  • ETW manifest base provider
  • TraceLogging
  • MOF for kernel log

How to use etl-parser?

etl-parser offers two scripts. The first script, etl2xml transforms all known ETL events into XML:

etl2xml -i example.etl -o example.xml

The second script, etl2pcap transforms network captures created through netsh into the pcap file format:

netsh start trace capture=yes
netsh stop trace

etl2pcap -i NetTrace.etl -o NetTrace.pcap

You can also use etl-parser as a library:

from etl.etl import IEtlFileObserver, build_from_stream
from etl.system import SystemTraceRecord
from etl.perf import PerfInfo
from etl.event import Event
from etl.trace import Trace
from etl.wintrace import WinTrace

class EtlFileLogger(IEtlFileObserver):
    def on_system_trace(self, event: SystemTraceRecord):
        """Mof kernel message with Process Id and Thread Id"""
        mof = event.get_mof() # Invoke MOF parser

    def on_perfinfo_trace(self, event: PerfInfo):
        """Mof kernel message with timestamp"""
        mof = event.get_mof() # Invoke MOF parser

    def on_trace_record(self, event: Trace):
        """unknown"""

    def on_event_record(self, event: Event):
        """ETW event this is what you search"""
        # Choose the "parse_" function which corresponds to your event
        message = event.parse_tracelogging() # Invoke TraceLogging parser
        message = event.parse_etw() # Invoke Manifest based parser

    def on_win_trace(self, event: WinTrace):
        """unknown"""
        etw = event.parse_etw()

with open("example.etl", "rb") as etl_file:
    etl_reader = build_from_stream(etl_file.read())
    etl_reader.parse(EtlFileLogger())

Installation

etl-parser is available from pip:

pip install etl-parser

Alternatively, you can install etl-parser using setup.py:

git clone https://github.com/airbus-cert/etl-parser.git
cd etl-parser
pip install -e .

Missing a parser?

If you encounter a parsing error, please open an issue on the Airbus CERT GitHub repository.

Why an ETL Parser?

The EVTX log format is fairly well documented, with lots of libraries and tools available today. This is not true for ETL: at time of development, there is no significant open-source project that we know of and the ETL format is not well documented.

ETL is massively used by Windows system programmers to log useful artifacts:

  • C:\Windows\System32\WDI\LogFiles\BootPerfDiagLogger.etl
  • C:\Windows\System32\WDI\LogFiles\ShutdownPerfDiagLogger.etl
  • NetTrace.etl via netsh
  • C:\Windows\System32\WDI\<GUID>\<GUID>\snapshot.etl
  • etc.

A lot of new APIs such as Tracelogging or WPP are based on ETW. These APIs are used extensively by Microsoft developers for Windows. Tracelogging is addressed by etl-parser, WPP will be addressed in a future release.

Microsoft offers a lot of consumers that create ETL traces, such as xperf.exe, logman.exe, netsh.exe, etc.

We believe it is a gold mine for DFIR analysts.

Credits

License

etl-parser is released under the Apache 2.0 license.

More Repositories

1

ttddbg

Time Travel Debugging IDA plugin
C++
551
star
2

Winshark

A wireshark plugin to instrument ETW
Lua
527
star
3

Yagi

Yet Another Ghidra Integration for IDA
C++
480
star
4

Invoke-Bof

Load any Beacon Object File using Powershell!
PowerShell
245
star
5

comida

An IDA Plugin that help analyzing module that use COM
Python
198
star
6

regrippy

A modern Python-3-based alternative to RegRipper
Python
184
star
7

yara-ttd

Use YARA rules on Time Travel Debugging traces
C
86
star
8

vbSparkle

VBScript & VBA source-to-source deobfuscator with partial-evaluation
C#
72
star
9

ntTraceControl

Powershell Event Tracing Toolbox
PowerShell
72
star
10

CVE-2024-4040

Scanner for CVE-2024-4040
Python
50
star
11

etwbreaker

An IDA plugin to deal with Event Tracing for Windows (ETW)
Python
49
star
12

minusone

Powershell Linter
Rust
46
star
13

PSTrace

Trace ScriptBlock execution for powershell v2
C
39
star
14

tree-sitter-powershell

Powershell grammar for tree-sitter
JavaScript
36
star
15

dnYara

A multi-platform .Net wrapper library for the native Yara library.
C#
35
star
16

timeliner

A rewrite of mactime, a bodyfile reader
Go
34
star
17

Splunk-ETW

A Splunk Technology Add-on to forward filtered ETW events.
C#
30
star
18

cacdec

The hidden mstsc recorder player
Python
28
star
19

ttd2mdmp

Extract data of TTD trace file to a minidump
C++
28
star
20

dirtypipe-ebpf_detection

An eBPF detection program for CVE-2022-0847
C
27
star
21

mispy

Another MISP module for Python
Python
17
star
22

mispgo

Golang library for MISP
Go
5
star
23

usnrs

USN Journal parsing software and library
Rust
5
star
24

bodyfile

A bodyfile parsing library
Go
3
star
25

nix-forensics

Reproducible forensics environment, 100% of the time
Nix
3
star
26

skyblue.team

Our website
HTML
1
star