• Stars
    star
    437
  • Rank 99,659 (Top 2 %)
  • Language
    C++
  • License
    Other
  • Created about 5 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Mercury: network metadata capture and analysis

Mercury: network metadata capture and analysis

This package contains two programs for fingerprinting network traffic and capturing and analyzing packet metadata: mercury, a Linux application that leverages the modern Linux kernel's high-performance networking capabilities (AF_PACKET and TPACKETv3), which is described below, and pmercury, a portable python application. There is also a User's Guide. While mercury is used in some production applications, please consider this software as a 'beta'. The CHANGELOG itemizes changes across different versions.

Overview

Mercury reads network packets, identifies metadata of interest, and writes out the metadata in JSON format. Alternatively, mercury can write out the packets that contain the metadata in the PCAP file format. Mercury can scale up to high data rates (40Gbps on server-class hardware); it uses zero-copy ring buffers to acquire packets, and packets are processed by independent worker threads. The amount of memory consumed by the ring buffers, and the number of worker threads, are configurable; this makes it easy to scale up (but be wary of using too much memory).

Mercury produces fingerprint strings for TLS, DTLS, SSH, HTTP, TCP, and other protocols; these fingerprints are formed by carefully selecting and normalizing metadata extracted from packets (as documented here). Fingerprint strings are reported in the "fingerprint" object in the JSON output. Optionally, mercury can perform process identification based on those fingerprints and the destination context; these results are reported in the "analysis" object.

Contents

Building and installing mercury

Mercury itself has minimal dependencies other than a g++ or llvm build environment, but to run the automated tests and ancillary programs in this package, you will need to install additional packages, as in the following Debian/Ubuntu example:

sudo apt install g++ jq git zlib1g-dev tcpreplay valgrind python3-pip libssl-dev clang
pip3 install jsonschema

To build mercury, in the root directory, run

./configure
make

to build the package (and check for the programs and python modules required to test it). TPACKETv3 is present in Linux kernels newer than 3.2.

Installation

In the root directory, edit mercury.cfg with the network interface you want to capture from, then run

./configure
make

sudo make install MERCURY_CFG=mercury.cfg

to install mercury and create and start a systemd service. If you don't want the mercury systemd service to be installed, then instead run

sudo make install-nosystemd

The default file and directory locations are

  • /usr/local/bin/mercury for the executable
  • /usr/local/share/mercury for the resource files
  • /usr/local/var/mercury for the output files
  • /etc/mercury/mercury.cfg for the configuration file
  • /etc/systemd/system/mercury.service for the systemd unit file

The output file directory is owned by the user mercury; this user is created by the 'make install' target, which must be run as root. The installation prefix /usr/local/ can be changed by running ./configure with the --prefix argument, for instance `--prefix=$HOME'. If you want to install the program somewhere in your home directory, you probably don't want to create the user mercury; you should use the 'make install-nonroot' target, which does not create a user, does not install anything into /etc, and does not install a systemd unit.

The easiest way to run mercury in capture mode is using systemd; the OS automatically starts the mercury systemd unit after each boot, and halts it when the OS is shut down. To check its status, run

systemctl status mercury

and the output should contain 'active (running)'. To view the log (stderr) output from the mercury unit, run

sudo journalctl -u mercury

To uninstall mercury, run

sudo make uninstall

which will remove the mercury program, resources directory, user, group, and systemd related files. The directory containing capture files will be retained, but its owner will be changed to root, to avoid unintentional data loss. All captured data files are retained across successive installs and uninstalls, and must be manually deleted.

Compile-time options

To create a debugging version of mercury, use the make debug-mercury target in the src/ subdirectory. Be sure to run make clean first.

There are compile-time options that can tune mercury for your hardware. Each of these options is set via a C/C++ preprocessor directive, which should be passed as an argument to "make" through the OPTFLAGS variable. Frst run make clean to remove the previous build, then run make "OPTFLAGS=". This runs make, telling it to pass to the C/C++ compiler. The available compile time options are:

  • -DDEBUG, which turns on debugging, and
  • -FBUFSIZE=16384, which sets the fwrite/fread buffer to 16,384 bytes (for instance). If multiple compile time options are used, then they must be passed to make together in the OPTFLAGS string, e.g. "OPTFLAGS=-DDEBUG -DFBUFSIZE=16384".

Running mercury

mercury: packet metadata capture and analysis
./src/mercury [INPUT] [OUTPUT] [OPTIONS]:
INPUT
   [-c or --capture] capture_interface   # capture packets from interface
   [-r or --read] read_file              # read packets from file
   no input option                       # read packets from standard input
OUTPUT
   [-f or --fingerprint] json_file_name  # write JSON fingerprints to file
   [-w or --write] pcap_file_name        # write packets to PCAP/MCAP file
   no output option                      # write JSON fingerprints to stdout
--capture OPTIONS
   [-b or --buffer] b                    # set RX_RING size to (b * PHYS_MEM)
   [-t or --threads] [num_threads | cpu] # set number of threads
   [-u or --user] u                      # set UID and GID to those of user u
   [-d or --directory] d                 # set working directory to d
GENERAL OPTIONS
   --config c                            # read configuration from file c
   [-a or --analysis]                    # analyze fingerprints
   --resources=f                         # use resource file f
   --stats=f                             # write stats to file f
   --stats-time=T                        # write stats every T seconds
   --stats-limit=L                       # limit stats to L entries
   [-s or --select] filter               # select traffic by filter (see --help)
   --nonselected-tcp-data                # tcp data for nonselected traffic
   --nonselected-udp-data                # udp data for nonselected traffic
   --tcp-reassembly                      # reassemble tcp data segments
   [-l or --limit] l                     # rotate output file after l records
   --output-time=T                       # rotate output file after T seconds
   --dns-json                            # output DNS as JSON, not base64
   --certs-json                          # output certs as JSON, not base64
   --metadata                            # output more protocol metadata in JSON
   [-v or --verbose]                     # additional information sent to stderr
   --license                             # write license information to stdout
   --version                             # write version information to stdout
   [-h or --help]                        # extended help, with examples

DETAILS
   "[-c or --capture] c" captures packets from interface c with Linux AF_PACKET
   using a separate ring buffer for each worker thread.  "[-t or --thread] t"
   sets the number of worker threads to t, if t is a positive integer; if t is
   "cpu", then the number of threads will be set to the number of available
   processors.  "[-b or --buffer] b" sets the total size of all ring buffers to
   (b * PHYS_MEM) where b is a decimal number between 0.0 and 1.0 and PHYS_MEM
   is the available memory; USE b < 0.1 EXCEPT WHEN THERE ARE GIGABYTES OF SPARE
   RAM to avoid OS failure due to memory starvation.

   "[-f or --fingerprint] f" writes a JSON record for each fingerprint observed,
   which incorporates the flow key and the time of observation, into the file f.
   With [-a or --analysis], fingerprints and destinations are analyzed and the
   results are included in the JSON output.

   "[-w or --write] w" writes packets to the file w, in PCAP format.  With the
   option [-s or --select], packets are filtered so that only ones with
   fingerprint metadata are written.

   "[r or --read] r" reads packets from the file r, in PCAP format.

   if neither -r nor -c is specified, then packets are read from standard input,
   in PCAP format.

   "[-s or --select] f" selects packets according to the metadata filter f, which
   is a comma-separated list of the following strings:
      dhcp              DHCP discover message
      dns               DNS messages
      dtls              DTLS clientHello, serverHello, and certificates
      http              HTTP request and response
      http.request      HTTP request
      http.response     HTTP response
      iec               IEC 60870-5-104
      mdns              multicast DNS
      nbns              NetBIOS Name Service
      openvpn_tcp       OpenVPN over TCP
      quic              QUIC handshake
      ssh               SSH handshake and KEX
      smb               SMB v1 and v2
      stun              STUN messages
      ssdp              SSDP (UPnP)
      tcp               TCP headers
      tcp.message       TCP initial message
      tls               TLS clientHello, serverHello, and certificates
      tls.client_hello  TLS clientHello
      tls.server_hello  TLS serverHello
      tls.certificates  TLS serverCertificates
      wireguard         WG handshake initiation message
      all               all of the above
      <no option>       all of the above
      none              none of the above

   --nonselected-tcp-data writes the first TCP Data field in a flow with
   nonzero length, for *non*-selected traffic, into JSON.  This option provides
   a view into the TCP data that the --select option does not recognize. The
   --select filter affects the TCP data written by this option; use
   '--select=none' to obtain the TCP data for each flow.

   --nonselected-udp-data writes the first UDP Data field in a flow with
   nonzero length, for *non*-selected traffic, into JSON.  This option provides
   a view into the UDP data that the --select option does not recognize. The
   --select filter affects the UDP data written by this option; use
   '--select=none' to obtain the UDP data for each flow.

   --tcp-reassembly enables the tcp reassembly
   This option allows mercury to keep track of tcp segment state and 
   and reassemble these segments based on the application in tcp payload

   "[-u or --user] u" sets the UID and GID to those of user u, so that
   output file(s) are owned by this user.  If this option is not set, then
   the UID is set to SUDO_UID, so that privileges are dropped to those of
   the user that invoked sudo.  A system account with username mercury is
   created for use with a mercury daemon.

   "[-d or --directory] d" sets the working directory to d, so that all output
   files are written into that location.  When capturing at a high data rate, a
   high performance filesystem and disk should be used, and NFS partitions
   should be avoided.

   "--config c" reads configuration information from the file c.

   [-a or --analysis] performs analysis and reports results in the "analysis"
   object in the JSON records.   This option only works with the option
   [-f or --fingerprint].

   "[-l or --limit] l" rotates output files so that each file has at most
   l records or packets; filenames include a sequence number, date and time.

   --dns-json writes out DNS responses as a JSON object; otherwise,
   that data is output in base64 format, as a string with the key "base64".

   --certs-json writes out certificates as JSON objects; otherwise,
    that data is output in base64 format, as a string with the key "base64".

   --metadata writes out additional metadata into the protocol JSON objects.

   [-v or --verbose] writes additional information to the standard error,
   including the packet count, byte count, elapsed time and processing rate, as
   well as information about threads and files.

   --license and --version write their information to stdout, then halt.

   [-h or --help] writes this extended help message to stdout.

SYSTEM

The directories used by the default install are as follows. Run mercury --help to see if the directories on your system differ.

   Resource files used in analysis: /usr/local/share/mercury
   Systemd service output:          /usr/local/var/mercury
   Systemd service configuration    /etc/mercury/mercury.cfg

EXAMPLES

   mercury -c eth0 -w foo.pcap           # capture from eth0, write to foo.pcap
   mercury -c eth0 -w foo.pcap -t cpu    # as above, with one thread per CPU
   mercury -c eth0 -w foo.mcap -t cpu -s # as above, selecting packet metadata
   mercury -r foo.mcap -f foo.json       # read foo.mcap, write fingerprints
   mercury -r foo.mcap -f foo.json -a    # as above, with fingerprint analysis
   mercury -c eth0 -t cpu -f foo.json -a # capture and analyze fingerprints

Ethics

Mercury is intended for defensive network monitoring, security research and forensics. Researchers, administrators, penetration testers, and security operations teams can use these tools to protect networks, detect vulnerabilities, and benefit the broader community through improved awareness and defensive posture. As with any packet monitoring tool, Mercury could potentially be misused. Do not run it on any network of which you are not the owner or the administrator.

Credits

Mercury and this package was developed by David McGrew, Brandon Enright, Blake Anderson, Lucas Messenger, Adam Weller, Andrew Chi, Shekhar Acharya, Anastasiia-Mariia Antonyk, Oleksandr Stepanov, Vigneshwari Viswanathan, and Apoorv Raj, with input from Brian Long, Bill Hudson, and others. Pmercury was developed by Blake Anderson, with input from others.

Acknowledgments

This package includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.

We make use of Mozilla's Public Suffix List which is subject to the terms of the Mozilla Public License, v. 2.0.

This package directly incorporates some software made by other developers, to make the package easier to build, deploy, and run. We are grateful to the copyright holders for making their excellent software available under licensing terms that allow its redistribution.

More Repositories

1

ChezScheme

Chez Scheme
Scheme
6,965
star
2

openh264

Open Source H.264 Codec
C++
5,527
star
3

joy

A package for capturing and analyzing network flow data and intraflow data, for network research, forensics, and security monitoring.
C
1,301
star
4

libsrtp

Library for SRTP (Secure Realtime Transport Protocol)
C
1,209
star
5

node-jose

JavaScript
700
star
6

thor

Thor Video Codec
C
697
star
7

mindmeld

An Open Source Conversational AI Platform for Deep-Domain Voice Interfaces and Chatbots.
Python
673
star
8

cisco-mibs

Various SNMP MIBs from Cisco
HTML
144
star
9

exanic-software

ExaNIC drivers, utilities and development libraries
C
138
star
10

mlspp

Implementation of Messaging Layer Security
C++
108
star
11

cjose

C library implementing the Javascript Object Signing and Encryption (JOSE)
C
105
star
12

libest

C
97
star
13

elsy

An opinionated, multi-language, build tool based on Docker and Docker Compose
Go
79
star
14

lal-build-manager

Project dependency manager
Rust
75
star
15

libacvp

The libacvp library is a client-side implementation of the draft ACVP protocol (github.com/usnistgov/ACVP).
C
65
star
16

ns3-802.11ax-simulator

NS3 Simulator of 802.11ax
C++
58
star
17

cisco-network-puppet-module

Ruby
55
star
18

open-nFAPI

An open source implementation of the Small Cell Forum's Network Functional API (nFAPI)
C
54
star
19

ns3-rmcat

Simulator of IETF RMCAT congestion control protocols
C++
50
star
20

go-mls

Message Layer Security
Go
48
star
21

cisco.github.io

Public Github Pages for Cisco
JavaScript
45
star
22

libfnr

FNR is a small domain block cipher to encrypt small objects ( < 128 bits )
C
40
star
23

senml

Tool to convert senml between formats and act as gateway server to other services
Go
39
star
24

opus

Cisco work on the Opus codec
C
31
star
25

hash-sigs

A full-featured implementation of of the LMS and HSS Hash Based Signature Schemes from draft-mcgrew-hash-sigs-07.
C
31
star
26

go-hpke

Implementation of draft-irtf-cfrg-hpke
Go
30
star
27

cisco-network-node-utils

Ruby
29
star
28

systemf

Prepared statement support for the system command
C
29
star
29

cisco-network-chef-cookbook

Ruby
27
star
30

OpenOSC

Open Object Size Checking: Library to detect buffer overflows in C/C++ code
C
27
star
31

NATTools

NAT Tools
C
26
star
32

ActionOrchestratorContent

A repository for Action Orchestrator content definitions. Includes workflows and Atomic Workflow Adapters as well as general adapter definitions.
25
star
33

exact-capture

Exablaze High Rate Capture Software
C
23
star
34

oraf

Optimized RAndom Forests
Scala
22
star
35

node-kms

node-scr
JavaScript
21
star
36

JabberWerxC

A C API for XMPP connectivity
C
18
star
37

zepster

Generate database schema, documentation, and other artifacts from an Entity-Relationship diagram, which is created as a GraphML file using the yEd graph editor.
Python
17
star
38

firepower-ngfw

Firepower Threat Defense Virtual templates and artifacts
13
star
39

cisco-yang-puppet-module

Puppet module to allow configuration of IOS-XR via YANG
Ruby
11
star
40

anyconnect-kdf

Cisco AnyConnect Linux Kernel Module
C
11
star
41

webex-assistant-sdk

An SDK for developing Webex Assistant Skills based on the MindMeld platform.
Python
11
star
42

xr-telemetry-m2m-web

A small web app to explore the IOS-XR internal data model, for streaming telemetry or other automation uses
Python
10
star
43

herisson

C++
10
star
44

goFish

Finding Fish
C++
9
star
45

cisco-nxapi

Ruby
8
star
46

asav

ASAv templates and artifacts
8
star
47

syncodecs

Synthetic codecs for evaluation of RMCAT work
C++
7
star
48

ciscohls

C
7
star
49

multilingual-speech-testing

Test software and data for evaluation of speech processing algorithms in multiple languages
Python
7
star
50

sframe

Implementation of draft-omara-sframe
C++
6
star
51

hyperledger-est-ca

EST CA for Hyperledger Fabric
Go
6
star
52

SSMAMTtools

C
6
star
53

ciscoacipuppet

Ruby
6
star
54

gse

Game State Encoder and Decoder for RTP
C++
6
star
55

go-tls-syntax

Encode and decode messages in TLS 1.3 syntax
Go
6
star
56

jfnr

Java bindings for FNR block cipher
Java
5
star
57

usnic_tools

usNIC tools
C
4
star
58

ampfsm

Cisco AMP Filesystem Module
C
4
star
59

arc

Go
4
star
60

xr-telemetry-m2m-lib

Libraries for interacting with the IOS-XR M2M service.
Python
3
star
61

JdbcDecorator

Java
3
star
62

gst-plugin-dlnasrc

A fork of the original gst-dlna-src plugin, these modifications make the plugin work with the RDK server
C
3
star
63

ampnetworkflow

Cisco AMP Device Flow Control
C
3
star
64

libamvp

A client-side implementation of the AMVP protocol.
C
2
star
65

guppi

Grand Unified Platform for Process Interactions: A DevOps Environment for Data Scientists
Python
2
star
66

cisco-otel-java

Cisco Distribution of OpenTelemetry Java
1
star
67

WXM-Invitations

Optional module for dispatching SMS and Email invitations for Cisco Webex Experience Management Surveys from your private cloud
C#
1
star
68

cgmi

C
1
star
69

libusnic_verbs

Perl
1
star
70

node-scr

JavaScript
1
star
71

pam_mpa

PAM module providing Multi-Person Authentication against local (/etc/shadow)
C
1
star