crowdsecurity/hub crowdsecurity/hub

  • Stars
    star
    147
  • Rank 249,949 (Top 5 %)
  • Language
    Python
  • Created over 4 years ago
  • Updated about 2 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
crowdsecurity/hub
github

crowdsecurity

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Main repository for crowdsec scenarios/parsers

CrowdSec


πŸ“š Documentation πŸ’  Configuration Hub πŸ’¬ Discourse (Forum) πŸ’¬ Gitter (Live chat)

CrowdSec Hub for parsers, enrichers and scenarios.

Foreword

This repository stores most of the official parsers/scenarios/collections for crowdsec.

The repository is not intended for use as-is, but rather as source of truth for the CrowdSec Hub and cscli.

Feel free to use the parsers/scenarios here as a source of inspiration.

Testing & Continuous integration

cscli provides a hubtest sub-command to help contributors to create tests for parsers and scenarios.

View & use existing tests

⚠️ most of cscli hubtest commands are expected to be run from the root directory of the hub. A git clone of this repository is the easier way to work ⚠️

list existing tests

cscli hubtest list

run a specific test

cscli hubtest run [test-name]

show current tests coverage

cscli hubtest coverage

Create your own (parser) test

We're going to create the CI tests for the dovecot-parser. Before you start :

  • you will need some actual logs
  • you'd better know if the service logs on its own or via syslog (we're in the later case here)
  1. Create a new test
β–Ά cscli hubtest create dovecot-logs --type syslog

  Test name                   :  dovecot-logs
  Test path                   :  .../github/hub/.tests/dovecot-logs
  Log file                    :  .../github/hub/.tests/dovecot-logs/dovecot-logs.log (please fill it with logs)
  Parser assertion file       :  .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion)
  Scenario assertion file     :  .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion)
  Configuration File          :  .../github/hub/.tests/dovecot-logs/config.yaml (please fill it with parsers, scenarios...)

What is relevant here is that every test is composed of :

  • A log file and it's associated type (same type as seen in acquis labels:type)
  • A configuration specifying which parsers and/or scenarios must be enabled for the test
  • A ultimately list of assertions that must be run against the parsers and/or scenarios output

Note: You can provide the parsers and scenarios you want in your test with --parsers and --scenarios (you can provide multiple parsers and scenarios)

If you want to test only a scenario, you can specify (--ignore-parsers) or set the ignore_parsers to true in the config.yaml

  1. Configure your test

We need to edit the test configuration to use the relevant parsers :

β–Ά cat .../github/hub/.tests/dovecot-logs/config.yaml
parsers:
- crowdsecurity/syslog-logs
- crowdsecurity/dovecot-logs
scenarios:
postoverflows:
- ""
log_file: dovecot-logs.log
log_type: syslog

note: the order doesn't matter. If the parser name is in the form author/parser it's from the hub, but relative paths are allowed for non-versioned parsers

Now we need to dump some actual logs into the test's log file :

β–Ά cat > .tests/dovecot-logs/dovecot-logs.log 
Jan 28 10:16:13 dovecot-box dovecot[7508]: imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<[email protected]>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>
Sep 8 07:16:29 canyon dovecot: auth-worker(24058): pam(toto,1.1.1.1,<youpi>): pam_authenticate() failed: Authentication failure (password mismatch?)
Sep 8 07:46:51 canyon dovecot: auth-worker(24544): pam(toto,1.1.1.1): unknown user
  1. Run the test for the first time

Now that we have config & logs, let's run it for the first time :

β–Ά cscli hubtest run dovecot-logs
INFO[27-09-2021 06:13:59 PM] Running test 'dovecot-logs'                  
INFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/dovecot-logs' installed succesfully in runtime environment 
INFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/syslog-logs' installed succesfully in runtime environment 
WARN[27-09-2021 06:14:02 PM] Assert file '/home/bui/github/hub/.tests/dovecot-logs/parser.assert' is empty, generating assertion: 

results["s00-raw"]["crowdsecurity/syslog-logs"][0].Success == true
...
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["pid"] == "7508"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["timestamp"] == "Jan 28 10:16:13"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_login_result"] == "Disconnected (auth failed, 1 attempts in 6 secs)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["message"] == "imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<[email protected]>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["protocol"] == "imap"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_local_ip"] == "7.7.7.7"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_remote_ip"] == "4.4.4.4"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_user"] == "[email protected]"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_type"] == "file"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["source_ip"] == "4.4.4.4"
...

As our parser.assert is empty, the tool is generating some "suggested" asserts for us. Your careful eye will keep only the ones relevant to the parser you're testing :

β–Ά cat > .tests/dovecot-logs/parser.assert 
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["pid"] == "7508"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["timestamp"] == "Jan 28 10:16:13"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_login_result"] == "Disconnected (auth failed, 1 attempts in 6 secs)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["message"] == "imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<[email protected]>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["protocol"] == "imap"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_local_ip"] == "7.7.7.7"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_remote_ip"] == "4.4.4.4"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_user"] == "[email protected]"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_type"] == "file"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["source_ip"] == "4.4.4.4"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_login_result"] == "Authentication failure (password mismatch?)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["message"] == "auth-worker(24058): pam(toto,1.1.1.1,<youpi>): pam_authenticate() failed: Authentication failure (password mismatch?)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["timestamp"] == "Sep 8 07:16:29"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_remote_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_user"] == "toto"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["datasource_type"] == "file"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["source_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_login_result"] == "unknown user"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_remote_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_user"] == "toto"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["message"] == "auth-worker(24544): pam(toto,1.1.1.1): unknown user"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["timestamp"] == "Sep 8 07:46:51"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["source_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["datasource_type"] == "file"
  1. Test your newly crafted test
β–Ά cscli hubtest run dovecot-logs                                
INFO[27-09-2021 06:19:33 PM] Running test 'dovecot-logs'                  
INFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/syslog-logs' installed succesfully in runtime environment 
INFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/dovecot-logs' installed succesfully in runtime environment 
Test 'dovecot-logs' passed successfully (39 assertions) 🟩

And be amazed.

Debug your own (parser) test

Things went wrong ? Don't panic

When working on a test, you can as well pass expressions directly to hubtest command and see the results :

β–Ά cscli hubtest  eval dovecot-logs -e 'results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed'             
dovecot_login_result: unknown user
dovecot_remote_ip: 1.1.1.1
dovecot_user: toto
facility: ""
logsource: syslog
message: 'auth-worker(24544): pam(toto,1.1.1.1): unknown user'
pid: ""
priority: ""
program: dovecot
timestamp: Sep 8 07:46:51
timestamp8601: ""

Open your PR

yes.

More Repositories

1

crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
Go
8,253
star
2

example-docker-compose

Example integration of crowdsec in docker-compose
Dockerfile
101
star
3

cs-firewall-bouncer

Crowdsec bouncer written in golang for firewalls
Go
100
star
4

opnsense-plugin-crowdsec

OPNsense plugin for CrowdSec
Volt
69
star
5

grafana-dashboards

Grafana dashboards for Crowdsec monitoring using Prometheus
61
star
6

home-assistant-addons

Home Assistant Crowdsec Addons
Dockerfile
54
star
7

cs-nginx-bouncer

CrowdSec bouncer for Nginx
Shell
49
star
8

pfSense-pkg-crowdsec

PHP
49
star
9

cs-cloudflare-bouncer

A CrowdSec Bouncer that syncs the decisions made by CrowdSec with CloudFlare's firewall. Manages multi user, multi account, multi zone setup. Supports IP, Country and AS scoped decisions.
Go
47
star
10

cs-wordpress-bouncer

CrowdSec is an open-source cyber security tool. This plugin blocks detected attackers or display them a captcha to check they are not bots.
PHP
34
star
11

helm-charts

CrowdSec community kubernetes helm charts
Shell
27
star
12

crowdsec-docs

CrowdSec Documentation: Comprehensive guides, tutorials, and references for installing, configuring, and using CrowdSec, an open-source cybersecurity platform. Contributions welcome!
MDX
22
star
13

cs-windows-firewall-bouncer

Crowdsec bouncer for the Windows Firewall
C#
20
star
14

cs-haproxy-bouncer

Crowdsec bouncer for HAProxy
Lua
20
star
15

cs-blocklist-mirror

Go
15
star
16

cs-cloud-firewall-bouncer

Crowdsec Cloud Firewall Bouncer
Go
15
star
17

spksrc-crowdsec

Makefile
14
star
18

php-cs-bouncer

The official PHP bouncers library for the CrowdSec LAPI
PHP
14
star
19

cs-custom-bouncer

CrowdSec bouncer to use custom scripts
Python
13
star
20

sec-lists

Repository to track keywords & patterns lists used by crowdsec parsers and scenarios
11
star
21

cs-standalone-php-bouncer

CrowdSec bouncer for PHP Website
PHP
11
star
22

cs-openresty-bouncer

CrowdSec bouncer for OpenResty
Shell
10
star
23

pycrowdsec

Python
10
star
24

lua-cs-bouncer

Lua
9
star
25

cs-express-bouncer

CrowdSec is an open-source cyber security tool. This Express middleware blocks detected attackers or display them a captcha to check they are not bots.
JavaScript
9
star
26

go-cs-bouncer

Go library to use crowdsec API.
Go
9
star
27

cs-aws-waf-bouncer

Crowdsec bouncer for AWS WAF
Go
7
star
28

cs-cloudflare-blocker

Block IPs/Ranges with cloudflare API
Go
7
star
29

cs-magento-bouncer

CrowdSec is an open-source cyber security tool. This module blocks detected attackers or display them a captcha to check they are not bots.
PHP
6
star
30

jupyter-ecs-spawner

ECS Spawner for JupyterHub
Python
5
star
31

cs-nginx-blocker

nginx blocker
Shell
4
star
32

php-capi-client

The official PHP client for the CrowdSec Central API (CAPI)
PHP
4
star
33

cs-netfilter-blocker

netfilter blocker
Go
4
star
34

cs-custom-blocker

CrowdSec blocker to call user scripts
Go
3
star
35

terraform-aws-crowdsec-serverless

HCL
3
star
36

misp-feed-generator

Python
3
star
37

cs-cloudflare-worker-bouncer

A CrowdSec Bouncer that syncs the decisions made by CrowdSec with CloudFlare's firewall using cloudflare workers. Manages multi user, multi account, multi zone setup. Supports IP, Country and AS scoped decisions.
Go
3
star
38

cs-lua-lib

Lua
2
star
39

ansible-collection-crowdsecurity.testing

Ansible collection for deployment and functional testing
2
star
40

php-remediation-engine

The official PHP remediation engine for CrowdSec
PHP
2
star
41

crowdsec_chrome_extension

Chrome extension which enables searching selected IP in CrowdSec's CTI
JavaScript
2
star
42

crowdsec-yaml-schemas

2
star
43

crowdsec-wasm-playground

JavaScript
2
star
44

php-lapi-client

The official PHP client for the CrowdSec Local API (LAPI)
PHP
2
star
45

python-capi-sdk

A Python SDK to do the heavy lifting of interacting with CAPI.
Python
2
star
46

pytest-cs

Pytest fixtures for crowdsec
Python
1
star
47

cs-nginx-njs-bouncer

CrowdSec NGINX remediation component (bouncer) using NJS
1
star
48

cs-wordpress-blocker

[DEPRECATED] Crowdsec Wordpress blocker
PHP
1
star
49

cs-thehive-cortex-analyzer

CrowdSec CTI enrichment integration into TheHive via Cortex Analyzer
HTML
1
star
50

magento-cs-extension

Magento extension allowing to share threat signal and benefit from the CrowdSec's community blocklist
PHP
1
star
51

fundraising-decks

Present the slides used for CrowdSec fundraising and explain the business model
1
star
52

packaging-debian

Python
1
star
53

crowdsec-service-api-sdk-python

Service API Python SDK
Python
1
star
54

go-cs-lib

Common library for crowdsec and go-based bouncers
Go
1
star