vulsio/go-cve-dictionary vulsio/go-cve-dictionary

  • Stars
    star
    365
  • Rank 116,851 (Top 3 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated 2 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
vulsio/go-cve-dictionary
github

vulsio

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Build a local copy of CVE (NVD and Japanese JVN). Server mode for easy querying.

go-cve-dictionary

This is tool to build a local copy of the NVD (National Vulnerabilities Database) [1] and the Japanese JVN [2], which contain security vulnerabilities according to their CVE identifiers [3] including exhaustive information and a risk score. The local copy is generated in sqlite format, and the tool has a server mode for easy querying.

[1] https://en.wikipedia.org/wiki/National_Vulnerability_Database
[2] https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures
[3] http://jvndb.jvn.jp/apis/termsofuse.html

Installation

Install requirements

go-cve-dictionary requires the following packages.

Here's an example for Amazon EC2 server.

$ ssh [email protected]  -i ~/.ssh/private.pem
$ sudo yum -y install sqlite git gcc
$ wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.7.1.linux-amd64.tar.gz
$ mkdir $HOME/go

Put these lines into /etc/profile.d/goenv.sh

export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Set the OS environment variable to current shell

$ source /etc/profile.d/goenv.sh

Deploy go-cve-dictionary

To install:

$ mkdir -p $GOPATH/src/github.com/vulsio
$ cd $GOPATH/src/github.com/vulsio
$ git clone https://github.com/vulsio/go-cve-dictionary.git
$ cd go-cve-dictionary
$ make install

Create a log output directory. You can use another directory on the command line option (--log-dir).

$ sudo mkdir /var/log/go-cve-dictionary
$ sudo chown ec2-user /var/log/go-cve-dictionary
$ sudo chmod 700 /var/log/go-cve-dictionary

Fetch vulnerability data from NVD.

$ go-cve-dictionary fetch nvd
... snip ...
$ ls -alh cve.sqlite3
-rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3

Now we have vulnerability data. Start go-cve-dictionary as server mode.

$ go-cve-dictionary server
[Mar 24 15:21:55]  INFO Opening DB. datafile: /home/ec2-user/cve.sqlite3
[Mar 24 15:21:55]  INFO Migrating DB
[Mar 24 15:21:56]  INFO Starting HTTP Sever...
[Mar 24 15:21:56]  INFO Listening on 127.0.0.1:1323

Update go-cve-dictionary

If the DB schema was changed, please specify new SQLite3, MySQL, PostgreSQL or Redis DB file.

$ cd $GOPATH/src/github.com/vulsio/go-cve-dictionary
$ git pull
$ rm -r vendor
$ make install

Binary files are created under $GOPATH/bin

Sample data sources

Hello HeartBleed

$ curl http://127.0.0.1:1323/cves/CVE-2014-0160 | jq "."
{
  "CveID": "CVE-2014-0160",
  "Nvd": {
    "Summary": "The (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug.",
    "Score": 5,
    "AccessVector": "NETWORK",
    "AccessComplexity": "LOW",
    "Authentication": "NONE",
    "ConfidentialityImpact": "PARTIAL",
    "IntegrityImpact": "NONE",
    "AvailabilityImpact": "NONE",
    "Cpes": null,
    "References": [
      {
        "Source": "CERT",
        "Link": "http://www.us-cert.gov/ncas/alerts/TA14-098A"
      },
      ...snip...
    ],
    "PublishedDate": "2014-04-07T18:55:03.893-04:00",
    "LastModifiedDate": "2015-10-22T10:19:38.453-04:00"
  },
  "Jvn": {
    "Title": "OpenSSL の heartbeat 拡張に情報漏えいの脆弱性",
    "Summary": "OpenSSL の heartbeat 拡張の実装には、情報漏えいの脆弱性が存在します。TLS や DTLS 通信において OpenSSL のコードを実行しているプロセスのメモリ内容が通信相手に漏えいする可能性があります。",
    "JvnLink": "http://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-001920.html",
    "JvnID": "JVNDB-2014-001920",
    "Score": 5,
    "Severity": "Medium",
    "Vector": "(AV:N/AC:L/Au:N/C:P/I:N/A:N)",
    "References": [
      {
        "Source": "AT-POLICE",
        "Link": "http://www.npa.go.jp/cyberpolice/detect/pdf/20140410.pdf"
      },
      ...snip...
    ],
    "Cpes": null,
    "PublishedDate": "2014-04-08T16:13:59+09:00",
    "LastModifiedDate": "2014-04-08T16:13:59+09:00"
  }
}

Hello Ruby on Rails 4.0.2

$ curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"name": "cpe:/a:rubyonrails:ruby_on_rails:4.0.2:-"}' http://localhost:1323/cpes | jq "."
[
  {
    "CveID": "CVE-2016-0751",
    "Nvd": {
      "CveDetailID": 345,
      "Summary": "actionpack/lib/action_dispatch/http/mime_type.rb in Action Pack in Ruby on Rails before 3.2.22.1, 4.0.x and 4.1.x before 4.1.14.1, 4.2.x before 4.2.5.1, and 5.x before 5.0.0.beta1.1 does not properly restrict use of the MIME type cache, which allows remote attackers to cause a denial of service (memory consumption) via a crafted HTTP Accept header.",
      "Score": 5,
      "AccessVector": "NETWORK",
      "AccessComplexity": "LOW",
      "Authentication": "NONE",
      "ConfidentialityImpact": "NONE",
      "IntegrityImpact": "NONE",
      "AvailabilityImpact": "PARTIAL",
      "Cpes": null,
      "References": [
        {
          "Source": "MLIST",
          "Link": "https://groups.google.com/forum/message/raw?msg=ruby-security-ann/9oLY_FCzvoc/5CDXbvpYEgAJ"
        },
        {
          "Source": "MLIST",
          "Link": "http://www.openwall.com/lists/oss-security/2016/01/25/9"
        }
      ],
      "PublishedDate": "2016-02-15T21:59:05.877-05:00",
      "LastModifiedDate": "2016-03-18T21:02:43.817-04:00"
    },
    "Jvn": {
      "Title": "",
      "Summary": "",
      "JvnLink": "",
      "JvnID": "",
      "Score": 0,
      "Severity": "",
      "Vector": "",
      "References": null,
      "Cpes": null,
      "PublishedDate": "0001-01-01T00:00:00Z",
      "LastModifiedDate": "0001-01-01T00:00:00Z"
    }
  },
  ... snip ...
]

Usage

$ go-cve-dictionary --help
GO CVE Dictionary

Usage:
  go-cve-dictionary [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  fetch       Fetch Vulnerability dictionary
  help        Help about any command
  server      Start CVE dictionary HTTP Server
  version     Show version

Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
  -h, --help                help for go-cve-dictionary
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary [command] --help" for more information about a command.

Usage: Fetch Command

$ go-cve-dictionary fetch --help
Fetch Vulnerability dictionary

Usage:
  go-cve-dictionary fetch [command]

Available Commands:
  jvn         Fetch Vulnerability dictionary from JVN
  nvd         Fetch Vulnerability dictionary from NVD

Flags:
      --batch-size int   The number of batch size to insert. NOTE: This Option does not work for dbtype: redis. (default 5)
  -h, --help             help for fetch

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary fetch [command] --help" for more information about a command.

Fetch NVD data

  • to fetch all years
$ go-cve-dictionary fetch nvd
  • to fetch specific years
$ go-cve-dictionary fetch nvd 2021

Fetch JVN data

  • to fetch all years
$ go-cve-dictionary fetch jvn
  • to fetch specific years
$ go-cve-dictionary fetch jvn 2021

Usage: Run HTTP Server

$ go-cve-dictionary server --help
Start CVE dictionary HTTP Server

Usage:
  go-cve-dictionary server [flags]

Flags:
      --bind string   HTTP server bind to IP address (default "127.0.0.1")
  -h, --help          help for server
      --port string   HTTP server port number (default "1323")

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Usage: Use MySQL as a DB storage back-end

  • fetch nvd

    $ go-cve-dictionary fetch nvd \
      --dbtype mysql \
      --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

  • fetch jvn

    $ go-cve-dictionary fetch jvn \
      --dbtype mysql \
      --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

  • server

    $ go-cve-dictionary server \
      --dbtype mysql \
      --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

Usage: Use Postgres as a DB storage back-end

  • fetch nvd

    $ go-cve-dictionary fetch nvd \
      --dbtype postgres \
      --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

  • fetch jvn

    $ go-cve-dictionary fetch jvn \
      --dbtype postgres \
      --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

  • server

    $ go-cve-dictionary server \
      --dbtype postgres \
      --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

Usage: Use Redis as a DB storage back-end

  • fetch nvd

    $ go-cve-dictionary fetch nvd \
      --dbtype redis \
      --dbpath "redis://localhost/0"

  • fetch jvn

    $ go-cve-dictionary fetch jvn \
      --dbtype redis \
      --dbpath "redis://localhost/0"

  • server

    $ go-cve-dictionary server \
      --dbtype redis \
      --dbpath "redis://localhost/0"

Misc

  • HTTP Proxy Support
    If your system at behind HTTP proxy, you have to specify -http-proxy option.

  • How to daemonize go-cve-dictionary
    Use Systemd, Upstart or supervisord, daemontools...

  • How to cross compile

    $ cd /path/to/your/local-git-repository/go-cve-dictionary
$ GOOS=linux GOARCH=amd64 go build -o cvedict.amd64

  • Logging
    go-cve-dictionary writes a log under -log-path specified directory (default is /var/log/go-cve-dictionary/).

  • Debug
    Run with --debug, --debug-sql option.

  • Completion Support

    • bash
    $ go-cve-dictionary completion bash > /usr/share/bash-completion/completions/go-cve-dictionary
    • zsh
    $ go-cve-dictionary completion zsh > ~/.zsh-completions/go-cve-dictionary
    • fish
    $ go-cve-dictionary completion fish > ~/.config/fish/completions/go-cve-dictionary.fish

Data Source

Authors

kotakanbe (@kotakanbe) created go-cve-dictionary and these fine people have contributed.

How to Contribute

  1. fork a repository: github.com/vulsio/go-cve-dictionary to github.com/you/repository
  2. get original code: github.com/vulsio/go-cve-dictionary
  3. work on original code
  4. add remote to your repository: git remote add myfork https://github.com/you/repo.git
  5. push your changes: git push myfork
  6. create a new Pull Request

Licence

Please see LICENSE.

Additional License

How can my organization use the NVD data within our own products and services? All NVD data is freely available from our XML Data Feeds. There are no fees, licensing restrictions, or even a requirement to register. All NIST publications are available in the public domain according to Title 17 of the United States Code. Acknowledgment of the NVD when using our information is appreciated. In addition, please email [email protected] to let us know how the information is being used.

More Repositories

1

go-exploitdb

Tool for searching Exploits from Exploit Databases, etc.
Go
235
star
2

gost

Build a local copy of Security Tracker. Notify via E-mail/Slack if there is an update.
Go
131
star
3

go-cpe-dictionary

Build a local copy of CPE(Common Platform Enumeration)
Go
92
star
4

goval-dictionary

Build a local copy of OVAL. Server mode for easy querying.
Go
89
star
5

vulsctl

Easy setup tool for Vuls
Shell
68
star
6

go-cti

Build a local copy of MITRE ATT&CK and CAPEC. Server mode for easy querying.
Go
29
star
7

go-msfdb

CVEs <--> Metasploit-Framework modules
Go
23
star
8

go-kev

Build a local copy of Known Exploited Vulnerabilities Catalog by CISA. Server mode for easy querying.
Go
17
star
9

vuln-list

6
star
10

vuls-data-raw-exploit-inthewild

6
star
11

licensecheck

Go
3
star
12

msfdb-list

3
star
13

msfdb-list-updater

Go
2
star
14

vuln-dictionary

Go
2
star
15

vuls-data-raw-nvd-api-cve

1
star
16

vuls-data-raw-fortinet

A collection of advisories for mechanically processing [PSIRT Advisories](https://www.fortiguard.com/psirt)
1
star