epi052/recursive-gobuster epi052/recursive-gobuster

  • This repository has been archived on 21/May/2021
  • Stars
    star
    116
  • Rank 303,894 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 6 years ago
  • Updated almost 4 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
epi052/recursive-gobuster
github

epi052

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

DEPRECATED - A wrapper around gobuster that automatically scans newly discovered directories.

Code style: black

DEPRECATED

DEPRECATED

DEPRECATED

This tool has been deprecated in favor of my new project feroxbuster. Feroxbuster is recursive, written in rust, and fast. It does all the things that recursive-gobuster does in addition to quite a bit more. If you came here looking for a recursive content discovery tool, please check out feroxbuster.

DEPRECATED

DEPRECATED

DEPRECATED

recursive-gobuster

A wrapper around gobuster that automatically scans newly discovered directories.

Why though?

@OJ designed gobuster to not be recursive. In fact, it's #3 in the list of reasons describing why he wrote gobuster in the first place. He's a much smarter man than I and I'm sure his reasoning for not including the capability is sound, but I wanted the ability to kick off a scan with my favorite scanner and walk away.

I started off looking for something ready-made. I found and really liked Ryan Wendel's shell script to accomplish recursive gobuster scanning. However, there were a few minor behavioral things I wanted to tweak. I also wanted to speed up execution a bit. His original script is what prompted me to write this one.

Dependencies

Install

recursive-gobuster is distributed with a pre-packaged executable zip file. This way, those who want to use this tool can do so without needing to manage their own virtualenv.

git clone https://github.com/epi052/recursive-gobuster.git

./recursive-gobuster/recursive-gobuster.pyz 
usage: recursive-gobuster.pyz [-h] [-t THREADS] [-x EXTENSIONS] [-w WORDLIST]
                              [-d] [-U USER] [-P PASSWORD] [-p PROXY] [-s]
                              target

Optionally, set it somewhere in your PATH

sudo ln -s $(pwd)/recursive-gobuster/recursive-gobuster.pyz /usr/local/bin

Build

There is a build.sh script included that performs the following actions:

  • installs pyinotify into the recursive-gobuster directory
  • removes pip metadata
  • creates an executable zip using python's zipapp library
  • ensures the executable zip has the executable bit set

Since there is a prebuilt package included in the repo, the build script is primarily included in case you want to modify the functionality of the script and afterwards would like to rebuild the executable zip. That workflow would look something like this:

  1. modify recursive-gobuster/__main__.py to your liking
  2. delete or backup current recursive-gobuster.pyz
  3. run build.sh

Tool Behavior

recursive-gobuster isn't recursive in the sense of how it works, only in that it will continue scanning directories until there are no more identified. These are the high-level steps it takes to scan:

  1. Creates a temporary directory and sets an inotify watch on the same directory.
  2. Begins a gobuster scan against the specified target url.
  3. Defines the scan's output file to be located within the watched directory.
  4. Watches the initial (and all subsequent) scan output file for two filesystem events IN_MODIFY and IN_DELETE. Each time a file is modified, any new sub-directory identified is used as the target url for a new scan. The new scans begin as soon as a new sub-directory is seen (i.e. asynchronously). When a file is closed, it signifies the end of that particular scan.
  5. Cleans up the temporary directory when all scans are complete.
  6. Creates a sorted log file of all results in the current working directory.

Considerations

I wrote this tool for me, to be integrated into my workflow with my preferences in mind. As such, there are some things to be aware of that you may or may not care for.

  • There is no limit placed on the number of concurrent scans and no option to do rate limiting. Each one is its own separate process, however this behavior may not be desirable, depending on your target and host machine.
  • The tool is pretty opinionated about options fed to gobuster. Removing the -e, or -n will likely break functionality of the tool. The other options are either configurable via command line options, or can be manipulated in the source code without any adverse side-effects
hard-coded options meaning
-q Don't print the banner and other noise
-e Expanded mode, print full URLs
-k Skip SSL certificate verification

Knowing all that, it's still just a python script and can easily be manipulated to your own preferences. I've included build.sh so you can make changes and easily generate a new packaged version.

Examples

usage: recursive-gobuster.pyz [-h] [-t THREADS] [-x EXTENSIONS] [-w WORDLIST]
                              [-d] [-U USER] [-P PASSWORD] [-p PROXY] [-s]
                              target

positional arguments:
  target                target to scan

optional arguments:
  -h, --help            show this help message and exit
  -t THREADS, --threads THREADS
                        # of threads for each spawned gobuster (default: 20)
  -x EXTENSIONS, --extensions EXTENSIONS
                        extensions passed to the -x option for spawned
                        gobuster
  -w WORDLIST, --wordlist WORDLIST
                        wordlist for each spawned gobuster (default:
                        /usr/share/seclists/Discovery/Web-Content/common.txt)
  -d, --devnull         send stderr to devnull
  -U USER, --user USER  Username for Basic Auth (dir mode only)
  -P PASSWORD, --password PASSWORD
                        Password for Basic Auth (dir mode only)
  -p PROXY, --proxy PROXY
                        Proxy to use for requests [http(s)://host:port] (dir
                        mode only)
  -s, --status          Include status code reporting (default: false)

Standard run with defaults

The default wordlist is seclists/Discovery/Web-Content/common.txt. The default number of threads is 20. There are no extensions by default.

time /opt/recursive_gobuster.pyz http://10.10.10.112
http://10.10.10.112/.hta/
http://10.10.10.112/.htpasswd/
http://10.10.10.112/.htaccess/
2019/01/17 05:49:34 [-] Wildcard response found: http://10.10.10.112/.htpasswd/a1612d11-2fda-486f-a039-d5aef1556386 => 403
2019/01/17 05:49:34 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:34 [-] Wildcard response found: http://10.10.10.112/.hta/b2b5e936-7574-41c8-bb60-022c81b0b325 => 403
2019/01/17 05:49:34 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:34 [-] Wildcard response found: http://10.10.10.112/.htaccess/5ae6f6d2-7269-48e4-af73-b6d604f6d3b2 => 403
2019/01/17 05:49:34 [!] To force processing of Wildcard responses, specify the '-fw' switch.
http://10.10.10.112/Images/
http://10.10.10.112/Images/.htaccess/
http://10.10.10.112/Images/.htpasswd/
http://10.10.10.112/Images/.hta/
2019/01/17 05:49:35 [-] Wildcard response found: http://10.10.10.112/Images/.htpasswd/12ec114e-3818-499e-bcb2-9b4139e6eb71 => 403
2019/01/17 05:49:35 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:35 [-] Wildcard response found: http://10.10.10.112/Images/.htaccess/003ce118-407c-483f-9edd-cb7cad646685 => 403
2019/01/17 05:49:35 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:35 [-] Wildcard response found: http://10.10.10.112/Images/.hta/f0f57fb2-93f5-4c5c-9cc9-84c09090e8ad => 403
2019/01/17 05:49:35 [!] To force processing of Wildcard responses, specify the '-fw' switch.
http://10.10.10.112/assets/
http://10.10.10.112/assets/.htaccess/
http://10.10.10.112/assets/.hta/
http://10.10.10.112/assets/.htpasswd/
2019/01/17 05:49:36 [-] Wildcard response found: http://10.10.10.112/assets/.htaccess/6fe86826-d623-4119-a1e4-b54edac070b3 => 403
2019/01/17 05:49:36 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:36 [-] Wildcard response found: http://10.10.10.112/assets/.hta/6259ad75-3d6a-4881-9a19-cad99623b204 => 403
2019/01/17 05:49:36 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:36 [-] Wildcard response found: http://10.10.10.112/assets/.htpasswd/5d2a8a43-9fe7-4f37-8fee-1498ae9048da => 403
2019/01/17 05:49:36 [!] To force processing of Wildcard responses, specify the '-fw' switch.
http://10.10.10.112/assets/css/
http://10.10.10.112/images/
http://10.10.10.112/assets/css/.htpasswd/
http://10.10.10.112/assets/css/.htaccess/
http://10.10.10.112/assets/css/.hta/
2019/01/17 05:49:41 [-] Wildcard response found: http://10.10.10.112/assets/css/.htpasswd/dc99e161-2379-4f98-930d-90248641fe73 => 403
2019/01/17 05:49:41 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:41 [-] Wildcard response found: http://10.10.10.112/assets/css/.htaccess/dd6b320c-a54b-4e29-9821-dd7b78b68c82 => 403
2019/01/17 05:49:41 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:41 [-] Wildcard response found: http://10.10.10.112/assets/css/.hta/917be959-5ac9-4201-b44f-c89dc1c9bcb5 => 403
2019/01/17 05:49:41 [!] To force processing of Wildcard responses, specify the '-fw' switch.
http://10.10.10.112/images/.htaccess/
http://10.10.10.112/images/.htpasswd/
http://10.10.10.112/images/.hta/
2019/01/17 05:49:41 [-] Wildcard response found: http://10.10.10.112/images/.hta/dfb25bd1-26c3-4fcd-9e0e-605fa3354505 => 403
2019/01/17 05:49:41 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:41 [-] Wildcard response found: http://10.10.10.112/images/.htpasswd/5ccb242b-0c03-4df1-9132-20fe8cc4b01b => 403
2019/01/17 05:49:41 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:41 [-] Wildcard response found: http://10.10.10.112/images/.htaccess/f64e7160-f6b6-4c89-a76f-1a34b3f879a5 => 403
2019/01/17 05:49:41 [!] To force processing of Wildcard responses, specify the '-fw' switch.
http://10.10.10.112/assets/fonts/
http://10.10.10.112/assets/fonts/.htaccess/
http://10.10.10.112/assets/fonts/.hta/
http://10.10.10.112/assets/fonts/.htpasswd/
2019/01/17 05:49:43 [-] Wildcard response found: http://10.10.10.112/assets/fonts/.htaccess/c29f202e-6bc1-47b0-bcaf-b450dbec87fc => 403
2019/01/17 05:49:43 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:43 [-] Wildcard response found: http://10.10.10.112/assets/fonts/.hta/d0df1e60-81bc-4516-bcae-ea03999acbdb => 403
2019/01/17 05:49:43 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:43 [-] Wildcard response found: http://10.10.10.112/assets/fonts/.htpasswd/03f5de0a-396b-4dc7-9d08-7b46202e3641 => 403
2019/01/17 05:49:43 [!] To force processing of Wildcard responses, specify the '-fw' switch.
http://10.10.10.112/assets/js/
http://10.10.10.112/assets/js/.hta/
http://10.10.10.112/assets/js/.htaccess/
http://10.10.10.112/assets/js/.htpasswd/
2019/01/17 05:49:44 [-] Wildcard response found: http://10.10.10.112/assets/js/.htpasswd/5c192b5e-8ef2-4a98-a850-64ff3308f82e => 403
2019/01/17 05:49:44 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:44 [-] Wildcard response found: http://10.10.10.112/assets/js/.hta/bf5b535a-ced5-4d4e-9de4-8f748473615c => 403
2019/01/17 05:49:44 [!] To force processing of Wildcard responses, specify the '-fw' switch.
2019/01/17 05:49:44 [-] Wildcard response found: http://10.10.10.112/assets/js/.htaccess/f1483aeb-8572-4186-8221-19bddcf57ef7 => 403
2019/01/17 05:49:44 [!] To force processing of Wildcard responses, specify the '-fw' switch.
All scans complete. Cleaning up.

real	0m37.033s
user	0m0.162s
sys	0m0.112s

# ls -l recursive*
-rw-r--r--  1 root root    962 Jan 11 20:42 recursive_gobuster_http:__10.10.10.112.log

Scan with STDERR sent to /dev/null

This option is included to suppress all the Wildcard response messages sent to STDERR.

/opt/recursive_gobuster.pyz -d http://10.10.10.112
http://10.10.10.112/.hta/
http://10.10.10.112/.htpasswd/
http://10.10.10.112/.htaccess/
http://10.10.10.112/Images/
http://10.10.10.112/Images/.htaccess/
http://10.10.10.112/Images/.htpasswd/
http://10.10.10.112/Images/.hta/
http://10.10.10.112/assets/
http://10.10.10.112/assets/.htaccess/
http://10.10.10.112/assets/.htpasswd/
http://10.10.10.112/assets/.hta/
http://10.10.10.112/assets/css/
http://10.10.10.112/images/
http://10.10.10.112/assets/css/.htaccess/
http://10.10.10.112/assets/css/.hta/
http://10.10.10.112/assets/css/.htpasswd/
http://10.10.10.112/images/.htaccess/
http://10.10.10.112/images/.htpasswd/
http://10.10.10.112/images/.hta/
http://10.10.10.112/assets/fonts/
http://10.10.10.112/assets/fonts/.hta/
http://10.10.10.112/assets/fonts/.htaccess/
http://10.10.10.112/assets/fonts/.htpasswd/
http://10.10.10.112/assets/js/
http://10.10.10.112/assets/js/.htaccess/
http://10.10.10.112/assets/js/.hta/
http://10.10.10.112/assets/js/.htpasswd/
All scans complete. Cleaning up.

real	0m37.141s
user	0m0.182s
sys	0m0.073s

Scan with extensions

/opt/recursive_gobuster.pyz -x php,html http://10.10.10.112
http://10.10.10.112/.htaccess/
-------- 8< --------

Scan with different wordlist

/opt/recursive_gobuster.pyz -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt http://10.10.10.112
http://10.10.10.112/.htaccess/
-------- 8< --------

More Repositories

1

feroxbuster

A fast, simple, recursive content discovery tool written in Rust.
Rust
5,788
star
2

osed-scripts

bespoke tooling for offensive security's Windows Usermode Exploit Dev course (OSED)
Python
491
star
3

recon-pipeline

An automated target reconnaissance pipeline.
Python
424
star
4

feroxfuzz

A structure-aware HTTP fuzzing library
Rust
200
star
5

fuzzing-101-solutions

Companion repository to the Fuzzing101 with LibAFL series of blog posts.
C
136
star
6

cve-2018-15473

Multi-threaded, IPv6 aware, wordlists/single-user username enumeration via CVE-2018-15473
Python
98
star
7

OSCE-exam-practice

Proof of Concept exploit scripts and fuzzing templates. Companion blog posts located at https://epi052.gitlab.io/notes-to-self/blog/2020-05-13-osce-exam-practice-part-one/
Python
56
star
8

rustdsplit

At some point, I learned about a method to perform a binary search on a file in order to identify its AV signature and change it to bypass signature-based AV. The tool I used back then is gone, so I wrote this.
Rust
34
star
9

htb-scripts-for-retired-boxes

Just a place to share some things I've written while participating in Hack The Box.
Python
19
star
10

CiscoNotes

Notes for Latest Cisco vulns
4
star
11

rust-i-choose-you-tcs-2021

Source code & slides for Rust, I choose you! A formative talk for the Rust-curious.
Rust
4
star
12

feroxbuster-docs

HTML
2
star
13

advent-of-code-2021

Solutions for AoC 2021
Rust
1
star
14

advent-of-code-2023

Rust
1
star
15

helpdesk

Python
1
star
16

slae-64-assignments

Assignments completed to garner the SLAE-64 certification. Primarily x86_64 asm, with some Python and C for good measure.
Assembly
1
star