Author: cyclone
URL: https://github.com/cyclone-github/spider
Description:
Spider is a web crawler and wordlist/ngram generator written in Go that crawls specified URLs to produce frequency-sorted wordlists and ngrams. Users can customize crawl depth, output files, and ngram options, making it ideal for web scraping to create targeted wordlists for tools like hashcat or John the Ripper. Spider combines the web scraping capabilities of CeWL and adds ngram generation, and since Spider is written in Go, it requires no additional libraries to download or install.
Spider just works.
Usage Instructions:
- To create a simple wordlist from a specified url (will save deduplicated wordlist to url_wordlist.txt):
./spider -url https://github.com/cyclone-github
- To set url crawl url depth of 2 and create ngrams len 1-5, use flag "-crawl 2" and "-ngram 1-5"
./spider -url https://github.com/cyclone-github -crawl 2 -ngram 1-5
- To set a custom output file, use flag "-o filename"
./spider -url https://github.com/cyclone-github -o wordlist.txt
- To set a delay to keep from being rate-limited, use flag "-delay nth" where nth is time in milliseconds
./spider -url https://github.com/cyclone-github -delay 100
- Run
./spider -help to see a list of all options
