@v1cvap0r

duplicut - dedupes a single wordlist without changing its order: ./duplicut wordlist.txt -o dedup_wordlist.txt rling - similar capability to duplicut: ./rling wordlist.txt dedup_wordlist.txt but also can remove dups from multiple files and save the dedup output to file like the example below from the GitHub page: ./rling big-file.txt new-file.txt /path/to/old-file.txt /path/to/others/*