Modern password wordlist creation usually implies concatenating multiple data sources.
Ideally, most probable passwords should stand at start of the wordlist, so most common passwords are cracked instantly.
With existing dedupe tools you are forced to choose if you prefer to preserve the order OR handle massive wordlists.
Unfortunately, wordlist creation requires both:
So i wrote duplicut in highly optimized C to address this very specific need 🤓 💻
git clone https://github.com/nil0x42/duplicut cd duplicut/ && make ./duplicut wordlist.txt -o clean-wordlist.txt
uint64 is enough to index lines in hashmap, by packing
size info within pointer's extra bits:
If you find a bug, or something doesn't work as expected, please compile duplicut in debug mode and post an issue with attached output:
# debug level can be from 1 to 4 make debug level=1 ./duplicut [OPTIONS] 2>&1 | tee /tmp/duplicut-debug.log