Subscraper

Perform subdomain enumeration through various techniques and retrieve detailed output to aid in further testing.
Alternatives To Subscraper
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Social Analyzer9,986
a month ago45May 24, 20225agpl-3.0JavaScript
API, CLI, and Web App for analyzing and finding a person's profile in 1000 social media \ websites
Spiderfoot9,571
a month ago133mitPython
SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
Red Teaming Toolkit7,614
5 days agogpl-3.0
This repository contains cutting-edge open-source security tools (OST) for a red teamer and threat hunter.
Osintgram6,771
12 days ago386gpl-3.0Python
Osintgram is a OSINT tool on Instagram. It offers an interactive shell to perform analysis on Instagram account of any users by its nickname
Rengine5,530
3 days ago142gpl-3.0JavaScript
reNgine is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines, recon data correlation and organization, continuous monitoring, backed by a database, and simple yet intuitive User Interface. reNgine makes it easy for penetration testers to gather reconnaissance with minimal configuration and with the help of reNgine's correlation, it just makes recon effortless.
Osmedeus4,465
2 days ago4mitGo
A Workflow Engine for Offensive Security
Reconftw4,304
2 days ago28gpl-3.0HTML
reconFTW is a tool designed to perform automated recon on a target domain by running the best set of tools to perform scanning and finding out vulnerabilities
Hakrawler3,540
3 months ago11February 22, 20214gpl-3.0Go
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Discover3,049
14 days agomitPowerShell
Custom bash scripts used to automate various penetration testing tasks including recon, scanning, enumeration, and malicious payload creation using Metasploit. For use with Kali Linux.
Raccoon2,571218 months ago33October 03, 201814mitPython
A high performance offensive security tool for reconnaissance and vulnerability scanning
Alternatives To Subscraper
Select To Compare


Alternative Project Comparisons
Readme

SubScraper

     
Overview   🔹   Usage   🔹   Contribute

💥 v3.0 now available! 💥

SubScraper is a fast subdomain enumeration tool that uses a variety of techniques to find subdomains of a given target. Subdomain enumeration is especially helpful during penetration testing and bug bounty hunting to uncover an organization's attack surface.

Depending on the CMD arguments applied, SubScraper can resolve DNS names, request HTTP(S) information, and perform CNAME lookups for takeover opportunities during the enumeration process. This can help identify next steps and discover patterns for exploitation.

Key Features

  • Modular design makes it easy to add new techniques/sources.
  • Various levels of enumeration for additional data gathering.
  • Allows for multiple target inputs or read targets from .txt file.
  • Windows CLI compatibility.
  • Generate output files in .txt or .csv format.

demo

Install

The following can be used to install SubScraper on Windows, Linux, & MacOs:

git clone https://github.com/m8sec/subscraper
cd subscraper
python3 setup.py install

Usage

Command Line Args

SubScraper Options:
  -T MAX_THREADS        Max threads for enumeration (Default: 55).
  -t TIMEOUT            Timeout [seconds] for search threads (Default: 25).
  -r REPORT             Output to specific file {txt*, csv}.
  target                Target domain.

Module Options:
  -L                    List SubScraper enumeration modules.
  -M MODULES            Execute module(s) by name or group (Default: all).
  -w WORDLIST           Custom wordlist for DNS brute force.
  --censys-id CENSYS_ID             Censys.io API ID.
  --censys-secret CENSYS_SECRET     Censys.io API Secret.

Enumeration Options:
  --dns                 Resolve DNS address for each subdomain identified.
  --http                Probe for active HTTP:80 & HTTPS:443 services.
  --takeover            Perform CNAME lookup & probe for HTTP(s) response.
  --all                 Perform all checks on enumerated subdomains.

Modules

Modules can be executed by name or by module groups:

  Module Name       Description

  archiveorg           - Use archive.org to find subdomains.
  certsh               - Subdomains enumeration using cert.sh.
  dnsbrute             - DNS bruteforce.
  threatcrowd          - Threadcrowd.org subdomain enumeration.
  dnsdumpster          - Use DNS dumpster to enumerate subdomains.
  bufferoverrun        - Bufferover.run passive enumeration.
  search               - Subdomain enumeration via search engine scraping.
  censys               - Gather subdomains through censys.io SSL cert Lookups.
    |_API_ID                   Censys.io API ID               (Required:True)
    |_API_SECRET               Censys.io API Secret           (Required:True)
  bevigil              - Gather subdomains through bevigil.com mobile app scan data
    |_API_Key                  BeVigil API Key                (Required:True)

Module Groups

  • all - Execute all modules (Default).
  • brute - Only execute DNS brute force techniques.
  • scrape - Only execute web scraping techniques.

Example Usage

subscraper example.com
subscraper targets.txt
cat targets.txt | subscraper pipe
subscraper -all -r enumeration.csv example.com
subscraper -M brute -w mywords.txt example.com
subscraper -M censys --censys-id abc123 --censys-secret xyz456 example.com

Execution Notes

  • SubScraper only uses PASSIVE enumeration techniques unless all, http, takeover arguments are applied.
  • API keys are required for the censys module, register for free at censys.io/register.
  • .txt reports will only include subdomains.
  • .csv reports, when paired with cmd args all, http, takeover, will provide additional HTTP data such as page size, title, and Server headers.

Contribute

Contribute to the project by:

  • Like and share the tool!
  • Create an issue to report new enumeration techniques or, better yet, develop a module and initiate a PR.
Popular Penetration Testing Projects
Popular Osint Projects
Popular Security Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Penetration Testing
Osint
Pentest Tool
Subdomain Scanner