Subscraper

Modular subdomain enumeration and target identification toolkit
Alternatives To Subscraper
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Spiderfoot10,330
14 days ago147mitPython
SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
Social Analyzer10,33012 months ago1June 25, 20217agpl-3.0JavaScript
API, CLI, and Web App for analyzing and finding a person's profile in 1000 social media \ websites
Red Teaming Toolkit7,614
4 months agogpl-3.0
This repository contains cutting-edge open-source security tools (OST) for a red teamer and threat hunter.
Osintgram7,385
12 days ago477gpl-3.0Python
Osintgram is a OSINT tool on Instagram. It offers an interactive shell to perform analysis on Instagram account of any users by its nickname
Rengine5,872
2 days ago185gpl-3.0JavaScript
reNgine is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines, recon data correlation and organization, continuous monitoring, backed by a database, and simple yet intuitive User Interface. reNgine makes it easy for penetration testers to gather reconnaissance with minimal configuration and with the help of reNgine's correlation, it just makes recon effortless.
Reconftw4,689
16 hours ago25mitHTML
reconFTW is a tool designed to perform automated recon on a target domain by running the best set of tools to perform scanning and finding out vulnerabilities
Osmedeus4,676
11 days ago9mitGo
A Workflow Engine for Offensive Security
Hakrawler3,820
2 months ago11February 22, 20218gpl-3.0Go
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Discover3,167
10 hours agomitPowerShell
Custom bash scripts used to automate various penetration testing tasks including recon, scanning, enumeration, and malicious payload creation using Metasploit. For use with Kali Linux.
Mosint3,023
7 days ago2mitGo
An automated e-mail OSINT tool
Alternatives To Subscraper
Select To Compare


Alternative Project Comparisons
Readme

SubScraper

     
Overview   🔹   Usage   🔹   Contribute

💥 v3.0 now available! 💥

SubScraper is a fast subdomain enumeration tool that uses a variety of techniques to find subdomains of a given target. Subdomain enumeration is especially helpful during penetration testing and bug bounty hunting to uncover an organization's attack surface.

Depending on the CMD arguments applied, SubScraper can resolve DNS names, request HTTP(S) information, and perform CNAME lookups for takeover opportunities during the enumeration process. This can help identify next steps and discover patterns for exploitation.

Key Features

  • Modular design makes it easy to add new techniques/sources.
  • Various levels of enumeration for additional data gathering.
  • Allows for multiple target inputs or read targets from .txt file.
  • Windows CLI compatibility.
  • Generate output files in .txt or .csv format.

demo

Install

The following can be used to install SubScraper on Windows, Linux, & MacOs:

git clone https://github.com/m8sec/subscraper
cd subscraper
python3 setup.py install

Docker

You can build a docker image and run subscraper from Docker:

git clone https://github.com/m8sec/subscraper.git
cd subscraper
docker build -t m8sec/subscraper .
# display help
docker run --rm m8sec/subscraper
# example scanning a site
docker run --rm m8sec/subscraper example.com

Usage

Command Line Args

SubScraper Options:
  -T MAX_THREADS        Max threads for enumeration (Default: 55).
  -t TIMEOUT            Timeout [seconds] for search threads (Default: 25).
  -r REPORT             Output to specific file {txt*, csv}.
  target                Target domain.

Module Options:
  -L                    List SubScraper enumeration modules.
  -M MODULES            Execute module(s) by name or group (Default: all).
  -w WORDLIST           Custom wordlist for DNS brute force.
  --censys-id CENSYS_ID             Censys.io API ID.
  --censys-secret CENSYS_SECRET     Censys.io API Secret.

Enumeration Options:
  --dns                 Resolve DNS address for each subdomain identified.
  --http                Probe for active HTTP:80 & HTTPS:443 services.
  --takeover            Perform CNAME lookup & probe for HTTP(s) response.
  --all                 Perform all checks on enumerated subdomains.

Modules

Modules can be executed by name or by module groups:

  Module Name       Description

  archiveorg           - Use archive.org to find subdomains.
  certsh               - Subdomains enumeration using cert.sh.
  dnsbrute             - DNS bruteforce.
  threatcrowd          - Threadcrowd.org subdomain enumeration.
  dnsdumpster          - Use DNS dumpster to enumerate subdomains.
  bufferoverrun        - Bufferover.run passive enumeration.
  search               - Subdomain enumeration via search engine scraping.
  censys               - Gather subdomains through censys.io SSL cert Lookups.
    |_API_ID                   Censys.io API ID               (Required:True)
    |_API_SECRET               Censys.io API Secret           (Required:True)
  bevigil              - Gather subdomains through bevigil.com mobile app scan data
    |_API_Key                  BeVigil API Key                (Required:True)

Module Groups

  • all - Execute all modules (Default).
  • brute - Only execute DNS brute force techniques.
  • scrape - Only execute web scraping techniques.

Example Usage

subscraper example.com
subscraper targets.txt
cat targets.txt | subscraper pipe
subscraper -all -r enumeration.csv example.com
subscraper -M brute -w mywords.txt example.com
subscraper -M censys --censys-id abc123 --censys-secret xyz456 example.com

Execution Notes

  • SubScraper only uses PASSIVE enumeration techniques unless all, http, takeover arguments are applied.
  • API keys are required for the censys module, register for free at censys.io/register.
  • .txt reports will only include subdomains.
  • .csv reports, when paired with cmd args all, http, takeover, will provide additional HTTP data such as page size, title, and Server headers.

Contribute

Contribute to the project by:

  • Like and share the tool!
  • Create an issue to report new enumeration techniques or, better yet, develop a module and initiate a PR.
Popular Penetration Testing Projects
Popular Osint Projects
Popular Security Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Penetration Testing
Osint
Pentest Tool
Subdomain Scanner