Spidy: Domain Name Collector
A lightweight Golang tool that crawls websites to extract domain names and checks their availability status.
Features
- Crawls websites up to a configurable depth
- Checks domain availability for specified TLDs (e.g.,
.com,.net,.org) - Configurable rate limits, user agents, and request timeouts
- Stores results locally for further analysis
Installation
git clone https://github.com/twiny/spidy.git
cd ./spidy
# Build
go build -o bin/spidy -v cmd/spidy/main.go
# Run
./bin/spidy -c config/config.yaml -u https://example.com Configuration
Example config.yaml file:
crawler:
max_depth: 10 # Max pages to crawl per site
rate_limit: "1/5s" # 1 request per 5 seconds
max_body_size: "20MB" # Max page size
user_agents:
- "Spidy/2.1 (+https://github.com/twiny/spidy)"
log:
rotate: 7 # Log rotation (days)
path: "./log" # Log directory
store:
ttl: "24h" # Cache retention time
path: "./store" # Storage directory
result:
path: "./result" # Output directory
parallel: 3 # Concurrent workers
timeout: "5m" # Request timeout
tlds: ["com", "net", "org"] # TLDs to check Usage
./bin/spidy -h