Domains
Understand domains, manage multiple websites, and organize your link health monitoring
Overview
Domains are the foundation of SEO Crawler. Each domain represents a website you want to monitor for broken links, SEO issues, and overall link health.
A domain in SEO Crawler:
- Represents a single website or subdomain
- Requires DNS verification before crawling
- Has its own crawl history and reports
- Tracks link health independently
- Can have custom crawl settings
Think of domains as individual projects in your SEO monitoring workspace.
What Counts as a Domain?
Each unique hostname counts as one domain toward your plan limit.
Examples
| Input | Counts As | Plan Impact |
|---|---|---|
example.com | 1 domain | Uses 1 slot |
www.example.com | Same as example.com | Uses 1 slot |
blog.example.com | 1 subdomain | Uses 1 slot |
shop.example.com | 1 subdomain | Uses 1 slot |
example.com + example.org | 2 separate domains | Uses 2 slots |
www is automatically normalized to the root domain. Both example.com and www.example.com are treated as the same domain.
Subdomains
Each subdomain is a separate domain:
mysite.com → Domain 1 blog.mysite.com → Domain 2 shop.mysite.com → Domain 3 api.mysite.com → Domain 4
If you need to monitor multiple subdomains, consider the Agency plan with unlimited domains.
Domain Lifecycle
Every domain goes through these stages:
Created
Domain is added to your account with a unique verification token.
Status: Unverified
Can Crawl: ❌ No
Verification Pending
You add the DNS TXT record to prove ownership.
Status: Pending Verification
Can Crawl: ❌ No
Verified
DNS record detected, domain ownership confirmed.
Status: Verified ✓
Can Crawl: ✅ Yes
Active Monitoring
Regular crawls run, link health tracked, reports generated.
Status: Active
Can Crawl: ✅ Yes
Removed (Optional)
Domain deleted from account, all data removed.
Status: Deleted
Can Crawl: ❌ No
Domain Status Indicators
Verification Status
| Badge | Status | Meaning | Action Required |
|---|---|---|---|
🟢 Verified | Active | Ready to crawl | None |
🟡 Pending | Waiting | DNS record not detected | Add TXT record |
🔴 Unverified | Failed | Verification incomplete | Re-verify |
⚪ Expired | Inactive | DNS record removed | Re-add TXT record |
Health Status
| Score | Status | Indicator | Meaning |
|---|---|---|---|
90-100% | Excellent | 🟢 | Few or no issues detected |
70-89% | Good | 🟢 | Minor issues present |
50-69% | Fair | 🟡 | Several issues need attention |
30-49% | Poor | 🟠 | Many broken links or issues |
0-29% | Critical | 🔴 | Severe issues detected |
N/A | Unknown | ⚪ | No crawls yet or crawl failed |
Domain Dashboard
The domains dashboard shows all your monitored websites at a glance.
Dashboard Columns
| Column | Information | Actions |
|---|---|---|
Domain | Domain name + verification badge | Click to view details |
Health Score | Current link health percentage | View breakdown |
Issues | Count of broken links and errors | Filter by type |
Last Crawl | When domain was last scanned | View history |
Next Crawl | Scheduled automation (if enabled) | Manage schedule |
Actions | Quick action menu | Start crawl, settings, delete |
Quick Actions
From the domain dashboard, you can:
- Start Crawl - Immediately trigger a new scan
- View Details - See comprehensive domain information
- Crawl History - Review all previous crawls
- Export Report - Download PDF or CSV
- Settings - Configure crawl options
- Delete - Remove domain and all data
Adding Domains
For detailed instructions, see the Adding Domains guide.
Quick Add Process
1. Click "Add Domain" button 2. Enter domain name (e.g., mysite.com) 3. Click "Continue" 4. Copy verification token 5. Add DNS TXT record 6. Click "Check DNS" 7. Start crawling! ✓
Domain Input Normalization
SEO Crawler automatically cleans your input:
# All of these become: example.com https://example.com http://www.example.com EXAMPLE.COM example.com/ example.com:443 www.example.com/homepage
Domain Verification
For detailed instructions, see the Verifying Domains guide.
Why Verification is Required
Domain verification ensures:
- ✅ You own or control the domain
- ✅ Prevents unauthorized crawling
- ✅ Protects website owners' privacy
- ✅ Complies with responsible crawling practices
Verification Methods
Currently supported: DNS TXT Record
Type: TXT Name: @ (or subdomain name) Value: seo-crawler-verify=your-unique-token
Coming Soon:
- HTML meta tag verification
- File upload verification
- Email verification for small sites
Re-verification
Domains require continuous verification. If you remove the DNS TXT record:
- Domain becomes unverified
- New crawls are blocked
- Existing data remains accessible
- Re-add TXT record to restore access
Domain Settings
Each domain has customizable settings for crawling behavior.
Crawl Configuration
Notification Settings
Configure how you're notified about domain issues:
| Setting | Options | When It Triggers |
|---|---|---|
Email Alerts | On/Off | Broken links detected |
Slack Notifications | On/Off | Crawl completed |
Webhook Events | Configure URLs | Real-time API events |
Report Frequency | Daily/Weekly/Monthly | Scheduled summaries |
Custom Settings
Agency Plan Features:
- Custom User-Agent - Set your own crawler identification
- Custom Headers - Add authentication or special headers
- IP Whitelist - Crawl from specific IP ranges
- Priority Crawling - Jump the queue for urgent scans
Managing Multiple Domains
Organizing Domains
Use these strategies to manage many domains:
Domain Limits by Plan
| Plan | Max Domains | Recommended Use Case |
|---|---|---|
Free | 1 domain | Personal site or testing |
Solo | 3 domains | Freelancers, small businesses |
Pro | 10 domains | Agencies, consultants |
Agency | Unlimited | Large agencies, enterprises |
Need more domains? Upgrade your plan or contact us for custom limits.
Domain Analytics
Per-Domain Metrics
Each domain tracks:
| Metric | Description | Updated |
|---|---|---|
Total URLs | Pages discovered on site | Per crawl |
Broken Links | 404s, timeouts, errors | Per crawl |
Health Score | Overall link health (0-100%) | Per crawl |
Response Times | Average page load time | Per crawl |
Crawl Frequency | How often scanned | Rolling 30 days |
Credit Usage | URLs crawled this month | Real-time |
Last Issues | Recent problems detected | Latest crawl |
Historical Data
View trends over time:
- Health score graph (30/60/90 days)
- Broken link trends
- New vs. fixed issues
- Crawl duration patterns
- Response time changes
Domain Security
Access Control
Who can access your domain data:
| Role | Permissions |
|---|---|
Owner (You) | Full access - view, crawl, edit, delete |
Team Members | Configure access per domain (Agency plan) |
API Keys | Scoped access based on key permissions |
Public Reports | Anyone with share link (if enabled) |
Sharing Crawl Results
Control who sees your data:
Domain Best Practices
Before Adding a Domain
Verify DNS Access
Ensure you can add TXT records to your DNS provider before adding the domain.
Review Robots.txt
Check if your `robots.txt` blocks crawlers. Add our user-agent if needed:
User-agent: SEOCrawlerBot Allow: /
Plan Your Crawl
Decide on max URLs and depth based on your site size and goals.
Ongoing Maintenance
- ✅ Keep DNS record active - Don't remove the verification TXT record
- ✅ Monitor health trends - Check dashboard weekly
- ✅ Fix broken links promptly - Address issues as they appear
- ✅ Update exclusions - Add new paths to skip (admin, APIs)
- ✅ Review crawl settings - Adjust as your site grows
- ✅ Use automation - Schedule regular crawls
Performance Optimization
Removing Domains
What Gets Deleted
When you delete a domain, we permanently remove:
- ❌ All crawl history and results
- ❌ All broken link data
- ❌ All reports and exports
- ❌ All automation schedules
- ❌ Verification status
- ❌ Custom settings and configurations
Domain deletion is permanent and cannot be undone. Export any data you need before deleting.
Before You Delete
- Export final report - Save crawl data as PDF/CSV
- Cancel automations - Stop scheduled crawls
- Revoke share links - Disable public access
- Download historical data - Archive what you need
- Inform team members - Notify anyone with access
How to Delete
Navigate to Domain
Go to the domain you want to delete.
Scroll to Danger Zone
At the bottom of the domain settings page.
Click Delete Domain
Review the warning and confirm deletion.
Type Domain Name
Enter the exact domain name to confirm.
Confirm
Click "Delete Permanently" - domain is immediately removed.
Re-adding a Deleted Domain
If you delete a domain and want to add it back:
- Same domain name allowed
- New verification token required
- All previous data is gone
- Starts fresh with new crawls
Troubleshooting
API Access
Manage domains programmatically with our API:
# List all domains GET /api/v1/domains # Get domain details GET /api/v1/domains/{domain_id} # Add new domain POST /api/v1/domains # Update domain settings PATCH /api/v1/domains/{domain_id} # Delete domain DELETE /api/v1/domains/{domain_id}
API access requires an Agency plan. See API Documentation for details.
Next Steps
Add Your First Domain
Step-by-step guide to adding domains.
<Card title="Verify Domain Ownership" icon="shield-check" href="/docs/getting-started/verifying-domains"
Complete DNS verification process.
<Card title="Start Your First Crawl" icon="spider-web" href="/docs/crawling/starting-crawls"
Run a link health scan.
<Card title="Automate Monitoring" icon="clock" href="/docs/features/automation"
Schedule recurring crawls.
Pro Tip: Set up automation for your most important domains to catch broken links before your users do!