Automated Crawls
Schedule automatic link health checks for your domains and receive email reports — with optional public share links for Agency plans.
Automated Crawls
Schedule automatic link health checks to run on your domains without manual intervention.
Overview
Automated crawls let you set it and forget it — configure a schedule for each domain and we'll automatically run crawls at your chosen frequency.
After each run, you'll receive an email summary with the results. Agency users can optionally include a public share link for clients or teammates.
Automation Flow
Automation Flow ═══════════════════════════════════════════════════════════ ┌─────────────────┐ │ Configure │ │ Schedule │───→ Every 6 hours └────────┬────────┘ │ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Crawl Runs │─────→│ Email Report │ │ Automatically │ │ Sent to You │ └────────┬────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Repeat at │ │ Next Interval │ └─────────────────┘
Available Schedules by Plan
The frequency options available depend on your subscription plan.
| Schedule | Free | Solo | Pro | Agency |
|---|---|---|---|---|
Off | ✓ | ✓ | ✓ | ✓ |
Weekly | — | ✓ | ✓ | ✓ |
Daily | — | — | ✓ | ✓ |
Every 12 Hours | — | — | — | ✓ |
Every 6 Hours | — | — | — | ✓ |
Hourly | — | — | — | ✓ |
Free plan users can run manual crawls, but don't have access to automation. Upgrade to Solo or higher to enable scheduled crawls.
Setting Up Automation
Go to Settings
Navigate to **Settings → Automation** in your dashboard.
Select a Domain
Find the domain you want to automate. Each **verified** domain has its own configuration card.
Choose a Schedule
Use the dropdown to select your preferred frequency (based on your plan).
Configure Scan Options
Choose which checks to run during each automated crawl.
Set Report Preferences (Agency)
Agency users can set a custom email recipient and enable a public report link.
Scan Options
When configuring automation, you can enable any combination of these checks:
| Option | Description | Available |
|---|---|---|
SSL Certificate | Check certificate expiry and warn about upcoming expirations | Solo+ (automation required) |
Anchor Text Analysis | Identify generic or non-descriptive link text | Solo+ (automation required) |
Response Time Trends | Compare response times with previous crawls | Solo+ (automation required) |
SEO Analysis | Analyze meta tags, headings, content, images, and markup | Solo+ (automation required) |
Sitemap & Orphan Pages | Compare sitemap.xml with crawled pages and detect orphan pages | Pro+ |
Enable only the checks you need. More checks can increase crawl duration, but produce more comprehensive reports.
Report Delivery
Email Reports
After each automated crawl completes, you'll receive an email summary containing:
- Total links checked
- Number of broken links found
- Health score percentage
- Link to view the full report
Email summaries are controlled by your notification preferences (Settings → Notifications).
Custom Recipients (Agency)
Agency plan users can send reports to any email address per domain — useful for:
- Clients
- Teammates
- Stakeholders
Public Report Links (Agency)
Enable Include public link to:
- Automatically make each crawl report publicly accessible
- Include the shareable link in the email
- Allow anyone with the URL to view the report (no login required)
Public links expose crawl data to anyone with the URL. Only enable this when you need to share externally.
How Automation Works
Scheduling Engine
Our scheduler runs in the background and:
- Checks regularly for domains that are due
- Only runs automation for verified domains
- Respects your daily crawl limit (automated crawls count)
- Prevents overlaps (won't start a new crawl if one is already running for that domain)
- Calculates the next run time after it triggers a crawl
Daily Crawl Limits
Automated crawls count toward your daily limit (manual + automated).
If you've used all your daily crawls, scheduled crawls will be skipped until the next day.
Best Practices
Choose the Right Frequency
| Site Type | Recommended Schedule | Reasoning |
|---|---|---|
Blog | Weekly | Content changes infrequently |
E-commerce | Daily | Products and pages change regularly |
News site | Every 6 hours | High content velocity |
Corporate | Weekly | Stable structure |
SaaS app | Daily | Docs and links change with releases |
Optimize Scan Options
Minimal (fastest):
- Link health only
Standard:
- SSL Certificate
- Response Time Trends
- SEO Analysis
Comprehensive (slowest):
- SSL Certificate
- Anchor Text Analysis
- Response Time Trends
- SEO Analysis
- Sitemap & Orphan Pages (Pro+)
Troubleshooting
Crawl Didn't Run
| Cause | Solution |
|---|---|
Domain not verified | Complete DNS verification first |
Daily limit reached | Wait until tomorrow or upgrade |
Crawl already running | The previous crawl is still in progress |
Schedule set to Off | Enable a schedule frequency |
Didn't Receive Email
| Cause | Solution |
|---|---|
Notifications disabled | Check Settings → Notifications |
Email in spam | Mark as not spam |
Wrong email (Agency) | Update the report recipient in Settings → Automation |