What Is AhrefsBot? User-Agent, IPs, and How to Block or Allow It

AhrefsBot is Ahrefs’ web crawler. Its data powers Ahrefs’ SEO tools and Yep, Ahrefs’ search engine. According to the official Ahrefs Bots page, it identifies as AhrefsBot/7.0, obeys robots.txt and Crawl-delay, crawls from IP ranges Ahrefs publishes, and its IPs always reverse-resolve to ahrefs.com or ahrefs.net. Blocking it does not change your Google rankings.

If you spotted “AhrefsBot” in server logs, a firewall event, or your CDN dashboard, it’s usually this normal crawler. But anyone can fake the user-agent, so verify the IP before you allowlist it. Below: how to verify it, then how to allow, slow, or block it.

What AhrefsBot is and what it does

AhrefsBot is Ahrefs’ main public web crawler. Its job is to discover pages and links across the open web, then store that data so Ahrefs can show backlink profiles, referring domains, anchor text, broken link opportunities, and more.

That activity can feel suspicious if you’re only seeing the bot in logs. But crawling isn’t “hacking.” A crawler requests URLs your server already serves to regular visitors, then reads responses the same way a browser would (just without the human).

A few practical reasons you’ll see AhrefsBot on your site:

• Link discovery. It finds new inbound and outbound links, which later appear in backlink indexes.
• Content discovery. It revisits pages to keep the index fresh, especially if your site updates often.
• Search engine coverage. Ahrefs also uses crawl data for Yep.com.
• Rules-based crawling. AhrefsBot is designed to follow standard crawl rules like robots.txt, and it supports controls like Crawl-delay.

If you publish public content and want accurate third-party SEO data about your domain, you usually want AhrefsBot to reach the pages you care about. This will allow Ahrefs to compute things like domain rating (DR) and backlink profiles. If your server is small or you’re under load, you might want to slow it down rather than block it outright.

If you’re working on your content and rankings, it also helps to pair crawl management with on-page improvements, for example using RightBlogger SEO Reports to tighten headings, keyword coverage, and structure while you keep crawl access clean.

How AhrefsBot compares to AI crawlers

AhrefsBot is the SEO crawler most site owners notice first, but AI crawlers now show up in the same logs. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot each have their own user-agent token, so the robots.txt and firewall patterns below work for them too. Google-Extended is different: it’s a robots.txt token that controls whether Google can use your content for AI training, not a separate crawler, so it never appears in your logs. Some AI bots fetch pages for training and others for AI search, so blocking the wrong one can cost you citations. Our AI crawler access guide sorts out which is which, and our GEO vs SEO guide covers why AI visibility matters.

Ahrefs bot names

Ahrefs runs two crawlers: AhrefsBot and AhrefsSiteAudit. Those are the names you’ll see in logs and the tokens you use in robots.txt. Here’s how they compare, based on ahrefs.com/robot and Ahrefs’ IP list help article:

AhrefsBot vs AhrefsSiteAudit

The two bots have different jobs:

• AhrefsBot is the global crawler that builds Ahrefs’ public web index of links and pages. This is the one that shows up even if you don’t use Ahrefs.
• AhrefsSiteAudit is the crawler used for Ahrefs Site Audit projects. This one typically appears when a site owner (or someone with access) sets up an audit for a domain inside Ahrefs.

Why the difference matters: you might be happy to let AhrefsBot crawl your public blog posts, but you may want tighter control over AhrefsSiteAudit if audits could hit heavy pages, staging URLs, or parameter-based URL traps. On the flip side, if you actively use Ahrefs tools for your own site, blocking AhrefsSiteAudit can lead to incomplete audit results.

Each bot has its own robots.txt token: AhrefsBot and AhrefsSiteAudit. A rule for one does not cover the other. Per the Ahrefs Bots page, AhrefsSiteAudit crawls at most 30 URLs per minute, and only verified site owners can raise that speed or let it ignore robots.txt on their own site.

Ahrefs user agent strings

These are the exact user agents Ahrefs lists on ahrefs.com/robot (checked November 2026). Use them to match requests in your server logs:

• AhrefsBot: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
• AhrefsSiteAudit (desktop): Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
• AhrefsSiteAudit (mobile): Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.173 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

For blocking or allowing, you rarely need the full string. Robots.txt and most firewall rules match on the short tokens AhrefsBot and AhrefsSiteAudit, which stay the same when Ahrefs bumps the version number or the mobile Chrome build.

Important: a user-agent string is easy to fake. Anyone can send a request that says “AhrefsBot.” Treat the user-agent as a clue, not proof.

Ahrefs IP addresses

Ahrefs publishes its crawler IP addresses in two places: a live JSON list of IP ranges (or individual IPs), and a plain-text copy in its IP list help article. Both are linked from ahrefs.com/robot. Every real Ahrefs IP also reverse-resolves to a hostname ending in ahrefs.com or ahrefs.net.

A few things to know before you copy the list:

• The list is IPv4 only. When we pulled the JSON range list in November 2026, it held 81 IPv4 prefixes and no IPv6 addresses.
• The help article also lists retired IPs that no longer belong to Ahrefs. Remove those from old allowlists, since traffic from them is not Ahrefs.
• Ahrefs asks you to allowlist all of its ranges, not a subset, so both bots can reach your site.

To keep an allowlist current, set up a cron job to pull IP addresses from the Ahrefs IP range list (or the Ahrefs individual IP list) on a schedule, so ranges Ahrefs adds or retires don’t leave you with a stale list.

Both endpoints are free and don’t need an API key. They return JSON by default, and the Ahrefs API docs list csv, xml, and php as other output options.

The API returns Ahrefs IP addresses as CIDR ranges in JSON format. A shortened response looks like this:

{
  "prefixes": [
    { "ipv4Prefix": "5.39.1.224/27" },
    { "ipv4Prefix": "5.39.109.160/27" },
    { "ipv4Prefix": "15.235.27.0/24" }
  ]
}

You can match incoming request IPs against these CIDR ranges to build an accurate allowlist or blocklist. If you’re on Cloudflare, you can paste these ranges directly into a WAF rule instead of managing them on your server.

How to verify the real AhrefsBot

If you’re going to allowlist AhrefsBot, don’t do it based on the user-agent alone. Your safest approach is to verify using multiple signals: user-agent, IP checks, and reverse DNS.

The three signals compare like this:

You can do these checks on most setups: Apache, Nginx, managed WordPress hosts, and CDNs like Cloudflare. Cloudflare also recognizes many major crawlers, and Ahrefs bots are included in Cloudflare’s verified bot ecosystem, which helps reduce guesswork when you’re filtering bot traffic in WAF logs.

Check your server logs for crawler patterns

Start with your raw access logs or request analytics (CDN logs work too). Our guide to finding wasted crawl budget in server logs shows how to pull and filter them. Ahrefs Bot Analytics shows which bots hit which pages without a JavaScript tag. It connects through Cloudflare, Vercel, Amazon CloudFront, or Fastly, and it’s free during its beta on paid Ahrefs plans. You’re looking for normal crawler behavior:

• Request rate that ramps up and down instead of constant spikes.
• Mostly public URLs, like blog posts, category pages, and sitemaps.
• Respect for obvious blocks. For example, not hammering pages that return 403 or 404.
• Reasonable status codes (mostly 200s and 304s, with the occasional 404 if your internal links aren’t perfect).

Red flags that often point to spoofed bots:

• Huge bursts that look like a stress test (hundreds of requests per second).
• Probing of sensitive paths like /wp-admin/, /xmlrpc.php, .env, or random PHP files that don’t exist.
• Ignoring your patterns. For example, repeatedly requesting the same broken URL, or crawling a blocked path nonstop.
• Odd geographic IP locations that don’t line up with the real crawler’s infrastructure footprint.

If you do see behavior that looks like scraping or probing, treat it like suspicious traffic first, then verify whether it’s truly Ahrefs before you block by user-agent.

Verify an IP with reverse DNS

Once you’ve identified a suspicious (or simply heavy) request, take the IP address and verify it.

A safe verification flow looks like this:

1. Copy the requesting IP from your logs (not from a third-party report).
2. Run a reverse DNS lookup on that IP.
3. Confirm the reverse hostname ends with ahrefs.com or ahrefs.net.
4. Cross-check the IP against the official IP list above.

This matters because IP ranges can change over time. Don’t rely on a random list from an old blog post or a firewall snippet you found in a forum. Always use the official source as your reference point, then build your allowlist or blocklist around what you confirm today.

If you’re doing content and link work, it can be helpful to compare what third-party crawlers see versus what your own tools show. For example, you can sanity-check your backlink visibility using RightBlogger’s Free Backlink Checker Tool while you decide whether to allow or restrict crawlers.

How to block, slow, or allow AhrefsBot

Managing AhrefsBot is really about choosing the lightest control that solves your problem.

A simple decision framework:

• Allow it if your site is public and you want accurate backlink and content discovery data (and possible visibility in Yep.com).
• Throttle it if crawling is real but your server resources are tight.
• Block it if the content is private, the site is staging, you’re running a paid community, or you’re under active strain.

Start with partial controls before full blocks. It’s like turning down a faucet before you shut off the water main.

Block or allow Ahrefs bots in robots.txt

Robots.txt is your first line of control because it’s simple and it’s reversible. You can:

• Block everything for AhrefsBot
• Block only sensitive areas
• Allow everything (by doing nothing special, assuming you’re not blocking it already)

Examples you can adapt:

Block AhrefsBot site-wide:

User-agent: AhrefsBot
Disallow: /

Block only sensitive paths:

User-agent: AhrefsBot
Disallow: /private/
Disallow: /wp-admin/

Block Ahrefs Site Audit crawls:

User-agent: AhrefsSiteAudit
Disallow: /

Allow AhrefsBot when other bots are blocked:

User-agent: *
Disallow: /

User-agent: AhrefsBot
Allow: /

Ahrefs says both bots respect Allow as well as Disallow rules. A bot follows the group that names it and ignores the * group, so the second group above lets AhrefsBot in while everything without its own group stays blocked. If nothing in your robots.txt blocks AhrefsBot, it’s already allowed and you don’t need to add anything.

A few practical notes:

• Robots.txt changes aren’t instant, they apply on the next crawl.
• These rules only cover AhrefsBot. To stop Ahrefs Site Audit crawls too, add a separate group that starts with User-agent: AhrefsSiteAudit.
• If your robots.txt has errors, Ahrefs says its bots can’t read your commands and keep crawling the way they did before. Keep the file clean.
• Blocking /wp-admin/ is common, but remember WordPress also needs admin-ajax.php for some public features. Don’t block files blindly if your front-end depends on them.

AhrefsBot crawl-delay

If your goal is “less load, same discovery,” a crawl-delay can help.

A common approach is:

User-agent: AhrefsBot
Crawl-delay: 10

The value is the minimum number of seconds between two requests. Both Ahrefs bots obey it, so add a matching User-agent: AhrefsSiteAudit group if Site Audit crawls are the problem. Site Audit is already capped at 30 URLs per minute, and only a verified owner can tell it to ignore crawl-delay on their own site.

One catch: Ahrefs says crawl-delay is strictly followed for HTML pages but cannot be followed when its bots render JavaScript, because a render loads several assets at once. You may still see bursts around CSS, JS, or images. If you’re seeing that kind of pattern, combine crawl-delay with caching and rate limiting at the edge.

For a short outage or maintenance window, Ahrefs says you can return 4xx or 5xx status codes. AhrefsBot detects the errors and backs off automatically, so you don’t need to edit robots.txt for a temporary slowdown.

Block or allow Ahrefs bots at the firewall

A firewall or CDN rule is the most reliable way to enforce whatever you decide about AhrefsBot.

Firewall and CDN controls are most useful when:

• You’re seeing fake bots spoofing the AhrefsBot user-agent.
• You want to allow only verified Ahrefs IPs and block everything else that claims to be Ahrefs.
• You need stronger enforcement than robots.txt (since robots is a request, not a lock).

Common options:

• Cloudflare or WAF rules. Create rules based on verified bot status, ASN, or IP ranges. If you allowlist by IP, keep it updated from Ahrefs’ official list.
• Server firewall rules. Useful for dedicated servers, but riskier if you don’t maintain them. A stale allowlist causes false blocks.
• Managed host bot controls. Some hosts offer toggles for “known bots” or custom rules at the edge.

In Cloudflare, you can paste either of these expressions into a WAF custom rule and set the action to Block. This one blocks fake AhrefsBot traffic but lets the verified crawler through:

(http.user_agent contains "AhrefsBot" and not cf.client.bot)

This one blocks both Ahrefs crawlers entirely:

(http.user_agent contains "AhrefsBot") or (http.user_agent contains "AhrefsSiteAudit")

The first rule relies on Cloudflare’s cf.client.bot field, which is true when a request comes from a known good bot. Ahrefs’ crawlers are on Cloudflare’s verified bot list, so real AhrefsBot requests pass and impostors get blocked.

To allow Ahrefs through a firewall that’s blocking it, Ahrefs’ Cloudflare help article recommends a custom rule with the field “IP Source Address,” the operator “is in,” and every range and single IP from its list. Set the action to Skip and skip all remaining custom rules, rate limiting rules, managed rules, and Super Bot Fight Mode rules. The expression starts like this:

(ip.src in {5.39.1.224/27 5.39.109.160/27 15.235.27.0/24 ...})

If you use Cloudflare’s older IP Access rules instead and they reject the /26 and /27 ranges, Ahrefs says to enter those as individual IPs (its help article lists them). On other firewalls the idea is the same: allow the published ranges, and keep them updated.

A good safety habit: if you decide to allowlist, only allowlist after verification (reverse DNS plus IP list). Otherwise, you can accidentally give a malicious scraper a free pass just because it set its user-agent to “AhrefsBot.”

Block AhrefsBot with .htaccess or Nginx rules

If robots.txt isn’t enough, you can block AhrefsBot at the server level. This stops the bot before it reaches your application.

For Apache servers, add this to your .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC]
RewriteRule .* - [F,L]

For Nginx, add this inside your server {} block:

if ($http_user_agent ~* "AhrefsBot") {
    return 403;
}

Server-level rules are stronger than robots.txt because they return a 403 before PHP even loads. Use .htaccess or Nginx rules when you’ve confirmed a bot is ignoring your robots.txt, or when you need to save resources on high-traffic servers.

Both examples match AhrefsBot only. To block Site Audit crawls as well, change the pattern to (AhrefsBot|AhrefsSiteAudit).

Troubleshooting AhrefsBot crawl problems

Once you change crawler access, you want to confirm two things: your site still performs well, and you didn’t block something you actually needed.

Fix server bottlenecks before blaming the bot

Crawler traffic often exposes weak spots you already had:

• No full-page caching, so every request hits PHP and the database.
• Expensive endpoints that should be protected or cached.
• Slow TTFB on category pages or search pages.
• Thin error handling that returns 500s during spikes.

A few fixes that pay off fast:

• Enable page caching (plugin, host cache, or edge cache).
• Put a CDN in front of heavy assets and set sane cache headers.
• Return correct status codes (don’t serve 200s for missing pages).
• Watch 4xx and 5xx spikes. Many crawlers slow down when they see lots of errors, so cleaning up error responses can reduce crawl pressure naturally.
• Block URL traps like calendar pages, endless filtered parameters, and internal search results.

Should you block AhrefsBot?

Use real-world scenarios to decide:

You should usually allow AhrefsBot if:

• You run a public blog, niche site, or marketing site.
• You want accurate backlink and content discovery data in third-party SEO platforms.
• You care about visibility in Yep.com.
• You use Ahrefs tools and want complete reports for your own domain.

You should throttle AhrefsBot if:

• You’re on a small VPS or shared hosting.
• Crawling triggers CPU spikes or database bottlenecks.
• You can’t upgrade hosting right now, but you can tune caching and crawl-delay.

You should block AhrefsBot if:

• The site is staging or development.
• Content is paid, private, or community-only.
• You’re dealing with an incident and need to reduce all non-human traffic until things stabilize.

If you do block it, do it intentionally, document it, and set a reminder to review later. Many “temporary” blocks become permanent by accident.

Check whether Ahrefs can crawl your site

If you want a quick yes or no on whether Ahrefs can reach your site, use Ahrefs’ Website status tool.

1. Open Ahrefs’ bot Website status page (the one shown in the screenshot).
2. Select AhrefsBot (for the public crawler) or AhrefsSiteAudit (for audit crawls).
3. Enter your domain (use the exact version you care about, like https://example.com).
4. Click Check status.
5. Read the result:
   • “This website can be crawled fully” means Ahrefs sees your robots.txt as allowing crawling.
   • If it shows blocked paths or errors, click Recrawl all robots.txt after you make changes to confirm the fix.

This is a clean troubleshooting step because it tells you what Ahrefs’ systems see, not just what you think you configured in robots.txt or your firewall.

Fix “robots.txt is not accessible” in Ahrefs

If Site Audit or the status check says your robots.txt is not accessible, Ahrefs couldn’t read the file. The request usually got blocked, timed out, or came back with an error instead of a 200. Work through these checks:

1. Open yoursite.com/robots.txt in a browser. It should load as plain text, not a login page, error, or redirect loop.
2. Request it with the Site Audit user-agent from your terminal: curl -I -A "Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)" https://example.com/robots.txt. A 403 here when your browser gets a 200 means a rule is matching the user-agent. (A fake-bot rule like the Cloudflare one above will also return 403, since your request doesn’t come from Ahrefs’ IPs.)
3. Check your firewall, CDN, and security plugin logs for blocked requests from Ahrefs’ IP ranges. Bot challenges and rate limits can block robots.txt even when the rest of the site loads fine.
4. Allowlist Ahrefs’ published IPs or its verified bots in whatever blocked them. Ahrefs’ IP list help article covers how.
5. Run the Website status check again and click Recrawl all robots.txt.

How to get AhrefsBot to crawl your site

If Ahrefs shows no data for your domain, a few common issues might be blocking the crawler. Work through these steps to allow AhrefsBot access.

1. Check your robots.txt and make sure you haven’t disallowed AhrefsBot with a Disallow: / rule. A valid XML sitemap referenced in robots.txt also helps crawlers discover your pages faster.
2. Remove IP-based blocks. Check your firewall, .htaccess, and security plugins for rules that silently block Ahrefs’ IP ranges.
3. Verify your site in Ahrefs Free, the free account formerly called Ahrefs Webmaster Tools. It’s free for sites you own and gives you Site Explorer and Site Audit data for your verified site.
4. Confirm your site has inbound links. AhrefsBot discovers pages through backlinks, and brand new sites with zero external links give the crawler no path to find them.
5. Be patient. AhrefsBot finds and revisits pages mostly through links, so new sites with few links get visited less often. See the FAQ below for the closest thing to a recrawl request.

Frequently asked questions about AhrefsBot

Is AhrefsBot malicious?

No. AhrefsBot is a legitimate SEO crawler operated by Ahrefs. That said, bad actors can spoof the AhrefsBot user-agent to disguise their own scrapers. Always verify suspicious requests by checking the source IP against Ahrefs’ published IP ranges and running a reverse DNS lookup.

How often does AhrefsBot crawl my site?

Ahrefs doesn’t publish a fixed schedule. Pages with more inbound links tend to get crawled more often. You can use a Crawl-delay directive in robots.txt to control how fast it requests pages.

What happens if I block AhrefsBot?

AhrefsBot stops crawling your pages once it rereads your robots.txt, which isn’t instant. Ahrefs says it begins dropping blocked sites from its index about 6 months after the block. Links pointing to your site from other domains still show in Ahrefs, because blocking only stops crawls of your own pages, so competitors can still see much of your backlink profile. Source: Ahrefs Help Center: How long does it take for my site to be removed from Ahrefs? Blocking does not affect your Google rankings.

Does blocking AhrefsBot affect my Google rankings?

No. Google uses Googlebot, not AhrefsBot. Blocking Ahrefs only affects Ahrefs’ own tools and the Yep.com search engine, which also relies on AhrefsBot data.

What is the AhrefsBot user-agent string?

The main user-agent is Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/). Site Audit currently uses Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) on desktop; the mobile string differs. See all user agent strings above.

Why do old AhrefsBot versions appear in logs?

Ahrefs’ current string is AhrefsBot/7.0. Archived copies of its bot page show AhrefsBot/5.2 in 2018 and AhrefsBot/6.1 in 2019. If a new request claims an old version, treat it as unverified until the IP and reverse DNS check out.

Can I force Ahrefs to recrawl my site?

Not on demand. AhrefsBot works through its own crawl schedule. Yep, Ahrefs’ search engine, takes part in IndexNow, and Ahrefs says that lets site owners notify it when content changes. For your own site, an Ahrefs Free account lets you verify the domain and run Site Audit on it.

Blog Automation That Keeps Your Posts Ranking

RightBlogger finds keywords, writes SEO articles, and publishes them to your blog. Then its Site Agent uses your Search Console data to keep older posts in shape, adding internal links, sharpening weak meta descriptions, and fixing stale years and broken links. Join 51,000+ bloggers using it to grow traffic from Google and ChatGPT.

Start for Free