There are more and more bulk SEO crawling tools hitting the SEO scene but aside from the big names, there are so many others that never get a look in, which is a real shame because there are some pretty amazing ones out there.
So this list is going to continue to expand as I find out about new tools – after all of these years, even I get to find new ones now and then.
I also am not going to suggest which are the best tools here, or even which I would consider the best as I don’t want this to turn into a popularity contest. Many of these I have used while conducting an SEO audit, while others are new to me.
While putting this list together, I have seen some of what these tools can do so I would urge you to take your time and see which one best fits your needs.
[su_note note_color=”#fddfa1″ radius=”7″]Got a tool to add? Leave a message or email me.[/su_note]
[su_divider top=”no” divider_color=”#d0d0d0″]
Agency Analytics Auditor – (from $49 pm)
Quickly identify and resolve on-site SEO issues. Improve rankings and search visibility.
Visit Agency Analytics
Ahrefs – (7-day $7 Trial then from $99 pm)
Tools to grow your search traffic, research your competitors and monitor your niche.
Alexa SEO Audit Tool – (14-day Free Trial then from $149 pm)
Find and fix technical issues that keep you from ranking. Our site audit tool scans your entire site to uncover ways to increase search visibility.
Beam Us Up – (Free)
Don’t waste time exporting and analyzing your SEO crawling software anymore. Get the FREE Beam Us Up SEO Crawler now. It is worth noting that the tool is no longer supported, but still has its usefulness.
Visit Beam Us Up
Botify – ($contact)
Founded in 2012, Botify was born out of a brand-new way of thinking about SEO. Instead of just identifying what’s wrong with a site, what if we had the data to help us understand the underlying issues behind why those issues were occurring?
Botify combines data from website crawls, log files, analytics, and Google Search Console performance data so you can dig deep and uncover correlations in your data (ex: How does page depth impact crawl frequency? Does content uniqueness correlate with organic traffic?). Botify gives you visibility into the entire search process, from crawling to conversions/revenue.
Today, hundreds of enterprise organizations in industries ranging from travel and e-commerce to publishers and agencies use Botify to drive sustainable and profitable outcomes from the organic search channel.
Botify is a fast-growing, VC-backed, SaaS company with $27M in funding and offices in New York, Seattle, Paris, and London.
Checkbot.io – (Freemium / £90 per year)
Fix broken links, duplicate titles, insecure pages, invalid HTML & more. Rated 4.9/5 with 20,000 users.
Clusteric – (From $97 pm)
Technical SEO AUDIT to improve rankings, site speed, visibility and fix crawling errors.
Cognitive SEO – (From £129 pm)
Our all-inclusive Site Audit tool is part of a complete SEO Software suite that covers all the needs an SEO Pro, webmaster or digital marketer might have.
Visit Cognitive SEO
ContentKing App – (From £19 pm)
Maximize your search engine visibility. Intelligent Auditing of your pages.
Visit Content King App
Deepcrawl – (From £63 pm)
Search Marketers experience a multitude of technical SEO issues and challenges which DeepCrawl can help solve.
Forecheck – (from $49 pm)
Forecheck is the fast SEO Crawler Software that will speed up many of your daily tasks. From standard SEO analysis to very unique reports.
Headmaster SEO – (Freemium / From £39 Lifetime)
HEADMasterSEO is a small desktop tool (PC or Mac) that can check very large lists of URLs for status code, redirect details, response time, response headers and HTTP header fields(X-Robots-Tag, Link, Vary). The program can import URLs from plain text, CSV and XML sitemap files. You can analyze, filter and sort the results in real-time or export to CSV.
The program can use up to 200 simultaneous URL requests and HEAD requests to speed up URL checking. It is designed to consume as little memory as possible and can process millions of URLs on computers with a modest amount of memory.
HEADMasterSEO can completely automate some technical SEO tasks such as auditing redirects in website migrations, testing canonical redirects and testing HSTS headers.
For testing website migrations you can setup URL mapping rules (similar to server URL rewriting) and the software can check if every URL redirects to the correct final destination using a single 301 redirect. It’s been successfully used to check website migrations with more than 4 million redirects.
HEADMasterSEO is free for checking small lists of up to 500 URLs and licenses are available for checking larger lists.
Visit Headmaster SEO
Jet Octopus – (From €20 pm)
Crawl 50 mln+ pages easily. Logs analysis overlapped with crawling data gives the most revealing site indexability picture. Look at your site with Google”s eyes.
Visit Jet Octopus
Lipperhey – (Freemium / $99 pm)
The complete SEO package for Free! Absolutely free (forever). No credit card, no commitment.
Moz – (From $99 pm)
We crawl your site on demand, uncover technical SEO issues, and recommend improvements and fixes.
Netpeak Spider – (From $19 pm)
Netpeak Spider is a desktop SEO crawler that checks 50+ on-page parameters detecting 60+ technical issues. The tool can be used to spot broken links, redirects, canonical issues, analyze meta tags, content, internal linking, page speed, and various other tasks.
It also has a bunch of built-in tools such as Web Scraper, Internal PageRank Calculator, XML Sitemap Generator and Validator, Source Code and HTTP Headers Analysis tool. It has a user-friendly interface, each issue contains a detailed description and tips on how to fix it.
Netpeak Spider allows exporting lots of custom and pre-defined reports including the Express Audit of the Optimization Quality in PDF.
Visit Netpeak Spider
Nightwatch.io – (From $19 pm)
With the introduction of Nightwatch’s Site Audit feature, users will be able to crawl their website directly in Nightwatch to quickly review if there are any SEO-related issues across website pages. Additionally, users can run the smart scheduler to schedule and save crawls automatically.
The Site Audit feature allows digital marketing professionals to set their insights on different SEO projects, create a project time frame, and compare data in the long run. It’s a quick way to perform an exhaustive website analysis to improve website ranking without doing too much heavy lifting.
OnCrawl – (From €39 pm)
OnCrawl is an award-winning technical SEO platform for enterprise SEO audits and daily monitoring that helps you make smarter SEO decisions. OnCrawl combines your content, log files and search data at scale so that you can open Google’s blackbox and build a SEO strategy with confidence.
Backed by a SEO crawler, a log analyzer and third-party integrations, OnCrawl currently works with over 800 clients in 66 countries including e-commerce websites, online publishers and agencies. Clients include Vistaprint, Canon, Lastminute.com, Forbes, and other major companies.
Driven by a strong technical spirit, OnCrawl gives search marketers easy access to the data they need by providing actionable dashboards and reports to support the entire SEO process to help you improve your rankings, traffic and revenues.
OnCrawl offers a free 14-day trial..
Pulno – (Freemium / From $12 pm)
Do you want to SEO like a pro? You can with Pulno! Check your website for issues with titles, headers, meta tags, broken links, broken images, page speed, encryption, redirects and more. Pulno will also let you optimize unused fonts, CSS and image files.
Moreover, the tool is equipped with a Unique Content Analyzer that looks for duplicate content both on-site and off-site. You can also now integrate Pulno with Google Search Console, meaning that you can see all the pages that are only one small improvement away from generating more search traffic to your website in one place.
Finally, you can create a comprehensive and easy to read report that will guide you as you fix your SEO issues.
Coupon code: IQSEO (10% lifetime discount).
Rank Active – (From $29 pm)
All on-page ranking factors in the palm of your hand. Look upon your website like search engines do.
Visit Rank Active
Raven Site Auditor – (From $109 pm)
Diagnose website issues and improve visibility to your site.
Visit Raven Site Auditor
Screaming Frog – (Freemium / £149 per year)
Screaming Frog is the original desktop SEO focused website crawler built for professional SEOs and regularly releases new advancements.
It has an advanced configurable hybrid storage engine which allows it to crawl millions of URLs and is extremely flexible, able to crawl small and large websites efficiently. It gathers key onsite data and provides hints to allow SEOs to make better, more-informed decisions.
It can be used for the basics, such as finding broken links, identifying duplicate content or improving page titles. But also the more advanced, such as integrating Google Search Console search performance or Google Analytics data, analysing hreflang annotations, creating beautiful visualisations, or performing XPath custom extraction and much more.
Visit Screaming Frog
Scrutiny (Mac Only) – ($115 USD Lifetime)
Scrutiny for Mac is a native Mac desktop app which means that it’s secure, fast and efficient and you own your own data.
It has many pages of settings and configuration. The basic scan settings and options allow you to fine-tune the engine for your site, including easy rate-limiting. Advanced settings include JS rendering, ‘on finish’ actions, authentication and scheduling.
When the scan is complete you have a large amount of information at your fingertips; Broken links / images / linked files, xml sitemap and sitemap visualisation with automatic ftp/sftp, spellcheck results, a large amount of SEO data, a summary of any areas of concern, reports for many SEO tests such as missing/long/short titles/descriptions, duplicate content, images without alt text, thin content, deep content and many more. Generate a customisable report for yourself or for your clients comprising a summary and a configurable selection of tables in csv or html format.
Scrutiny can help your migration to https by reporting pages with mixed content, and pages with links to the http site.
Search Console – (Free)
This is used by almost all site owners or their SEO consultants but is often forgotten about just what information you can get from it.
If you aren’t making use of Search Console, you really should change that.
Visit Search Console
SE Optimer – (From $19 pm)
DIY SEO crawls your whole site for problems, recommends clear, easy to follow tasks with guides for your website type.
Visit SE Optimer
SE Ranking – (From £23 pm)
SE Ranking offers a complete set of tools for a comprehensive site audit, SEO/PPC competitor analysis, website ranking, keyword suggestion and grouping, backlink monitoring, automated professional reporting, and much more.
Besides standard SEO tools the system offers a set of additional features such as Page changes monitor, Lead generator and Marketing Plan aimed at bringing your professional digital services to a whole new level. All on one platform within the same user-friendly interface
Visit SE Ranking
SEMRush – (From $99 pm)
Complete workflow for any SEO professional
SEO Mator – (From $19 pm)
Seomator crawls your website and gives you a full and comprehensive report based on your website’s SEO, internal links, HTML tags, backlinks, page speed, mobile usability, text statistics, social media, organic presence and content quality.
Visit SEO Mator
SEO Monitor – (From €49 pm)
The SEO platform that actively works for you & your agency.
Visit SEO Monitor
SEO Profiler – (From $69 pm)
SEOprofiler is a full featured website promotion tool that offers SEO tools for keyword research, web page optimization, link analysis, link building, ranking checks, SEO audits, website monitoring, website analytics, and much more.
Visit SEO Profiler
SEObility – (Freemium / £39 pm)
Online SEO Software for better website optimization. Seobility comes with all SEO analysis and tools you need.
Serped.net – (From $79 pm)
From keyword research and backlink analysis to rank tracking and reporting: everything you need to outrank competition and generate more traffic
Serpstat – (From $69 pm)
Stay in the loop of your website’s stats. Using Serpstat On-site audit allows you to perform an instant in-depth analysis of web pages by addressing multiple ranking factors
Sistrix – (From €100 pm)
OnPage-Optimisation using the SISTRIX Optimizer. Analyse a site for technical SEO issues. We’ll give you a prioritised list of actions.
Site Analyzer – (From €25 pm)
Site Analyzer is the 360° SEO analyzer for any website. Try it for free and analyze your SEO!
Visit Site Analyzer
Sitebulb – (£25 pm)
Sitebulb is a desktop website crawler, that is much more than just a crawler. It audits your website and produces actionable recommendations and beautiful visualizations. It will take your technical SEO audits to the next level, and save you time in the process.
In its debut year, Sitebulb won ‘Best Search Software Tool’ at both the UK and US Search awards, and with their Crawl Maps, an innovative solution to visualizing website architecture, they won the ‘Best Software Innovation’ at the US Search Awards.
Spotibo – (Freemium / From $9 pm)
Find duplicate content on your site, error pages, wrong anchor texts, redirect chains and much more.
Surfer SEO – (From $29 pm)
Surfer makes on-page SEO productive and ridiculously easy. It analyses 500+ signals from your organic competitors, so you get the quickest bump in search results. Once you enter your keyword, you can check how your page is doing against top performers.
If you’re missing some important terms, Surfer will let you know. If your content is too short or your keyword density is way below average, you will see that right away. If your meta tags are not optimized, you’ll get recommendations on how to fix that.
The best part? Basically, anyone with basic SEO knowledge can use Surfer and implement its suggestions.
Visit Surfer SEO
Visual SEO Studio – (Freemium / €149 per year)
Visual SEO Studio, the SEO Analysis Tool for Windows & Mac.
Visit Visual SEO Studio
Webbee – (From $15 pm)
Webbee is a desktop based SEO spider that crawl your website following the pattern of major search engine bots. It searches every nook and corner of your website and collects data for you to spot fruitful opportunities and critical issues that can be turned into major benefits.
Website Auditor – (From £24 pm)
WebSite Auditor is a site audit tool that allows you to crawl websites and analyse technical, onsite and onpage SEO. It’s a cross-platform app, so you can run it on Windows, Mac, and Linux systems. With WebSite Auditor, you can: –
- Analyse site structure (+ build visualisations).
- Customise crawl settings.
- Check sites for indexing issues, redirects, encoding factors, broken links, page speed, broken images, duplicate meta tags, etc.
- Analyse page titles and meta data.
- Manage robots.txt files and XML sitemaps.
- Analyse internal links (using the InLink rank metric)
- Audit hreflang implementation.
- Analyse on-page optimisation of landing pages.
- Check the number of visitors every page gets.
- Use TF-IDF for content optimisation.
- Check for duplicate content issues.
- Schedule checks.
- Create custom reports in HTML/PDF/CSV.
- Store project data and reports in SEO PowerSuite Cloud or Dropbox
You can have all interface and reports in several languages: English, German, Spanish, French, Dutch, Polish, Russian, and Slovak.
Visit Website Auditor
Wildshark Spider – ($contact)
Identify Errors And Improve Usability With The WildShark SEO Spider Tool.
Visit Wildshark Spider
WooRank – (From £59 pm)
Instant SEO Checker & SEO Audit Tool. Simple, easy-to-use website analysis tool to track and connect your marketing data
Xenu Link Sleuth – (Free)
Xenu’s Link Sleuth (TM) checks Web sites for broken links. It displays a continously updated list of URLs which you can sort by different criteria.
Visit Xenu Link Sleuth