Spider Crawler Software
Screaming Frog SEO Spider Website Crawler
The industry leading website crawler for Windows, macOS and Ubuntu, trusted by thousands of SEOs and agencies worldwide for technical SEO site audits.
Download
Pricing
Buy & Renew
Overview
User Guide
Tutorials
FAQ
Support
SEO Spider Tool
The Screaming Frog SEO Spider is a website crawler that helps you improve onsite SEO, by extracting data & auditing for common SEO issues. Download & crawl 500 URLs for free, or buy a licence to remove the limit & access advanced features.
Free Vs Paid
What can you do with the SEO Spider Tool?
The SEO Spider is a powerful and flexible site crawler, able to crawl both small and very large websites efficiently, while allowing you to analyse the results in real-time. It gathers key onsite data to allow SEOs to make informed decisions.
Features
Find Broken Links, Errors & Redirects
Analyse Page Titles & Meta Data
Review Meta Robots & Directives
Audit hreflang Attributes
Discover Exact Duplicate Pages
Generate XML Sitemaps
Site Visualisations
Crawl Limit
Scheduling
Crawl Configuration
Save Crawls & Re-Upload
JavaScript Rendering
Crawl Comparison
Near Duplicate Content
Custom
AMP Crawling & Validation
Structured Data & Validation
Spelling & Grammar Checks
Custom Source Code Search
Custom Extraction
Google Analytics Integration
Search Console Integration
PageSpeed Insights Integration
Link Metrics Integration
Forms Based Authentication
Store & View Raw & Rendered HTML
Free Technical Support
Price per licence
Licences last 1 year. After that you will be required to renew your licence.
Free Version
Crawl Limit – 500 URLs
Paid Version
Crawl Limit – Unlimited*
* The maximum number of URLs you can crawl is dependent on allocated memory and storage. Please see our FAQ.
” Out of the myriad of tools we use at iPullRank I can definitively say that I only use the Screaming Frog SEO Spider every single day. It’s incredibly feature-rich, rapidly improving and I regularly find a new use case. I can’t endorse it strongly enough. ”
Mike King
Founder, iPullRank
” The Screaming Frog SEO Spider is my “go to” tool for initial SEO audits and quick validations: powerful, flexible and low-cost. I couldn’t recommend it more. ”
Aleyda Solis
Owner, Orainti
The SEO Spider Tool Crawls & Reports On…
The Screaming Frog SEO Spider is an SEO auditing tool,
built by real SEOs with thousands of users worldwide. A quick summary of some of the data collected in a crawl
include –
Errors – Client errors such as broken links & server errors (No responses, 4XX client & 5XX server errors).
Redirects – Permanent, temporary, JavaScript redirects & meta refreshes.
Blocked URLs – View & audit URLs disallowed by the protocol.
Blocked Resources – View & audit blocked resources in rendering mode.
External Links – View all external links, their status codes and source pages.
Security – Discover insecure pages, mixed content, insecure forms, missing security headers and more.
URI Issues – Non ASCII characters, underscores, uppercase characters, parameters, or long URLs.
Duplicate Pages – Discover exact and near duplicate pages using advanced algorithmic checks.
Page Titles – Missing, duplicate, long, short or multiple title elements.
Meta Description – Missing, duplicate, long, short or multiple descriptions.
Meta Keywords – Mainly for reference or regional search engines, as they are not used by Google, Bing or Yahoo.
File Size – Size of URLs & Images.
Response Time – View how long pages take to respond to requests.
Last-Modified Header – View the last modified date in the HTTP header.
Crawl Depth – View how deep a URL is within a website’s architecture.
Word Count – Analyse the number of words on every page.
H1 – Missing, duplicate, long, short or multiple headings.
H2 – Missing, duplicate, long, short or multiple headings
Meta Robots – Index, noindex, follow, nofollow, noarchive, nosnippet etc.
Meta Refresh – Including target page and time delay.
Canonicals – Link elements & canonical HTTP headers.
X-Robots-Tag – See directives issued via the HTTP Headder.
Pagination – View rel=“next” and rel=“prev” attributes.
Follow & Nofollow – View meta nofollow, and nofollow link attributes.
Redirect Chains – Discover redirect chains and loops.
hreflang Attributes – Audit missing confirmation links, inconsistent & incorrect languages codes, non canonical hreflang and more.
Inlinks – View all pages linking to a URL, the anchor text and whether the link is follow or nofollow.
Outlinks – View all pages a URL links out to, as well as resources.
Anchor Text – All link text. Alt text from images with links.
Rendering – Crawl JavaScript frameworks like AngularJS and React, by crawling the rendered HTML after JavaScript has executed.
AJAX – Select to obey Google’s now deprecated AJAX Crawling Scheme.
Images – All URLs with the image link & all images from a given page. Images over 100kb, missing alt text, alt text over 100 characters.
User-Agent Switcher – Crawl as Googlebot, Bingbot, Yahoo! Slurp, mobile user-agents or your own custom UA.
Custom HTTP Headers – Supply any header value in a request, from Accept-Language to cookie.
Custom Source Code Search – Find anything you want in the source code of a website! Whether that’s Google Analytics code, specific text, or code etc.
Custom Extraction – Scrape any data from the HTML of a URL using XPath, CSS Path selectors or regex.
Google Analytics Integration – Connect to the Google Analytics API and pull in user and conversion data directly during a crawl.
Google Search Console Integration – Connect to the Google Search Analytics API and collect impression, click and average position data against URLs.
PageSpeed Insights Integration – Connect to the PSI API for Lighthouse metrics, speed opportunities, diagnostics and Chrome User Experience Report (CrUX) data at scale.
External Link Metrics – Pull external link metrics from Majestic, Ahrefs and Moz APIs into a crawl to perform content audits or profile links.
XML Sitemap Generation – Create an XML sitemap and an image sitemap using the SEO spider.
Custom – Download, edit and test a site’s using the new custom
Rendered Screen Shots – Fetch, view and analyse the rendered pages crawled.
Store & View HTML & Rendered HTML – Essential for analysing the DOM.
AMP Crawling & Validation – Crawl AMP URLs and validate them, using the official integrated AMP Validator.
XML Sitemap Analysis – Crawl an XML Sitemap independently or part of a crawl, to find missing, non-indexable and orphan pages.
Visualisations – Analyse the internal linking and URL structure of the website, using the crawl and directory tree force-directed diagrams and tree graphs.
Structured Data & Validation – Extract & validate structured data against specifications and Google search features.
Spelling & Grammar – Spell & grammar check your website in over 25 different languages.
Crawl Comparison – Compare crawl data to see changes in issues and opportunities to track technical SEO progress. Compare site structure, detect changes in key elements and metrics and use URL mapping to compare staging against production sites.
” I’ve tested nearly every SEO tool that has hit the market, but I can’t think of any I use more often than Screaming Frog. To me, it’s the Swiss Army Knife of SEO Tools. From uncovering serious technical SEO problems to crawling top landing pages after a migration to uncovering JavaScript rendering problems to troubleshooting international SEO issues, Screaming Frog has become an invaluable resource in my SEO arsenal. I highly recommend Screaming Frog for any person involved in SEO. ”
” Screaming Frog Web Crawler is one of the essential tools I turn to when performing a site audit. It saves time when I want to analyze the structure of a site, or put together a content inventory for a site, where I can capture how effective a site might be towards meeting the informational or situation needs of the audience of that site. I usually buy a new edition of Screaming Frog on my birthday every year, and it is one of the best birthday presents I could get myself. ”
Bill Slawski
Director, Go Fish Digital
About The Tool
The Screaming Frog SEO Spider is a fast and advanced SEO site audit tool. It can be used to crawl both small and very large websites, where manually checking every page would be extremely labour intensive, and where you can easily miss a redirect, meta refresh or duplicate page issue. You can view, analyse and filter the crawl data as it’s gathered and updated continuously in the program’s user interface.
The SEO Spider allows you to export key onsite SEO elements (URL, page title, meta description, headings etc) to a spread sheet, so it can easily be used as a base for SEO recommendations. Check our out demo video above.
Crawl 500 URLs For Free
The ‘lite’ version of the tool is free to download and use. However, this version is restricted to crawling up to 500 URLs in a single crawl and it does not give you full access to the configuration, saving of crawls, or advanced features such as JavaScript rendering, custom extraction, Google Analytics integration and much more. You can crawl 500 URLs from the same website, or as many websites as you like, as many times as you like, though!
For just £149 per year you can purchase a licence, which removes the 500 URL crawl limit, allows you to save crawls, and opens up the spider’s configuration options and advanced features.
Alternatively hit the ‘buy a licence’ button in the SEO Spider to buy a licence after downloading and trialing the software.
FAQ & User Guide
The SEO Spider crawls sites like Googlebot discovering hyperlinks in the HTML using a breadth-first algorithm. It uses a configurable hybrid storage engine, able to save data in RAM and disk to crawl large websites. By default it will only crawl the raw HTML of a website, but it can also render web pages using headless Chromium to discover content and links.
For more guidance and tips on our to use the Screaming Frog SEO crawler –
Please read our quick-fire getting started guide.
Please see our recommended hardware, user guide, tutorials and FAQ. Please also watch the demo video embedded above!
Check out our tutorials, including how to use the SEO Spider as a broken link checker, duplicate content checker, website spelling & grammar checker, generating XML Sitemaps, crawling JavaScript, testing, web scraping, crawl comparison and crawl visualisations.
Updates
Keep updated with future releases by subscribing to RSS feed, our mailing list below and following us on Twitter @screamingfrog.
Support & Feedback
If you have any technical problems, feedback or feature requests for the SEO Spider, then please just contact us via our support. We regularly update the SEO Spider and currently have lots of new features in development!
Back to top
What is a web crawler? | How web spiders work | Cloudflare
What is a web crawler bot?
A web crawler, spider, or search engine bot downloads and indexes content from all over the Internet. The goal of such a bot is to learn what (almost) every webpage on the web is about, so that the information can be retrieved when it’s needed. They’re called “web crawlers” because crawling is the technical term for automatically accessing a website and obtaining data via a software program.
These bots are almost always operated by search engines. By applying a search algorithm to the data collected by web crawlers, search engines can provide relevant links in response to user search queries, generating the list of webpages that show up after a user types a search into Google or Bing (or another search engine).
A web crawler bot is like someone who goes through all the books in a disorganized library and puts together a card catalog so that anyone who visits the library can quickly and easily find the information they need. To help categorize and sort the library’s books by topic, the organizer will read the title, summary, and some of the internal text of each book to figure out what it’s about.
However, unlike a library, the Internet is not composed of physical piles of books, and that makes it hard to tell if all the necessary information has been indexed properly, or if vast quantities of it are being overlooked. To try to find all the relevant information the Internet has to offer, a web crawler bot will start with a certain set of known webpages and then follow hyperlinks from those pages to other pages, follow hyperlinks from those other pages to additional pages, and so on.
It is unknown how much of the publicly available Internet is actually crawled by search engine bots. Some sources estimate that only 40-70% of the Internet is indexed for search – and that’s billions of webpages.
What is search indexing?
Search indexing is like creating a library card catalog for the Internet so that a search engine knows where on the Internet to retrieve information when a person searches for it. It can also be compared to the index in the back of a book, which lists all the places in the book where a certain topic or phrase is mentioned.
Indexing focuses mostly on the text that appears on the page, and on the metadata* about the page that users don’t see. When most search engines index a page, they add all the words on the page to the index – except for words like “a, ” “an, ” and “the” in Google’s case. When users search for those words, the search engine goes through its index of all the pages where those words appear and selects the most relevant ones.
*In the context of search indexing, metadata is data that tells search engines what a webpage is about. Often the meta title and meta description are what will appear on search engine results pages, as opposed to content from the webpage that’s visible to users.
How do web crawlers work?
The Internet is constantly changing and expanding. Because it is not possible to know how many total webpages there are on the Internet, web crawler bots start from a seed, or a list of known URLs. They crawl the webpages at those URLs first. As they crawl those webpages, they will find hyperlinks to other URLs, and they add those to the list of pages to crawl next.
Given the vast number of webpages on the Internet that could be indexed for search, this process could go on almost indefinitely. However, a web crawler will follow certain policies that make it more selective about which pages to crawl, in what order to crawl them, and how often they should crawl them again to check for content updates.
The relative importance of each webpage: Most web crawlers don’t crawl the entire publicly available Internet and aren’t intended to; instead they decide which pages to crawl first based on the number of other pages that link to that page, the amount of visitors that page gets, and other factors that signify the page’s likelihood of containing important information.
The idea is that a webpage that is cited by a lot of other webpages and gets a lot of visitors is likely to contain high-quality, authoritative information, so it’s especially important that a search engine has it indexed – just as a library might make sure to keep plenty of copies of a book that gets checked out by lots of people.
Revisiting webpages: Content on the Web is continually being updated, removed, or moved to new locations. Web crawlers will periodically need to revisit pages to make sure the latest version of the content is indexed.
requirements: Web crawlers also decide which pages to crawl based on the protocol (also known as the robots exclusion protocol). Before crawling a webpage, they will check the file hosted by that page’s web server. A file is a text file that specifies the rules for any bots accessing the hosted website or application. These rules define which pages the bots can crawl, and which links they can follow. As an example, check out the file.
All these factors are weighted differently within the proprietary algorithms that each search engine builds into their spider bots. Web crawlers from different search engines will behave slightly differently, although the end goal is the same: to download and index content from webpages.
Why are web crawlers called ‘spiders’?
The Internet, or at least the part that most users access, is also known as the World Wide Web – in fact that’s where the “www” part of most website URLs comes from. It was only natural to call search engine bots “spiders, ” because they crawl all over the Web, just as real spiders crawl on spiderwebs.
Should web crawler bots always be allowed to access web properties?
That’s up to the web property, and it depends on a number of factors. Web crawlers require server resources in order to index content – they make requests that the server needs to respond to, just like a user visiting a website or other bots accessing a website. Depending on the amount of content on each page or the number of pages on the site, it could be in the website operator’s best interests not to allow search indexing too often, since too much indexing could overtax the server, drive up bandwidth costs, or both.
Also, developers or companies may not want some webpages to be discoverable unless a user already has been given a link to the page (without putting the page behind a paywall or a login). One example of such a case for enterprises is when they create a dedicated landing page for a marketing campaign, but they don’t want anyone not targeted by the campaign to access the page. In this way they can tailor the messaging or precisely measure the page’s performance. In such cases the enterprise can add a “no index” tag to the landing page, and it won’t show up in search engine results. They can also add a “disallow” tag in the page or in the file, and search engine spiders won’t crawl it at all.
Website owners may not want web crawler bots to crawl part or all of their sites for a variety of other reasons as well. For instance, a website that offers users the ability to search within the site may want to block the search results pages, as these are not useful for most users. Other auto-generated pages that are only helpful for one user or a few specific users should also be blocked.
What is the difference between web crawling and web scraping?
Web scraping, data scraping, or content scraping is when a bot downloads the content on a website without permission, often with the intention of using that content for a malicious purpose.
Web scraping is usually much more targeted than web crawling. Web scrapers may be after specific pages or specific websites only, while web crawlers will keep following links and crawling pages continuously.
Also, web scraper bots may disregard the strain they put on web servers, while web crawlers, especially those from major search engines, will obey the file and limit their requests so as not to overtax the web server.
How do web crawlers affect SEO?
SEO stands for search engine optimization, and it is the discipline of readying content for search indexing so that a website shows up higher in search engine results.
If spider bots don’t crawl a website, then it can’t be indexed, and it won’t show up in search results. For this reason, if a website owner wants to get organic traffic from search results, it is very important that they don’t block web crawler bots.
What web crawler bots are active on the Internet?
The bots from the major search engines are called:
Google: Googlebot (actually two crawlers, Googlebot Desktop and Googlebot Mobile, for desktop and mobile searches)
Bing: Bingbot
Yandex (Russian search engine): Yandex Bot
Baidu (Chinese search engine): Baidu Spider
There are also many less common web crawler bots, some of which aren’t associated with any search engine.
Why is it important for bot management to take web crawling into account?
Bad bots can cause a lot of damage, from poor user experiences to server crashes to data theft. However, in blocking bad bots, it’s important to still allow good bots, such as web crawlers, to access web properties. Cloudflare Bot Management allows good bots to keep accessing websites while still mitigating malicious bot traffic. The product maintains an automatically updated allowlist of good bots, like web crawlers, to ensure they aren’t blocked. Smaller organizations can gain a similar level of visibility and control over their bot traffic with Super Bot Fight Mode, available on Cloudflare Pro and Business plans.
What is a web crawler and how does it work? – Ryte
A crawler is a computer program that automatically searches documents on the Web. Crawlers are primarily programmed for repetitive actions so that browsing is automated. Search engines use crawlers most frequently to browse the internet and build an index. Other crawlers search different types of information such as RSS feeds and email addresses. The term crawler comes from the first search engine on the Internet: the Web Crawler. Synonyms are also “Bot” or “Spider. ” The most well known webcrawler is the Googlebot.
Contents
1 How does a crawler work?
2 Applications
3 Examples of a crawler
4 Crawler vs. Scraper
5 Blocking a crawler
6 Significance for search engine optimization
7 References
8 Web Links
How does a crawler work? [edit]
In principle, a crawler is like a librarian. It looks for information on the Web, which it assigns to certain categories, and then indexes and catalogues it so that the crawled information is retrievable and can be evaluated.
The operations of these computer programs need to be established before a crawl is initiated. Every order is thus defined in advance. The crawler then executes these instructions automatically. An index is created with the results of the crawler, which can be accessed through output software.
The information a crawler will gather from the Web depends on the particular instructions.
This graphic visualize the link relationships that are uncovered by a crawler:
Applications[edit]
The classic goal of a crawler is to create an index. Thus crawlers are the basis for the work of search engines. They first scour the Web for content and then make the results available to users. Focused crawlers, for example, focus on current, content-relevant websites when indexing.
Web crawlers are also used for other purposes:
Price comparison portals search for information on specific products on the Web, so that prices or data can be compared accurately.
In the area of data mining, a crawler may collect publicly available e-mail or postal addresses of companies.
Web analysis tools use crawlers or spiders to collect data for page views, or incoming or outbound links.
Crawlers serve to provide information hubs with data, for example, news sites.
Examples of a crawler[edit]
The most well known crawler is the Googlebot, and there are many additional examples as search engines generally use their own web crawlers. For example
Bingbot
Slurp Bot
DuckDuckBot
Baiduspider
Yandex Bot
Sogou Spider
Exabot
Alexa Crawler[1]
Crawler vs. Scraper[edit]
Unlike a scraper, a crawler only collects and prepares data. Scraping is, however, a black hat technique, which aims to copy data in the form of content from other sites to place it that way or a slightly modified form of it on one’s own website. While a crawler mostly deals with metadata that is not visible to the user at first glance, a scraper extracts tangible content.
Blocking a crawler[edit]
If you don’t want certain crawlers to browse your website, you can exclude their user agent using However, that cannot prevent content from being indexed by search engines. The noindex meta tag or the canonical tag serves better for this purpose.
Significance for search engine optimization[edit]
Webcrawlers like the Googlebot achieve their purpose of ranking websites in the SERP through crawling and indexing. They follow permanent links in the WWW and on websites. Per website, every crawler has a limited timeframe and budget available. Website owners can utilize the crawl budget of the Googlebot more effectively by optimizing the website structure such as the navigation. URLs deemed more important due to a high number of sessions and trustworthy incoming links are usually crawled more often. There are certain measures for controlling crawlers like the Googlebot such as the, which can provide concrete instructions not to crawl certain areas of a website, and the XML sitemap. This is stored in the Google Search Console, and provides a clear overview of the structure of a website, making it clear which areas should be crawled and indexed.
References[edit]
↑ Web Crawlers. Accessed on May 28, 2019
Web Links[edit]
Google Support – Googlebot
JavaScript Crawling with Ryte
Frequently Asked Questions about spider crawler software
What is spider or crawler software?
A web crawler, or spider, is a type of bot that is typically operated by search engines like Google and Bing. Their purpose is to index the content of websites all across the Internet so that those websites can appear in search engine results.
What is a crawler software?
A crawler is a computer program that automatically searches documents on the Web. Crawlers are primarily programmed for repetitive actions so that browsing is automated. Search engines use crawlers most frequently to browse the internet and build an index.
Is crawling legal?
Web data scraping and crawling aren’t illegal by themselves, but it is important to be ethical while doing it. Don’t tread onto other people’s sites without being considerate.Nov 17, 2017
