ProxyBoys

Prevent Web Scraping

• January 12, 2022
• 0

Prevent Web Scraping – A Step by Step Guide – DataDome

Who uses web scraper bots, and why?
Your content is gold, and it’s the reason visitors come to your website. Threat actors also want your gold, and use scraper bot attacks to gather and exploit your web content—to republish content with no overhead, or to undercut your prices automatically, for example.
Online retailers often hire professional web scrapers or use web scraping tools to gather competitive intelligence to craft future retail pricing strategies and product catalogs.
Threat actors try their best to disguise their bad web scraping bots as good ones, such as the ubiquitous Googlebots. DataDome identifies over 1 million hits per day from fake Googlebots on all customer websites.
Read more: TheFork (TripAdvisor) blocks scraping on its applications
The anatomy of a scraping attack
Scraping attacks contain three main phases:
Target URL address and parameter values: Web scrapers identify their targets and make preparations to limit scraping attack detection by creating fake user accounts, masking their malicious scraper bots as good ones, obfuscating their source IP addresses, and more.
Run scraping tools & processes: The army of scraper bots run on the target website, mobile app or API. The often intense level of bot traffic will often overload servers and result in poor website performance or even downtime.
Extract content and data: Web scrapers extract proprietary content and database records from the target and store it in their database for later analysis and abuse.
Figure 1: OAT-011 indicative diagram. Source: OWASP.
Common protection strategies against web scraping
Common anti crawler protection strategies include:
Monitoring new or existing user accounts with high levels of activity and no purchases
Detecting abnormally high volumes of product views as a sign of non-human activity
Tracking the activity of competitors for signs of price and product catalog matching
Enforcing site terms and conditions that stop malicious web scraping
Employing bot protection capabilities with deep behavioral analysis to pinpoint bad bots and prevent web scraping
Site owners commonly use “” files to communicate their intentions when it comes to scraping. files permit scraping bots to traverse specific pages; however, malicious bots don’t care about files (which serve as a “no trespassing” sign).
A clear, binding terms of use agreement that dictates permitted and non-permitted activity can potentially help in litigation. Check out our terms and conditions template for precise, enforceable anti-scraping wording.
Scrapers will do everything in their power to disguise scraping bots as genuine users. The ability to scrape publicly available content, register fake user accounts for malicious bots, and pass valid HTTP requests from randomly generated device IDs and IP addresses, deems traditional rule-based security measures, such as WAFs, ineffective against sophisticated scraping attacks.
How DataDome protects against website and content scraping
A good bot detection solution will be able to identify visitor behavior that shows signs of web scraping in real time, and automatically block malicious bots before scraping attacks unravel while maintaining a smooth experience for real human users. To correctly identify fraudulent traffic and block web scraping tools, a bot protection solution must be able to analyze both technical and behavioral data.
“Bots were scraping our website in order to steal our content and then sell it to third parties. Since we’ve activated the [DataDome bot] protection, web scraper bots are blocked and cannot access the website. Our data are secured and no longer accessible to bots. We are also now able to monitor technical logs in order to detect abnormal behaviors such as aggressive IP addresses or unusual queries. ”
Head of Technical Dept., Enterprise (1001-5000 employees)
DataDome employs a two-layer bot detection engine to help CTOs and CISOs protect their websites, mobile apps, and APIs from malicious scraping bots & block web scraping tools. It compares every site hit with a massive in-memory pattern database, and uses a blend of AI and machine learning to decide in less than 2 milliseconds whether to grant access to your pages or not.
DataDome is the only bot protection solution delivered as-a-service. It deploys in minutes on any web architecture, is unmatched in brute force attack detection speed and accuracy, and runs on autopilot. You will receive real-time notifications whenever your site is under scraping attack, but no intervention is required. Once you have set up a whitelist of trusted partner bots, DataDome will take care of all unwanted traffic and stop malicious bots from crawling your site in order to prevent website scraping.
Want to see is scraper bots are on your site? You can test your site today. (It’s easy & free. )

How do I prevent site scraping? [closed] – Stack Overflow

Note: Since the complete version of this answer exceeds Stack Overflow’s length limit, you’ll need to head to GitHub to read the extended version, with more tips and details.
In order to hinder scraping (also known as Webscraping, Screenscraping, Web data mining, Web harvesting, or Web data extraction), it helps to know how these scrapers work, and, by extension, what prevents them from working well.
There’s various types of scraper, and each works differently:
Spiders, such as Google’s bot or website copiers like HTtrack, which recursively follow links to other pages in order to get data. These are sometimes used for targeted scraping to get specific data, often in combination with a HTML parser to extract the desired data from each page.
Shell scripts: Sometimes, common Unix tools are used for scraping: Wget or Curl to download pages, and Grep (Regex) to extract the data.
HTML parsers, such as ones based on Jsoup, Scrapy, and others. Similar to shell-script regex based ones, these work by extracting data from pages based on patterns in HTML, usually ignoring everything else.
For example: If your website has a search feature, such a scraper might submit a request for a search, and then get all the result links and their titles from the results page HTML, in order to specifically get only search result links and their titles. These are the most common.
Screenscrapers, based on eg. Selenium or PhantomJS, which open your website in a real browser, run JavaScript, AJAX, and so on, and then get the desired text from the webpage, usually by:
Getting the HTML from the browser after your page has been loaded and JavaScript has run, and then using a HTML parser to extract the desired data. These are the most common, and so many of the methods for breaking HTML parsers / scrapers also work here.
Taking a screenshot of the rendered pages, and then using OCR to extract the desired text from the screenshot. These are rare, and only dedicated scrapers who really want your data will set this up.
Webscraping services such as ScrapingHub or Kimono. In fact, there’s people whose job is to figure out how to scrape your site and pull out the content for others to use.
Unsurprisingly, professional scraping services are the hardest to deter, but if you make it hard and time-consuming to figure out how to scrape your site, these (and people who pay them to do so) may not be bothered to scrape your website.
Embedding your website in other site’s pages with frames, and embedding your site in mobile apps.
While not technically scraping, mobile apps (Android and iOS) can embed websites, and inject custom CSS and JavaScript, thus completely changing the appearance of your pages.
Human copy – paste: People will copy and paste your content in order to use it elsewhere.
There is a lot overlap between these different kinds of scraper, and many scrapers will behave similarly, even if they use different technologies and methods.
These tips mostly my own ideas, various difficulties that I’ve encountered while writing scrapers, as well as bits of information and ideas from around the interwebs.
How to stop scraping
You can’t completely prevent it, since whatever you do, determined scrapers can still figure out how to scrape. However, you can stop a lot of scraping by doing a few things:
Monitor your logs & traffic patterns; limit access if you see unusual activity:
Check your logs regularly, and in case of unusual activity indicative of automated access (scrapers), such as many similar actions from the same IP address, you can block or limit access.
Specifically, some ideas:
Rate limiting:
Only allow users (and scrapers) to perform a limited number of actions in a certain time – for example, only allow a few searches per second from any specific IP address or user. This will slow down scrapers, and make them ineffective. You could also show a captcha if actions are completed too fast or faster than a real user would.
Detect unusual activity:
If you see unusual activity, such as many similar requests from a specific IP address, someone looking at an excessive number of pages or performing an unusual number of searches, you can prevent access, or show a captcha for subsequent requests.
Don’t just monitor & rate limit by IP address – use other indicators too:
If you do block or rate limit, don’t just do it on a per-IP address basis; you can use other indicators and methods to identify specific users or scrapers. Some indicators which can help you identify specific users / scrapers include:
How fast users fill out forms, and where on a button they click;
You can gather a lot of information with JavaScript, such as screen size / resolution, timezone, installed fonts, etc; you can use this to identify users.
HTTP headers and their order, especially User-Agent.
As an example, if you get many request from a single IP address, all using the same User Agent, screen size (determined with JavaScript), and the user (scraper in this case) always clicks on the button in the same way and at regular intervals, it’s probably a screen scraper; and you can temporarily block similar requests (eg. block all requests with that user agent and screen size coming from that particular IP address), and this way you won’t inconvenience real users on that IP address, eg. in case of a shared internet connection.
You can also take this further, as you can identify similar requests, even if they come from different IP addresses, indicative of distributed scraping (a scraper using a botnet or a network of proxies). If you get a lot of otherwise identical requests, but they come from different IP addresses, you can block. Again, be aware of not inadvertently blocking real users.
This can be effective against screenscrapers which run JavaScript, as you can get a lot of information from them.
Related questions on Security Stack Exchange:
How to uniquely identify users with the same external IP address? for more details, and
Why do people use IP address bans when IP addresses often change? for info on the limits of these methods.
Instead of temporarily blocking access, use a Captcha:
The simple way to implement rate-limiting would be to temporarily block access for a certain amount of time, however using a Captcha may be better, see the section on Captchas further down.
Require registration & login
Require account creation in order to view your content, if this is feasible for your site. This is a good deterrent for scrapers, but is also a good deterrent for real users.
If you require account creation and login, you can accurately track user and scraper actions. This way, you can easily detect when a specific account is being used for scraping, and ban it. Things like rate limiting or detecting abuse (such as a huge number of searches in a short time) become easier, as you can identify specific scrapers instead of just IP addresses.
In order to avoid scripts creating many accounts, you should:
Require an email address for registration, and verify that email address by sending a link that must be opened in order to activate the account. Allow only one account per email address.
Require a captcha to be solved during registration / account creation.
Requiring account creation to view content will drive users and search engines away; if you require account creation in order to view an article, users will go elsewhere.
Block access from cloud hosting and scraping service IP addresses
Sometimes, scrapers will be run from web hosting services, such as Amazon Web Services or GAE, or VPSes. Limit access to your website (or show a captcha) for requests originating from the IP addresses used by such cloud hosting services.
Similarly, you can also limit access from IP addresses used by proxy or VPN providers, as scrapers may use such proxy servers to avoid many requests being detected.
Beware that by blocking access from proxy servers and VPNs, you will negatively affect real users.
Make your error message nondescript if you do block
If you do block / limit access, you should ensure that you don’t tell the scraper what caused the block, thereby giving them clues as to how to fix their scraper. So a bad idea would be to show error pages with text like:
Too many requests from your IP address, please try again later.
Error, User Agent header not present!
Instead, show a friendly error message that doesn’t tell the scraper what caused it. Something like this is much better:
Sorry, something went wrong. You can contact support via, should the problem persist.
This is also a lot more user friendly for real users, should they ever see such an error page. You should also consider showing a captcha for subsequent requests instead of a hard block, in case a real user sees the error message, so that you don’t block and thus cause legitimate users to contact you.
Use Captchas if you suspect that your website is being accessed by a scraper.
Captchas (“Completely Automated Test to Tell Computers and Humans apart”) are very effective against stopping scrapers. Unfortunately, they are also very effective at irritating users.
As such, they are useful when you suspect a possible scraper, and want to stop the scraping, without also blocking access in case it isn’t a scraper but a real user. You might want to consider showing a captcha before allowing access to the content if you suspect a scraper.
Things to be aware of when using Captchas:
Don’t roll your own, use something like Google’s reCaptcha: It’s a lot easier than implementing a captcha yourself, it’s more user-friendly than some blurry and warped text solution you might come up with yourself (users often only need to tick a box), and it’s also a lot harder for a scripter to solve than a simple image served from your site
Don’t include the solution to the captcha in the HTML markup: I’ve actually seen one website which had the solution for the captcha in the page itself, (although quite well hidden) thus making it pretty useless. Don’t do something like this. Again, use a service like reCaptcha, and you won’t have this kind of problem (if you use it properly).
Captchas can be solved in bulk: There are captcha-solving services where actual, low-paid, humans solve captchas in bulk. Again, using reCaptcha is a good idea here, as they have protections (such as the relatively short time the user has in order to solve the captcha). This kind of service is unlikely to be used unless your data is really valuable.
Serve your text content as an image
You can render text into an image server-side, and serve that to be displayed, which will hinder simple scrapers extracting text.
However, this is bad for screen readers, search engines, performance, and pretty much everything else. It’s also illegal in some places (due to accessibility, eg. the Americans with Disabilities Act), and it’s also easy to circumvent with some OCR, so don’t do it.
You can do something similar with CSS sprites, but that suffers from the same problems.
Don’t expose your complete dataset:
If feasible, don’t provide a way for a script / bot to get all of your dataset. As an example: You have a news site, with lots of individual articles. You could make those articles be only accessible by searching for them via the on site search, and, if you don’t have a list of all the articles on the site and their URLs anywhere, those articles will be only accessible by using the search feature. This means that a script wanting to get all the articles off your site will have to do searches for all possible phrases which may appear in your articles in order to find them all, which will be time-consuming, horribly inefficient, and will hopefully make the scraper give up.
This will be ineffective if:
The bot / script does not want / need the full dataset anyway.
Your articles are served from a URL which looks something like This (and similar things) which will allow scrapers to simply iterate over all the articleIds and request all the articles that way.
There are other ways to eventually find all the articles, such as by writing a script to follow links within articles which lead to other articles.
Searching for something like “and” or “the” can reveal almost everything, so that is something to be aware of. (You can avoid this by only returning the top 10 or 20 results).
You need search engines to find your content.
Don’t expose your APIs, endpoints, and similar things:
Make sure you don’t expose any APIs, even unintentionally. For example, if you are using AJAX or network requests from within Adobe Flash or Java Applets (God forbid! ) to load your data it is trivial to look at the network requests from the page and figure out where those requests are going to, and then reverse engineer and use those endpoints in a scraper program. Make sure you obfuscate your endpoints and make them hard for others to use, as described.
To deter HTML parsers and scrapers:
Since HTML parsers work by extracting content from pages based on identifiable patterns in the HTML, we can intentionally change those patterns in oder to break these scrapers, or even screw with them. Most of these tips also apply to other scrapers like spiders and screenscrapers too.
Frequently change your HTML
Scrapers which process HTML directly do so by extracting contents from specific, identifiable parts of your HTML page. For example: If all pages on your website have a div with an id of article-content, which contains the text of the article, then it is trivial to write a script to visit all the article pages on your site, and extract the content text of the article-content div on each article page, and voilà, the scraper has all the articles from your site in a format that can be reused elsewhere.
If you change the HTML and the structure of your pages frequently, such scrapers will no longer work.
You can frequently change the id’s and classes of elements in your HTML, perhaps even automatically. So, if your ticle-content becomes something like div. a4c36dda13eaf0, and changes every week, the scraper will work fine initially, but will break after a week. Make sure to change the length of your ids / classes too, otherwise the scraper will use div. [any-14-characters] to find the desired div instead. Beware of other similar holes too..
If there is no way to find the desired content from the markup, the scraper will do so from the way the HTML is structured. So, if all your article pages are similar in that every div inside a div which comes after a h1 is the article content, scrapers will get the article content based on that. Again, to break this, you can add / remove extra markup to your HTML, periodically and randomly, eg. adding extra divs or spans. With modern server side HTML processing, this should not be too hard.
Things to be aware of:
It will be tedious and difficult to implement, maintain, and debug.
You will hinder caching. Especially if you change ids or classes of your HTML elements, this will require corresponding changes in your CSS and JavaScript files, which means that every time you change them, they will have to be re-downloaded by the browser. This will result in longer page load times for repeat visitors, and increased server load. If you only change it once a week, it will not be a big problem.
Clever scrapers will still be able to get your content by inferring where the actual content is, eg. by knowing that a large single block of text on the page is likely to be the actual article. This makes it possible to still find & extract the desired data from the page. Boilerpipe does exactly this.
Essentially, make sure that it is not easy for a script to find the actual, desired content for every similar page.
See also How to prevent crawlers depending on XPath from getting page contents for details on how this can be implemented in PHP.
Change your HTML based on the user’s location
This is sort of similar to the previous tip. If you serve different HTML based on your user’s location / country (determined by IP address), this may break scrapers which are delivered to users. For example, if someone is writing a mobile app which scrapes data from your site, it will work fine initially, but break when it’s actually distributed to users, as those users may be in a different country, and thus get different HTML, which the embedded scraper was not designed to consume.
Frequently change your HTML, actively screw with the scrapers by doing so!
An example: You have a search feature on your website, located at, which returns the following HTML:

(And so on, lots more identically structured divs with search results)
As you may have guessed this is easy to scrape: all a scraper needs to do is hit the search URL with a query, and extract the desired data from the returned HTML. In addition to periodically changing the HTML as described above, you could also leave the old markup with the old ids and classes in, hide it with CSS, and fill it with fake data, thereby poisoning the scraper. Here’s how the search results page could be changed: