Proxy For Crawling
How to use proxies for web scraping – Zyte
If you are serious about web scraping you’ll quickly realize that proxy management is a critical component of any web scraping project.
When scraping the web at any reasonable scale, using proxies is an absolute must. However, it is common for managing and troubleshooting proxy issues to consume more time than building and maintaining the spiders themselves.
In this guide, we will cover everything you need to know about proxies for web scraping and how they will make your life easier.
What are proxies and why do you need them when web scraping?
Before we discuss what a proxy is we first need to understand what an IP address is and how they work.
An IP address is a numerical address assigned to every device that connects to an Internet Protocol network like the internet, giving each device a unique identity. Most IP addresses look like this:
207. 148. 1. 212
A proxy is a 3rd party server that enables you to route your request through their servers and use their IP address in the process. When using a proxy, the website you are making the request to no longer sees your IP address but the IP address of the proxy, giving you the ability to scrape the web anonymously if you choose.
Currently, the world is transitioning from IPv4 to a newer standard called IPv6. This newer version will allow for the creation of more IP addresses. However, in the proxy business IPv6 is still not a big thing so most IPs still use the IPv4 standard.
When scraping a website, we recommend that you use a 3rd party proxy and set your company name as the user agent so the website owner can contact you if your scraping is overburdening their servers or if they would like you to stop scraping the data displayed on their website.
There are a number of reasons why proxies are important for data web scraping:
Using a proxy (especially a pool of proxies – more on this later) allows you to crawl a website much more reliably. Significantly reducing the chances that your spider will get banned or a proxy enables you to make your request from a specific geographical region or device (mobile IPs for example) which enables you to see the specific content that the website displays for that given location or device. This is extremely valuable when scraping product data from online a proxy pool allows you to make a higher volume of requests to a target website without being a proxy allows you to get around blanket IP bans some websites impose. Example: it is common for websites to block requests from AWS because there is a track record of some malicious actors overloading websites with large volumes of requests using AWS a proxy enables you to make unlimited concurrent sessions to the same or different websites.
Why use a proxy pool?
Ok, we now know what proxies are, but how do you use them as part of your web scraping?
In a similar way to if we only use our own IP address to scrape a website, if you only use one proxy to scrape a website this will reduce your crawling reliability, geotargeting options, and the number of concurrent requests you can make.
As a result, you need to build a pool of proxies that you can route your requests through. Splitting the amount of traffic over a large number of proxies.
The size of your proxy pool will depend on a number of factors:
The number of requests you will be making per target websites – larger websites with more sophisticated anti-bot countermeasures will require a larger proxy type of IPs you are using as proxies – datacenter, residential or mobile quality of the IPs you are using as proxies – are they public proxies, shared, or private dedicated proxies? Are they datacenter, residential, or mobile IPs? (data center IPs are typically lower quality than residential IPs and mobile IPs, but are often more stable than residential/mobile IPs due to the nature of the network) sophistication of your proxy management system – proxy rotation, throttling, session management, etc.
All five of these factors have a big impact on the effectiveness of your proxy pool. If you don’t properly configure your pool of proxies for your specific web scraping project you can often find that your proxies are being blocked and you’re no longer able to access the target website.
In the next section, we will look at the different types of IPs you can use as proxies.
What are your proxy options?
If you’ve done any level of research into your proxy options you will have probably realized that this can be a confusing topic. Every proxy provider is shouting from the rafters that they have the best website proxy IPs, with very little explanation as to why. Making it very hard to assess which is the best proxy solution for your particular project.
So in this section of the guide, we will break down the key differences between the available proxy solutions and help you decide which solution is best for your needs. First, let’s talk about the fundamentals of proxies – the underlying IPs.
As mentioned already, a proxy is just a 3rd party IP address that you can route your request through. However, there are 3 main types of IPs to choose from. Each type with its own pros and cons.
Datacenter IPs
Datacenter IPs are the most common type of proxy IP. They are the IPs of servers housed in data centers. These IPs are the most commonplace and the cheapest to buy. With the right proxy management solution, you can build a very robust web crawling solution for your business.
Residential IPs
Residential IPs are the IPs of private residences, enabling you to route your request through a residential network. As residential IPs are harder to obtain, they are also much more expensive. In a lot of situations, they are overkill as you could easily achieve the same results with cheaper data center IPs. They also raise legal/consent issues due to the fact you are using a person’s personal network to scrape the web.
Mobile IPs
Mobile IPs are the IPs of private mobile devices. As you can imagine, acquiring the IPs of mobile devices is quite difficult so they are very expensive. For most web scraping projects mobile IPs are overkill unless you want to only scrape the results shown to mobile users. But more significantly they raise even trickier legal/consent issues as oftentimes the device owner isn’t fully aware that you are using their GSM network for web scraping.
Our recommendation is to go with data center IPs and put in place a robust proxy management solution. In the vast majority of cases, this approach will generate the best results for the lowest cost. With proper proxy management, data center IPs give similar results as residential or mobile IPs without legal concerns and at a fraction of the cost.
Public, shared, or dedicated proxies
The other consideration we need to discuss is whether you should use public, shared, or dedicated proxies.
As a general rule, you always stay well clear of public proxies, or “open proxies”. Not only are these proxies of very low quality, but they can also be very dangerous. These proxies are open for anyone to use, so they quickly get used to slam websites with huge amounts of dubious requests. Inevitably resulting in them getting blacklisted and blocked by websites very quickly. What makes them even worse though is that these proxies are often infected with malware and other viruses. As a result, when using a public proxy you run the risk of spreading any malware that is present, infecting your own machines, and even making public your web scraping activities if you haven’t properly configured your security (SSL certs, etc. ).
The decision between shared or dedicated proxies is a bit more intricate. Depending on the size of your project, your need for performance and your budget using a web scraping IP rotation service where you pay for access to a shared pool of IPs might be the right option for you. However, if you have a larger budget and where performance is a high priority for you then paying for a dedicated pool of proxies might be the better option.
Ok, by now you should have a good idea of what proxies are and what are the pros and cons of the different types of IPs you can use in your proxy pool. However, picking the right type of proxy is only part of the battle, the real tricky part is managing your pool of proxies so they don’t get banned.
How to manage your proxy pool
If you are planning on scraping at any reasonable scale, just purchasing a pool of proxies and routing your requests through them likely won’t be sustainable long term. Your proxies will inevitably get banned and stop returning high-quality data.
Here are some of the main challenges that you will face when managing your proxy pool:
Identify Bans – Your proxy solution needs to be able to detect numerous types of bans so that you can troubleshoot and fix the underlying problem – i. e. captchas, redirects, blocks, ghosting, Errors – If your proxies experience any errors, bans, timeouts, etc. they need to be able to retry the request with different – Managing user agents is crucial to having a healthy ntrol Proxies – Some scraping projects require you to keep a session with the same proxy, so you’ll need to configure your proxy pool to allow for Delays – Randomize delays and apply good throttling to help cloak the fact that you are ographical Targeting – Sometimes you’ll need to be able to configure your pool so that only some proxies will be used on certain websites.
Managing a pool of 5-10 proxies is ok, but when you have 100s or 1, 000s it can get messy fast. To overcome these challenges you have three core solutions: Do It Yourself, Proxy Rotators, and Done For You Solutions.
Do it yourself
In this situation, you purchase a pool of shared or dedicated proxies, then build and tweak a proxy management solution yourself to overcome all the challenges you run into. This can be the cheapest option but can be the most wasteful in terms of time and resources. Often it is best to only take this option if you have a dedicated web scraping team who have the bandwidth to manage your proxy pool, or if you have zero budget and can’t afford anything better.
Proxy rotators
The middle-of-the-park solution is to purchase your proxies from a provider that also provides proxy rotation and geographical targeting. In this situation, the solution will take care of the more basic proxy management issues. Leaving you to develop and manage session management, throttling, ban identification logic, etc.
Done for you
The final solution is to completely outsource the management of your proxy management. Solutions such as Zyte Smart Proxy Manager (formerly Crawlera), which is basically a rotating proxy for scraping, are designed as smart downloaders, where your spiders just have to make a request to its API and it will return the data you require. Managing all the proxy rotation, throttling, blacklists, session management, etc. under the hood so you don’t have to.
Each one of these approaches has its own pros and cons, so the best solution will depend on your specific priorities and constraints.
Learn more about rotating proxies for web scraping
Here at Zyte (formerly Scrapinghub), we have been in the web scraping industry for 12 years. We have helped extract web data for more than 1, 000 clients ranging from Government agencies and Fortune 100 companies to early-stage startups and individuals. During this time we gained a tremendous amount of experience and expertise in web data extraction.
Here are some of our best resources if you want to deepen your proxy management knowledge:
Crawlera (Now Zyte Smart Proxy Manager) webinar series: Proxy management done rightHow to scrape the web without getting blockedHow to use Smart Proxy Manager (formerly Crawlera) with ScrapyDeveloper tools that make web scraping a breezeProxy management: Should I build my proxy infrastructure in-house or use an off-the-shelf proxy solution?
1/3 Marketer, 1/3 Ops, 1/3 Techie.
Currently Demand Generation Manager at Zyte.
Data analytics, knowledge graph enthusiast with a particular taste for its applications in financial services, cybersecurity, law enforcement & intelligence sectors.
ProxyCrawl: Anonymous proxy scraping and leading web …
All-In-One data crawling and scraping platform for business internet data at scaleScraping websites content on demandStart crawling and scraping websites in minutes thanks to our tools created to open your doors to internet data awling APIEasy to use crawler API built from developers to developers.
Bypass blocks and captchas and scrape any website without maintaining awling API informationCrawlerFor large scale projects that require large amounts of data delivered to their servers.
Crawler takes care of internet crawling following your needs and awler informationScraper APIGet structured data for your business.
Scraper API to get scraped data directly for your business raper API informationLeads APIAccess trustful company emails for your business.
Leads API crawls the web in real-time and extracts company emails from any API informationCloud StorageMove your crawled and scraped data to the cloud with ProxyCrawl cloud storage designed for Storage informationScreenshots APITake screenshots of websites with an easy to use API
Get screenshots of the entire pages in JPEG format on different devices and screen reenshots API informationSupporting all kind of crawling projectsUsed by the world’s most innovative businesses – big and smallTrusted by more than 32, 000 paying customersWhich crawled more than 1490 billion unique pages anonymouslyStart crawling the web todayTry it free. No credit card required. Instant set-up.
Universal HTTP proxy for web scraping and crawling – Apify
Apify Proxy is a multi-purpose HTTP proxy service that improves data throughput and efficiency of your web scraping and automation bots, and enables access to websites from various service provides access to Apify’s pool of datacenter and residential IP addresses, monitors the health of the IP pool, and smartly rotates addresses to ensure stable and reliable nefitsCombines datacenter and residential IPsApify Proxy provides access to both residential and datacenter IP addresses. Datacenter IPs are fast and cheap, but might be blocked by target websites. Residential IPs are more expensive and harder to the right balance between performance and cost for your web crawling telligent proxy rotationApify Proxy uses machine learning to rotate and select optimal IP address for the specific target website. Dead or burned IPs are automatically removed from the pool to reduce stable long-term performance for your web crawling bots with little SERPsApify Proxy lets you download and extract data from Google Search engine result pages (SERPs), including Google Shopping. Select country and language to get localized with a low upfront commitment, and then pay as you go as you scale. Try Google Search scraper actor to quickly get started. FeaturesHTTPS supportSecurely access websites protected with SSL/TLS encryption without installing self-signed monitoringApify Proxy periodically checks that all the IP addresses are working on selected target websites to reduce error endpointAccess Apify Proxy on a single hostname, which makes it easy to use from any HTTP proxy-enabled affic statisticsEasily track which domains and websites were accessed by the proxy and how much data was geolocationSelect arbitrary country for residential IP addresses, in order to obtain country-specific versions of target IP sessionsRetain the same IP address for long periods of time, for example to perform operations on a website after Proxy pricingShared datacenter IPsDedicated datacenter IPsResidential IPsSERPsDatacenter IP addresses that are shared with other users. They represent the most cost-effective option for many target websites, although there is a chance the IPs might be blocked due to activity of other users. Note that there is an additional charge for data datacenter IPs are included in paid plans by default. You can increase this number by upgrading your expected monthly data transferUp to 100 IPsUp to 500 IPsUp to 1, 000 IPsAbove 1, 000 IPsAny questions? Please contact to start using Apify Proxy?
Frequently Asked Questions about proxy for crawling
Why is proxy used in crawling?
Use a proxy server Using an intermediary between your device and the target website reduces IP address blocks, ensures anonymity, and allows you to access websites that might be unavailable in your region.Sep 16, 2021
What is proxy crawl?
What is Proxy Crawl? It is a top web scraping tool for developers. Get data for SEO or data mining projects without worrying about worldwide proxies. Scrape Amazon, FB, Yahoo, and thousands of websites. Proxy Crawl is a tool in the Web Scraping API category of a tech stack.
Is proxy scraping legal?
The courts determined that scraping public data is legal. As long as the data is available on the public domain and it is not copyright protected then it can be legally scraped. The data scraped should, however, be used within the confines of the law. Web scraped data has limitations in commercial applications.Aug 26, 2020
