Ip Rotation Python
How To Rotate Proxies and change IP Addresses using …
A common problem faced by web scrapers is getting blocked by websites while scraping them. There are many techniques to prevent getting blocked, like
Rotating IP addresses
Using Proxies
Rotating and Spoofing user agents
Using headless browsers
Reducing the crawling rate
What is a rotating proxy?
A rotating proxy is a proxy server that assigns a new IP address from the proxy pool for every connection. That means you can launch a script to send 1, 000 requests to any number of sites and get 1, 000 different IP addresses. Using proxies and rotating IP addresses in combination with rotating user agents can help you get scrapers past most of the anti-scraping measures and prevent being detected as a scraper.
The concept of rotating IP addresses while scraping is simple – you can make it look to the website that you are not a single ‘bot’ or a person accessing the website, but multiple ‘real’ users accessing the website from multiple locations. If you do it right, the chances of getting blocked are minimal.
In this blog post, we will show you how to send your requests to a website using a proxy, and then we’ll show you how to send these requests through multiple IP addresses or proxies.
How to send requests through a Proxy in Python 3 using Requests
If you are using Python-Requests, you can send requests through a proxy by configuring the proxies argument. For example
import requests
proxies = {
”: ”,
”: ”, }
(”, proxies=proxies)
We’ll show how to send a real request through a free proxy.
Let’s find a proxy
There are many websites dedicated to providing free proxies on the internet. One such site is. Let’s go there and pick a proxy that supports (as we are going to test this on an website).
Here is our proxy –
IP: 207. 148. 1. 212 Port: 8080
Note:
This proxy might not work when you test it. You should pick another proxy from the website if it doesn’t work.
Now let’s make a request to HTTPBin’s IP endpoint and test if the request went through the proxy
url = ”
“”: ”,
“”: ”}
response = (url, proxies=proxies)
print(())
{‘origin’: ‘209. 50. 52. 162’}
You can see that the request went through the proxy. Let’s get to sending requests through a pool of IP addresses.
Rotating Requests through a pool of Proxies in Python 3
We’ll gather a list of some active proxies from. You can also use private proxies if you have access to them.
You can make this list by manually copy and pasting, or automate this by using a scraper (If you don’t want to go through the hassle of copy and pasting every time the proxies you have gets removed). You can write a script to grab all the proxies you need and construct this list dynamically every time you initialize your web scraper. Once you have the list of Proxy IPs to rotate, the rest is easy.
We have written some code to pick up IPs automatically by scraping. (This code could change when the website updates its structure)
from import fromstring
def get_proxies():
response = (url)
parser = fromstring()
proxies = set()
for i in (‘//tbody/tr’)[:10]:
if (‘. //td[7][contains(text(), “yes”)]’):
#Grabbing IP and corresponding PORT
proxy = “:”([(‘. //td[1]/text()’)[0], (‘. //td[2]/text()’)[0]])
(proxy)
return proxies
The function get_proxies will return a set of proxy strings that can be passed to the request object as proxy config.
proxies = get_proxies()
print(proxies)
{‘121. 129. 127. 209:80’, ‘124. 41. 215. 238:45169’, ‘185. 93. 3. 123:8080’, ‘194. 182. 64. 67:3128’, ‘106. 0. 38. 174:8080’, ‘163. 172. 175. 210:3128′, ’13. 92. 196. 150:8080’}
Now that we have the list of Proxy IP Addresses in a variable proxies, we’ll go ahead and rotate it using a Round Robin method.
from itertools import cycle
import traceback
#If you are copy pasting proxy ips, put in the list below
#proxies = [‘121. 150:8080’]
proxy_pool = cycle(proxies)
for i in range(1, 11):
#Get a proxy from the pool
proxy = next(proxy_pool)
print(“Request #%d”%i)
try:
response = (url, proxies={“”: proxy, “”: proxy})
except:
#Most free proxies will often get connection errors. You will have retry the entire request using another proxy to work.
#We will just skip retries as its beyond the scope of this tutorial and we are only downloading a single url
print(“Skipping. Connnection error”)
Request #1
{‘origin’: ‘121. 209’}
Request #2
{‘origin’: ‘124. 238’}
Request #3
{‘origin’: ‘185. 123’}
Request #4
{‘origin’: ‘194. 67’}
Request #5
Skipping. Connnection error
Request #6
{‘origin’: ‘163. 210’}
Request #7
{‘origin’: ’13. 150′}
Request #8
Request #9
Request #10
Okay – it worked. Request #5 had a connection error probably because the free proxy we grabbed was overloaded with users trying to get their proxy traffic through. Below is the full code to do this.
Full Code
Rotating Proxies in Scrapy
Scrapy does not have built in proxy rotation. There are many middlewares in scrapy for rotating proxies or ip address in scrapy. We have found scrapy-rotating-proxies to be the most useful among them.
Install scrapy-rotating-proxies using
pip install scrapy-rotating-proxies
In your scrapy project’s add,
DOWNLOADER_MIDDLEWARES = {
‘tatingProxyMiddleware’: 610, }
ROTATING_PROXY_LIST = [
”,
#… ]
As an alternative to ROTATING_PROXY_LIST, you can specify a ROTATING_PROXY_LIST_PATH options with a path to a file with proxies, one per line:
ROTATING_PROXY_LIST_PATH = ‘/my/path/’
You can read more about this middleware on its github repo.
5 Things to keep in mind while using proxies and rotating IP addresses
Here are a few tips that you should remember:
Do not rotate IP Address when scraping websites after logging in or using Sessions
We don’t recommend rotating IPs if you are logging into a website. The website already knows who you are when you log in, through the session cookies it sets. To maintain the logged-in state, you need to keep passing the Session ID in your cookie headers. The servers can easily tell that you are bot when the same session cookie is coming from multiple IP addresses and block you.
A similar logic applies if you are sending back that session cookie to a website. The website already knows this session is using a certain IP and a User-Agent. Rotating these two fields would do you more harm than good in these cases.
In these situations, it’s better just to use a single IP address and maintain the same request headers for each unique login.
Avoid Using Proxy IP addresses that are in a sequence
Even the simplest anti-scraping plugins can detect that you are a scraper if the requests come from IP addresses that are continuous or belong to the same range like this:
64. 233. 160. 0
64. 1
64. 2
64. 3
Some websites have gone as far as blocking the entire providers like AWS and have even blocked entire countries.
If you are using free proxies – automate
Free proxies tend to die out soon, mostly in days or hours and would expire before the scraping even completes. To prevent that from disrupting your scrapers, write some code that would automatically pick up and refresh the proxy list you use for scraping with working IP addresses. This will save you a lot of time and frustration.
Use Elite Proxies whenever possible if you are using Free Proxies ( or even if you are paying for proxies)
All proxies aren’t the same. There are mainly three types of proxies available in the internet.
Transparent Proxy – A transparent proxy is a server that sits between your computer and the internet and redirects your requests and responses without modifying them. It sends your real IP address in the HTTP_X_FORWARDED_FOR header, this means a website that does not only determine your REMOTE_ADDR but also checks for specific proxy headers that will still know your real IP address. The HTTP_VIA header is also sent, revealing that you are using a proxy server.
Anonymous Proxy – An anonymous proxy does not send your real IP address in the HTTP_X_FORWARDED_FOR header, instead, it submits the IP address of the proxy or it’ll just be blank. The HTTP_VIA header is sent with a transparent proxy, which would reveal you are using a proxy server. An anonymous proxy server does not tell websites your real IP address anymore. This can be helpful to just keep your privacy on the internet. The website can still see you are using a proxy server, but in the end, it does not really matter as long as the proxy server does not disclose your real IP address. If someone really wants to restrict page access, an anonymous proxy server will be detected and blocked.
Elite Proxy – An elite proxy only sends REMOTE_ADDR header while the other headers are empty. It will make you seem like a regular internet user who is not using a proxy at all. An elite proxy server is ideal to pass any restrictions on the internet and to protect your privacy to the fullest extent. You will seem like a regular internet user who lives in the country that your proxy server is running in.
Elite Proxies are your best option as they are hard to be detected. Use anonymous proxies if it’s just to keep your privacy on the internet. Lastly, use transparent proxies – although the chances of success are very low.
Get Premium Proxies if you are Scraping Thousands of Pages
Free proxies available on the internet are always abused and end up being in blacklists used by anti-scraping tools and web servers. If you are doing serious large-scale data extraction, you should pay for some good proxies. There are many providers who would even rotate the IPs for you.
Use IP Rotation in combination with Rotating User Agents
IP rotation on its own can help you get past some anti-scraping measures. If you find yourself being banned even after using rotating proxies, a good solution is adding header spoofing and rotation.
That’s all we’ve got to say. Happy Scraping
Having problems collecting the data you need? We can help
Are your periodic data extraction jobs interrupted due to website blocking or other IT infrastructural issues? Using ScrapeHero’s data extraction service will make it hassle-free for you.
How to Use Proxies to Rotate IP Addresses in Python
·
7 min read
· Updated
sep 2021
· Web Scraping
· Sponsored
A proxy is a server application that acts as an intermediary for requests between a client and the server from which the client is requesting a certain service (HTTP, SSL, etc. ).
When using a proxy server, instead of directly connecting to the target server and requesting whatever that is you wanna request, you direct the request to the proxy server which evaluates the request and performs it, and returns the response, here is a simple Wikipedia demonstration of proxy servers:
Web scraping experts often use more than one proxy to prevent websites to ban their IP addresses. Proxies have several other benefits, including bypassing filters and censorship, hiding your real IP address, etc.
In this tutorial, you will learn how you can use proxies in Python using requests library, we will be also using stem library which is a Python controller library for Tor, let’s install them:
pip3 install bs4 requests stem
Related: How to Make a Subdomain Scanner in Python.
Using Free Available Proxies
First, there are some websites that offer free proxy list to use, I have built a function to automatically grab this list:
import requests
import random
from bs4 import BeautifulSoup as bs
def get_free_proxies():
url = ”
# get the HTTP response and construct soup object
soup = bs((url). content, “”)
proxies = []
for row in (“table”, attrs={“id”: “proxylisttable”}). find_all(“tr”)[1:]:
tds = nd_all(“td”)
try:
ip = tds[0]()
port = tds[1]()
host = f”{ip}:{port}”
(host)
except IndexError:
continue
return proxies
However, when I tried to use them, most of them were timing out, I filtered some working ones:
proxies = [
‘167. 172. 248. 53:3128’,
‘194. 226. 34. 132:5555’,
‘203. 202. 245. 62:80’,
‘141. 0. 70. 211:8080’,
‘118. 69. 50. 155:80’,
‘201. 55. 164. 177:3128′,
’51. 15. 166. 107:3128′,
’91. 205. 218. 64:80’,
‘128. 199. 237. 57:8080’, ]
This list may not be viable forever, in fact, most of these will stop working when you read this tutorial (so you should execute the above function each time you want to use fresh proxy servers).
The below function accepts a list of proxies and creates a requests session that randomly selects one of the proxies passed:
def get_session(proxies):
# construct an HTTP session
session = ssion()
# choose one random proxy
proxy = (proxies)
oxies = {“”: proxy, “”: proxy}
return session
Let’s test this by making a request to a website that returns our IP address:
for i in range(5):
s = get_session(proxies)
print(“Request page with IP:”, (“, timeout=1. 5)())
except Exception as e:
Here is my output:
Request page with IP: 45. 64. 134. 198
Request page with IP: 141. 211
Request page with IP: 94. 250. 230
Request page with IP: 46. 173. 219. 2
Request page with IP: 201. 177
As you can see, these are some IP addresses of the working proxy servers and not our real IP address (try to visit this website in your browser and you’ll see your real IP address).
Free proxies tend to die very quickly, mostly in days or even hours, and would often die before our scraping project ends. To prevent that, you need to use premium proxies for large-scale data extraction projects, there are many providers out there who rotate IP addresses for you. One of the well-known solutions is Crawlera. We will talk more about it in the last section of this tutorial.
Using Tor as a Proxy
You can also use the Tor network to rotate IP addresses:
from ntrol import Controller
from stem import Signal
def get_tor_session():
# initialize a requests Session
# setting the proxy of both & to the localhost:9050
# this requires a running Tor service in your machine and listening on port 9050 (by default)
oxies = {“”: “socks5localhost:9050”, “”: “socks5localhost:9050”}
def renew_connection():
with om_port(port=9051) as c:
thenticate()
# send NEWNYM signal to establish a new clean connection through the Tor network
()
if __name__ == “__main__”:
s = get_tor_session()
ip = (“)
print(“IP:”, ip)
renew_connection()
Note: The above code should work only if you have Tor installed in your machine (head to this link to properly install it) and well configured (ControlPort 9051 is enabled, check this stackoverflow answer for further details).
This will create a session with a Tor IP address and make an HTTP request, and then renew the connection by sending NEWNYM signal (which tells Tor to establish a new clean connection) to change the IP address and make another request, here is the output:
IP: 185. 220. 101. 49
IP: 109. 100. 21
Great! However, when you experience web scraping using the Tor network, you’ll soon realize it’s pretty slow most of the time, that is why the recommended way is below.
Using Crawlera
Scrapinghub’s Crawlera allows you to crawl quickly and reliably, it manages and rotates proxies internally, so if you’re banned, it will automatically detect that and rotates the IP address for you.
Crawlera is a smart proxy network, specifically designed for web scraping and crawling. Its job is clear: making your life easier as a web scraper. It helps you get successful requests and extract data at scale from any website using any web scraping tool.
With its simple API, the request you make when scraping will be routed through a pool of high-quality proxies. When necessary, it automatically introduces delays between requests and removes/adds IP addresses to overcome different crawling challenges.
Here is how you can use Crawlera with requests library in Python:
proxy_host = “”
proxy_port = “8010”
proxy_auth = “
proxies = {
“”: f”{proxy_auth}@{proxy_host}:{proxy_port}/”,
“”: f”{proxy_auth}@{proxy_host}:{proxy_port}/”}
r = (url, proxies=proxies, verify=False)
Once you register for a plan, you’ll be provided with an API key in which you’ll replace proxy_auth.
So, here is what Crawlera does for you:
You send the HTTP request using its single endpoint API.
It automatically selects, rotates, throttles, and blacklists IPs to retrieve the target data.
It handles request headers and maintains sessions.
You receive a successful request in response.
Conclusion
There are several proxy types including transparent proxies, anonymous proxies, elite proxies. If your goal of using proxies is to prevent websites from banning your scrapers, then elite proxies are your optimal choice, it will make you seem like a regular internet user who is not using a proxy at all.
Furthermore, an extra anti-scraping measure is using rotating user agents, in which you send a changing spoofed header each time saying that you’re a regular browser.
Finally, Crawlera saves your time and energy by automatically managing proxies for you, it also provides a 14-day free trial, so you can just try it out without any risk. If you need a proxy solution, I highly suggest you should try Crawlera.
Want to Learn More about Web Scraping?
Finally, if you want to dig more into web scraping with different Python libraries, not just BeautifulSoup, the below courses will definitely be valuable for you:
Modern Web Scraping with Python using Scrapy Splash Selenium.
Web Scraping and API Fundamentals in Python 2021.
Learn also: How to Extract All Website Links in Python.
Happy Coding ♥
View Full Code
Read Also
Comment panel
Scraping in Python – Preventing IP ban – Stack Overflow
I am using Python to scrape pages. Until now I didn’t have any complicated issues.
The site that I’m trying to scrape uses a lot of security checks and have some mechanism to prevent scraping.
Using Requests and lxml I was able to scrape about 100-150 pages before getting banned by IP. Sometimes I even get ban on first request (new IP, not used before, different C block). I have tried with spoofing headers, randomize time between requests, still the same.
I have tried with Selenium and I got much better results. With Selenium I was able to scrape about 600-650 pages before getting banned. Here I have also tried to randomize requests (between 3-5 seconds, and make (300) call on every 300th request). Despite that, Im getting banned.
From here I can conclude that site have some mechanism where they ban IP if it requested more than X pages in one open browser session or something like that.
Based on your experience what else should I try?
Will closing and opening browser in Selenium help (for example after every 100th requests close and open browser). I was thinking about trying with proxies but there are about million of pages and it will be very expansive.
theAlse5, 30711 gold badges62 silver badges105 bronze badges
asked Feb 1 ’16 at 14:36
I had this problem too. I used urllib with tor in python3.
download and install tor browser
testing tor
open terminal and type:
curl –socks5-hostname localhost:9050 <>
if you see result it’s worked.
Now we should test in python. Now run this code
import socks
import socket
from quest import Request, urlopen
from bs4 import BeautifulSoup
#set socks5 proxy to use tor
t_default_proxy(CKS5, “localhost”, 9050)
= cksocket
req = Request(”, headers={‘User-Agent’: ‘Mozilla/5. 0’, })
html = urlopen(req)()
soup = BeautifulSoup(html, ”)
print(soup(‘title’)[0]. get_text())
if you see
Congratulations. This browser is configured to use Tor.
it worked in python too and this means you are using tor for web scraping.
Gruber1, 8494 gold badges24 silver badges42 bronze badges
answered Mar 2 ’18 at 8:02
3
You could use proxies.
You can buy several hundred IPs for very cheap, and use selenium as you previously have done.
Furthermore I suggest varying the browser your use and other user-agent parameters.
You could iterate over using a single IP address to load only x number of pages and stopping prior to getting banned.
def load_proxy(PROXY_HOST, PROXY_PORT):
fp = refoxProfile()
t_preference(“”, 1)
t_preference(“”, PROXY_HOST)
t_preference(“”, int(PROXY_PORT))
t_preference(“eragent. override”, “whater_useragent”)
fp. update_preferences()
return refox(firefox_profile=fp)
Rahul9, 1334 gold badges44 silver badges81 bronze badges
answered Feb 1 ’16 at 14:48
ParsaParsa2, 5752 gold badges16 silver badges30 bronze badges
5
Not the answer you’re looking for? Browse other questions tagged python selenium web-scraping screen-scraping or ask your own question.
