Python Requests Timeout Example
Timeout for python requests.get entire response – Stack Overflow
I’m gathering statistics on a list of websites and I’m using requests for it for simplicity. Here is my code:
data=[]
websites=[”, ”]
for w in websites:
r= (w, verify=False)
( (, len(ntent), tal_seconds(), str([(atus_code, ) for l in r. history]), str(r. ()), str(r. ())))
Now, I want to timeout after 10 seconds so the loop doesn’t get stuck.
This question has been of interest before too but none of the answers are clean. I will be putting some bounty on this to get a nice answer.
I hear that maybe not using requests is a good idea but then how should I get the nice things requests offer. (the ones in the tuple)
asked Feb 23 ’14 at 7:36
5
What about using eventlet? If you want to timeout the request after 10 seconds, even if data is being received, this snippet will work for you:
import requests
import eventlet
nkey_patch()
with eventlet. Timeout(10):
(“, verify=False)
answered Feb 28 ’14 at 13:43
AlvaroAlvaro2, 0991 gold badge13 silver badges11 bronze badges
22
UPDATE: In new version of requests:
If you specify a single value for the timeout, like this:
r = (”, timeout=5)
The timeout value will be applied to both the connect and the read timeouts. Specify a tuple if you would like to set the values separately:
r = (”, timeout=(3. 05, 27))
If the remote server is very slow, you can tell Requests to wait forever for a response, by passing None as a timeout value and then retrieving a cup of coffee.
r = (”, timeout=None)
My old (probably outdated) answer (which was posted long time ago):
There are other ways to overcome this problem:
1. Use the TimeoutSauce internal class
From:
import requests from apters import TimeoutSauce
class MyTimeout(TimeoutSauce):
def __init__(self, *args, **kwargs):
connect = (‘connect’, 5)
read = (‘read’, connect)
super(MyTimeout, self). __init__(connect=connect, read=read)
apters. TimeoutSauce = MyTimeout
This code should cause us to set the read timeout as equal to the
connect timeout, which is the timeout value you pass on your
() call. (Note that I haven’t actually tested this code, so
it may need some quick debugging, I just wrote it straight into the
GitHub window. )
2. Use a fork of requests from kevinburke: From its documentation:
The timeout value will be applied to both the connect and the read
timeouts. Specify a tuple if you would like to set the values
separately:
kevinburke has requested it to be merged into the main requests project, but it hasn’t been accepted yet.
H6. 27. 8k12 gold badges72 silver badges79 bronze badges
answered Mar 13 ’14 at 11:42
HieuHieu5, 8642 gold badges37 silver badges32 bronze badges
7
Since requests >= 2. 4. 0, you can use the timeout argument, i. e:
(”, timeout=10)
Note:
timeout is not a time limit on the entire response download; rather,
an exception is raised if the server has not issued a response for
timeout seconds ( more precisely, if no bytes have been received on the
underlying socket for timeout seconds). If no timeout is specified
explicitly, requests do not time out.
answered Oct 30 ’18 at 14:15
Pedro LobitoPedro Lobito79. 7k27 gold badges219 silver badges238 bronze badges
2
To create a timeout you can use signals.
The best way to solve this case is probably to
Set an exception as the handler for the alarm signal
Call the alarm signal with a ten second delay
Call the function inside a try-except-finally block.
The except block is reached if the function timed out.
In the finally block you abort the alarm, so it’s not singnaled later.
Here is some example code:
import signal
from time import sleep
class TimeoutException(Exception):
“”” Simple Exception to be called on timeouts. “””
pass
def _timeout(signum, frame):
“”” Raise an TimeoutException.
This is intended for use as a signal handler.
The signum and frame arguments passed to this are ignored.
“””
# Raise TimeoutException with system default timeout message
raise TimeoutException()
# Set the handler for the SIGALRM signal:
(GALRM, _timeout)
# Send the SIGALRM signal in 10 seconds:
(10)
try:
# Do our code:
print(‘This will take 11 seconds… ‘)
sleep(11)
print(‘done! ‘)
except TimeoutException:
print(‘It timed out! ‘)
finally:
# Abort the sending of the SIGALRM signal:
(0)
There are some caveats to this:
It is not threadsafe, signals are always delivered to the main thread, so you can’t put this in any other thread.
There is a slight delay after the scheduling of the signal and the execution of the actual code. This means that the example would time out even if it only slept for ten seconds.
But, it’s all in the standard python library! Except for the sleep function import it’s only one import. If you are going to use timeouts many places You can easily put the TimeoutException, _timeout and the singaling in a function and just call that. Or you can make a decorator and put it on functions, see the answer linked below.
You can also set this up as a “context manager” so you can use it with the with statement:
class Timeout():
“”” Timeout for use with the `with` statement. “””
raise Timeout. TimeoutException()
def __init__(self, timeout=10):
self. timeout = timeout
(GALRM, Timeout. _timeout)
def __enter__(self):
(self. timeout)
def __exit__(self, exc_type, exc_value, traceback):
return exc_type is Timeout. TimeoutException
# Demonstration:
print(‘This is going to take maximum 10 seconds… ‘)
with Timeout(10):
sleep(15)
print(‘No timeout? ‘)
print(‘Done’)
One possible down side with this context manager approach is that you can’t know if the code actually timed out or not.
Sources and recommended reading:
The documentation on signals
This answer on timeouts by @David Narayan. He has organized the above code as a decorator.
Alex Peters2, 2301 gold badge17 silver badges25 bronze badges
answered Mar 3 ’14 at 20:23
totokakatotokaka1, 8721 gold badge16 silver badges29 bronze badges
Try this request with timeout & error handling:
url = ”
r = (url, timeout=10)
except requests. exceptions. Timeout as e:
print e
answered Jul 8 ’19 at 19:11
DaWeDaWe8119 silver badges19 bronze badges
0
Set stream=True and use er_content(1024). Yes, eventlet. Timeout just somehow doesn’t work for me.
start = time()
timeout = 5
with get(config[‘source’][‘online’], stream=True, timeout=timeout) as r:
r. raise_for_status()
content = bytes()
content_gen = er_content(1024)
while True:
if time()-start > timeout:
raise TimeoutError(‘Time out! ({} seconds)'(timeout))
content += next(content_gen)
except StopIteration:
break
data = ()(‘\n’)
if len(data) in [0, 1]:
raise ValueError(‘Bad requests data’)
except (questException, ValueError, IndexError, KeyboardInterrupt,
TimeoutError) as e:
print(e)
with open(config[‘source’][‘local’]) as f:
data = [() for line in adlines()]
The discussion is here
answered Feb 28 ’18 at 3:28
PolvPolv1, 3981 gold badge14 silver badges28 bronze badges
1
The connect timeout is the number of seconds Requests will wait for your client to establish a connection to a remote machine (corresponding to the connect()) call on the socket. It’s a good practice to set connect timeouts to slightly larger than a multiple of 3, which is the default TCP packet retransmission window.
Once your client has connected to the server and sent the HTTP request, the read timeout started. It is the number of seconds the client will wait for the server to send a response. (Specifically, it’s the number of seconds that the client will wait between bytes sent from the server. In 99. 9% of cases, this is the time before the server sends the first byte).
answered Jul 8 at 16:45
Alexander CAlexander C2, 5711 gold badge20 silver badges30 bronze badges
This may be overkill, but the Celery distributed task queue has good support for timeouts.
In particular, you can define a soft time limit that just raises an exception in your process (so you can clean up) and/or a hard time limit that terminates the task when the time limit has been exceeded.
Under the covers, this uses the same signals approach as referenced in your “before” post, but in a more usable and manageable way. And if the list of web sites you are monitoring is long, you might benefit from its primary feature — all kinds of ways to manage the execution of a large number of tasks.
answered Feb 27 ’14 at 5:47
Chris JohnsonChris Johnson18. 2k5 gold badges70 silver badges75 bronze badges
I believe you can use multiprocessing and not depend on a 3rd party package:
import multiprocessing
def call_with_timeout(func, args, kwargs, timeout):
manager = nager()
return_dict = ()
# define a wrapper of `return_dict` to store the result.
def function(return_dict):
return_dict[‘value’] = func(*args, **kwargs)
p = ocess(target=function, args=(return_dict, ))
()
# Force a max. `timeout` or wait for the process to finish
(timeout)
# If thread is still active, it didn’t finish: raise TimeoutError
if _alive():
p. terminate()
raise TimeoutError
else:
return return_dict[‘value’]
call_with_timeout(, args=(url, ), kwargs={‘timeout’: 10}, timeout=60)
The timeout passed to kwargs is the timeout to get any response from the server, the argument timeout is the timeout to get the complete response.
answered Oct 28 ’15 at 19:24
Jorge LeitaoJorge Leitao16. 1k16 gold badges76 silver badges111 bronze badges
In case you’re using the option stream=True you can do this:
r = (
‘url_to_large_file’,
timeout=1, # relevant only for underlying socket
stream=True)
with open(‘/tmp/’), ‘wb’) as f:
start_time = ()
for chunk in er_content(chunk_size=1024):
if chunk: # filter out keep-alive new chunks
(chunk)
if () – start_time > 8:
raise Exception(‘Request took longer than 8s’)
The solution does not need signals or multiprocessing.
answered Apr 24 ’18 at 13:17
ub_marcoub_marco1301 silver badge5 bronze badges
Just another one solution (got it from)
Before upload you can find out the content size:
TOO_LONG = 10*1024*1024 # 10 Mb
big_url = ”
r = (big_url, stream=True)
print (r. headers[‘content-length’])
# 1073741824
if int(r. headers[‘content-length’]) < TOO_LONG:
# upload content:
content = ntent
But be careful, a sender can set up incorrect value in the 'content-length' response field.
answered Jul 9 '18 at 9:56
Denis KuzinDenis Kuzin7931 gold badge8 silver badges17 bronze badges
timeout = (connection timeout, data read timeout) or give a single argument(timeout=1)
req = quest('GET', '', timeout=(1, 1))
print(req)
except adTimeout:
print("READ TIME OUT")
answered Feb 8 '19 at 11:07
this code working for socketError 11004 and 10060......
# -*- encoding:UTF-8 -*-
__author__ = 'ACE'
from import *
class TimeOutModel(QThread):
Existed = pyqtSignal(bool)
TimeOut = pyqtSignal()
def __init__(self, fun, timeout=500, parent=None):
@param fun: function or lambda
@param timeout: ms
super(TimeOutModel, self). __init__(parent)
= fun
= QTimer(self)
(self. time_timeout)
nnect()
tTerminationEnabled(True)
def time_timeout(self):
self. terminate()
def run(self):
bb = lambda: (")
a = QApplication([])
z = TimeOutModel(bb, 500)
print 'timeout'
a. exec_()
answered Oct 11 '16 at 6:14
ACEEACEE137 bronze badges
Despite the question being about requests, I find this very easy to do with pycurl CURLOPT_TIMEOUT or CURLOPT_TIMEOUT_MS.
No threading or signaling required:
import pycurl
import StringIO
url = ''
timeout_ms = 1000
raw = ringIO()
c = ()
(pycurl. TIMEOUT_MS, timeout_ms) # total timeout in milliseconds
(pycurl. WRITEFUNCTION, )
(SIGNAL, 1)
(, url)
(TPGET, 1)
rform()
except
int_exc() # error generated on timeout
pass # or just pass if you don't want to print the error
answered Oct 28 '16 at 9:46
Well, I tried many solutions on this page and still faced instabilities, random hangs, poor connections performance.
I'm now using Curl and i'm really happy about it's "max time" functionnality and about the global performances, even with such a poor implementation:
toutput('curl -m6 -Ss "')
Here, I defined a 6 seconds max time parameter, englobing both connection and transfer time.
I'm sure Curl has a nice python binding, if you prefer to stick to the pythonic syntax:)
answered Nov 9 '17 at 23:14
technicotechnico1, 1641 gold badge10 silver badges22 bronze badges
There is a package called timeout-decorator that you can use to time out any python function.
@timeout_decorator. timeout(5)
def mytest():
print("Start")
for i in range(1, 10):
(1)
print("{} seconds have passed"(i))
It uses the signals approach that some answers here suggest. Alternatively, you can tell it to use multiprocessing instead of signals (e. g. if you are in a multi-thread environment).
answered Sep 13 '18 at 19:37
Christian LongChristian Long7, 3782 gold badges54 silver badges53 bronze badges
If it comes to that, create a watchdog thread that messes up requests' internal state after 10 seconds, e. :
closes the underlying socket, and ideally
triggers an exception if requests retries the operation
Note that depending on the system libraries you may be unable to set deadline on DNS resolution.
answered Feb 25 '14 at 22:40
Dima TisnekDima Tisnek9, 7774 gold badges53 silver badges109 bronze badges
I'm using requests 2. 2. 1 and eventlet didn't work for me. Instead I was able use gevent timeout instead since gevent is used in my service for gunicorn.
import gevent
import
(subprocess=True)
with gevent. Timeout(5):
ret = (url)
print atus_code, ntent
except gevent. timeout. Timeout as e:
print "timeout: {}"(ssage)
Please note that gevent. Timeout is not caught by general Exception handling.
So either explicitly catch gevent. Timeout
or pass in a different exception to be used like so: with gevent. Timeout(5, requests. Timeout): although no message is passed when this exception is raised.
answered May 23 '20 at 22:03
xsdfxsdf3852 silver badges14 bronze badges
I came up with a more direct solution that is admittedly ugly but fixes the real problem. It goes a bit like this:
resp = (some_url, stream=True)
(read_timeout)
# This will load the entire response even though stream is set
You can read the full explanation here
answered Dec 3 '15 at 18:29
RealisticRealistic9689 silver badges18 bronze badges
Not the answer you're looking for? Browse other questions tagged python timeout python-requests or ask your own question.
Python’s Requests Library (Guide)
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Making HTTP Requests With Python
The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.
Throughout this article, you’ll see some of the most useful features that requests has to offer as well as how to customize and optimize those features for different situations you may come across. You’ll also learn how to use requests in an efficient way as well as how to prevent requests to external services from slowing down your application.
In this tutorial, you’ll learn how to:
Make requests using the most common HTTP methods
Customize your requests’ headers and data, using the query string and message body
Inspect data from your requests and responses
Make authenticated requests
Configure your requests to help prevent your application from backing up or slowing down
Though I’ve tried to include as much information as you need to understand the features and examples included in this article, I do assume a very basic general knowledge of HTTP. That said, you still may be able to follow along fine anyway.
Now that that is out of the way, let’s dive in and see how you can use requests in your application!
Getting Started With requests
Let’s begin by installing the requests library. To do so, run the following command:
If you prefer to use Pipenv for managing Python packages, you can run the following:
$ pipenv install requests
Once requests is installed, you can use it in your application. Importing requests looks like this:
Now that you’re all set up, it’s time to begin your journey through requests. Your first goal will be learning how to make a GET request.
The GET Request
HTTP methods such as GET and POST, determine which action you’re trying to perform when making an HTTP request. Besides GET and POST, there are several other common methods that you’ll use later in this tutorial.
One of the most common HTTP methods is GET. The GET method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET request, invoke ().
To test this out, you can make a GET request to GitHub’s Root REST API by calling get() with the following URL:
>>>>>> (”)
Congratulations! You’ve made your first request. Let’s dive a little deeper into the response of that request.
The Response
A Response is a powerful object for inspecting the results of the request. Let’s make that same request again, but this time store the return value in a variable so that you can get a closer look at its attributes and behaviors:
>>>>>> response = (”)
In this example, you’ve captured the return value of get(), which is an instance of Response, and stored it in a variable called response. You can now use response to see a lot of information about the results of your GET request.
Status Codes
The first bit of information that you can gather from Response is the status code. A status code informs you of the status of the request.
For example, a 200 OK status means that your request was successful, whereas a 404 NOT FOUND status means that the resource you were looking for was not found. There are many other possible status codes as well to give you specific insights into what happened with your request.
By accessing. status_code, you can see the status code that the server returned:
>>>>>> atus_code
200. status_code returned a 200, which means your request was successful and the server responded with the data you were requesting.
Sometimes, you might want to use this information to make decisions in your code:
if atus_code == 200:
print(‘Success! ‘)
elif atus_code == 404:
print(‘Not Found. ‘)
With this logic, if the server returns a 200 status code, your program will print Success!. If the result is a 404, your program will print Not Found.
requests goes one step further in simplifying this process for you. If you use a Response instance in a conditional expression, it will evaluate to True if the status code was between 200 and 400, and False otherwise.
Therefore, you can simplify the last example by rewriting the if statement:
if response:
else:
print(‘An error has occurred. ‘)
Keep in mind that this method is not verifying that the status code is equal to 200. The reason for this is that other status codes within the 200 to 400 range, such as 204 NO CONTENT and 304 NOT MODIFIED, are also considered successful in the sense that they provide some workable response.
For example, the 204 tells you that the response was successful, but there’s no content to return in the message body.
So, make sure you use this convenient shorthand only if you want to know if the request was generally successful and then, if necessary, handle the response appropriately based on the status code.
Let’s say you don’t want to check the response’s status code in an if statement. Instead, you want to raise an exception if the request was unsuccessful. You can do this using. raise_for_status():
import requests
from requests. exceptions import HTTPError
for url in [”, ”]:
try:
response = (url)
# If the response was successful, no Exception will be raised
response. raise_for_status()
except HTTPError as _err:
print(f’HTTP error occurred: {_err}’) # Python 3. 6
except Exception as err:
print(f’Other error occurred: {err}’) # Python 3. 6
If you invoke. raise_for_status(), an HTTPError will be raised for certain status codes. If the status code indicates a successful request, the program will proceed without that exception being raised.
Now, you know a lot about how to deal with the status code of the response you got back from the server. However, when you make a GET request, you rarely only care about the status code of the response. Usually, you want to see more. Next, you’ll see how to view the actual data that the server sent back in the body of the response.
Content
The response of a GET request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of Response, you can view the payload in a variety of different formats.
To see the response’s content in bytes, you use. content:
>>> ntent
b'{“current_user_url”:”, “current_user_authorizations_html_url”:”/client_id}”, “authorizations_url”:”, “code_search_url”:”query}{&page, per_page, sort, order}”, “commit_search_url”:”query}{&page, per_page, sort, order}”, “emails_url”:”, “emojis_url”:”, “events_url”:”, “feeds_url”:”, “followers_url”:”, “following_url”:”/target}”, “gists_url”:”/gist_id}”, “hub_url”:”, “issue_search_url”:”query}{&page, per_page, sort, order}”, “issues_url”:”, “keys_url”:”, “notifications_url”:”, “organization_repositories_url”:”org}/repos{? type, page, per_page, sort}”, “organization_url”:”org}”, “public_gists_url”:”, “rate_limit_url”:”, “repository_url”:”owner}/{repo}”, “repository_search_url”:”query}{&page, per_page, sort, order}”, “current_user_repositories_url”:”? type, page, per_page, sort}”, “starred_url”:”/owner}{/repo}”, “starred_gists_url”:”, “team_url”:”, “user_url”:”user}”, “user_organizations_url”:”, “user_repositories_url”:”user}/repos{? type, page, per_page, sort}”, “user_search_url”:”query}{&page, per_page, sort, order}”}’
While. content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. response will do that for you when you access
>>>>>>
‘{“current_user_url”:”, “current_user_authorizations_html_url”:”/client_id}”, “authorizations_url”:”, “code_search_url”:”query}{&page, per_page, sort, order}”, “commit_search_url”:”query}{&page, per_page, sort, order}”, “emails_url”:”, “emojis_url”:”, “events_url”:”, “feeds_url”:”, “followers_url”:”, “following_url”:”/target}”, “gists_url”:”/gist_id}”, “hub_url”:”, “issue_search_url”:”query}{&page, per_page, sort, order}”, “issues_url”:”, “keys_url”:”, “notifications_url”:”, “organization_repositories_url”:”org}/repos{? type, page, per_page, sort}”, “organization_url”:”org}”, “public_gists_url”:”, “rate_limit_url”:”, “repository_url”:”owner}/{repo}”, “repository_search_url”:”query}{&page, per_page, sort, order}”, “current_user_repositories_url”:”? type, page, per_page, sort}”, “starred_url”:”/owner}{/repo}”, “starred_gists_url”:”, “team_url”:”, “user_url”:”user}”, “user_organizations_url”:”, “user_repositories_url”:”user}/repos{? type, page, per_page, sort}”, “user_search_url”:”query}{&page, per_page, sort, order}”}’
Because the decoding of bytes to a str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting. encoding before accessing
>>>>>> response. encoding = ‘utf-8’ # Optional: requests infers this internally
>>>
If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str you retrieved from and deserialize it using (). However, a simpler way to accomplish this task is to use ():
>>>>>> ()
{‘current_user_url’: ”, ‘current_user_authorizations_html_url’: ‘/client_id}’, ‘authorizations_url’: ”, ‘code_search_url’: ‘query}{&page, per_page, sort, order}’, ‘commit_search_url’: ‘query}{&page, per_page, sort, order}’, ’emails_url’: ”, ’emojis_url’: ”, ‘events_url’: ”, ‘feeds_url’: ”, ‘followers_url’: ”, ‘following_url’: ‘/target}’, ‘gists_url’: ‘/gist_id}’, ‘hub_url’: ”, ‘issue_search_url’: ‘query}{&page, per_page, sort, order}’, ‘issues_url’: ”, ‘keys_url’: ”, ‘notifications_url’: ”, ‘organization_repositories_url’: ‘org}/repos{? type, page, per_page, sort}’, ‘organization_url’: ‘org}’, ‘public_gists_url’: ”, ‘rate_limit_url’: ”, ‘repository_url’: ‘owner}/{repo}’, ‘repository_search_url’: ‘query}{&page, per_page, sort, order}’, ‘current_user_repositories_url’: ‘? type, page, per_page, sort}’, ‘starred_url’: ‘/owner}{/repo}’, ‘starred_gists_url’: ”, ‘team_url’: ”, ‘user_url’: ‘user}’, ‘user_organizations_url’: ”, ‘user_repositories_url’: ‘user}/repos{? type, page, per_page, sort}’, ‘user_search_url’: ‘query}{&page, per_page, sort, order}’}
The type of the return value of () is a dictionary, so you can access values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.
Query String Parameters
One common way to customize a GET request is to pass values through query string parameters in the URL. To do this using get(), you pass data to params. For example, you can use GitHub’s Search API to look for the requests library:
# Search GitHub’s repositories for requests
response = (
”,
params={‘q’: ‘requests+language:python’}, )
# Inspect some attributes of the `requests` repository
json_response = ()
repository = json_response[‘items’][0]
print(f’Repository name: {repository[“name”]}’) # Python 3. 6+
print(f’Repository description: {repository[“description”]}’) # Python 3. 6+
By passing the dictionary {‘q’: ‘requests+language:python’} to the params parameter of (), you are able to modify the results that come back from the Search API.
You can pass params to get() in the form of a dictionary, as you have just done, or as a list of tuples:
>>>>>> (… ”,… params=[(‘q’, ‘requests+language:python’)],… )
You can even pass the values as bytes:
>>>>>> (… params=b’q=requests+language:python’,… )
Query strings are useful for parameterizing GET requests. You can also customize your requests by adding or modifying the headers you send.
Other HTTP Methods
Aside from GET, other popular HTTP methods include POST, PUT, DELETE, HEAD, PATCH, and OPTIONS. requests provides a method, with a similar signature to get(), for each of these HTTP methods:
>>>>>> (”, data={‘key’:’value’})
>>> (”, data={‘key’:’value’})
>>> (”)
>>> requests. options(”)
Each function call makes a request to the bin service using the corresponding HTTP method. For each method, you can inspect their responses in the same way you did before:
>>> response. headers[‘Content-Type’]
‘application/json’
>>> response = (”)
>>> json_response = ()
>>> json_response[‘args’]
{}
Headers, response bodies, status codes, and more are returned in the Response for each method. Next you’ll take a closer look at the POST, PUT, and PATCH methods and learn how they differ from the other request types.
The Message Body
According to the HTTP specification, POST, PUT, and the less common PATCH requests pass their data through the message body rather than through parameters in the query string. Using requests, you’ll pass the payload to the corresponding function’s data parameter.
data takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data you send in the body of your request to the specific needs of the service you’re interacting with.
For example, if your request’s content type is application/x-www-form-urlencoded, you can send the form data as a dictionary:
You can also send that same data as a list of tuples:
>>>>>> (”, data=[(‘key’, ‘value’)])
If, however, you need to send JSON data, you can use the json parameter. When you pass JSON data via json, requests will serialize your data and add the correct Content-Type header for you.
is a great resource created by the author of requests, Kenneth Reitz. It’s a service that accepts test requests and responds with data about the requests. For instance, you can use it to inspect a basic POST request:
>>>>>> response = (”, json={‘key’:’value’})
>>> json_response[‘data’]
‘{“key”: “value”}’
>>> json_response[‘headers’][‘Content-Type’]
You can see from the response that the server received your request data and headers as you sent them. requests also provides this information to you in the form of a PreparedRequest.
Inspecting Your Request
When you make a request, the requests library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.
You can view the PreparedRequest by accessing. request:
>>> quest. headers[‘Content-Type’]
”
b'{“key”: “value”}’
Inspecting the PreparedRequest gives you access to all kinds of information about the request being made such as payload, URL, headers, authentication, and more.
So far, you’ve made a lot of different kinds of requests, but they’ve all had one thing in common: they’re unauthenticated requests to public APIs. Many services you may come across will want you to authenticate in some way.
Authentication
Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the Authorization header or a custom header defined by the service. All the request functions you’ve seen to this point provide a parameter called auth, which allows you to pass your credentials.
One example of an API that requires authentication is GitHub’s Authenticated User API. This endpoint provides information about the authenticated user’s profile. To make a request to the Authenticated User API, you can pass your GitHub username and password in a tuple to get():
>>>>>> from getpass import getpass
>>> (”, auth=(‘username’, getpass()))
The request succeeded if the credentials you passed in the tuple to auth are valid. If you try to make this request with no credentials, you’ll see that the status code is 401 Unauthorized:
When you pass your username and password in a tuple to the auth parameter, requests is applying the credentials using HTTP’s Basic access authentication scheme under the hood.
Therefore, you could make the same request by passing explicit Basic authentication credentials using HTTPBasicAuth:
>>>>>> from import HTTPBasicAuth
>>> from getpass import getpass
>>> (… auth=HTTPBasicAuth(‘username’, getpass())… )
Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. requests provides other methods of authentication out of the box such as HTTPDigestAuth and HTTPProxyAuth.
You can even supply your own authentication mechanism. To do so, you must first create a subclass of AuthBase. Then, you implement __call__():
from import AuthBase
class TokenAuth(AuthBase):
“””Implements a custom authentication scheme. “””
def __init__(self, token):
= token
def __call__(self, r):
“””Attach an API token to a custom auth header. “””
r. headers[‘X-TokenAuth’] = f'{}’ # Python 3. 6+
return r
(”, auth=TokenAuth(‘12345abcde-token’))
Here, your custom TokenAuth mechanism receives a token, then includes that token in the X-TokenAuth header of your request.
Bad authentication mechanisms can lead to security vulnerabilities, so unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like Basic or OAuth.
While you’re thinking about security, let’s consider dealing with SSL Certificates using requests.
SSL Certificate Verification
Any time the data you are trying to send or receive is sensitive, security is important. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using SSL, which means that verifying the target server’s SSL Certificate is critical.
The good news is that requests does this for you by default. However, there are some cases where you might want to change this behavior.
If you want to disable SSL Certificate verification, you pass False to the verify parameter of the request function:
>>>>>> (”, verify=False)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: InsecureRequestWarning)
requests even warns you when you’re making an insecure request to help you keep your data safe!
Performance
When using requests, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.
Timeouts
When you make an inline request to an external service, your system will need to wait upon the response before moving on. If your application waits too long for that response, requests to your service could back up, your user experience could suffer, or your background jobs could hang.
By default, requests will wait indefinitely on the response, so you should almost always specify a timeout duration to prevent these things from happening. To set the request’s timeout, use the timeout parameter. timeout can be an integer or float representing the number of seconds to wait on a response before timing out:
>>>>>> (”, timeout=1)
>>> (”, timeout=3. 05)
In the first request, the request will timeout after 1 second. In the second request, the request will timeout after 3. 05 seconds.
You can also pass a tuple to timeout with the first element being a connect timeout (the time it allows for the client to establish a connection to the server), and the second being a read timeout (the time it will wait on a response once your client has established a connection):
>>>>>> (”, timeout=(2, 5))
If the request establishes a connection within 2 seconds and receives data within 5 seconds of the connection being established, then the response will be returned as it was before. If the request times out, then the function will raise a Timeout exception:
from requests. exceptions import Timeout
response = (”, timeout=1)
except Timeout:
print(‘The request timed out’)
print(‘The request did not time out’)
Your program can catch the Timeout exception and respond accordingly.
The Session Object
Until now, you’ve been dealing with high level requests APIs such as get() and post(). These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them.
Underneath those abstractions is a class called Session. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a Session instance directly.
Sessions are used to persist parameters across requests. For example, if you want to use the same authentication across multiple requests, you could use a session:
from getpass import getpass
# By using a context manager, you can ensure the resources used by
# the session will be released after use
with ssion() as session:
= (‘username’, getpass())
# Instead of (), you’ll use ()
response = (”)
# You can inspect the response just like you did before
print(response. headers)
print(())
Each time you make a request with session, once it has been initialized with authentication credentials, the credentials will be persisted.
The primary performance optimization of sessions comes in the form of persistent connections. When your app makes a connection to a server using a Session, it keeps that connection around in a connection pool. When your app wants to connect to the same server again, it will reuse a connection from the pool rather than establishing a new one.
Max Retries
When a request fails, you may want your application to retry the same request. However, requests will not do this for you by default. To apply this functionality, you need to implement a custom Transport Adapter.
Transport Adapters let you define a set of configurations per service you’re interacting with. For example, let’s say you want all requests to to retry three times before finally raising a ConnectionError. You would build a Transport Adapter, set its max_retries parameter, and mount it to an existing Session:
from apters import HTTPAdapter
from requests. exceptions import ConnectionError
github_adapter = HTTPAdapter(max_retries=3)
session = ssion()
# Use `github_adapter` for all requests to endpoints that start with this URL
(”, github_adapter)
(”)
except ConnectionError as ce:
print(ce)
When you mount the HTTPAdapter, github_adapter, to session, session will adhere to its configuration for each request to Timeouts, Transport Adapters, and sessions are for keeping your code efficient and your application resilient.
Conclusion
You’ve come a long way in learning about Python’s powerful requests library.
You’re now able to:
Make requests using a variety of different HTTP methods such as GET, POST, and PUT
Customize your requests by modifying headers, authentication, query strings, and message bodies
Inspect the data you send to the server and the data the server sends back to you
Work with SSL Certificate verification
Use requests effectively using max_retries, timeout, Sessions, and Transport Adapters
Because you learned how to use requests, you’re equipped to explore the wide world of web services and build awesome applications using the fascinating data they provide.
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Making HTTP Requests With Python
Example – Python Set Request Timeout – techEplanet
When we invoke an api, it is always a good practice to set a request timeout. In case the api you are accessing is not responding you are looking at a hanging call if there are no timeouts. In this article we will see how to set request timeout in Python.
Example – Python Set Request Timeout
Let us take a look at the below code. Here we are making a GET request and setting a timeout of 5 seconds. The requests module takes an argument “timeout” in its methods which is used for timeouts in seconds. If the request didn’t get a response within the specified timeout, then it will throw an exception.
import requests
#Setting a timeout for 5 seconds
url = ‘localhost/techeplanet’
response = (url, timeout=5)
