Curl Proxy
performing HTTP requests with cURL (using PROXY) – Stack …
I have this proxy address: 125. 119. 175. 48:8909
How can I perform a HTTP request using cURL like curl, but specifying the proxy address of my network?
ivanleoncz6, 3093 gold badges49 silver badges45 bronze badges
asked Feb 25 ’12 at 15:47
0
From man curl:
-x, –proxy <[protocol][user:password@]proxyhost[:port]>
Use the specified HTTP proxy.
If the port number is not specified, it is assumed at port 1080.
bgcode24. 8k30 gold badges95 silver badges158 bronze badges
answered Feb 25 ’12 at 15:51
Karl BarkerKarl Barker10. 3k3 gold badges18 silver badges24 bronze badges
5
General way:
export _proxy=
Then you can connect through proxy from (many) application.
And, as per comment below, for:
answered Feb 25 ’12 at 19:51
airweenairween5, 4001 gold badge12 silver badges17 bronze badges
8
The above solutions might not work with some curl versions I tried them for myself(curl 7. 22. 0). But what worked for me was:
curl -x proxy_server:proxy_port –proxy-user username:password -L url
Hope it solves the issue better!
answered Feb 11 ’14 at 10:14
AmarAmar1, 9721 gold badge11 silver badges15 bronze badges
2
Beware that if you are using a SOCKS proxy, instead of a HTTP/HTTPS proxy, you will need to use the –socks5 switch instead:
curl –socks5 125. 48:8909
You can also use –socks5-hostname instead of –socks5 to resolve DNS on the proxy side.
answered Aug 28 ’14 at 16:57
Filipe CorreiaFilipe Correia4, 7454 gold badges26 silver badges46 bronze badges
4
as an adition to airween, another good idea is to add this into your, so you’ll be able to switch from non proxied to proxied environment:
alias proxyon=”export _proxy=’YOURPROXY:YOURPORT’;export _proxy=’YOURPROXY:YOURPORT'”
alias proxyoff=”export _proxy=”;export _proxy=””
WHERE YOURPROXY:YOURPORT is exactly that, your ip and port proxy:-).
Then, simply doing
proxyon
your system will start to use the proxy, and just the opposite with:
proxyoff
answered May 22 ’14 at 8:03
Alejandro MorenoAlejandro Moreno5, 1622 gold badges29 silver badges28 bronze badges
1
use the following
curl -I -x 192. 168. X. X:XX
192. X:XX put your proxy server ip and port.
-v verbose mode it will give more details including headers and response.
answered Jan 29 ’15 at 16:48
13krn13krn5014 silver badges9 bronze badges
I like using this in order to get the IP under which I am seen
curl -x proxy_server:proxy_port && echo
Hope this helps someone.
answered Sep 4 ’17 at 14:11
bmetgebmetge3113 silver badges3 bronze badges
For curl you can configure proxy in your ~/ (_curlrc on Windows) file by adding proxy value, the syntax is:
proxy = username:password@proxy-host:port
answered Mar 5 ’16 at 14:16
kenorbkenorb125k69 gold badges606 silver badges641 bronze badges
C:\Desktop>curl –head –proxy 1. 1. 1:8080
Shree19. 9k28 gold badges89 silver badges134 bronze badges
answered Sep 6 ’19 at 9:52
Just summarizing all great mentioned answers:
curl -x
M. MashayeM. Mashaye87411 silver badges21 bronze badges
With a proxy with authentication I use:
curl -x
because, I don’t know why curl doesn’t use/catch [s]_proxy environment variables.
answered May 31 ’17 at 6:39
You don’t need to export the [s]_proxy shell variable if you’re just setting the proxy for a one off command. e. g.
_proxy= curl
That said, I’d prefer curl -x if I knew I was always going to use a proxy.
answered Mar 27 ’17 at 18:03
overthinkoverthink22. 9k3 gold badges63 silver badges68 bronze badges
sudo curl -x -fsSL
This worked perfectly for me, the error comes because curl need to set
the proxy
Remmember replace the proxy with your proxy, mine, “example” was
answered Jun 11 ’19 at 16:26
Depending on your workplace, you may also need to specify the -k or the –insecure option for curl in order to get past potential issues with CA certificates.
curl -x
curl -vv -k -x
iamtheexp01iamtheexp013, 2968 gold badges30 silver badges35 bronze badges
In case the proxy is using automatic proxy with PAC file. We can find the actual proxy from the javascript from the PAC URL.
And if the proxy needs authentication, we can first use a normal web-browser to access the website which will promote authentication dialog. After authentication, we can use wireshark to capture the package sends to the proxy server, from the package, we can get the auth token from header: Proxy-Authorization
Then we can set the _proxy environment variable and also include auth token in the header: Proxy-Authorization
export _proxy=proxyserver:port
curl -H “Proxy-Authorization: xxxx” targetURL
answered Jun 24 ’17 at 5:49
Jianwu ChenJianwu Chen3, 8943 gold badges21 silver badges28 bronze badges
Not the answer you’re looking for? Browse other questions tagged linux curl proxy or ask your own question.
How to use cURL with proxy? – Blog | Oxylabs
This step-by-step guide will explain how to use cURL or simply, curl, with proxy servers. It covers all the aspects, beginning from installation to explaining various options to set the proxy.
We did not target any specific proxy service. Therefore this tutorial should work with all proxy servers. All you need to know is the server details and credentials.
This is a fairly technical tutorial and expects readers to have a basic understanding of what a proxy is. It would be especially interesting and useful for those starting with web scraping.
Navigation:
What is cURL?
Installation
What you need to connect to a proxy
Using cURL with HTTP/HTTPS proxy
Command line argument to set proxy in cURL
Using environment variables
Configure cURL to always use proxy
Ignore or override proxy for one request
Bonus tip – turning proxies off and on quickly
cURL socks proxy
Summary
cURL is a command line tool for sending and receiving data using the url. Let’s look at the simplest example of using curl. Open your terminal or command prompt and type in this command and press Enter:
curl This will get the HTML of the page and print it on the console.
curl -I
This will print the document information.
HTTP/1. 1 200 OK
Content-Type: text/html; charset=ISO-8859-1
The question “what is cURL? ” is also answered in one of our previous articles. We recommend reading it if you want to learn how it became such a universal asset.
cURL is provided with many Linux distributions and with MacOS. Now it is provided with Windows 10 as well.
If your Linux distribution is not provided with it, you can install it by running the install command. For example, on Ubuntu, open Terminal and run this command:
sudo apt install curl
If you are running an older version of Windows, or if you want to install an alternate version, you can download curl from the official download page.
Irrespective of which proxy service you use, you will need the following information to use a:
proxy server addressportprotocolusername (if authentication is required)password (if authentication is required)
In this tutorial, we are going to assume that the proxy server is 127. 0. 1, the port is 1234, the user name is user, and the password is pwd. We will look into multiple examples covering various protocols.
NOTE. If you are on a network that uses NTLM authentication, you can use the switch –proxy-ntlm while running curl. Similarly, –proxy-digest can be used for digest authentication. You can look at all the available options by running curl –help. This tutorial will have examples for the scenario when a username and password has to be specified.
The next section will cover the first curl proxy scenario, which happens to be the most common one – HTTP and HTTPS proxy with curl.
If you recall, we looked at using curl without proxy like this:
curl This particular website is especially useful for testing out proxies as the output of this page is the origin IP address. If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address.
There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.
NOTE. All the command line options, or switches, are case sensitive. For example, -f instructs curl to fail silently, while -F denotes a form to be submitted.
Open terminal and type the following command, and press Enter:
curl –help
The output is going to be a huge list of options. One of them is going to look like this:
-x, –proxy [protocol]host[:port]
Note that x is small, and it is case-sensitive. The proxy details can be supplied using -x or –proxy switch. Both mean the same thing. Bot of the curl with proxy commands are same:
curl -x “user:pwd@127. 1:1234″ ”
or
curl –proxy “user:pwd@127. 1:1234″ ”
NOTE. If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.
curl –proxy “user:pwd@127. 1:1234″ ” -k
You may have noticed that both the proxy url and target url are surrounded in double quotes. This is a recommended practice to handle special characters in the url.
Another interesting thing to note here is that the default proxy protocol is. Thus, following two commands will do exactly the same:
Another way to use proxy with curl is to set the environment variables _proxy and _proxy.
Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc file.
If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.
_proxy – the proxy will be used to access addresses that use protocol
Simply set the variables _proxy to proxy address and _proxy to set proxy address. Open terminal and run these two commands.
export _proxy=”user:pwd@127. 1:1234″
After running these two commands, run curl normally.
curl ”
If you see SSL Certificate errors, add -k to ignore these errors.
Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:
unset _proxy
See the next section to set default proxy only for curl and not system wide.
If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.
For Linux and MacOS, open terminal and navigate to your home directory. If there is already a file, open it. If there is none, create a new file. Here are the set of commands that can be run:
cd ~
nano
In this file, add this line:
proxy=”user:pwd@127. 1:1234″
Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from file.
On Windows, the file is named _curlrc. This file can be placed in the%APPDATA% directory.
To find the exact path of%APPDATA%, open command prompt and run the following command:
echo%APPDATA%
This directory will be something like C:\Users\
This works exactly the same way in Linux, MacOS, and Windows.
If the proxy is set globally, or by modifying the file, this can still be overridden to set another proxy or even bypass it.
To override proxy for one request, set the new proxy using -x or –proxy switch as usual:
curl –proxy “user:pwd@1. 1:8090″ ”
If you want to bypass proxy altogether for a request, you can pass –noproxy followed by “*”. This instructs curl to not use proxy for all URLs.
curl –noproxy “*” ”
If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.
This tip is dedicated only for advanced users. If you do not know what a file is, you may skip this section.
You can create an alias in your file to set proxies and unset proxies. For example, open file using any editor and add these lines:
alias proxyon=”export _proxy=’ user:pwd@127. 1:1234′;export _proxy=’ user:pwd@127. 1:1234′”
alias proxyoff=”unset _proxy;unset _proxy”
After adding these lines, save the and update the shell to read this To do this, run this this command in the terminal:. ~/
Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:
proxyon
proxyoff
If the proxy server is using socks protocol, the syntax remains the same:
curl -x “socks5user:pwd@127. 1:1234″ ”
Similarly, socks4, socks4a, socks5 or socks5h can be used depending on the socks version.
Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.
curl –socks5 “127. 1:1234″ ” –proxy-user user:pwd
Again, –socks4, –socks4a or –socks5 can be used, depending on the version.
cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.
You can learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.
Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you’ll probably find her just chillin’ while listening to her favorite music or playing board games with friends.
All information on Oxylabs Blog is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.
Proxies – Everything curl
A proxy is a machine or software that does something on behalf of you, the can also see it as a middle man that sits between you and the server you want to work with, a middle man that you connect to instead of the actual remote server. You ask the proxy to perform your desired operation for you and then it will run off and do that and then return the data to are several different types of proxies and we shall list and discuss them further down in this networks are setup to require a proxy in order for you to reach the Internet or perhaps that special network you are interested in. The use of proxies are introduced on your network by the people and management that run your network for policy or technical the networking space there are a few methods for the automatic detection of proxies and how to connect to them, but none of those methods are truly universal and curl supports none of them. Furthermore, when you communicate to the outside world through a proxy that often means that you have to put a lot of trust on the proxy as it will be able to see and modify all the non-secure network traffic you send or get through it. That trust is not easy to assume you check your browser’s network settings, sometimes under an advanced settings tab, you can learn what proxy or proxies your browser is configured to use. Chances are big that you should use the same one or ones when you use an example, you can find proxy settings for Firefox browser in Preferences => General => Network Settings as shown below:proxy settings for FirefoxSome network environments provides several different proxies that should be used in different situations, and a customizable way to handle that is supported by the browsers. This is called “proxy auto-config”, or PAC. A PAC file contains a JavaScript function that decides which proxy a given network connection (URL) should use, and even if it should not use a proxy at all. Browsers most typically read the PAC file off a URL on the local curl has no JavaScript capabilities, curl does not support PAC files. If your browser and network use PAC files, the easiest route forward is usually to read the PAC file manually and figure out the proxy you need to specify to run curl are not proxies but they’re blocking the way between you and the server you want to access. A “captive portal” is one of these systems that are popular to use in hotels, airports and for other sorts of network access to a larger audience. The portal will “capture” all network traffic and redirect you to a login web page until you have either clicked OK and verified that you have read their conditions or perhaps even made sure that you have paid plenty of money for the right to use the ‘s traffic will of course also captured by such portals and often the best way is to use a browser to accept the conditions and “get rid of” the portal since from then on they often allow all other traffic originating from that same machine (MAC address) for a period of often you can use curl too to submit that “ok” affirmation, if you just figure out how to submit the form and what fields to include in it. If this is something you end up doing many times, it may be worth supports several different types of default proxy type is HTTP so if you specify a proxy host name (or IP address) without a scheme part (the part that is often written as “”) curl goes with assuming it’s an HTTP also allows a number of different options to set the proxy type instead of using the scheme prefix. See the SOCKS section HTTP proxy is a proxy that the client speaks HTTP with to get the transfer done. curl will, by default, assume that a host you point out with -x or –proxy is an HTTP proxy, and unless you also specify a port number it will default to port 1080 (and the reason for that particular port number is purely historical) you want to request the web page using a proxy on 192. 168. 0. 1 port 8080, a command line could look like:curl -x 192. 1:8080 that the proxy receives your request, forwards it to the real server, then reads the response from the server and then hands that back to the you enable verbose mode with -v when talking to a proxy, you will see that curl connects to the proxy instead of the remote server, and you will see that it uses a slightly different request was designed to allow and provide secure and safe end-to-end privacy from the client to the server (and back). In order to provide that when speaking to an HTTP proxy, the HTTP protocol has a special request that curl uses to setup a tunnel through the proxy that it then can encrypt and verify. This HTTP method is known as the proxy tunnels encrypted data through to the remote server after a CONNECT method sets it up, the proxy cannot see nor modify the traffic without breaking the encryption:curl -x means Man-In-The-Middle. MITM-proxies are usually deployed by companies in “enterprise environments” and elsewhere, where the owners of the network have a desire to investigate even TLS encrypted do this, they require users to install a custom “trust root” (Certificate Authority (CA) certificate) in the client, and then the proxy terminates all TLS traffic from the client, impersonates the remote server and acts like a proxy. The proxy then sends back a generated certificate signed by the custom CA. Such proxy setups usually transparently capture all traffic from clients to TCP port 443 on a remote machine. Running curl in such a network would also get its HTTPS traffic practice, of course, allows the middle man to decrypt and snoop on all TLS “HTTP proxy” means the proxy itself speaks HTTP. HTTP proxies are primarily used to proxy HTTP but it is also fairly common that they support other protocols as well. In particular, FTP is fairly commonly talking FTP “over” an HTTP proxy, it is usually done by more or less pretending the other protocol works like HTTP and asking the proxy to “get this URL” even if the URL is not using HTTP. This distinction is important because it means that when sent over an HTTP proxy like this, curl does not really speak FTP even though given an FTP URL; thus FTP-specific features will not work:curl -x you can do instead then, is to “tunnel through” the HTTP proxy! Most HTTP proxies allow clients to “tunnel through” it to a server on the other side. That’s exactly what’s done every time you use HTTPS through the HTTP tunnel through an HTTP proxy with curl using -p or you do HTTPS through a proxy you normally connect through to the default HTTPS remote TCP port number 443, so therefore you will find that most HTTP proxies white list and allow connections only to hosts on that port number and perhaps a few others. Most proxies will deny clients from connecting to just any random port (for reasons only the proxy administrators know), assuming that the HTTP proxy allows it, you can ask it to tunnel through to a remote server on any port number so you can do other protocols “normally” even when tunneling. You can do FTP tunneling like this:curl -p -x can tell curl to use HTTP/1. 0 in its CONNECT request issued to the HTTP proxy by using –proxy1. 0 [proxy] instead of is a protocol used for proxies and curl supports it. curl supports both SOCKS version 4 as well as version 5, and both versions come in two can select the specific SOCKS version to use by using the correct scheme part for the given proxy host with -x, or you can specify it with a separate option instead of CKS4 is for the version 4 and SOCKS4a is for the version 4 without resolving the host name locally:curl -x socks4 curl –socks4 SOCKS4a versions:curl -x socks4a curl –socks4a is for the version 5 and SOCKS5-hostname is for the version 5 without resolving the host name locally:curl -x socks5 curl –socks5 SOCKS5-hostname versions. This sends the host name to the server so there’s no name resolving done locally:curl -x socks5h curl –socks5-hostname proxies can require authentication, so curl then needs to provide the proper credentials to the proxy to be allowed to use it, and failing to do will only make the proxy return HTTP responses using code thentication for proxies is similar to “normal” HTTP authentication. It is separate from the server authentication to allow clients to independently use both normal host authentication as well as proxy curl, you set the user name and password for the proxy authentication with the -U user:password or –proxy-user user:password option:curl -U daniel:secr3t -x myproxy:80 example will default to using the Basic authentication scheme. Some proxies will require another authentication scheme (and the headers that are returned when you get a 407 response will tell you which) and then you can ask for a specific method with –proxy-digest, –proxy-negotiate, –proxy-ntlm. The above example command again, but asking for NTLM auth with the proxy:curl -U daniel:secr3t -x myproxy:80 –proxy-ntlmThere’s also the option that asks curl to figure out which method the proxy wants and supports and then go with that (with the possible expense of extra roundtrips) using –proxy-anyauth. Asking curl to use any method the proxy wants is then like this:curl -U daniel:secr3t -x myproxy:80 –proxy-anyauthAll the previously mentioned protocols to speak with the proxy are clear text protocols, HTTP and the SOCKS versions. Using these methods could allow someone to eavesdrop on your traffic the local network where you or the proxy solution for that is to use HTTPS to the proxy, which then establishes a secure and encrypted connection that is safe from easy a HTTPS proxy is specified, the default port will be checks for the existence of specially named environment variables before it runs to see if a proxy is requested to get specify the proxy by setting a variable named [scheme]_proxy to hold the proxy host name (the same way you would specify the host with -x). So if you want to tell curl to use a proxy when access a HTTP server, you set the ‘_proxy’ environment variable. Like this__proxy= -v the above example shows HTTP, you can, of course, also set ftp_proxy, _proxy, and so on. All these proxy environment variable names except _proxy can also be specified in uppercase, like set a single variable that controls all protocols, the ALL_PROXY exists. If a specific protocol variable one exists, such a one will take using environment variables to set a proxy, you could easily end up in a situation where one or a few host names should be excluded from going through the proxy. This is then done with the NO_PROXY variable. Set that to a comma- separated list of host names that should not use a proxy when being accessed. You can set NO_PROXY to be a single asterisk (‘*’) to match all an alternative to the NO_PROXY variable, there’s also a –noproxy command line option that serves the same purpose and works the same HTTP version of the proxy environment variables is treated differently than the others. It is only accepted in its lower case version because of the CGI protocol, which lets users run scripts in a server when invoked by an HTTP server. When a CGI script is invoked by a server, it automatically creates environment variables for the script based on the incoming headers in the request. Those environment variables are prefixed with uppercase HTTP_! An incoming request to a HTTP server using a request header like Proxy: yada will therefore create the environment variable HTTP_PROXY set to contain yada before the CGI script is started. If that CGI script runs cepting the upper case version of this environment variable has been the source for many security problems in lots of software through you want to add HTTP headers meant specifically for a proxy and not for the remote server, the –header option falls example, if you issue a HTTPS request through a HTTP proxy, it will be done by first issuing a CONNECT to the proxy that establishes a tunnel to the remote server and then it sends the request to that server. That first CONNECT is only issued to the proxy and you may want to make sure only that receives your special header, and send another set of custom headers to the remote a specific different User-Agent: only to the proxy:curl –proxy-header “User-Agent: magic/3000” -x proxy
Frequently Asked Questions about curl proxy
What is curl proxy?
An HTTP proxy is a proxy that the client speaks HTTP with to get the transfer done. curl will, by default, assume that a host you point out with -x or –proxy is an HTTP proxy, and unless you also specify a port number it will default to port 1080 (and the reason for that particular port number is purely historical).
Is curl using proxy?
Using cURL with HTTP/HTTPS proxy If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address. There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.Aug 9, 2021
How do I use a proxy to curl?
To use a proxy with Curl, you must pass the required proxy address using the -x (or –proxy) command-line option and proxy credentials using the -U (or –proxy-user) command-line switch. Proxy credentials may also be passed in the proxy string and will be URL decoded by Curl.Oct 16, 2021
