Crawlera Api Key
Smart Proxy Manager API – Zyte documentation
Note
To fetch HTTPS web pages you need to download and install the certificate file or disable SSL certificate verification. See Smart Proxy Manager with HTTPS in our Knowledge Base.
Warning
Proxy-Authorization header is required on 8010, 8011 and 8014 ports otherwise HTTP 407 response is returned.
Proxy API¶
Smart Proxy Manager works with a standard HTTP web proxy API, where you only need an API
key for authentication. This is the standard way to perform a request via
Smart Proxy Manager:
curl -vx -U
Errors¶
See Errors Reference.
Sessions¶
See Sessions
Using Smart Proxy Manager with headless browsers¶
See our articles in the Knowledge Base:
Using Smart Proxy Manager Headless Proxy
Using Smart Proxy Manager with Splash
Using Smart Proxy Manager with Selenium
Using Smart Proxy Manager with Selenium Wire
Using Smart Proxy Manager with PhantomJS
Using Smart Proxy Manager with Puppeteer
Using Smart Proxy Manager with Playwright
Using Smart Proxy Manager from different languages¶
Use our code examples provided in Smart Proxy Manager Integrations.
Accessing HTTPS URLs¶
See Smart Proxy Manager with HTTPS in our Knowledge Base
Working with Cookies¶
See Smart Proxy Manager and Cookies in our Knowledge Base
HTTPS Proxy Endpoint¶
As described in curl:An HTTPS proxy receives all transactions over an SSL/TLS connection.
Once a secure connection with the proxy is established, the user agent
uses the proxy as usual, including sending CONNECT requests to instruct
the proxy to establish a [usually secure] TCP tunnel with an origin
server. HTTPS proxies protect nearly all aspects of user-proxy
communications as opposed to HTTP proxies that receive all requests
(including CONNECT requests) in vulnerable clear text.
Smart Proxy Manager supports a HTTPS Proxy interface on port 8014.
Here is an example of how to use it with curl:
You can use Smart Proxy Manager HTTPS Proxy with all HTTP clients that support HTTPS proxies.
In case of requiring certificate with PKCS#12 format, you can generate it with the following openssl command:
openssl pkcs12 -export -nokeys -password pass:changeit -in -out zyte-smartproxy-ca. p12
Settings — scrapy-crawlera 1.5.1 documentation
This Middleware adds some settings to configure how to work with Crawlera.
CRAWLERA_APIKEY¶
Default: None
Unique Crawlera API Key provided for authentication.
CRAWLERA_URL¶
Default: ”
Crawlera instance url, it varies depending on adquiring a private or dedicated instance. If Crawlera didn’t provide
you with a private instance url, you don’t need to specify it.
CRAWLERA_MAXBANS¶
Default: 400
Number of consecutive bans from Crawlera necessary to stop the spider.
CRAWLERA_DOWNLOAD_TIMEOUT¶
Default: 190
Timeout for processing Crawlera requests. It overrides Scrapy’s DOWNLOAD_TIMEOUT.
CRAWLERA_PRESERVE_DELAY¶
Default: False
If False Sets Scrapy’s DOWNLOAD_DELAY to 0, making the spider to crawl faster. If set to True, it will
respect the provided DOWNLOAD_DELAY from Scrapy.
CRAWLERA_BACKOFF_STEP¶
Default: 15
Step size used for calculating exponential backoff according to the formula: random. uniform(0, min(max, step * 2 ** attempt)).
CRAWLERA_BACKOFF_MAX¶
Default: 180
Max value for exponential backoff as showed in the formula above.
How to use Zyte Smart Proxy Manager (formerly Crawlera …
Zyte Smart Proxy Manager (formerly Crawlera) is a proxy service, specifically designed for web scraping. In this article, you are going to learn how to use Zyte Smart Proxy Manager inside your Scrapy Smart Proxy Manager is a smart HTTP/HTTPS downloader. When you make requests using Zyte Smart Proxy Manager it routes them through a pool of IP addresses. When necessary, it automatically introduces delays between requests and discards IP addresses to help manage crawling challenges. And simply like that, it makes a successful request order to use Zyte Smart Proxy Manager, you need to have an account with a Zyte Smart Proxy Manager subscription. If you haven’t signed up yet you can sign up here, it’s free. When you subscribe to a plan you will get an API key. You will need to use this API key in your Scrapy project to use Zyte Smart Proxy thing you need to do is to install the Zyte Smart Proxy Manager middleware:pip install scrapy-crawleraNext, add these lines to the project settings:# enable the middleware DOWNLOADER_MIDDLEWARES = {‘awleraMiddleware’: 610} # enable crawlera CRAWLERA_ENABLED = True # the APIkey you get with your subscription CRAWLERA_APIKEY = ‘
