Using Scrapy Python
Scrapy Tutorial — Scrapy 2.5.1 documentation
In this tutorial, we’ll assume that Scrapy is already installed on your system.
If that’s not the case, see Installation guide.
We are going to scrape, a website
that lists quotes from famous authors.
This tutorial will walk you through these tasks:
Creating a new Scrapy project
Writing a spider to crawl a site and extract data
Exporting the scraped data using the command line
Changing spider to recursively follow links
Using spider arguments
Scrapy is written in Python. If you’re new to the language you might want to
start by getting an idea of what the language is like, to get the most out of
Scrapy.
If you’re already familiar with other languages, and want to learn Python quickly, the Python Tutorial is a good resource.
If you’re new to programming and want to start with Python, the following books
may be useful to you:
Automate the Boring Stuff With Python
How To Think Like a Computer Scientist
Learn Python 3 The Hard Way
You can also take a look at this list of Python resources for non-programmers,
as well as the suggested resources in the learnpython-subreddit.
Creating a project¶
Before you start scraping, you will have to set up a new Scrapy project. Enter a
directory where you’d like to store your code and run:
scrapy startproject tutorial
This will create a tutorial directory with the following contents:
tutorial/
# deploy configuration file
tutorial/ # project’s Python module, you’ll import your code from here
# project items definition file
# project middlewares file
# project pipelines file
# project settings file
spiders/ # a directory where you’ll later put your spiders
Our first Spider¶
Spiders are classes that you define and that Scrapy uses to scrape information
from a website (or a group of websites). They must subclass
Spider and define the initial requests to make,
optionally how to follow links in the pages, and how to parse the downloaded
page content to extract data.
This is the code for our first Spider. Save it in a file named
under the tutorial/spiders directory in your project:
import scrapy
class QuotesSpider():
name = “quotes”
def start_requests(self):
urls = [
”,
”, ]
for url in urls:
yield quest(url=url, )
def parse(self, response):
page = (“/”)[-2]
filename = f’quotes-{page}’
with open(filename, ‘wb’) as f:
()
(f’Saved file {filename}’)
As you can see, our Spider subclasses
and defines some attributes and methods:
name: identifies the Spider. It must be
unique within a project, that is, you can’t set the same name for different
Spiders.
start_requests(): must return an iterable of
Requests (you can return a list of requests or write a generator function)
which the Spider will begin to crawl from. Subsequent requests will be
generated successively from these initial requests.
parse(): a method that will be called to handle
the response downloaded for each of the requests made. The response parameter
is an instance of TextResponse that holds
the page content and has further helpful methods to handle it.
The parse() method usually parses the response, extracting
the scraped data as dicts and also finding new URLs to
follow and creating new requests (Request) from them.
How to run our spider¶
To put our spider to work, go to the project’s top level directory and run:
This command runs the spider with name quotes that we’ve just added, that
will send some requests for the domain. You will get an output
similar to this:… (omitted for brevity)
2016-12-16 21:24:05 [] INFO: Spider opened
2016-12-16 21:24:05 [scrapy. extensions. logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:24:05 [] DEBUG: Telnet console listening on 127. 0. 1:6023
2016-12-16 21:24:05 [] DEBUG: Crawled (404)
2016-12-16 21:24:05 [] DEBUG: Crawled (200)
2016-12-16 21:24:05 [quotes] DEBUG: Saved file
2016-12-16 21:24:05 [] INFO: Closing spider (finished)…
Now, check the files in the current directory. You should notice that two new
files have been created: and, with the content
for the respective URLs, as our parse method instructs.
Note
If you are wondering why we haven’t parsed the HTML yet, hold
on, we will cover that soon.
What just happened under the hood? ¶
Scrapy schedules the quest objects
returned by the start_requests method of the Spider. Upon receiving a
response for each one, it instantiates Response objects
and calls the callback method associated with the request (in this case, the
parse method) passing the response as argument.
A shortcut to the start_requests method¶
Instead of implementing a start_requests() method
that generates quest objects from URLs,
you can just define a start_urls class attribute
with a list of URLs. This list will then be used by the default implementation
of start_requests() to create the initial requests
for your spider:
start_urls = [
The parse() method will be called to handle each
of the requests for those URLs, even though we haven’t explicitly told Scrapy
to do so. This happens because parse() is Scrapy’s
default callback method, which is called for requests without an explicitly
assigned callback.
Storing the scraped data¶
The simplest way to store the scraped data is by using Feed exports, with the following command:
scrapy crawl quotes -O
That will generate a file containing all scraped items,
serialized in JSON.
The -O command-line switch overwrites any existing file; use -o instead
to append new content to any existing file. However, appending to a JSON file
makes the file contents invalid JSON. When appending to a file, consider
using a different serialization format, such as JSON Lines:
scrapy crawl quotes -o
The JSON Lines format is useful because it’s stream-like, you can easily
append new records to it. It doesn’t have the same problem of JSON when you run
twice. Also, as each record is a separate line, you can process big files
without having to fit everything in memory, there are tools like JQ to help
doing that at the command-line.
In small projects (like the one in this tutorial), that should be enough.
However, if you want to perform more complex things with the scraped items, you
can write an Item Pipeline. A placeholder file
for Item Pipelines has been set up for you when the project is created, in
tutorial/ Though you don’t need to implement any item
pipelines if you just want to store the scraped items.
Following links¶
Let’s say, instead of just scraping the stuff from the first two pages
from, you want quotes from all the pages in the website.
Now that you know how to extract data from pages, let’s see how to follow links
from them.
First thing is to extract the link to the page we want to follow. Examining
our page, we can see there is a link to the next page with the following
markup:
We can try extracting it in the shell:
>>> (‘ a’)()
‘Next ‘
This gets the anchor element, but we want the attribute href. For that,
Scrapy supports a CSS extension that lets you select the attribute contents,
like this:
>>> (‘ a::attr(href)’)()
‘/page/2/’
There is also an attrib property available
(see Selecting element attributes for more):
>>> (‘ a’)[‘href’]
Let’s see now our spider modified to recursively follow the link to the next
page, extracting data from it:
for quote in (”):
yield {
‘text’: (”)(),
‘author’: (”)(),
‘tags’: (‘ ‘)(), }
next_page = (‘ a::attr(href)’)()
if next_page is not None:
next_page = response. urljoin(next_page)
yield quest(next_page, )
Now, after extracting the data, the parse() method looks for the link to
the next page, builds a full absolute URL using the
urljoin() method (since the links can be
relative) and yields a new request to the next page, registering itself as
callback to handle the data extraction for the next page and to keep the
crawling going through all the pages.
What you see here is Scrapy’s mechanism of following links: when you yield
a Request in a callback method, Scrapy will schedule that request to be sent
and register a callback method to be executed when that request finishes.
Using this, you can build complex crawlers that follow links according to rules
you define, and extract different kinds of data depending on the page it’s
visiting.
In our example, it creates a sort of loop, following all the links to the next page
until it doesn’t find one – handy for crawling blogs, forums and other sites with
pagination.
A shortcut for creating Requests¶
As a shortcut for creating Request objects you can use
‘author’: (‘span small::text’)(),
yield (next_page, )
Unlike quest, supports relative URLs directly – no
need to call urljoin. Note that just returns a Request
instance; you still have to yield this Request.
You can also pass a selector to instead of a string;
this selector should extract necessary attributes:
for href in (‘ a::attr(href)’):
yield (href, )
For elements there is a shortcut: uses their href
attribute automatically. So the code can be shortened further:
for a in (‘ a’):
yield (a, )
To create multiple requests from an iterable, you can use
llow_all instead:
anchors = (‘ a’)
yield from llow_all(anchors, )
or, shortening it further:
yield from llow_all(css=’ a’, )
More examples and patterns¶
Here is another spider that illustrates callbacks and following links,
this time for scraping author information:
class AuthorSpider():
name = ‘author’
start_urls = [”]
author_page_links = (‘ + a’)
yield from llow_all(author_page_links, rse_author)
pagination_links = (‘ a’)
yield from llow_all(pagination_links, )
def parse_author(self, response):
def extract_with_css(query):
return (query)(default=”)()
‘name’: extract_with_css(”),
‘birthdate’: extract_with_css(”),
‘bio’: extract_with_css(”), }
This spider will start from the main page, it will follow all the links to the
authors pages calling the parse_author callback for each of them, and also
the pagination links with the parse callback as we saw before.
Here we’re passing callbacks to
llow_all as positional
arguments to make the code shorter; it also works for
Request.
The parse_author callback defines a helper function to extract and cleanup the
data from a CSS query and yields the Python dict with the author data.
Another interesting thing this spider demonstrates is that, even if there are
many quotes from the same author, we don’t need to worry about visiting the
same author page multiple times. By default, Scrapy filters out duplicated
requests to URLs already visited, avoiding the problem of hitting servers too
much because of a programming mistake. This can be configured by the setting
DUPEFILTER_CLASS.
Hopefully by now you have a good understanding of how to use the mechanism
of following links and callbacks with Scrapy.
As yet another example spider that leverages the mechanism of following links,
check out the CrawlSpider class for a generic
spider that implements a small rules engine that you can use to write your
crawlers on top of it.
Also, a common pattern is to build an item with data from more than one page,
using a trick to pass additional data to the callbacks.
Using spider arguments¶
You can provide command line arguments to your spiders by using the -a
option when running them:
scrapy crawl quotes -O -a tag=humor
These arguments are passed to the Spider’s __init__ method and become
spider attributes by default.
In this example, the value provided for the tag argument will be available
via You can use this to make your spider fetch only quotes
with a specific tag, building the URL based on the argument:
url = ”
tag = getattr(self, ‘tag’, None)
if tag is not None:
url = url + ‘tag/’ + tag
yield quest(url, )
‘author’: (”)(), }
If you pass the tag=humor argument to this spider, you’ll notice that it
will only visit URLs from the humor tag, such as
You can learn more about handling spider arguments here.
Next steps¶
This tutorial covered only the basics of Scrapy, but there’s a lot of other
features not mentioned here. Check the What else? section in
Scrapy at a glance chapter for a quick overview of the most important ones.
You can continue from the section Basic concepts to know more about the
command-line tool, spiders, selectors and other things the tutorial hasn’t covered like
modeling the scraped data. If you prefer to play with an example project, check
the Examples section.
Making Web Crawlers Using Scrapy for Python – DataCamp
If you would like an overview of web scraping in Python, take DataCamp’s Web Scraping with Python course.
In this tutorial, you will learn how to use Scrapy which is a Python framework using which you can handle large amounts of data! You will learn Scrapy by building a web scraper for which is an e-commerce website. Let’s get scrapping!
Scrapy Overview
Scrapy Vs. BeautifulSoup
Scrapy Installation
Scrapy Shell
Creating a project and Creating a custom spider
A basic HTML and CSS knowledge will help you understand this tutorial with greater ease and speed. Read this article for a fresher on HTML and CSS.
Source
Web scraping has become an effective way of extracting information from the web for decision making and analysis. It has become an essential part of the data science toolkit. Data scientists should know how to gather data from web pages and store that data in different formats for further analysis.
Any web page you see on the internet can be crawled for information and anything visible on a web page can be extracted [2]. Every web page has its own structure and web elements that because of which you need to write your web crawlers/spiders according to the web page being extracted.
Scrapy provides a powerful framework for extracting the data, processing it and then save it.
Scrapy uses spiders, which are self-contained crawlers that are given a set of instructions [1]. In Scrapy it is easier to build and scale large crawling projects by allowing developers to reuse their code.
In this section, you will have an overview of one of the most popularly used web scraping tool called BeautifulSoup and its comparison to Scrapy.
Scrapy is a Python framework for web scraping that provides a complete package for developers without worrying about maintaining code.
Beautiful Soup is also widely used for web scraping. It is a Python package for parsing HTML and XML documents and extract data from them. It is available for Python 2. 6+ and Python 3.
Here are some differences between them in a nutshell:
Scrapy
BeautifulSoup
Functionality
—
Scrapy is the complete package for downloading web pages, processing them and save it in files and databases
BeautifulSoup is basically an HTML and XML parser and requires additional libraries such as requests, urlib2 to open URLs and store the result [6]
Learning Curve
Scrapy is a powerhouse for web scraping and offers a lot of ways to scrape a web page. It requires more time to learn and understand how Scrapy works but once learned, eases the process of making web crawlers and running them from just one line of command. Becoming an expert in Scrapy might take some practice and time to learn all functionalities.
BeautifulSoup is relatively easy to understand for newbies in programming and can get smaller tasks done in no time
Speed and Load
Scrapy can get big jobs done very easily. It can crawl a group of URLs in no more than a minute depending on the size of the group and does it very smoothly as it uses Twister which works asynchronously (non-blocking) for concurrency.
BeautifulSoup is used for simple scraping jobs with efficiency. It is slower than Scrapy if you do not use multiprocessing.
Extending functionality
Scrapy provides Item pipelines that allow you to write functions in your spider that can process your data such as validating data, removing data and saving data to a database. It provides spider Contracts to test your spiders and allows you to create generic and deep crawlers as well. It allows you to manage a lot of variables such as retries, redirection and so on.
If the project does not require much logic, BeautifulSoup is good for the job, but if you require much customization such as proxys, managing cookies, and data pipelines, Scrapy is the best option.
Information: Synchronous means that you have to wait for a job to finish to start a new job while Asynchronous means you can move to another job before the previous job has finished
Here is an interesting DataCamp BeautifulSoup tutorial to learn.
With Python 3. 0 (and onwards) installed, if you are using anaconda, you can use conda to install scrapy. Write the following command in anaconda prompt:
conda install -c conda-forge scrapy
To install anaconda, look at these DataCamp tutorials for Mac and Windows.
Alternatively, you can use Python Package Installer pip. This works for Linux, Mac, and Windows:
pip install scrapy
Scrapy also provides a web-crawling shell called as Scrapy Shell, that developers can use to test their assumptions on a site’s behavior. Let us take a web page for tablets at AliExpress e-commerce website. You can use the Scrapy shell to see what components the web page returns and how you can use them to your requirements.
Open your command line and write the following command:
scrapy shell
If you are using anaconda, you can write the above command at the anaconda prompt as well. Your output on the command line or anaconda prompt will be something like this:
You have to run a crawler on the web page using the fetch command in the Scrapy shell. A crawler or spider goes through a webpage downloading its text and metadata.
fetch()
Note: Always enclose URL in quotes, both single and double quotes work
The output will be as follows:
The crawler returns a response which can be viewed by using the view(response) command on shell:
view(response)
And the web page will be opened in the default browser.
You can view the raw HTML script by using the following command in Scrapy shell:
print()
You will see the script that’s generating the webpage. It is the same content that when you left right-click any blank area on a webpage and click view source or view page source. Since, you need only relevant information from the entire script, using browser developer tools you will inspect the required element. Let us take the following elements:
Tablet name
Tablet price
Number of orders
Name of store
Right-click on the element you want and click inspect like below:
Developer tools of the browser will help you a lot with web scraping. You can see that it is an tag with a class product and the text contains the name of the product:
You can extract this using the element attributes or the css selector like classes. Write the following in the Scrapy shell to extract the product name:
(“. product::text”). extract_first()
The output will be:
extract_first() extract the first element that satisfies the css selector. If you want to extract all the product names use extract():
(“. extract()
Following code will extract price range of the products:
(“”). extract()
Similarly, you can try with a number of orders and the name of the store.
XPath is a query language for selecting nodes in an XML document [7]. You can navigate through an XML document using XPath. Behind the scenes, Scrapy uses Xpath to navigate to HTML document items. The CSS selectors you used above are also converted to XPath, but in many cases, CSS is very easy to use. But you should know how the XPath in Scrapy works.
Go to your Scrapy Shell and write fetch() the same way as before. Try out the following code snippets [3]:
(‘/html’). extract()
This will show you all the code under the tag. / means direct child of the node. If you want to get the
(‘/html//div’). extract()
For XPath, you must learn to understand the use of / and // to know how to navigate through child and descendent nodes. Here is a helpful tutorial for XPath Nodes and some examples to try out.
If you want to get all
(“//div”). extract()
You can further filter your nodes that you start from and reach your desired nodes by using attributes and their values. Below is the syntax to use classes and their values.
(“//div[@class=’quote’]/span[@class=’text’]”). extract()
(“//div[@class=’quote’]/span[@class=’text’]/text()”). extract()
Use text() to extract all text inside nodes
Consider the following HTML code:
You want to get the text inside the tag, which is child node of
(‘//div[@class=”site-notice-container container”]/a[@class=”notice-close”]/text()’). extract()
Creating a Scrapy project and Custom Spider
Web scraping can be used to make an aggregator that you can use to compare data. For example, you want to buy a tablet, and you want to compare products and prices together you can crawl your desired pages and store in an excel file. Here you will be scraping for tablets information.
Now, you will create a custom spider for the same page. First, you need to create a Scrapy project in which your code and results will be stored. Write the following command in the command line or anaconda prompt.
scrapy startproject aliexpress
This will create a hidden folder in your default python or anaconda installation. aliexpress will be the name of the folder. You can give any name. You can view the folder contents directly through explorer. Following is the structure of the folder:
file/folder
Purpose
deploy configuration file
aliexpress/
Project’s Python module, you’ll import your code from here
__
Initialization file
project items file
project pipelines file
project settings file
spiders/
a directory where you’ll later put your spiders
Once you have created the project you will change to the newly created directory and write the following command:
[scrapy genspider aliexpress_tablets]()
This creates a template file named in the spiders directory as discussed above. The code in that file is as below:
import scrapy
class AliexpressTabletsSpider():
name = ‘aliexpress_tablets’
allowed_domains = [”]
start_urls = [”]
def parse(self, response):
pass
In the above code you can see name, allowed_domains, sstart_urls and a parse function.
name: Name is the name of the spider. Proper names will help you keep track of all the spider’s you make. Names must be unique as it will be used to run the spider when scrapy crawl name_of_spider is used.
allowed_domains (optional): An optional python list, contains domains that are allowed to get crawled. Request for URLs not in this list will not be crawled. This should include only the domain of the website (Example:) and not the entire URL specified in start_urls otherwise you will get warnings.
start_urls: This requests for the URLs mentioned. A list of URLs where the spider will begin to crawl from, when no particular URLs are specified [4]. So, the first pages downloaded will be those listed here. The subsequent Request will be generated successively from data contained in the start URLs [4].
parse(self, response): This function will be called whenever a URL is crawled successfully. It is also called the callback function. The response (used in Scrapy shell) returned as a result of crawling is passed in this function, and you write the extraction code inside it!
Information: You can use BeautifulSoup inside parse() function of the Scrapy spider to parse the html document.
Note: You can extract data through css selectors using () as discussed in scrapy shell section but also using XPath (XML) that allows you to access child elements. You will see the example of () in the code edited in pass() function.
You will make changes to the file. I have added another URL in start_urls. You can add the extraction logic to the pass() function as below:
# -*- coding: utf-8 -*-
start_urls = [”,
”]
print(“procesing:”)
#Extract data using css selectors
(‘. product::text’). extract()
(”). extract()
#Extract data using xpath
(“//em[@title=’Total Orders’]/text()”). extract()
(“//a[@class=’store $p4pLog’]/text()”). extract()
row_data=zip(product_name, price_range, orders, company_name)
#Making extracted data row wise
for item in row_data:
#create a dictionary to store the scraped info
scraped_info = {
#key:value
‘page’,
‘product_name’: item[0], #item[0] means product in the list and so on, index tells what value to assign
‘price_range’: item[1],
‘orders’: item[2],
‘company_name’: item[3], }
#yield or give the scraped info to scrapy
yield scraped_info
Information: zip() takes n number of iterables and returns a list of tuples. ith element of the tuple is created using the ith element from each of the iterables. [8]
The yield keyword is used whenever you are defining a generator function. A generator function is just like a normal function except it uses yield keyword instead of return. The yield keyword is used whenever the caller function needs a value and the function containing yield will retain its local state and continue executing where it left off after yielding value to the caller function. Here yield gives the generated dictionary to Scrapy which will process and save it!
Now you can run the spider:
scrapy crawl aliexpress_tablets
You will see a long output at the command line like below:
Exporting data
You will need data to be presented as a CSV or JSON so that you can further use the data for analysis. This section of the tutorial will take you through how you can save CSV and JSON file for this data.
To save a CSV file, open from the project directory and add the following lines:
FEED_FORMAT=”csv”
FEED_URI=””
After saving the, rerun the scrapy crawl aliexpress_tablets in your project directory.
The CSV file will look like:
Note: Everytime you run the spider it will append the file.
FEED_FORMAT [5]: This sets the format you want to store the data. Supported formats are: + JSON
+ CSV
+ JSON Lines
+ XML
FEED_URI [5]: This gives the location of the file. You can store a file on your local file storage or an FTP as well.
Scrapy’s Feed Export can also add a timestamp and the name of spider to your file name, or you can use these to identify a directory in which you want to store. %(time)s: gets replaced by a timestamp when the feed is being created [5]%(name)s: gets replaced by the spider name [5]
For Example:
Store in FTP using one directory per spider [5]:
ftpuser:[email protected]/scraping/feeds/%(name)s/%(time)
The Feed changes you make in will apply to all spiders in the project. You can also set custom settings for a particular spider that will override the settings in the file.
custom_settings={ ‘FEED_URI’: “aliexpress_%(time)”,
‘FEED_FORMAT’: ‘json’}
#yield or give the scraped info to Scrapy
returns the URL of the page from which response is generated. After running the crawler using scrapy crawl aliexpress_tablets you can view the json file:
Following Links
You must have noticed, that there are two links in the start_urls. The second link is the page 2 of the same tablets search results. It will become impractical to add all links. A crawler should be able to crawl by itself through all the pages, and only the starting point should be mentioned in the start_urls.
If a page has subsequent pages, you will see a navigator for it at the end of the page that will allow moving back and forth the pages. In the case you have been implementing in this tutorial, you will see it like this:
Here is the code that you will see:
As you can see that under there is a tag with class class that is the current page you are on, and under that are all tags with links to the next page. Everytime you will have to get the tags after this tag. Here comes a little bit of CSS! In this, you have to get sibling node and not a child node, so you have to make a css selector that tells the crawler to find tags that are after tag with class.
Remember! Each web page has its own structure. You will have to study the structure a little bit on how you can get the desired element. Always try out (SELECTOR) on Scrapy Shell before writing them in code.
Modify your as below:
‘FEED_FORMAT’: ‘csv’}
NEXT_PAGE_SELECTOR = ‘ + a::attr(href)’
next_page = (NEXT_PAGE_SELECTOR). extract_first()
if next_page:
yield quest(
response. urljoin(next_page), )
In the above code:
you first extracted the link of the next page using next_page = (NEXT_PAGE_SELECTOR). extract_first() and then if the variable next_page gets a link and is not empty, it will enter the if body.
response. urljoin(next_page): The parse() method will use this method to build a new url and provide a new request, which will be sent later to the callback. [9]
After receiving the new URL, it will scrape that link executing the for body and again look for the next page. This will continue until it doesn’t get a next page link.
Here you might want to sit back and enjoy your spider scraping all the pages. The above spider will extract from all subsequent pages. That will be a lot of scraping! But your spider will do it! Below you can see the size of the file has reached 1. 1MB.
Scrapy does it for you!
In this tutorial, you have learned about Scrapy, how it compares to BeautifulSoup, Scrapy Shell and how to write your own spiders in Scrapy. Scrapy handles all the heavy load of coding for you, from creating project files and folders till handling duplicate URLs it helps you get heavy-power web scraping in minutes and provides you support for all common data formats that you can further input in other programs. This tutorial will surely help you understand Scrapy and its framework and what you can do with it. To become a master in Scrapy, you will need to go through all the fantastic functionalities it has to provide, but this tutorial has made you capable of scraping groups of web pages in an efficient way.
For further reading, you can refer to Offical Scrapy Docs.
Also, don’t forget to check out DataCamp’s Web Scraping with Python course.
References
[1] [2] [3] [4] [5] [6] [7] [8] [9]
Implementing Web Scraping in Python with Scrapy – GeeksforGeeks
Nowadays data is everything and if someone wants to get data from webpages then one way to use an API or implement Web Scraping techniques. In Python, Web scraping can be done easily by using scraping tools like BeautifulSoup. But what if the user is concerned about performance of scraper or need to scrape data overcome this problem, one can make use of MultiThreading/Multiprocessing with BeautifulSoup module and he/she can create spider, which can help to crawl over a website and extract data. In order to save the time one use the help of Scrapy one can:
1. Fetch millions of data efficiently
2. Run it on server
3. Fetching data
4. Run spider in multiple processesScrapy comes with whole new features of creating spider, running it and then saving data easily by scraping it. At first it looks quite confusing but it’s for the ’s talk about the installation, creating a spider and then testing 1: Creating virtual environmentIt is good to create one virtual environment as it isolates the program and doesn’t affect any other programs present in the machine. To create virtual environment first install it by using:sudo apt-get install python3-venvCreate one folder and then activate it:mkdir scrapy-project && cd scrapy-project
python3 -m venv myvenv
If above command gives Error then try this:python3. 5 -m venv myvenvAfter creating virtual environment activate it by using:source myvenv/bin/activate Step 2: Installing Scrapy moduleInstall Scrapy by using:pip install scrapyTo install scrapy for any specific version of python:python3. 5 -m pip install scrapyReplace 3. 5 version with some other version like 3. 6. Step 3: Creating Scrapy projectWhile working with Scrapy, one needs to create scrapy startproject gfgIn Scrapy, always try to create one spider which helps to fetch data, so to create one, move to spider folder and create one python file over there. Create one spider with name python file. Step 4: Creating SpiderMove to the spider folder and create While creating spider, always create one class with unique name and define requirements. First thing is to name the spider by assigning it with name variable and then provide the starting URL through which spider will start crawling. Define some methods which helps to crawl much deeper into that website. For now, let’s scrap all the URL present and store all those scrapyclass ExtractUrls(): name = “extract” def start_requests(self): for url in urls: yield quest(url = url, callback =)Main motive is to get each url and then request it. Fetch all the urls or anchor tags from it. To do this, we need to create one more method parse, to fetch data from the given url. Step 5: Fetching data from given pageBefore writing parse function, test few things like how to fetch any data from given page. To do this make use of scrapy shell. It is just like python interpreter but with the ability to scrape data from the given url. In short, its a python interpreter with Scrapy shell URLNote: Make sure to in the same directory where is present, else it will not shell for fetching data from the given page, use selectors. These selectors can be either from CSS or from Xpath. For now, let’s try to fetch all url by using CSS get anchor tag (‘a’)To extract the data:links = (‘a’). extract()For example, links[0] will show something like this:’GeeksforGeeks‘To get href attribute, use attributes = (‘a::attr(href)’). extract()This will get all the href data which is very useful. Make use of this link and start requesting, let’s create parse method and fetch all the urls and then yield it. Follow that particular URL and fetch more links from that page and this will keep on happening again and again. In short, we are fetching all url present on that, by default, filters those url which has already been visited. So it will not crawl the same url path again. But it’s possible that in two different pages there are two or more than two similar links. For example, in each page, the header link will be available which means that this header link will come in each page request. So try to exclude it by checking parse(self, response): title = (‘title::text’). extract_first() links = (‘a::attr(href)’). extract() for link in links: yield { ‘title’: title, ‘links’: link} if ‘geeksforgeeks’ in link: yield quest(url = link, callback =) Below is the implementation of scraper:import scrapyclass ExtractUrls(): name = “extract” def start_requests(self): for url in urls: yield quest(url = url, callback =) def parse(self, response): title = (‘title::text’). extract() for link in links: yield { ‘title’: title, ‘links’: link} if ‘geeksforgeeks’ in link: yield quest(url = link, callback =) Step 6: In last step, Run the spider and get output in simple json filescrapy crawl NAME_OF_SPIDER -o links. jsonHere, name of spider is “extract” for given example. It will fetch loads of data within few: Note: Scraping any web page is not a legal activity. Don’t perform any scraping operation without permission. Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics. To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course
Frequently Asked Questions about using scrapy python
How do you use Scrapy in python?
While working with Scrapy, one needs to create scrapy project. In Scrapy, always try to create one spider which helps to fetch data, so to create one, move to spider folder and create one python file over there. Create one spider with name gfgfetch.py python file. Move to the spider folder and create gfgfetch.py .Nov 8, 2019
Is Scrapy better than Beautifulsoup?
Performance. Due to the built-in support for generating feed exports in multiple formats, as well as selecting and extracting data from various sources, the performance of Scrapy can be said to be faster than Beautiful Soup. Working with Beautiful Soup can speed up with the help of Multithreading process.Apr 8, 2020
How do you use Scrapy tutorial?
Scrapy TutorialCreating a new Scrapy project.Writing a spider to crawl a site and extract data.Exporting the scraped data using the command line.Changing spider to recursively follow links.Using spider arguments.Apr 7, 2021
