How To Program A Web Crawler
How to build a web crawler? – Scraping-bot.io
At the era of big data, web scraping is a life saver. To save even more time, you can couple ScrapingBot to a web crawling bot.
What is a web crawler?
A crawler, or spider, is an internet bot indexing and visiting every URLs it encounters. Its goal is to visit a website from end to end, know what is on every webpage and be able to find the location of any information. The most known web crawlers are the search engine ones, the GoogleBot for example. When a website is online, those crawlers will visit it and read its content to display it in the relevant search result pages.
How does a web crawler work?
Starting from the root URL or a set of entries, the crawler will fetch the webpages and find other URLs to visit, called seeds, in this page. All the seeds found on this page will be added on its list of URLs to be visited. This list is called the horizon. The crawler organises the links in two threads: ones to visit, and already visited ones. It will keep visiting the links until the horizon is empty.
Because the list of seeds can be very long, the crawler has to organise those following several criterias, and prioritise which ones to visit first and revisit. To know which pages are more important to crawl, the bot will consider how many links go to this URL, how often it is visited by regular users.
What is the difference between a web scraper and a web crawler?
Crawling, by definition, always implies the web. A crawler’s purpose is to follow links to reach numerous pages and analyze their meta data and content.
Scraping is possible out of the web. For example you can retrieve some information from a database. Scraping is pulling data from the web or a database.
Why do you need a web crawler?
With web scraping, you gain a huge amount of time, by automatically retrieving the information you need instead of looking for it and copying it manually. However, you still need to scrape page after page. Web crawling allows you to collect, organize and visit all the pages present on the root page, with the possibility to exclude some links. The root page can be a search result or category.
For example, you can pick a product category or a search result page from amazon as an entry, and crawl it to scrape all the product details, and limit it to the first 10 pages with the suggested products as well.
How to build a web crawler?
The first thing you need to do is threads:
Visited URLsURLs to be visited (queue)
To avoid crawling the same page over and over, the URL needs to automatically move to the visited URLs thread once you’ve finished crawling it. In each webpage, you will find new URLs. Most of them will be added to the queue, but some of them might not add any value for your purpose. Hence why you also need to set rules for URLs you’re not interested in.
Deduplication is a critical part of web crawling. On some websites, and particularly on e-commerce ones, a single webpage can have multiple URLs. As you want to scrape this page only once, the best way to do so is to look for the canonical tag in the code. All the pages with the same content will have this common canonical URL, and this is the only link you will have to crawl and scrape.
Here’s an example of a canonical tag in HTML:
previousDepth) {
previousDepth =;
(`——- CRAWLING ON DEPTH LEVEL ${previousDepth} ——–`);}
return nextLink;}
function peekInQueue() {
return linksQueue[0];}
//Adds links we’ve visited to the seenList
function addToSeen(linkObj) {
seenLinks[] = linkObj;}
//Returns whether the link has been seen.
function linkInSeenListExists(linkObj) {
return seenLinks[] == null? false: true;}
How to Build a Web Crawler – A Guide for Beginners
As a newbie, I built a web crawler and extracted 20k data successfully from the Amazon Career website. How can you set up a crawler and create a database which eventually turns to your asset at No Cost? Let’s dive right in.
What is a web crawler?
A web crawler is an internet bot that indexes the content of a website on the internet. It then extracts target information and data automatically. As a result, it exports the data into a structured format (list/table/database).
Why do you need a Web Crawler, especially for Enterprises?
Imagine Google Search doesn’t exist. How long will it take you to get the recipe for chicken nuggets without typing in the keyword? There are 2. 5 quintillion bytes of data created each day. That said, without Google Search, it’s impossible to find the information.
From Hackernoon by Ethan Jarrell
Google Search is a unique web crawler that indexes the websites and finds the page for us. Besides the search engine, you can build a web crawler to help you achieve:
1. Content aggregation: it works to compile information on niche subjects from various resources into one single platform. As such, it is necessary to crawl popular websites to fuel your platform in time.
2. Sentiment Analysis: it is also called opinion mining. As the name indicates, it is the process to analyze public attitudes towards one product and service. It requires a monotonic set of data to evaluate accurately. A web crawler can extract tweets, reviews, and comments for analysis.
3. Lead generation: Every business needs sales leads. That’s how they survive and prosper. Let’s say you plan to make a marketing campaign targeting a specific industry. You can scrape email, phone number, and public profiles from an exhibitor or attendee list of Trade Fairs, like attendees of the 2018 Legal Recruiting Summit.
How to build a web crawler as a beginner?
A. Scraping with a programming language
writing scripts with computer languages are predominantly used by programmers. It can be as powerful as you create it to be. Here is an example of a snippet of bot code.
From Kashif Aziz
Web scraping using Python involves three main steps:
1. Send an HTTP request to the URL of the webpage. It responds to your request by returning the content of web pages.
2. Parse the webpage. A parser will create a tree structure of the HTML as the webpages are intertwined and nested together. A tree structure will help the bot follow the paths that we created and navigate through to get the information.
3. Using python library to search the parse tree.
Among the computer languages for a web crawler, Python is easy-to-implement comparing to PHP and Java. It still has a steep learning curve that prevents many non-tech professionals from using it. Even though it is an economic solution to write your own, it’s still not sustainable regard to the extended learning cycle within a limited time frame.
However, there is a catch! What if there is a method that can get you the same results without writing a single line of code?
B. Web scraping tool comes in handy as a great alternative.
There are many options, but I use Octoparse. Let’s go back to the Amazon Career webpage as an example:
Goal: build a crawler to extract administrative job opportunities including Job title, Job ID, description, basic qualification, preferred qualification, and page URL.
URL:
1. Open Octoparse and select “Advanced Mode”. Enter the above URL to set up a new task.
2. As one can expect, the job listings include detail-pages that spread over to multiple pages. As such, we need to set up pagination so that the crawler can navigate through it. To this, click the “Next Page” button and choose “Look click Single Button” from the Action Tip Panel
3. As we want to click through each listing, we need to create a loop item. To do this, click one job listing. Octoparse will work its magic and identify all other job listings from the page. Choose the “Select All” command from the Action Tip Panel, then choose “Loop Click Each Element” command.
4. Now, we are on the detail page, and we need to tell the crawler to get the data. In this case, click “Job Title” and select “Extract the text of the selected element” command from the Action Tip Panel. As follows, repeat this step and get “Job ID”, “Description, ” “Basic Qualification”, “Preferred Qualification” and Page URL.
5. Once you finish setting up the extraction fields, click “Start Extraction” to execute.
However, that’s not All!
For SaaS software, it requires new users to take a considerable amount of training before thoroughly enjoy the benefits. To eliminate the difficulties to set up and use. Octoparse adds “Task Templates” covering over 30 websites for starters to grow comfortable with the software. They allow users to capture the data without task configuration.
As you gain confidence, you can use Wizard Mode to build your crawler. It has step-by-step guides to facilitate you to develop your task. For experienced experts, “Advanced Mode” should be able to extract the enterprise volume of data. Octoparse also provides rich training materials for you and your employees to get most of the software.
Final thoughts
Writing scripts can be painful as it has high initial and maintenance costs. No single web page is identical, and we need to write a script for every single site. It is not sustainable if you need to crawl many websites. Besides, websites likely changes its layout and structure. As a result, we have to debug and adjust the crawler accordingly. The web scraping tool is more practical for enterprise-level data extraction with fewer efforts and costs.
Consider you may have difficulties to find a web scraping tool, I compile a list of most popular scraping tools. This video can walk you through to get your device that fits your needs! Feel free to take advantage of it.
Author: Ashley Ng
Ashley is a data enthusiast and passionate blogger with hands-on experience in web scraping. She focuses on capturing web data and analyzing in a way that empowers companies and businesses with actionable insights. Read her blog here to discover practical tips and applications on web data extraction.
日本語記事：ゼロからWebクローラーを構築する方法Webスクレイピングについての記事は 公式サイトでも読むことができます。Artículo en español: Cómo Construir Un Web Rastreador (Crawler) Desde Cero: Una Guía para PrincipiantesTambién puede leer artículos de web scraping en el Website Oficial
Making Web Crawlers Using Scrapy for Python – DataCamp
If you would like an overview of web scraping in Python, take DataCamp’s Web Scraping with Python course.
In this tutorial, you will learn how to use Scrapy which is a Python framework using which you can handle large amounts of data! You will learn Scrapy by building a web scraper for which is an e-commerce website. Let’s get scrapping!
Scrapy Overview
Scrapy Vs. BeautifulSoup
Scrapy Installation
Scrapy Shell
Creating a project and Creating a custom spider
A basic HTML and CSS knowledge will help you understand this tutorial with greater ease and speed. Read this article for a fresher on HTML and CSS.
Source
Web scraping has become an effective way of extracting information from the web for decision making and analysis. It has become an essential part of the data science toolkit. Data scientists should know how to gather data from web pages and store that data in different formats for further analysis.
Any web page you see on the internet can be crawled for information and anything visible on a web page can be extracted [2]. Every web page has its own structure and web elements that because of which you need to write your web crawlers/spiders according to the web page being extracted.
Scrapy provides a powerful framework for extracting the data, processing it and then save it.
Scrapy uses spiders, which are self-contained crawlers that are given a set of instructions [1]. In Scrapy it is easier to build and scale large crawling projects by allowing developers to reuse their code.
In this section, you will have an overview of one of the most popularly used web scraping tool called BeautifulSoup and its comparison to Scrapy.
Scrapy is a Python framework for web scraping that provides a complete package for developers without worrying about maintaining code.
Beautiful Soup is also widely used for web scraping. It is a Python package for parsing HTML and XML documents and extract data from them. It is available for Python 2. 6+ and Python 3.
Here are some differences between them in a nutshell:
Scrapy
BeautifulSoup
Functionality
—
Scrapy is the complete package for downloading web pages, processing them and save it in files and databases
BeautifulSoup is basically an HTML and XML parser and requires additional libraries such as requests, urlib2 to open URLs and store the result [6]
Learning Curve
Scrapy is a powerhouse for web scraping and offers a lot of ways to scrape a web page. It requires more time to learn and understand how Scrapy works but once learned, eases the process of making web crawlers and running them from just one line of command. Becoming an expert in Scrapy might take some practice and time to learn all functionalities.
BeautifulSoup is relatively easy to understand for newbies in programming and can get smaller tasks done in no time
Speed and Load
Scrapy can get big jobs done very easily. It can crawl a group of URLs in no more than a minute depending on the size of the group and does it very smoothly as it uses Twister which works asynchronously (non-blocking) for concurrency.
BeautifulSoup is used for simple scraping jobs with efficiency. It is slower than Scrapy if you do not use multiprocessing.
Extending functionality
Scrapy provides Item pipelines that allow you to write functions in your spider that can process your data such as validating data, removing data and saving data to a database. It provides spider Contracts to test your spiders and allows you to create generic and deep crawlers as well. It allows you to manage a lot of variables such as retries, redirection and so on.
If the project does not require much logic, BeautifulSoup is good for the job, but if you require much customization such as proxys, managing cookies, and data pipelines, Scrapy is the best option.
Information: Synchronous means that you have to wait for a job to finish to start a new job while Asynchronous means you can move to another job before the previous job has finished
Here is an interesting DataCamp BeautifulSoup tutorial to learn.
With Python 3. 0 (and onwards) installed, if you are using anaconda, you can use conda to install scrapy. Write the following command in anaconda prompt:
conda install -c conda-forge scrapy
To install anaconda, look at these DataCamp tutorials for Mac and Windows.
Alternatively, you can use Python Package Installer pip. This works for Linux, Mac, and Windows:
pip install scrapy
Scrapy also provides a web-crawling shell called as Scrapy Shell, that developers can use to test their assumptions on a site’s behavior. Let us take a web page for tablets at AliExpress e-commerce website. You can use the Scrapy shell to see what components the web page returns and how you can use them to your requirements.
Open your command line and write the following command:
scrapy shell
If you are using anaconda, you can write the above command at the anaconda prompt as well. Your output on the command line or anaconda prompt will be something like this:
You have to run a crawler on the web page using the fetch command in the Scrapy shell. A crawler or spider goes through a webpage downloading its text and metadata.
fetch()
Note: Always enclose URL in quotes, both single and double quotes work
The output will be as follows:
The crawler returns a response which can be viewed by using the view(response) command on shell:
view(response)
And the web page will be opened in the default browser.
You can view the raw HTML script by using the following command in Scrapy shell:
print()
You will see the script that’s generating the webpage. It is the same content that when you left right-click any blank area on a webpage and click view source or view page source. Since, you need only relevant information from the entire script, using browser developer tools you will inspect the required element. Let us take the following elements:
Tablet name
Tablet price
Number of orders
Name of store
Right-click on the element you want and click inspect like below:
Developer tools of the browser will help you a lot with web scraping. You can see that it is an tag with a class product and the text contains the name of the product:
You can extract this using the element attributes or the css selector like classes. Write the following in the Scrapy shell to extract the product name:
(“. product::text”). extract_first()
The output will be:
extract_first() extract the first element that satisfies the css selector. If you want to extract all the product names use extract():
(“. extract()
Following code will extract price range of the products:
(“”). extract()
Similarly, you can try with a number of orders and the name of the store.
XPath is a query language for selecting nodes in an XML document [7]. You can navigate through an XML document using XPath. Behind the scenes, Scrapy uses Xpath to navigate to HTML document items. The CSS selectors you used above are also converted to XPath, but in many cases, CSS is very easy to use. But you should know how the XPath in Scrapy works.
Go to your Scrapy Shell and write fetch() the same way as before. Try out the following code snippets [3]:
(‘/html’). extract()
This will show you all the code under the tag. / means direct child of the node. If you want to get the
(‘/html//div’). extract()
For XPath, you must learn to understand the use of / and // to know how to navigate through child and descendent nodes. Here is a helpful tutorial for XPath Nodes and some examples to try out.
If you want to get all
(“//div”). extract()
You can further filter your nodes that you start from and reach your desired nodes by using attributes and their values. Below is the syntax to use classes and their values.
(“//div[@class=’quote’]/span[@class=’text’]”). extract()
(“//div[@class=’quote’]/span[@class=’text’]/text()”). extract()
Use text() to extract all text inside nodes
Consider the following HTML code:
You want to get the text inside the tag, which is child node of
(‘//div[@class=”site-notice-container container”]/a[@class=”notice-close”]/text()’). extract()
Creating a Scrapy project and Custom Spider
Web scraping can be used to make an aggregator that you can use to compare data. For example, you want to buy a tablet, and you want to compare products and prices together you can crawl your desired pages and store in an excel file. Here you will be scraping for tablets information.
Now, you will create a custom spider for the same page. First, you need to create a Scrapy project in which your code and results will be stored. Write the following command in the command line or anaconda prompt.
scrapy startproject aliexpress
This will create a hidden folder in your default python or anaconda installation. aliexpress will be the name of the folder. You can give any name. You can view the folder contents directly through explorer. Following is the structure of the folder:
file/folder
Purpose
deploy configuration file
aliexpress/
Project’s Python module, you’ll import your code from here
__
Initialization file
project items file
project pipelines file
project settings file
spiders/
a directory where you’ll later put your spiders
Once you have created the project you will change to the newly created directory and write the following command:
[scrapy genspider aliexpress_tablets]()
This creates a template file named in the spiders directory as discussed above. The code in that file is as below:
import scrapy
class AliexpressTabletsSpider():
name = ‘aliexpress_tablets’
allowed_domains = [”]
start_urls = [”]
def parse(self, response):
pass
In the above code you can see name, allowed_domains, sstart_urls and a parse function.
name: Name is the name of the spider. Proper names will help you keep track of all the spider’s you make. Names must be unique as it will be used to run the spider when scrapy crawl name_of_spider is used.
allowed_domains (optional): An optional python list, contains domains that are allowed to get crawled. Request for URLs not in this list will not be crawled. This should include only the domain of the website (Example:) and not the entire URL specified in start_urls otherwise you will get warnings.
start_urls: This requests for the URLs mentioned. A list of URLs where the spider will begin to crawl from, when no particular URLs are specified [4]. So, the first pages downloaded will be those listed here. The subsequent Request will be generated successively from data contained in the start URLs [4].
parse(self, response): This function will be called whenever a URL is crawled successfully. It is also called the callback function. The response (used in Scrapy shell) returned as a result of crawling is passed in this function, and you write the extraction code inside it!
Information: You can use BeautifulSoup inside parse() function of the Scrapy spider to parse the html document.
Note: You can extract data through css selectors using () as discussed in scrapy shell section but also using XPath (XML) that allows you to access child elements. You will see the example of () in the code edited in pass() function.
You will make changes to the file. I have added another URL in start_urls. You can add the extraction logic to the pass() function as below:
# -*- coding: utf-8 -*-
start_urls = [”,
”]
print(“procesing:”)
#Extract data using css selectors
(‘. product::text’). extract()
(”). extract()
#Extract data using xpath
(“//em[@title=’Total Orders’]/text()”). extract()
(“//a[@class=’store $p4pLog’]/text()”). extract()
row_data=zip(product_name, price_range, orders, company_name)
#Making extracted data row wise
for item in row_data:
#create a dictionary to store the scraped info
scraped_info = {
#key:value
‘page’,
‘product_name’: item[0], #item[0] means product in the list and so on, index tells what value to assign
‘price_range’: item[1],
‘orders’: item[2],
‘company_name’: item[3], }
#yield or give the scraped info to scrapy
yield scraped_info
Information: zip() takes n number of iterables and returns a list of tuples. ith element of the tuple is created using the ith element from each of the iterables. [8]
The yield keyword is used whenever you are defining a generator function. A generator function is just like a normal function except it uses yield keyword instead of return. The yield keyword is used whenever the caller function needs a value and the function containing yield will retain its local state and continue executing where it left off after yielding value to the caller function. Here yield gives the generated dictionary to Scrapy which will process and save it!
Now you can run the spider:
scrapy crawl aliexpress_tablets
You will see a long output at the command line like below:
Exporting data
You will need data to be presented as a CSV or JSON so that you can further use the data for analysis. This section of the tutorial will take you through how you can save CSV and JSON file for this data.
To save a CSV file, open from the project directory and add the following lines:
FEED_FORMAT=”csv”
FEED_URI=””
After saving the, rerun the scrapy crawl aliexpress_tablets in your project directory.
The CSV file will look like:
Note: Everytime you run the spider it will append the file.
FEED_FORMAT [5]: This sets the format you want to store the data. Supported formats are: + JSON
+ CSV
+ JSON Lines
+ XML
FEED_URI [5]: This gives the location of the file. You can store a file on your local file storage or an FTP as well.
Scrapy’s Feed Export can also add a timestamp and the name of spider to your file name, or you can use these to identify a directory in which you want to store. %(time)s: gets replaced by a timestamp when the feed is being created [5]%(name)s: gets replaced by the spider name [5]
For Example:
Store in FTP using one directory per spider [5]:
ftpuser:[email protected]/scraping/feeds/%(name)s/%(time)
The Feed changes you make in will apply to all spiders in the project. You can also set custom settings for a particular spider that will override the settings in the file.
custom_settings={ ‘FEED_URI’: “aliexpress_%(time)”,
‘FEED_FORMAT’: ‘json’}
#yield or give the scraped info to Scrapy
returns the URL of the page from which response is generated. After running the crawler using scrapy crawl aliexpress_tablets you can view the json file:
Following Links
You must have noticed, that there are two links in the start_urls. The second link is the page 2 of the same tablets search results. It will become impractical to add all links. A crawler should be able to crawl by itself through all the pages, and only the starting point should be mentioned in the start_urls.
If a page has subsequent pages, you will see a navigator for it at the end of the page that will allow moving back and forth the pages. In the case you have been implementing in this tutorial, you will see it like this:
Here is the code that you will see:
As you can see that under there is a tag with class class that is the current page you are on, and under that are all tags with links to the next page. Everytime you will have to get the tags after this tag. Here comes a little bit of CSS! In this, you have to get sibling node and not a child node, so you have to make a css selector that tells the crawler to find tags that are after tag with class.
Remember! Each web page has its own structure. You will have to study the structure a little bit on how you can get the desired element. Always try out (SELECTOR) on Scrapy Shell before writing them in code.
Modify your as below:
‘FEED_FORMAT’: ‘csv’}
NEXT_PAGE_SELECTOR = ‘ + a::attr(href)’
next_page = (NEXT_PAGE_SELECTOR). extract_first()
if next_page:
yield quest(
response. urljoin(next_page), )
In the above code:
you first extracted the link of the next page using next_page = (NEXT_PAGE_SELECTOR). extract_first() and then if the variable next_page gets a link and is not empty, it will enter the if body.
response. urljoin(next_page): The parse() method will use this method to build a new url and provide a new request, which will be sent later to the callback. [9]
After receiving the new URL, it will scrape that link executing the for body and again look for the next page. This will continue until it doesn’t get a next page link.
Here you might want to sit back and enjoy your spider scraping all the pages. The above spider will extract from all subsequent pages. That will be a lot of scraping! But your spider will do it! Below you can see the size of the file has reached 1. 1MB.
Scrapy does it for you!
In this tutorial, you have learned about Scrapy, how it compares to BeautifulSoup, Scrapy Shell and how to write your own spiders in Scrapy. Scrapy handles all the heavy load of coding for you, from creating project files and folders till handling duplicate URLs it helps you get heavy-power web scraping in minutes and provides you support for all common data formats that you can further input in other programs. This tutorial will surely help you understand Scrapy and its framework and what you can do with it. To become a master in Scrapy, you will need to go through all the fantastic functionalities it has to provide, but this tutorial has made you capable of scraping groups of web pages in an efficient way.
For further reading, you can refer to Offical Scrapy Docs.
Also, don’t forget to check out DataCamp’s Web Scraping with Python course.
References
[1] [2] [3] [4] [5] [6] [7] [8] [9]
Frequently Asked Questions about how to program a web crawler
How do you create a web crawler in Python?
Making Web Crawlers Using Scrapy for PythonScrapy Overview.Scrapy Vs. BeautifulSoup.Scrapy Installation.Scrapy Shell.Creating a project and Creating a custom spider.Jan 11, 2019
What is the main process of a web crawler program?
Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code.
What is the best programming language for developing a web crawler?
Python is mostly known as the best web scraper language. It’s more like an all-rounder and can handle most of the web crawling related processes smoothly. Beautiful Soup is one of the most widely used frameworks based on Python that makes scraping using this language such an easy route to take.Aug 9, 2017
