How To Scrape Prices From Websites
How to scrape Prices from any eCommerce website
Price Scraping involves gathering price information of a product from an eCommerce website using web scraping. A price scraper can help you easily scrape prices from website for price monitoring purposes of your competitor and your products.
How to Scrape Prices
1. Create your own Price Monitoring Tool to Scrape Prices
There are plenty of web scraping tutorials on the internet where you can learn how to create your own price scraper to gather pricing from eCommerce websites. However, writing a new scraper for every different eCommerce site could get very expensive and tedious. Below we demonstrate some advanced techniques to build a basic web scraper that could scrape prices from any eCommerce page.
2. Web Scraping using Price Scraping Tools
Web scraping tools such as ScrapeHero Cloud can help you scrape prices without coding, downloading and learning how to use a tool. ScrapeHero Cloud has pre-built crawlers that can help you scrape popular eCommerce websites such as Amazon, Walmart, Target easily. ScrapeHero Cloud also has scraping APIs to help you scrape prices from Amazon and Walmart in real-time, web scraping APIs can help you get pricing details within seconds.
3. Custom Price Monitoring Solution
ScrapeHero Price Monitoring Solutions are cost-effective and can be built within weeks and in some cases days. Our price monitoring solution can easily be scaled to include multiple websites and/or products within a short span of time. We have considerable experience in handling all the challenges involved in price monitoring and have the sufficient know-how about the essentials of product monitoring.
How to Build a Price Scraper
In this tutorial, we will show you how to build a basic web scraper which will help you in scraping prices from eCommerce websites by taking a few common websites as an example.
Let’s start by taking a look at a few product pages, and identify certain design patterns on how product prices are displayed on the websites.
Observations and Patterns
Some patterns that we identified by looking at these product pages are:
Price appears as currency figures (never as words)
The price is the currency figure with the largest font size
Price comes inside first 600 pixels height
Usually the price comes above other currency figures
Of course, there could be exceptions to these observations, we’ll discuss how to deal with exceptions later in this article. We can combine these observations to create a fairly effective and generic crawler for scraping prices from eCommerce websites.
Implementation of a generic eCommerce scraper to scrape prices
Step 1: Installation
This tutorial uses the Google Chrome web browser. If you don’t have Google Chrome installed, you can follow the installation instructions.
Instead of Google Chrome, advanced developers can use a programmable version of Google Chrome called Puppeteer. This will remove the necessity of a running GUI application to run the scraper. However, that is beyond the scope of this tutorial.
Step 2: Chrome Developer Tools
The code presented in this tutorial is designed for scraping prices as simple as possible. Therefore, it will not be capable of fetching the price from every product page out there.
For now, we’ll visit an Amazon product page or a Sephora product page in Google Chrome.
Visit the product page in Google Chrome
Right-click anywhere on the page and select ‘Inspect Element’ to open up Chrome DevTools
Click on the Console tab of DevTools
Inside the Console tab, you can enter any JavaScript code. The browser will execute the code in the context of the web page that has been loaded. You can learn more about DevTools using their official documentation.
Step 3: Run the JavaScript snippet
Copy the following JavaScript snippet and paste it into the console.
let elements = [
cument. querySelectorAll(‘ body *’)]
function createRecordFromElement(element) {
const text = ()
var record = {}
const bBox = tBoundingClientRect()
if( <= 30 &&! (bBox. x == 0 && bBox. y == 0)) {
record['fontSize'] = parseInt(getComputedStyle(element)['fontSize'])}
record['y'] = bBox. y
record['x'] = bBox. x
record['text'] = text
return record}
let records = (createRecordFromElement)
function canBePrice(record) {
if( record['y'] > 600 ||
record[‘fontSize’] == undefined ||! record[‘text’](/(^(US){0, 1}(rs\. |Rs\. |RS\. |\$|₹|INR|USD|CAD|C\$){0, 1}(\s){0, 1}[\d, ]+(\. \d+){0, 1}(\s){0, 1}(AED){0, 1}$)/))
return false
else return true}
let possiblePriceRecords = (canBePrice)
let priceRecordsSortedByFontSize = (function(a, b) {
if (a[‘fontSize’] == b[‘fontSize’]) return a[‘y’] > b[‘y’]
return a[‘fontSize’] < b['fontSize']})
(priceRecordsSortedByFontSize[0]['text']);
Press ‘Enter’ and you should now be seeing the price of the product displayed on the console.
If you don’t, then you have probably visited a product page which is an exception to our observations. This is completely normal, we’ll discuss how we can expand our script to cover more product pages of these kinds. You could try one of the sample pages provided in step 2.
The animated GIF below shows how we get the price from
How it works
First, we have to fetch all the HTML DOM elements in the page.
We need to convert each of these elements to simple JavaScript objects which stores their XY position values, text content and font size, which looks something like {'text':'Tennis Ball', 'fontSize':'14px', 'x':100, 'y':200}. So we have to write a function for that, as follows.
const text = () // Fetches text content of the element
var record = {} // Initiates a simple JavaScript object
// getBoundingClientRect is a function provided by Google Chrome, it returns
// an object which contains x, y values, height and width
// getComputedStyle is a function provided by Google Chrome, it returns an
// object with all its style information. Since this function is relatively
// time-consuming, we are only collecting the font size of elements whose
// text content length is atmost 30 and whose x and y coordinates are not 0
Now, convert all the elements collected to JavaScript objects by applying our function on all elements using the JavaScript map function.
Remember the observations we made regarding how a price is displayed. We can now filter just those records which match our design observations. So we need a function that says whether a given record matches with our design observations.
if(
record['y'] > 600 ||
We have used a Regular Expression to check if a given text is a currency figure or not. You can modify this regular expression in case it doesn’t cover any web pages that you’re experimenting with.
Now we can filter just the records that are possibly price records
Finally, as we’ve observed, the Price comes as the currency figure having the highest font size. If there are multiple currency figures with equally high font size, then Price probably corresponds to the one residing at a higher position. We are going to sort out our records based on these conditions, using the JavaScript sort function.
Now we just need to display it on the console
(priceRecordsSortedByFontSize[0][‘text’])
Taking it further
Moving to a GUI-less based scalable program
You can replace Google Chrome with a headless version of it called Puppeteer. Puppeteer is arguably the fastest option for headless web rendering. It works entirely based on the same ecosystem provided in Google Chrome. Once Puppeteer is set up, you can inject our script programmatically to the headless browser, and have the price returned to a function in your program. To learn more, visit our tutorial on Puppeteer.
Improving and enhancing this script
You will quickly notice that some product pages will not work with such a script because they don’t follow the assumptions we have made about how the product price is displayed and the patterns we identified.
Unfortunately, there is no “holy grail” or a perfect solution to this problem. It is possible to generalize more web pages and identify more patterns and enhance this scraper.
A few suggestions for enhancements are:
Figuring out more features, such as font-weight, font color, etc.
Class names or IDs of the elements containing price would probably have the word price. You could figure out such other commonly occurring words.
Currency figures with strike-through are probably regular prices, those could be ignored.
There could be pages that follow some of our design observations but violates some others. The snippet provided above strictly filters out elements that violate even one of the observations. In order to deal with this, you can try creating a score based system. This would award points for following certain observations and penalize for violating certain observations. Those elements scoring above a particular threshold could be considered as price.
The next significant step that you would use to handle other pages is to employ Artificial Intelligence/Machine Learning based techniques. You can identify and classify patterns and automate the process to a larger degree this way. However, this field is an evolving field of study and we at ScrapeHero are using such techniques already with varying degrees of success.
If you need help to scrape prices from you can check out our tutorial specifically designed for
We can help with your data or automation needs
Turn the Internet into meaningful, structured and usable data
Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.
How to scrape Prices from any eCommerce website
Price Scraping involves gathering price information of a product from an eCommerce website using web scraping. A price scraper can help you easily scrape prices from website for price monitoring purposes of your competitor and your products.
How to Scrape Prices
1. Create your own Price Monitoring Tool to Scrape Prices
There are plenty of web scraping tutorials on the internet where you can learn how to create your own price scraper to gather pricing from eCommerce websites. However, writing a new scraper for every different eCommerce site could get very expensive and tedious. Below we demonstrate some advanced techniques to build a basic web scraper that could scrape prices from any eCommerce page.
2. Web Scraping using Price Scraping Tools
Web scraping tools such as ScrapeHero Cloud can help you scrape prices without coding, downloading and learning how to use a tool. ScrapeHero Cloud has pre-built crawlers that can help you scrape popular eCommerce websites such as Amazon, Walmart, Target easily. ScrapeHero Cloud also has scraping APIs to help you scrape prices from Amazon and Walmart in real-time, web scraping APIs can help you get pricing details within seconds.
3. Custom Price Monitoring Solution
ScrapeHero Price Monitoring Solutions are cost-effective and can be built within weeks and in some cases days. Our price monitoring solution can easily be scaled to include multiple websites and/or products within a short span of time. We have considerable experience in handling all the challenges involved in price monitoring and have the sufficient know-how about the essentials of product monitoring.
How to Build a Price Scraper
In this tutorial, we will show you how to build a basic web scraper which will help you in scraping prices from eCommerce websites by taking a few common websites as an example.
Let’s start by taking a look at a few product pages, and identify certain design patterns on how product prices are displayed on the websites.
Observations and Patterns
Some patterns that we identified by looking at these product pages are:
Price appears as currency figures (never as words)
The price is the currency figure with the largest font size
Price comes inside first 600 pixels height
Usually the price comes above other currency figures
Of course, there could be exceptions to these observations, we’ll discuss how to deal with exceptions later in this article. We can combine these observations to create a fairly effective and generic crawler for scraping prices from eCommerce websites.
Implementation of a generic eCommerce scraper to scrape prices
Step 1: Installation
This tutorial uses the Google Chrome web browser. If you don’t have Google Chrome installed, you can follow the installation instructions.
Instead of Google Chrome, advanced developers can use a programmable version of Google Chrome called Puppeteer. This will remove the necessity of a running GUI application to run the scraper. However, that is beyond the scope of this tutorial.
Step 2: Chrome Developer Tools
The code presented in this tutorial is designed for scraping prices as simple as possible. Therefore, it will not be capable of fetching the price from every product page out there.
For now, we’ll visit an Amazon product page or a Sephora product page in Google Chrome.
Visit the product page in Google Chrome
Right-click anywhere on the page and select ‘Inspect Element’ to open up Chrome DevTools
Click on the Console tab of DevTools
Inside the Console tab, you can enter any JavaScript code. The browser will execute the code in the context of the web page that has been loaded. You can learn more about DevTools using their official documentation.
Step 3: Run the JavaScript snippet
Copy the following JavaScript snippet and paste it into the console.
let elements = [
cument. querySelectorAll(‘ body *’)]
function createRecordFromElement(element) {
const text = ()
var record = {}
const bBox = tBoundingClientRect()
if( <= 30 &&! (bBox. x == 0 && bBox. y == 0)) {
record['fontSize'] = parseInt(getComputedStyle(element)['fontSize'])}
record['y'] = bBox. y
record['x'] = bBox. x
record['text'] = text
return record}
let records = (createRecordFromElement)
function canBePrice(record) {
if( record['y'] > 600 ||
record[‘fontSize’] == undefined ||! record[‘text’](/(^(US){0, 1}(rs\. |Rs\. |RS\. |\$|₹|INR|USD|CAD|C\$){0, 1}(\s){0, 1}[\d, ]+(\. \d+){0, 1}(\s){0, 1}(AED){0, 1}$)/))
return false
else return true}
let possiblePriceRecords = (canBePrice)
let priceRecordsSortedByFontSize = (function(a, b) {
if (a[‘fontSize’] == b[‘fontSize’]) return a[‘y’] > b[‘y’]
return a[‘fontSize’] < b['fontSize']})
(priceRecordsSortedByFontSize[0]['text']);
Press ‘Enter’ and you should now be seeing the price of the product displayed on the console.
If you don’t, then you have probably visited a product page which is an exception to our observations. This is completely normal, we’ll discuss how we can expand our script to cover more product pages of these kinds. You could try one of the sample pages provided in step 2.
The animated GIF below shows how we get the price from
How it works
First, we have to fetch all the HTML DOM elements in the page.
We need to convert each of these elements to simple JavaScript objects which stores their XY position values, text content and font size, which looks something like {'text':'Tennis Ball', 'fontSize':'14px', 'x':100, 'y':200}. So we have to write a function for that, as follows.
const text = () // Fetches text content of the element
var record = {} // Initiates a simple JavaScript object
// getBoundingClientRect is a function provided by Google Chrome, it returns
// an object which contains x, y values, height and width
// getComputedStyle is a function provided by Google Chrome, it returns an
// object with all its style information. Since this function is relatively
// time-consuming, we are only collecting the font size of elements whose
// text content length is atmost 30 and whose x and y coordinates are not 0
Now, convert all the elements collected to JavaScript objects by applying our function on all elements using the JavaScript map function.
Remember the observations we made regarding how a price is displayed. We can now filter just those records which match our design observations. So we need a function that says whether a given record matches with our design observations.
if(
record['y'] > 600 ||
We have used a Regular Expression to check if a given text is a currency figure or not. You can modify this regular expression in case it doesn’t cover any web pages that you’re experimenting with.
Now we can filter just the records that are possibly price records
Finally, as we’ve observed, the Price comes as the currency figure having the highest font size. If there are multiple currency figures with equally high font size, then Price probably corresponds to the one residing at a higher position. We are going to sort out our records based on these conditions, using the JavaScript sort function.
Now we just need to display it on the console
(priceRecordsSortedByFontSize[0][‘text’])
Taking it further
Moving to a GUI-less based scalable program
You can replace Google Chrome with a headless version of it called Puppeteer. Puppeteer is arguably the fastest option for headless web rendering. It works entirely based on the same ecosystem provided in Google Chrome. Once Puppeteer is set up, you can inject our script programmatically to the headless browser, and have the price returned to a function in your program. To learn more, visit our tutorial on Puppeteer.
Improving and enhancing this script
You will quickly notice that some product pages will not work with such a script because they don’t follow the assumptions we have made about how the product price is displayed and the patterns we identified.
Unfortunately, there is no “holy grail” or a perfect solution to this problem. It is possible to generalize more web pages and identify more patterns and enhance this scraper.
A few suggestions for enhancements are:
Figuring out more features, such as font-weight, font color, etc.
Class names or IDs of the elements containing price would probably have the word price. You could figure out such other commonly occurring words.
Currency figures with strike-through are probably regular prices, those could be ignored.
There could be pages that follow some of our design observations but violates some others. The snippet provided above strictly filters out elements that violate even one of the observations. In order to deal with this, you can try creating a score based system. This would award points for following certain observations and penalize for violating certain observations. Those elements scoring above a particular threshold could be considered as price.
The next significant step that you would use to handle other pages is to employ Artificial Intelligence/Machine Learning based techniques. You can identify and classify patterns and automate the process to a larger degree this way. However, this field is an evolving field of study and we at ScrapeHero are using such techniques already with varying degrees of success.
If you need help to scrape prices from you can check out our tutorial specifically designed for
We can help with your data or automation needs
Turn the Internet into meaningful, structured and usable data
Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.
Web Scraping 101: 10 Myths that Everyone Should Know | Octoparse
1. Web Scraping is illegal
Many people have false impressions about web scraping. It is because there are people don’t respect the great work on the internet and use it by stealing the content. Web scraping isn’t illegal by itself, yet the problem comes when people use it without the site owner’s permission and disregard of the ToS (Terms of Service). According to the report, 2% of online revenues can be lost due to the misuse of content through web scraping. Even though web scraping doesn’t have a clear law and terms to address its application, it’s encompassed with legal regulations. For example:
Violation of the Computer Fraud and Abuse Act (CFAA)
Violation of the Digital Millennium Copyright Act (DMCA)
Trespass to Chattel
Misappropriation
Copy right infringement
Breach of contract
Photo by Amel Majanovic on Unsplash
2. Web scraping and web crawling are the same
Web scraping involves specific data extraction on a targeted webpage, for instance, extract data about sales leads, real estate listing and product pricing. In contrast, web crawling is what search engines do. It scans and indexes the whole website along with its internal links. “Crawler” navigates through the web pages without a specific goal.
3. You can scrape any website
It is often the case that people ask for scraping things like email addresses, Facebook posts, or LinkedIn information. According to an article titled “Is web crawling legal? ” it is important to note the rules before conduct web scraping:
Private data that requires username and passcodes can not be scrapped.
Compliance with the ToS (Terms of Service) which explicitly prohibits the action of web scraping.
Don’t copy data that is copyrighted.
One person can be prosecuted under several laws. For example, one scraped some confidential information and sold it to a third party disregarding the desist letter sent by the site owner. This person can be prosecuted under the law of Trespass to Chattel, Violation of the Digital Millennium Copyright Act (DMCA), Violation of the Computer Fraud and Abuse Act (CFAA) and Misappropriation.
It doesn’t mean that you can’t scrape social media channels like Twitter, Facebook, Instagram, and YouTube. They are friendly to scraping services that follow the provisions of the file. For Facebook, you need to get its written permission before conducting the behavior of automated data collection.
4. You need to know how to code
A web scraping tool (data extraction tool) is very useful regarding non-tech professionals like marketers, statisticians, financial consultant, bitcoin investors, researchers, journalists, etc. Octoparse launched a one of a kind feature – web scraping templates that are preformatted scrapers that cover over 14 categories on over 30 websites including Facebook, Twitter, Amazon, eBay, Instagram and more. All you have to do is to enter the keywords/URLs at the parameter without any complex task configuration. Web scraping with Python is time-consuming. On the other side, a web scraping template is efficient and convenient to capture the data you need.
5. You can use scraped data for anything
It is perfectly legal if you scrape data from websites for public consumption and use it for analysis. However, it is not legal if you scrape confidential information for profit. For example, scraping private contact information without permission, and sell them to a 3rd party for profit is illegal. Besides, repackaging scraped content as your own without citing the source is not ethical as well. You should follow the idea of no spamming, no plagiarism, or any fraudulent use of data is prohibited according to the law.
Check Below Video: 10 Myths About Web Scraping!
6. A web scraper is versatile
Maybe you’ve experienced particular websites that change their layouts or structure once in a while. Don’t get frustrated when you come across such websites that your scraper fails to read for the second time. There are many reasons. It isn’t necessarily triggered by identifying you as a suspicious bot. It also may be caused by different geo-locations or machine access. In these cases, it is normal for a web scraper to fail to parse the website before we set the adjustment.
Read this article: How to Scrape Websites Without Being Blocked in 5 Mins?
7. You can scrape at a fast speed
You may have seen scraper ads saying how speedy their crawlers are. It does sound good as they tell you they can collect data in seconds. However, you are the lawbreaker who will be prosecuted if damages are caused. It is because a scalable data request at a fast speed will overload a web server which might lead to a server crash. In this case, the person is responsible for the damage under the law of “trespass to chattels” law (Dryer and Stockton 2013). If you are not sure whether the website is scrapable or not, please ask the web scraping service provider. Octoparse is a responsible web scraping service provider who places clients’ satisfaction in the first place. It is crucial for Octoparse to help our clients get the problem solved and to be successful.
8. API and Web scraping are the same
API is like a channel to send your data request to a web server and get desired data. API will return the data in JSON format over the HTTP protocol. For example, Facebook API, Twitter API, and Instagram API. However, it doesn’t mean you can get any data you ask for. Web scraping can visualize the process as it allows you to interact with the websites. Octoparse has web scraping templates. It is even more convenient for non-tech professionals to extract data by filling out the parameters with keywords/URLs.
9. The scraped data only works for our business after being cleaned and analyzed
Many data integration platforms can help visualize and analyze the data. In comparison, it looks like data scraping doesn’t have a direct impact on business decision making. Web scraping indeed extracts raw data of the webpage that needs to be processed to gain insights like sentiment analysis. However, some raw data can be extremely valuable in the hands of gold miners.
With Octoparse Google Search web scraping template to search for an organic search result, you can extract information including the titles and meta descriptions about your competitors to determine your SEO strategies; For retail industries, web scraping can be used to monitor product pricing and distributions. For example, Amazon may crawl Flipkart and Walmart under the “Electronic” catalog to assess the performance of electronic items.
10. Web scraping can only be used in business
Web scraping is widely used in various fields besides lead generation, price monitoring, price tracking, market analysis for business. Students can also leverage a Google scholar web scraping template to conduct paper research. Realtors are able to conduct housing research and predict the housing market. You will be able to find Youtube influencers or Twitter evangelists to promote your brand or your own news aggregation that covers the only topics you want by scraping news media and RSS feeds.
Source:
Dryer, A. J., and Stockton, J. 2013. “Internet ‘Data Scraping’: A Primer for Counseling Clients, ” New York Law Journal. Retrieved from
Frequently Asked Questions about how to scrape prices from websites
How do you scrape a website price?
How to scrape Prices from any eCommerce websiteCreate your own Price Monitoring Tool to Scrape Prices.Web Scraping using Price Scraping Tools.Custom Price Monitoring Solution.Sep 13, 2018
Is it legal to scrape website data?
It is perfectly legal if you scrape data from websites for public consumption and use it for analysis. However, it is not legal if you scrape confidential information for profit. For example, scraping private contact information without permission, and sell them to a 3rd party for profit is illegal.Aug 16, 2021
Is it legal to scrape prices?
Screen-scraping is legal as long as the information you’re taking from other websites is strictly factual. However, if a website’s terms of use ban you from screen-scraping (even if the data is just facts), you should not go ahead with scraping data as you could be sued for breach of contract.May 2, 2019