Scrape Search Results Python
How to scrape Google search results using Python – Practical …
Stallions… Picture by Alex Kotliarksyi, Unsplash.
9 minutes to read
Although I suspect you are probably not technically allowed to do it, I doubt there’s an SEO in the land who hasn’t scraped Google search engine results to analyse them, or used an SEO tool that does the same thing. It’s much more convenient than picking through the SERPs to extract links by hand.
In this project, I’ll show you how you can build a relatively robust (but also slightly flawed) web scraper using
Requests-HTML that can return a list of URLs from a Google search, so you can analyse the URLs in your technical
SEO projects.
If you just want a quick, free way to scrape Google search results using Python, without paying for a SERP API
service, then give my EcommerceTools package a try. It lets you scrape Google search results in three lines of
code. Here’s how it’s done.
Load the packages
First, open up a Jupyter notebook and import the below packages. You’ll likely already have requests, urllib, and pandas, but you can install requests_html by entering pip3 install requests_html, if you don’t already have it.
import requests
import urllib
import pandas as pd
from requests_html import HTML
from requests_html import HTMLSession
Get the page source
Next, we’ll write a little function to pass our URL to Requests-HTML and return the source code of the page. This first creates a session, then fetches the response, or throws an exception if something goes wrong. We’ll scrape the interesting bits in the next step.
def get_source(url):
“””Return the source code for the provided URL.
Args:
url (string): URL of the page to scrape.
Returns:
response (object): HTTP response object from requests_html.
“””
try:
session = HTMLSession()
response = (url)
return response
except questException as e:
print(e)
Scrape the results
This is the bit where things get interesting, and slightly hacky. I suspect Google does not like people scraping their search results, so you’ll find that there are no convenient CSS class names we can tap into. Those that are present, seem to change, causing scrapers to break. To work around this I’ve used an alternate approach, which is more robust, but does have a limitation.
First, we’re using () to URL encode our search query. This will add + characters where spaces sit and ensure that the search term used doesn’t break the URL when we append it. After that, we’ll combine it with the Google search URL and get back the page source using get_source().
Rather than using the current CSS class or XPath to extract the links, I’ve just exported all the absolute URLs from the page using This is more resistant to changes in Google’s source code, but it means there will be Google URLs also present.
Since it’s only non-Google content in which I’m interested, I’ve removed any URLs with a Google-related URL prefix. The downside is that it will remove legitimate Google URLs in the SERPs.
def scrape_google(query):
query = (query)
response = get_source(” + query)
links = list()
google_domains = (‘. ‘,
‘google. ‘,
‘. ‘)
for url in links[:]:
if artswith(google_domains):
(url)
return links
Running the function gives us a list of URLs that were found on the Google search results for our chosen term, with any Google-related URLs removed. This obviously isn’t a perfect match for the actual results, however, it does return the non-Google domains in which I’m interested.
scrape_google(“data science blogs”)
[”,
”,
”]
You can tweak the code accordingly to extract only the links from certain parts of the SERPs, but you’ll find that you’ll need to update the code regular as the source code is changed frequently. For what I needed, this did the job fine.
Want the text instead?
If you’re after the title, snippet, and the URL for each search engine result, try this approach instead. First, create a function to format and URL encode the query, send it to Google and show the output.
def get_results(query):
Next, we’ll parse the response HTML. I’ve pored over the obfuscated HTML and extracted the current CSS values that hold the values for the result, the title, the link, and the snippet text. These change frequently, so this may not work in the future without adjusting these values.
def parse_results(response):
css_identifier_result = “. tF2Cxc”
css_identifier_title = “h3″
css_identifier_link = ” a”
css_identifier_text = “”
results = (css_identifier_result)
output = []
for result in results:
item = {
‘title’: (css_identifier_title, first=True),
‘link’: (css_identifier_link, first=True)[‘href’],
‘text’: (css_identifier_text, first=True)}
(item)
return output
Finally, we’ll wrap up the functions in a google_search() function, which will put everything above together and return a neat list of dictionaries containing the results.
def google_search(query):
response = get_results(query)
return parse_results(response)
results = google_search(“web scraping”)
results
[{‘title’: ‘What is Web Scraping and What is it Used For? | ParseHub’,
‘link’: ”,
‘text’: ”},
{‘title’: ‘Web scraping – Wikipedia’,
‘text’: ‘Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may access the World\xa0… \n\u200eHistory · \u200eTechniques · \u200eSoftware · \u200eLegal issues’},
{‘title’: ‘Web Scraper – The #1 web scraping extension’,
‘text’: ‘The most popular web scraping extension. Start scraping in minutes. Automate your tasks with our Cloud Scraper. No software to download, no coding needed. \n\u200eWeb Scraper · \u200eCloud · \u200eTest Sites · \u200eDocumentation’},
{‘title’: ‘Web Scraper – Free Web Scraping’,
‘text’: ’23 Sept 2020 — With a simple point-and-click interface, the ability to extract thousands of records from a website takes only a few minutes of scraper setup. Web\xa0… ‘},
{‘title’: ‘Python Web Scraping Tutorials – Real Python’,
‘text’: ‘Web scraping is about downloading structured data from the web, selecting some of that data, and passing along what you selected to another process. ‘},
{‘title’: ‘ParseHub | Free web scraping – The most powerful web scraper’,
‘text’: ‘ParseHub is a free web scraping tool. Turn any site into a spreadsheet or API. As easy as clicking on the data you want to extract. ‘},
{‘title’: ‘Web Scraping Explained – WebHarvy’,
‘text’: ‘Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting etc. ) is a technique employed to extract large amounts of data from websites\xa0… ‘},
{‘title’: ‘What Is Web Scraping And How Does Web Crawling Work? ‘,
‘text’: ‘Web scraping, also called web data extraction, is the process of extracting or scraping data from websites. Learn about web crawling and how it works. ‘},
{‘title’: “A beginner’s guide to web scraping with Python | Opensource… “,
‘text’: “22 May 2020 — Setting a goal for our web scraping project. Now we have our dependencies installed, but what does it take to scrape a webpage? Let’s take a\xa0… “}]
If you want to quickly scrape several pages of Google search results, rather than just the first page of results,
check out EcommerceTools instead, or adapt the code above to support pagination.
Matt Clarke, Saturday, March 13, 2021
Scrape Google Search Results using Python BeautifulSoup
In this article, we are going to see how to Scrape Google Search Results using Python Needed:bs4: Beautiful Soup(bs4) is a Python library for pulling data out of HTML and XML files. This module does not come built-in with Python. To install this type the below command in the install bs4
requests: Requests allows you to send HTTP/1. 1 requests extremely easily. This module also does not come built-in with Python. To install this type the below command in the install requests
Approach:Import the beautifulsoup and request two strings with the default Google search URL, ‘ and our customized search ncatenate these two strings to get our search the URL data using (url), store it in a variable, a string and store the result of our fetched request, using we use BeautifulSoup to analyze the extracted page. We can simply create an object to perform those operations but beautifulsoup comes with a lot of in-built features to scrape the web. We have created a soup object first using beautifulsoup from the request-response We can do (h3) to grab all major headings of our search result, Iterate through the object and print it as a string. Example 1: Below is the implementation of the above thon3import requestsimport bs4text= “geeksforgeeks”( url)soup = autifulSoup(, “”)print(soup)Output:Let’s We can do (h3) to grab all major headings of our search result, Iterate through the object and print it as a nd_all( ‘h3′)for info in heading_object: print(tText()) print(“——“)Output:Example 2: Below is the implementation. In the form of extracting the city temperature using Google search:Pythonimport requests import bs4 city = “Imphal”request_result = ( url)soup = autifulSoup(, “”)temp = ( “div”, class_=’BNeawe’) print( temp) Output: Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics. To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course
How to scrape 1,000 Google search result links in 5 minutes. – LinkedIn
Graham Onak
Owner at GainTap
Published Jun 26, 2015
This is the best way to scrape Google search results quickly, easily and for free.
In this video I show you how to use a free Chrome extension called Linkclump to quickly copy Google search results to a Google sheet. This is the best way I know how to copy links from Google.
Most crawlers don’t pull Google results, here’s why.
Scraping Google is against their terms of service. They go so far as to block your IP if you automate scraping of their search results. I’ve tried great scraping tools like with no luck. This is especially the case if you’re trying to pull search results from pages that Google hides as duplicates.
The best way to scrape Google is manually.
It may not be as fast as using a web crawler, but the fact is – it’s safe, easy and fast. I’ve used the above web scraping technique to pull 1, 000 links in 5 minutes on the couch. Here’s the rundown on what you need to do.
Download Linkclump for Chrome
Adjust your Linkclump settings – set them to “Copy to Clipboard” on action.
Open a spreadsheet
Search for a term
Right click and drag to copy all links in the selection
Copy and paste to a spreadsheet
Go to the next page of search results
Rinse and repeat
That’s it! Super easy and fast. If you don’t have the time, this makes for an excellent project to outsource to a virtual assistant.
Frequently Asked Questions about scrape search results python
How do I scrape search results?
The best way to scrape Google is manually.Download Linkclump for Chrome.Adjust your Linkclump settings – set them to “Copy to Clipboard” on action.Open a spreadsheet.Search for a term.Right click and drag to copy all links in the selection.Copy and paste to a spreadsheet.Go to the next page of search results.More items…•Jun 26, 2015
How do you scrape data from Google search in Python?
Approach:Import the beautifulsoup and request libraries.Concatenate these two strings to get our search URL.Fetch the URL data using requests. … Create a string and store the result of our fetched request, using request_result. … Now we use BeautifulSoup to analyze the extracted page. … We can do soup.Dec 29, 2020
Is it possible to scrape Google search results?
Although Google does not take legal action against scraping, it uses a range of defensive methods that makes scraping their results a challenging task, even when the scraping tool is realistically spoofing a normal web browser: … Network and IP limitations are as well part of the scraping defense systems.
