Serp Data
SERP 101: All About Search Engine Results Pages
There are millions of pages on the web, but none are more important to digital marketers than search engine results pages, or SERPs.
Search engine optimization specialists and PPC advertisers alike vie for the same precious real estate in the most prominent parts of the SERPs, but competition is fierce and technological developments in search mean it’s more important than ever for digital marketers to know how search works and what they can do to maximize their visibility.
Page one of the SERP is where you want to be!
Search Engine Results Pages: What They Are and How They Work
Search engine results pages are web pages served to users when they search for something online using a search engine, such as Google. The user enters their search query (often using specific terms and phrases known as keywords), upon which the search engine presents them with a SERP.
Every SERP is unique, even for search queries performed on the same search engine using the same keywords or search queries. This is because virtually all search engines customize the experience for their users by presenting results based on a wide range of factors beyond their search terms, such as the user’s physical location, browsing history, and social settings. Two SERPs may appear identical, and contain many of the same results, but will often feature subtle differences.
The appearance of search engine results pages is constantly in flux due to experiments conducted by Google, Bing, and other search engine providers to offer their users a more intuitive, responsive experience. This, combined with emerging and rapidly developing technologies in the search space, mean that the SERPs of today differ greatly in appearance from their older predecessors.
Organic Results
SERPs typically contain two types of content – “organic” results and paid results. Organic results are listings of web pages that appear as a result of the search engine’s algorithm (more on this shortly). Search engine optimization professionals, commonly known as SEOs, specialize in optimizing web content and websites to rank more highly in organic search results.
In the following figure, the highlighted results are all organic results:
Organic results on the SERP
The box on the right side of this SERP is known as the Knowledge Graph (also sometimes called the Knowledge Box). This is a feature that Google introduced in 2012 that pulls data to commonly asked questions from sources across the web to provide concise answers to questions in one central location on the SERP. In this case, you can see a wide range of information about Abraham Lincoln, such as the date and place of his birth, his height, the date on which he was assassinated, his political affiliation, and the names of his children – many of which facts have their own links to the relevant pages.
Some SERPs will feature significantly more organic results than others, such as the example above. This is due to the differing intent of various searches. There are three primary types of Internet search:
Informational
Navigational
Transactional
Informational searches are those in which the user hopes to find information on a given topic, such as Abraham Lincoln. It wouldn’t make much sense to place ads or other types of paid results on a SERP like this, as the search query “Abraham Lincoln” has very low commercial intent; the vast majority of searchers using this search query are not looking to buy something, and as such only informational results are displayed on the SERP.
Navigational queries are those in which the user hopes to locate a specific website through their search. This may be the case for individuals searching for a specific website, trying to locate a website whose URL they can no longer remember, or another type of navigational objective.
Finally, transactional searches are those in which paid results are most likely to be displayed on the SERP. Transactional searches have high commercial intent, and search queries leading to transactional SERPs may include keywords such as “buy” and other terms that suggest a strong desire to make a purchase.
Find the best keywords to target for your business with our Free Keyword Tool.
Paid Results
In contrast to organic results, paid results are those that have been paid to be displayed by an advertiser. In the past, paid results were almost exclusively limited to small, text-based ads that were typically displayed above and to the right of the organic results. Today, however, paid results can take a wide range of forms, and there are dozens of advertising formats that cater to the needs of advertisers.
Some paid results on the Google SERP
In the example above (a SERP for the search query “lawnmowers”), all of the results on the SERP – with the exception of the map and business listing beneath it – are paid results. The three large text-based ads at the top of the SERP (considered prime positioning for advertisers) are typical PPC ads. Of those three ads, the lower two (for and) both feature ad extensions allowing prospective customers to navigate to specific pages on their websites directly from the ads.
The image-based ads on the right of the page are Shopping ads, a feature offered on the Google AdWords platform that allows ecommerce retailers’ product information to be displayed alongside other results on the SERP. Shopping ads can contain a wide range of information, such as product availability, user reviews, special offers, and more.
There are two additional PPC ads directly beneath the Shopping ads (as indicated by the yellow “Ads” flag above them) that also feature the user review ad extensions, indicated by the star ratings directly beneath the destination URL.
The map and business listing are the only results on this SERP that are not explicitly paid results. This map is shown based on a user’s location, and feature listings for local businesses that have set up their free Google My Business listing. Google My Business is a free directory of companies that can help smaller local businesses increase their visibility to searchers based on geolocation, a particularly important feature on mobile.
Ranking Signals and Search Algorithms
Organic results are listings that have been indexed by the search engine based on a number of factors, also known as “ranking signals. ”
For example, the search algorithm used by Google features hundreds of ranking factors, and while nobody outside of Google knows precisely what they are, some are thought to be more important than others. In the past, the link profile of a site – the number of external links that link to a specific website or web page from other websites – was an important ranking signal. It still is to some extent (which is why Wikipedia ranks so prominently in organic results for so many queries), though search advances at such a rapid pace that ranking signals that were once crucial to the search algorithm may be less important today, a source of constant frustration to SEOs.
Search Engine Optimization
As its name implies, search engine optimization is the practice of optimizing websites and web pages for discovery in search engines and, as a result, more visible placement on search engine results pages. This is accomplished through a variety of means, from what is known as “on-page” SEO to “off-page” techniques.
On-Page SEO
On-page SEO refers to best practices that web content creators and site owners can follow to ensure their content is as easily discoverable as possible. This includes the creation of detailed page metadata (data about data) for each page and elements such as images, the use of unique, static URLs, the inclusion of keywords in relevant headings and subheadings, and the use of clean HTML code, to name a few.
Off-Page SEO
Unlike on-page SEO techniques, off-page SEO refers to strategies that affect the site as a whole. Common off-page SEO techniques include link building and exchange, social bookmarking, content marketing, submissions to directories and search engine indexes, and the creation of online communities on social media.
Although the full scope of SEO is far too broad to cover comprehensively here, all you need to know is that SEO focuses primarily on achieving higher rankings from an organic perspective. Businesses may hire an agency or SEO professional to perform their SEO work, but beyond this investment, no money changes hands and all the emphasis is placed on ranking more highly in organic search.
PPC and Search Engine Results Pages
Unlike SEO, pay-per-click marketing focuses solely on the investment of advertising budget to achieve prominent positioning on search engine results pages. However, it’s not as simple as merely throwing more money at a campaign – advertisers must think strategically to achieve their goals.
Paid search functions as an auction. Advertisers bid on keywords that are relevant to their business that can trigger the display of their ads when users search for those terms. A wide range of factors determine where an ad will be shown on the SERP. Some ads might be displayed above the organic search results (such as the Lowe’s, Craftsman, and Husqvarna examples in the “lawnmowers” SERP example above), whereas others may be shown to the right of the organic results. Some advertisers choose to limit the display of their ads to mobile searches only, whereas others exclude mobile results altogether. Some ads feature extensions, and some do not.
Like SEO, paid search is a complex topic, but for now, just remember that paid search focuses on optimizing ads to be shown in as prominent a position on the SERP as possible.
Own the SERP with WordStream
Whether you’re hoping to optimize a PPC campaign for prime positioning or want to identify new opportunities for SEO through comprehensive keyword research, WordStream can help.
WordStream Advisor, our intuitive, centralized digital marketing management platform, makes online advertising easy. With full integration with Facebook Advertising, a suite of specialized keyword research and diagnostic tools, and intuitive, customized reporting, WordStream Advisor gives you everything you need to own the SERP and grow your business through digital marketing.
Try a risk-free, no-obligation free trial of WordStream Advisor today.
10 Best SERP API to Scrape Real-time Search Engine Results …
Google constantly keeps on changing its SERP structure and overall algorithm, so it’s essential to scrape search results via accurate sources.
And it’s all the same with other search engines as well. Most of the things that work right now will soon become a thing of the past. In that case, if you’ll keep on relying on an outdated method of scraping SERP data, you’ll be lost among the trenches.
To avoid that from happening, you need to jump on the solution that works now and in the future.
Before we get started, let’s get the basics right.
What is SERP?
SERP stands for Search Engine Results Pages.
Take this for an example: Whenever you search for something on Google (or other search engines), you get a page full of helpful information related to your query. If you’re searching for cat photos, you’ll get cat photos are shown and relevant articles.
Google SERP has evolved recently. You get the results on the Google search page. This means you don’t need to visit another site to find out what you are looking for. Ex: if you search for “open source vulnerability scanner, ” you get the list of a scanner on the page as shown below.
This is called featured snippet, one of the SERP features. There are many other SERP features available like knowledge panel, site links, image pack, a tweet box, etc.
How to check SERP manually?
Manually checking SERP data used to be easy and reliable in the past. However, these days, it’s not the same anymore. You don’t often get accurate results from it because a lot of elements like your search history, device, and location affect the process.
And no, you can’t even rely on the glorious incognito mode.
However, following the below actions might possibly land you with accurate data:
Sign out of your Gmail account
Clear cookies and history
Change your location to the one you’re searching in
Open a new tab or window for every new search
Even if it does throw up accurate results, it’s still a lot of manual work. I am a huge fan of saving time, so here’s what you need to know about using SERP API.
Saves time
Imagine spending half of your day’s time trying to do things manually and ending up not focusing on aspects that are more important for your business. That’d be terrible.
Fortunately, using SERP API can ridiculously save you a ton of time and cut down on the labor work.
Accurate results
As I mentioned earlier, checking SERP manually can be a hit or miss sometimes. There are a lot of factors that you need to take care of to make sure you’re getting the right results. However, such is not the case with SERP API. You’re guaranteed only to receive the most accurate data, every time.
No rocket science
Using a SERP API does not require you to have any hardcore coding knowledge. Most of the technical stuff is already done by the source from which you’re getting the API. This makes sure you don’t have to touch those “complicated” areas.
Enough of theory. Let’s explore the API solutions.
Serpstack
With real-time and super accurate Google search results, Serpstack is hands down one of my favorites in this list. It is completed based on JSON REST API and goes well with every programming language out there.
Moreover, it covers all of the critical search engine elements like:
Web
Images
Videos
News
Shopping
Some of the features that are worth talking about are:
No-waiting processing of high-volume API requests
Various customization options
Robust proxy network enabled
Super tight security with 256-bit SSL encryption
Straightforward integration
Lightning-fast speed of API requests
You can use this API free for the first 100 requests/month after that pricing plans start from just $29. 99 per month.
Oxylabs
With a Trustpilot score of 4. 5/5 stars, Oxylabs’s SERP Scraper API does an excellent job of giving you public data from the main search engines out there.
Whether you want to use it for brand monitoring, ads data tracking, keyword data mining, or other reason, it lets you collect data effortlessly. Here are some main features of it:
Receive data from both, organic and paid results
100% data delivery
Structured data in CSV and as well as JSON format
Highly scalable and easy to integrate
Hassle-free data extraction on their part so you don’t have to do much
On top of that, you get your own dedicated account manager, 24/7 support, and any other necessary help to ensure a smooth flow of the process.
Zenserp
Built with the intention of “speed” in mind, Zenserp is another popular choice that makes scraping Google search results a breeze. You can easily integrate this solution via browser, CURL, Python,, or PHP.
It lets you scrape through the latest SERP elements like:
Organic results
Paid results
Answer box
Featured snippet
Maps
Zenserp comes with some really impressive features like:
Real-time results
Ability to view location-based results
Super accurate data
Ability to handle high volume requests
This API also costs $29. 99 for the starting plan and comes with a free version as well with limitations.
SERPMaster
SERPMaster prides itself as being the lowest cost Google SERP scraper with guaranteed data delivery for each request sent. Getting started with their API is easy, as code examples are provided for cURL, Python, PHP, and integrations, as well as making requests from one’s web browser.
Features
Data from all SERP features (organic, ads, featured snippet, etc)
Global location support
SERPs from any device or browser
Real-time data scraping
JavaScript rendering
Batch requests
24/7 support
SERPMaster plans provide the best value for money on the market. You can start using their API for just $20 per month.
SerpWow
With over 620 satisfied customers, SerpWow can be a great choice for reliable SERP data. It lets you scrape various search engines like:
Google
Yahoo
Bing
Baidu
Yandex
Naver
It allows scraping desktop, mobile, and as well as tablet results. Other features include:
Worldwide locations
Structured data via JSON, CSV or HTML
Easy integration via Python, PHP,, CURL or Galang
Works seamlessly without proxies
Can handle high volume API requests with ease
You can make use of their free trial first and then pick one of their premium plans starting from just $9 per month.
SerpAPI
Trusted by companies like IBM, Harvard University, and The Hoth, SerpAPI is packed with advanced features that can make scraping SERP super smooth. Here are some of them:
No waiting, real-time results
Solves CAPTCHAs automatically
Ability to get results from any location
JSON Results like maps, shopping and knowledge graph
Super security with Legal US Shield
Premium plans start from just $50/month, and you can test it out with the free trial available.
DataForSeo
Another brilliant option for SERP API, DataForSEO supports most of the major search results like:
Featured snippets
Rich snippets
Answer boxes
Top stories
Knowledge graph
This API has one of the most support teams that’ll help you throughout the process of testing and integrating. Some features that are worth mentioning are:
Accurate data
Fast processing of results
All worldwide locations results
Postback/Pingback options
Organized view of top 100 organic and paid results
Very easy to integrate
Informational stats that indicate the usage of your API
DataForSEO uses the “Pay Per Use” pricing structure, which means you only pay for what you’ve used. Nothing more, nothing less. The entry-level payment amount starts at $50-$100 and has a free trial as well.
I recommend checking out their pricing page for a better understanding.
Apify
Apify’s Google search scraper crawls the SERPs and returns back various data in the groups of:
Ads
Organic
Product ads
Related queries
It has some pretty useful features like the ability to search within a specific location and extract custom attributes. In addition, you can keep a watch on what your competitors are ranking and also analyze ads for your chosen keywords.
There is a lot more to it, and you can find everything out by taking their free trial. Premium pricing plans start from just $49/month.
GeoRanker
With the ability to scrape through maps, organic, or ad results, GeoRanker is definitely a great contender that can simplify the task of checking the SERP. Some of the main features include:
Simple integration method
Works for any search engines
Supports all locations
And much more!
They have a free trial that you can get into and then book one of their paid plans by talking to their representatives.
SERPhouse
Last but not least, SERPhouse API works for Google and Bing and is very easy to use since it displays results in the REST JSON format. You can scrape results like:
Carousel
People also ask
They also recently rolled out the ability to fetch images, which is pretty cool. With this API, you get access to features like:
Multiple device selection
Postback/Pingback option
Location and language-specific option
Automatically solves CAPTCHAs
No waiting request processing
It comes with a free plan with limitations, and as well as paid plans starting at just $9. 99/month.
Conclusion
There you have it. I hope the above API will help you save time by automating the process of checking SERP and that, too, accurately. If you are interested in building your own scraper then check out these popular frameworks.
Search engine results page – Wikipedia
Search Engine Results Pages (SERP) are the pages displayed by search engines in response to a query by a user. The main component of the SERP is the listing of results that are returned by the search engine in response to a keyword query.
The results are of two general types:
organic search: retrieved by the search engine’s algorithm
sponsored search: advertisements.
The results are normally ranked by relevance to the query. Each result displayed on the SERP normally includes a title, a link that points to the actual page on the Web, and a short description showing where the keywords have matched content within the page for organic results. For sponsored results, the advertiser chooses what to display.
Due to the huge number of items that are available or related to the query, there are usually several pages in response to a single search query as the search engine or the user’s preferences restrict viewing to a subset of results per page. Each succeeding page will tend to have lower ranking or lower relevancy results. Just like the world of traditional print media and its advertising, this enables competitive pricing for page real estate, but is complicated by the dynamics of consumer expectations and intent— unlike static print media where the content and the advertising on every page is the same all of the time for all viewers, despite such hard copy being localized to some degree, usually geographic, like state, metro-area, city, or neighbourhood, search engine results can vary based on individual factors such as browsing habits. [1]
Components[edit]
The organic search results, query, and advertisements are the three main components of the SERP, However, the SERP of major search engines, like Google, Yahoo!, Bing, Petal, Sogou may include many different types of enhanced results (organic search, and sponsored) such as rich snippets, images, maps, definitions, answer boxes, videos or suggested search refinements. A recent study revealed that 97% of queries in Google returned at least one rich feature. [2]
The major search engines visually differentiate specific content types such as images, news, and blogs. Many content types have specialized SERP templates and visual enhancements on the first search results page.
Search query[edit]
Also known as ‘user search string’, this is the word or set of words that are typed by the user in the search bar of the search engine. The search box is located on all major search engines like Google, Yahoo, Bing, Petal, Sogou. Users indicate the topic desired based on the keywords they enter into the search box in the search engine.
In the competition between search engines to draw the attention of more users and advertisers, consumer satisfaction has been a driving force in the evolution of the search algorithm applied to better filter the results by relevancy.
Search queries are no longer successful based upon merely finding words that match purely by spelling. Intent and expectations have to be derived to determine whether the appropriate result is a match based upon the broader meanings drawn from context.
And that sense of context has grown from simple matching of words, and then of phrases, to the matching of ideas. And the meanings of those ideas change over time and context. Successful matching can be crowdsourced, what are others currently searching for and clicking on, when one enters keywords related to those other searches. And the crowdsourcing may be focused based upon one’s own social networking.
With the advent of portable devices, smartphones, and wearable devices, watches and various sensors, these provide ever more contextual dimensions for consumer and advertiser to refine and maximize relevancy using such additional factors that may be gleaned like: a person’s relative health, wealth, and various other status, time of day, personal habits, mobility, location, weather, and nearby services and opportunities, whether urban or suburban, like events, food, recreation, and business. Social context and crowdsourcing influences can also be pertinent factors.
The move away from keyboard input and the search box to voice access, aside from convenience, also makes other factors available to varying degrees of accuracy and pertinence, like a person’s character, intonation, mood, accent, ethnicity, and even elements overheard from nearby people and the background environment.
Searching is changing from explicit keywords: on TV show w, did x marry y or z, or election results for candidate x in county y for this date z, or final scores for team x in game y for this date z to vocalizing from a particular time and location: hey, so who won. And getting the results that one expects.
Organic results[edit]
Organic SERP listings are the natural listings generated by search engines based on a series of metrics that determine their relevance to the searched term. Webpages that score well on a search engine’s algorithmic test show in this list. These algorithms are generally based upon factors such as quality and relevance of the content, expertise, authoritativeness, and trustworthiness of the website and author on a given topic, good user experience and backlinks. [3]
People tend to view the first results on the first page. [4] Each page of search engine results usually contains 10 organic listings (however some results pages may have fewer organic listings). According to a 2019 study, [5] the CTR’s for the first page goes as follows:
Position 1: 31. 7%
Position 2: 24. 7%
Position 3: 18. 7%
Position 4: 13. 6%
Position 5: 9. 5%
Position 6: 6. 2%
Position 7: 4. 2%
Position 8: 3. 1%
Position 9: 3%
Position 10: 3. 1%
[edit]
Several major search engines offer “sponsored results” to companies, who may pay the search engine to have their products or services appear above other search hits. This is often done in the form of bidding between companies, where the highest bidder gets the top result. A 2018 report from the European Commission showed that consumers generally avoid these top results, as there is an expectation that the topmost results on a search engine page will be sponsored, and thus less relevant. [6]
Rich snippets[edit]
Rich snippets are displayed by Google in the search results pages when a website contains content in structured data markup. Structured data markup helps the Google algorithm to index and understand the content better.
Google supports rich snippets for the following data types:[7]
Product – Information about a product, including price, availability, and review ratings.
Recipe – Recipes that can be displayed in web searches and Recipe View.
Review – A review of an item such as a restaurant, movie, or store.
Event – An organized event, such as musical concerts or art festivals, that people may attend at a particular time and place.
Software Application – Information about a software app, including its URL, review ratings, and price.
Video – An online video, including a description and thumbnail and It shows the three resources of answers image, content and URL in one box.
News article – A news article, including the headline, images, and publisher info.
Science datasets
Job-related content
Mark up “how-to” pages
Sitelinks Search Box
Frequently asked questions (FAQ)
Movie
Job training & job posting
Breadcrumb
Featured snippets[edit]
Featured Snippet is a summary of an answer to user’s query. This snippet appears at the top of the list of search hits. Google supports the following types of featured snippets:[8]
Knowledge graph[edit]
Search engines like Google, Bing, Sogou have started to expand their data into Encyclopedia and other rich sources of information.
Google for example calls this sort of information “Google Knowledge Graph”, if a search query matches it will display an additional sub-window on right hand side with information from its sources. [9][10]
Google Discover[edit]
Google discover formerly known as Google feed is a way of getting topics and news information to users on the homepage below the search box. [11]
Generation[edit]
Major search engines like Google, Yahoo!, Bing, Petal, Sogou primarily use content contained within the page and fallback to metadata tags of a web page to generate the content that makes up a search snippet. [12] Generally, the HTML title tag will be used as the title of the snippet while the most relevant or useful contents of the web page (description tag or page copy) will be used for the description.
Scraping and automated access[edit]
Search engine result pages are protected from automated access by a range of defensive mechanisms and the terms of service. [13] These result pages are the primary data source for Search engine optimization, the website placement for competitive keywords that has become an important field of business and interest. Google has even used Twitter to warn users against this practice[14]
The sponsored (creative) results on Google can cost a large amount of money for advertisers. The most expensive keywords are for legal services, especially personal injury lawyers in highly competitive markets. These keywords range in the hundreds of USD, while the most expensive is nearly 1000 USD for each sponsored click.
The process of harvesting search engine result pages data is usually called “search engine scraping” or in a general form “web crawling” and generates the data SEO related companies need to evaluate website competitive organic and sponsored rankings. This data can be used to track the position of websites and show the effectiveness of SEO as well as keywords that may need more SEO investment to rank higher.
See also[edit]
User intent
References[edit]
^ “Measuring the Filter Bubble: How Google is influencing what you click”. DuckDuckGo Blog. 2018-12-04. Retrieved 2019-12-04.
^ “Google Glossary: Revenge of Mega-SERP”. Moz. Retrieved 2016-05-25.
^ “How Search algorithms work”. Google.
^, Jansen, B. J., and Spink, A. 2004. An Analysis of Documents Viewing Patterns of Web Search Engine Users. In Web Mining: Applications and Techniques. Editor: Anthony Scime. p. 339-354.
^ Dean, Brian (27 August 2019). “We Analyzed 5 Million Google Search Results. Here’s What We Learned About Organic Click Through Rate”. Backlinko.
^ “Behavioural Study on the Transparency of Online Platforms” (PDF). European Union. 2018. p. 19. Retrieved 22 April 2021.
^ “Enable Search result features for your site”.
^ Willson, Amelia (20 October 2017). “A Complete List of the Different Types of Featured Snippets”.
^ “Knowledge – Inside Search – Google”.. Retrieved 2017-04-04.
^ “Bing for Partners helps businesses and developers succeed”.. Retrieved 2017-04-04.
^ “Enable Search result features for your site | Google Search Central”. Google Developers. Retrieved 2021-03-03.
^ Cutts, M. (2007) Video: anatomy of a search snippet. Matt Cutts: Gadgets, Google, and SEO.
^ “Google Terms of Service – Privacy & Terms – Google”.. Retrieved 2017-04-04.
^ “Google Warns: Automated Queries On Google Is Against The Terms Of Service”. Retrieved 2017-04-04.
