| gl | searchTerms | rank | title | snippet | displayLink | link | queryTime | |
|---|---|---|---|---|---|---|---|---|
| 0 | us | what is bitcoin | 1 | Bitcoin - Open source P2P money | Bitcoin uses peer-to-peer technology to operat... | bitcoin.org | https://bitcoin.org/ | 2022-04-24 18:10:18.884769+00:00 |
| 1 | us | what is bitcoin | 2 | Bitcoin Definition | Bitcoin is a digital or virtual currency creat... | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | 2022-04-24 18:10:18.884769+00:00 |
| 2 | us | what is bitcoin | 3 | What is bitcoin and how does it work? | New Sc... | Bitcoin is a digital currency which operates f... | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | 2022-04-24 18:10:18.884769+00:00 |
| 20 | us | what is litecoin | 1 | Litecoin - Open source P2P digital currency | What is Litecoin? ... Litecoin is a peer-to-pe... | litecoin.org | https://litecoin.org/ | 2022-04-24 18:10:18.910371+00:00 |
| 21 | us | what is litecoin | 2 | Litecoin (LTC) Definition | Litecoin (LTC) is a cryptocurrency created fro... | www.investopedia.com | https://www.investopedia.com/articles/investin... | 2022-04-24 18:10:18.910371+00:00 |
| 22 | us | what is litecoin | 3 | Litecoin price today, LTC to USD live, marketc... | The live Litecoin price today is $104.96 USD w... | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | 2022-04-24 18:10:18.910371+00:00 |
The SERP heatmap
This heatmap shows how often each domain appeared on each position while checking the rankings. The frequency of appearances andn the domains will depend on the queries you select. Intuitively, you should immediately be able to spot where the the most frequent positions are in the heatmap, and which domains they belong to.
In the video we discuss how to make this chart, how to interpret it, and how to use it.
Our video today:
Analyzing Google SERPs with advertools
We’ll delve into analyzing SERPs (Search Engine Results Pages) on Google using a function called serp_goog in advertools. I’ll show you how it works in the simplest way and gradually move into more advanced usages.
Requirements
You need the following setup:
Create a custom search engine: At first, you might be asked to enter a site to search. Enter any domain, then go to the control panel and remove it. Make sure you enable “Search the entire web” and image search. You will also need to get your search engine ID, which you can find on the control panel page.
Enable the custom search API: The service will allow you to retrieve and display search results from your custom search engine programmatically. You will need to create a project for this first.
Create credentials for this project: so you can get your key.
Enable billing for your project if you want to run more than 100 queries per day. The first 100 queries are free; then for each additional 1,000 queries, you pay $5.
Basic Usage
Here’s the basic usage of the function:
import advertools as adv
import pandas as pd
cx = "your_custom_search_engine_id"
key = "your_developer_key"
query = "What is Bitcoin?"
serp_data = adv.serp_goog(cx=cx, key=key, q=query)This will return a data frame showing the top 10 results for the query “What is Bitcoin”, including the rank, title, snippet, display link, and actual URL.
Summarizing Results
The function also returns metadata such as query time, total results, and many more.
You can visualize this data using a simple SERP heat map function which in its basic form:
Running Queries in Bulk
Running queries in bulk can provide more interesting insights. For example:
Analyzing Across Different Dimensions
To analyze across other dimensions, such as countries, you can run the same queries in different geolocations:
gl = ["us", "uk"] # geo-location
queries = ["What is Bitcoin?", "What is Ethereum?"]
by_country = adv.serp_goog(cx=cx, key=key, q=queries, gl=gl)This expands your dataset and allows for more comprehensive analysis such as how rankings differ between countries.
Practical Applications
Analyzing Search Terms
You can analyze various search term variations across different countries and industries. For example, comparing who ranks for “What is Bitcoin” and “Price of Bitcoin”:
Strategic Analysis
Use the data for strategic analysis like: - Word frequency in titles and snippets. - Crawling landing pages for further SEO analysis. - Examining metadata and SERP features.
Image Search
Track image search results by setting the search type to image:
query = "Pink dresses"
searchType = "image"
pink_dress = adv.serp_goog(cx=cx, key=key, q=query, searchType=searchType)You can track image URLs and analyze which domains dominate the image search results.
Advanced Features
The advertools supports additional parameters like: - Date restrict - Image color type - Dominant color - And more…
Explore the documentation for an explanation of the parameters and their usage.
adv.serp_goog(
q,
cx,
key,
c2coff=None,
cr=None,
dateRestrict=None,
exactTerms=None,
excludeTerms=None,
fileType=None,
filter=None,
gl=None,
highRange=None,
hl=None,
hq=None,
imgColorType=None,
imgDominantColor=None,
imgSize=None,
imgType=None,
linkSite=None,
lowRange=None,
lr=None,
num=None,
orTerms=None,
relatedSite=None,
rights=None,
safe=None,
searchType=None,
siteSearch=None,
siteSearchFilter=None,
sort=None,
start=None,
)