Blog – advertools Blog

Using the crawl meta parameter while crawling

crawling

scraping

advertools

scrapy

An overview of how to use the new meta parameter while crawling with advertools. Set arbitrary metadata, set custom request headers per URL, and limited support for crawling JavaScript websites.

Word Co-Occurrence Matrix

data visualization

plotly

analytics

text analysis

keywords

How to create a co-occurrence matrix of words/tokens in a list of documents (keywords, page titles, product descriptions, etc.)

Zooming in to a plot for clearer interpretation

data visualization

plotly

analytics

How to create a zoomed-in sub-chart to focus on a certain region of the chart, while keeping the original chart for context.

Evergreen Crawling Using XML Sitemaps

crawling

scraping

advertools

sitemaps

An approach to updating a crawl file by crawling only new and/or updated URLs.

XML Sitemap Request Headers

sitemaps

advertools

v0.15

Setting custom request headers while fetching and parsing XML sitemaps.

Comparing Crawls

crawling

scraping

advertools

v0.15

Various ways to compare two crawls of the same website based on the common URLs and your choice of page element (title, h1, meta description, etc).

Doing Data Work Using Code

programming

code

work

A comparison between doing work with code and doing work by moving the mouse and clicking on buttons. The advantages and opportunities.

Log File Analysis at the Command Line

analytics

CLI

log files

A collection of recipes for various log file analysis tasks like getting requests per day, Google bot visits, and more.

Audit and Analyze Internal Links

links

analytics

python

advertools

adviz

crawling

How to map internal links of a website, count in/out/all/links, calculate Pagerank, cluster and visualize by PR.

The Structure of an advertools Crawl File

crawling

scraping

advertools

Understanding how the crawl file is created, how its columns are structured, and building a deep understanding of how the various columns relate to one another.

Risk of Trusting AI Too Much

AI

LLMs

There are many risks associated with the use of artificial intelligence. One of these risks is people trusting it more than humans, particularly regarding areas where we are (so far) better.

Should You Make an LLM App? - Slides

An attempt to create simple guidelines to help decide which are the ideal situations to create an LLM app, using a programmatic approach (API and/or finetuning).

Should You Make an LLM App?

An attempt to create simple guidelines to help decide which are the ideal situations to create an LLM app, using a programmatic approach (API and/or finetuning).

Body Text XPath Selector

crawling

scraping

advertools

xpath

How to extract/scrape the main content of a webpage while crawling. A function that allows you to customize which tags to include/exclude.

Evaluating Content in Bulk with LLMs

LLMs

python

AI

Generative AI

How to ask structured questions in bulk to develop an evaluation of content based on standard criteria (using Google’s content quality guidlines).

Bulk Prompting Workflow

LLMs

python

AI

Generative AI

How to create bulk prompts for LLMs using a structured dataset, a prompt template, and two lines of Python code.

Google Search Console Audit & Analysis Template - Slides

auditing

analysis

python

advertools

adviz

racing chart

maps

gsc

A Python template for importing, processing, visualizing, and analyzing a GSC account using various libraries and techniques.

XML Sitemap Dashboard

sitemaps

data visualization

python

advertools

adviz

dashboards

Prompt Engineering vs. Prompt Prototyping

LLMs

AI

Generative AI

A comparison between trial and error for creating useful prompts and the process of prompting on a large scale.

SERP Analysis

data science

python

seo

serp

Analyzing SERPs on a large scale. An approach and a flexible visualization for better understanding SERPs across an industry, topic, or set of keyword variants.

Automating Python Scripts

data science

python

seo

automation

Running automated scripts on a server on continuous basis, instead of manually.

Programming vs Software Development/Engineering

python

programming

software

An important difference that is meant to be good news for people who want to learn programming. Also make a clear distinction with software development, and clarifies the differences.

Inverting a mapping

python

programming

content

data manipulation

A simple and powerful script to invert a mapping where each key is mapped to multiple values. The result is that each value will be mapped to multiple keys.

Audit and Analyze External Links

links

analytics

python

advertools

adviz

gsc

crawling

Getting the number of unique domains that a website links to can give a good idea on which other website they give importance. Next, we can check status codes, identify and locate any broken links, and take a look at redirects.

Auditing vs. Analysis

auditing

analysis

strategy

The main differences between making sure things are running correctly and legally, and trying to understand things from a strategic point of view.

Google Search Console Analysis Template

auditing

analysis

python

advertools

adviz

racing chart

maps

gsc

A template for importing, processing, visualizing, and analyzing a GSC account using various Python libraries and techniques.