Body Text XPath Selector

How to extract/scrape the main content of a webpage while crawling. A function that allows you to customize which tags to include/exclude.
crawling
scraping
advertools
xpath
Author

Elias Dabbas

Published

June 26, 2024

How to extract body text content from a webpage

Usually when crawling we are interested in extracting and analyzin the main content of a webapage. Whether this is a blog post, news article, product description, or an opinion piece, we want to get that text without the boilerplate (navigation, footers, etc.).

This is a flexible function that generates that XPath selector, and it can be modified easily. The main idea is based on the fact that some HTML tags are primarily used to display text (h1, h2, span, bold, etc), and others that aren’t (script, img, video).

Text tags

You can easily change which tags you want to consider as text tags and which you don’t, depending on your the particular website you are crawling.

Non-text tags

In some cases there are tags that might contain text that you might be interested in. For example iframes, which might include interesting content in some cases. Using “text” to differentiate between the two categories of tags might not be ideal in some cases. Usually nav elements contain text and links, but not as part of the main content of a page.

In any case you can modify the elements you want and get the final XPath expression.

def bodytext_xpath(
    text_tags=[
        "a",
        "abbr",
        "address",
        "b",
        "blockquote",
        "cite",
        "code",
        "dd",
        "del",
        "div",
        "dl",
        "dt",
        "em",
        "h1",
        "h2",
        "h3",
        "h4",
        "h5",
        "h6",
        "header",
        "i",
        "ins",
        "kbd",
        "li",
        "mark",
        "ol",
        "p",
        "pre",
        "q",
        "section",
        "small",
        "span",
        "strong",
        "sub",
        "sup",
        "time",
        "u",
        "ul",
    ],
    non_text_tags=[
        "area",
        "aside",
        "audio",
        "button",
        "caption",
        "col",
        "colgroup",
        "datalist",
        "details",
        "embed",
        "fieldset",
        "footer",
        "form",
        "head",
        "iframe",
        "img",
        "input",
        "label",
        "legend",
        "link",
        "map",
        "meta",
        "nav",
        "noscript",
        "object",
        "optgroup",
        "option",
        "output",
        "param",
        "picture",
        "script",
        "select",
        "source",
        "style",
        "svg",
        "table",
        "tbody",
        "td",
        "textarea",
        "tfoot",
        "th",
        "thead",
        "title",
        "tr",
        "track",
        "video",
    ],
):
    """Create an XPath expression to exract body text form a web page.

    This approximates the page content that you are probably interested in (article text, product description, blog post, etc.)
    You can change which tags are included and which are not by modifying the available parameters.

    Parameters
    ----------
    text_tags : list
        A list of tags that typically contain text elements. You can add to it (if you want to include text in iframes, or td for example.)
    non_text_tags : list
        A list of tags that typically don't contain text (video, image, etc.). Make sure to remove the ones you don't want from this list if you want to modify the first one.
    """
    text_tags_expr = "".join([f"self::{tag} or " for tag in text_tags])[:-4]
    non_text_tags_expr = "".join(
        [f"not(ancestor::{tag}) and " for tag in non_text_tags]
    )[:-5]
    final_exp = "//body//*[" + text_tags_expr + "][" + non_text_tags_expr + "]/text()"
    return final_exp

Running this function give us the XPath expression that can be used with any SEO crawler if you want to try it:

bodytext_xpath()
//body//*[self::a or self::abbr or self::address or self::b or self::blockquote or self::cite or self::code or self::dd or self::del or self::div or self::dl or self::dt or self::em or self::h1 or self::h2 or self::h3 or self::h4 or self::h5 or self::h6 or self::header or self::i or self::ins or self::kbd or self::li or self::mark or self::ol or self::p or self::pre or self::q or self::section or self::small or self::span or self::strong or self::sub or self::sup or self::time or self::u or self::ul][not(ancestor::area) and not(ancestor::aside) and not(ancestor::audio) and not(ancestor::button) and not(ancestor::caption) and not(ancestor::col) and not(ancestor::colgroup) and not(ancestor::datalist) and not(ancestor::details) and not(ancestor::embed) and not(ancestor::fieldset) and not(ancestor::footer) and not(ancestor::form) and not(ancestor::head) and not(ancestor::iframe) and not(ancestor::img) and not(ancestor::input) and not(ancestor::label) and not(ancestor::legend) and not(ancestor::link) and not(ancestor::map) and not(ancestor::meta) and not(ancestor::nav) and not(ancestor::noscript) and not(ancestor::object) and not(ancestor::optgroup) and not(ancestor::option) and not(ancestor::output) and not(ancestor::param) and not(ancestor::picture) and not(ancestor::script) and not(ancestor::select) and not(ancestor::source) and not(ancestor::style) and not(ancestor::svg) and not(ancestor::table) and not(ancestor::tbody) and not(ancestor::td) and not(ancestor::textarea) and not(ancestor::tfoot) and not(ancestor::th) and not(ancestor::thead) and not(ancestor::title) and not(ancestor::tr) and not(ancestor::track) and not(ancestor::video)]/text()