HTML Parsing

One verb, routed by the qualifier

Parse decides what to do from the result qualifier, never by sniffing the input. That makes the statement readable on its own:

(indexPage: Crawler) {
    Request the <page> from the <http-client> with "https://example.com".
    Parse the <links: links> from the <page: body>.
    Return an <OK: status> with <links>.
}

An unqualified Parse is an alias of Extract — reading a field off an already-structured value is Extract's job, not a parse format. Any other qualifier is an error naming the valid ones, rather than a guess.

Reading attributes with select

The fixed qualifiers answer the four questions their names ask. select answers the rest. The with clause is an ordinary CSS selector, optionally followed by @attribute:

Parse the <hrefs: select> from the <page> with "a@href".
Parse the <srcs: select>  from the <page> with "img@src".
Parse the <desc: select>  from the <page> with "meta[name=description]@content".
Parse the <headings: select> from the <page> with "h2".

• With @ — the attribute's value, per matching element.
• Without — the element's text.

One qualifier covers both, so nobody has to remember two names for "the link texts" and "the link targets".

An element without the attribute contributes nothing

Given <a href="/one">One</a> <a>Not a link</a> <a href="/two">Two</a>:

Parse the <texts: select> from the <page> with "a".        (* 3 items *)
Parse the <hrefs: select> from the <page> with "a@href".   (* 2 items *)

The result is the attributes that exist, not a list with holes where they do not. A list with holes is only useful if you are correlating it against another list by index — and for that, CSS already says what you meant:

Parse the <texts: select> from the <page> with "a[href]".
Parse the <hrefs: select> from the <page> with "a[href]@href".
(* same length, same order, aligned by construction *)

An attribute that is present but empty — <input value=""> — is reported as the empty string rather than dropped. Dropping it would conflate "no value" with "an empty value", and only the first is a missing attribute.

Where the content comes from

content, markdown and page take the document's main body in priority order — <main>, then <article>, then <body>. Whitespace is normalised, empty segments are dropped, and script and style elements are ignored.

markdown preserves document structure: headings become #…######, links become [text](url), images ![alt](src), <strong> becomes **bold**, <pre><code> becomes a fenced block, and lists, tables, blockquotes and rules all carry over.

A crawler, end to end

(crawl: Crawler takes <url>) {
    Extract the <start> from the <parameter: url>.
    Request the <response> from the <http-client> with <start>.

    (* title and body in one parse *)
    Parse the <page: page> from the <response: body>.
    Extract the <title> from the <page: title>.

    (* relative hrefs are resolved against the page, then normalised *)
    Parse the <hrefs: select> from the <response: body> with "a[href]@href".
    for each <href> in <hrefs> {
        Compute the <absolute: url-resolve> from <href> with { base: <start> }.
        Compute the <canonical: url-normalize> from <absolute>.
        Store the <canonical> into the <visited-repository>.
    }

    Return an <OK: status> with { title: <title>, found: <hrefs> }.
}

url-resolve and url-normalize are what make a crawler's "have I seen this?" answerable — two spellings of one address compare equal after normalising. See Computations.