mdl_extract.h File Reference

Extract image URLs and anchor links from an HTML page (v1 scanner). More...

#include <stddef.h>
#include <stdint.h>
#include "ra8_err.h"

Include dependency graph for mdl_extract.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	mdl_url_list_t
	Bounded list of absolute URLs found on a page. More...
struct	mdl_hit_t
	One discovery hit: a human-facing title paired with a series URL. More...
struct	mdl_hit_list_t
	Bounded list of titled hits plus the raw anchor tally. More...

Enumerations
enum	mdl_extract_limits_t : uint16_t {   k_mdl_max_urls = 2048 ,   k_mdl_url_max = 512 }
	Fixed capacities for an extracted URL list (zero dynamic alloc). More...
enum	mdl_hit_limits_t : uint16_t {   k_mdl_max_hits = 128 ,   k_mdl_hit_title_max = 256 }
	Fixed capacities for a titled-anchor hit list (search/browse). More...

Functions
bool	mdl_extract_resolve_url (const char *base, const char *raw, char *out, size_t out_cap)
	Resolve a possibly relative HTTP(S) URL against an absolute base URL.
ra8_err_t	mdl_extract_selector (const char *html, size_t html_len, const char *selector, char *out, size_t out_cap)
	Extract one bounded metadata value using a site-descriptor selector.
ra8_err_t	mdl_extract_images (const char *html, size_t html_len, const char *base_url, const char *prefer_attr, const char *url_contains, mdl_url_list_t *out)
	Scan html for <img> image URLs, resolved to absolute form.
ra8_err_t	mdl_extract_anchors (const char *html, size_t html_len, const char *base_url, const char *href_contains, mdl_url_list_t *out)
	Scan html for <a href> links, resolved to absolute form.
ra8_err_t	mdl_extract_hits (const char *html, size_t html_len, const char *base_url, const char *url_contains, mdl_hit_list_t *out)
	Scan html for <a href> links, keeping each hit's title and URL.

Detailed Description

Extract image URLs and anchor links from an HTML page (v1 scanner).

A deliberately small tag scanner, NOT a DOM parser: it finds <img> / <a> tags, reads an attribute, resolves relative URLs against the page URL, and filters by a substring. It is enough to drive real sites host-side. On-device this is replaced by litehtml (already vendored) behind these signatures, and the per-site match strings come from the config descriptor – neither change touches callers.

Copyright

Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Definition in file mdl_extract.h.

Enumeration Type Documentation

mdl_extract_limits_t

enum mdl_extract_limits_t : uint16_t

Fixed capacities for an extracted URL list (zero dynamic alloc).

Enumerator
k_mdl_max_urls	Max URLs captured per page (chapters or images).
k_mdl_url_max	Max bytes per URL, including the NUL.

Definition at line 23 of file mdl_extract.h.

mdl_hit_limits_t

enum mdl_hit_limits_t : uint16_t

Fixed capacities for a titled-anchor hit list (search/browse).

Enumerator
k_mdl_max_hits	Max titled hits captured per results page.
k_mdl_hit_title_max	Max title bytes per hit, including the NUL.

Definition at line 35 of file mdl_extract.h.

Function Documentation

mdl_extract_anchors()

ra8_err_t mdl_extract_anchors	(	const char *	html,
		size_t	html_len,
		const char *	base_url,
		const char *	href_contains,
		mdl_url_list_t *	out )

Scan html for <a href> links, resolved to absolute form.

Parameters

[in]	html	HTML bytes (need not be NUL-terminated).
[in]	html_len	Length of html in bytes.
[in]	base_url	Absolute URL of the page.
[in]	href_contains	If non-NULL and non-empty, keep only hrefs whose absolute URL contains this substring.
[out]	out	List to fill; out->count is reset first.

Return values

k_ra8_ok	Scan complete (count may be 0).
k_ra8_err_invalid_arg	NULL argument.
k_ra8_err_no_mem	Reached k_mdl_max_urls; remainder skipped.

Uses caller-owned fixed-capacity results and performs no allocation. Inputs stay borrowed and appended values are complete and terminated.

Returns

Operation status.

Precondition

Every required pointer is non-null and remains valid for the call.

Lengths and capacities describe complete referenced objects without overflow.

Postcondition

Documented outputs and the return value describe the same outcome.

A rejected or failed operation is never reported as successful.

Note

Thread safety follows ownership of the supplied context; no synchronization is added.

Since

Version 0.1.0

Definition at line 342 of file mdl_extract.c.

References internal_scan_tags().

Referenced by priv_mdl_app_prepare_chapters().

mdl_extract_hits()

ra8_err_t mdl_extract_hits	(	const char *	html,
		size_t	html_len,
		const char *	base_url,
		const char *	url_contains,
		mdl_hit_list_t *	out )

Scan html for <a href> links, keeping each hit's title and URL.

The discovery counterpart to mdl_extract_anchors: instead of a bare URL list it yields (title, URL) pairs suitable for a numbered search/browse listing. For every <a> whose resolved absolute href contains url_contains it records the URL and a best-effort title – the anchor's title= attribute, else its inner text with nested tags stripped, whitespace collapsed and the common HTML entities decoded, else the URL's last path segment. Duplicate URLs are merged, and a later occurrence carrying a real title upgrades an earlier slug-only fallback (a results card is often a thumbnail link followed by a titled text link to the same series). The total number of resolvable anchors scanned – before the filter – is reported in out->anchors_seen so the caller can distinguish "no match" from "no links".

Parameters

[in]	html	HTML bytes (need not be NUL-terminated).
[in]	html_len	Length of html in bytes.
[in]	base_url	Absolute URL of the page (for relative resolution).
[in]	url_contains	If non-NULL and non-empty, keep only hits whose absolute URL contains this substring.
[out]	out	List to fill; out->count and out->anchors_seen are reset first.

Returns

An ra8_err_t scan result.

Return values

k_ra8_ok	Scan complete (count may be 0).
k_ra8_err_invalid_arg	A NULL html, base_url or out.
k_ra8_err_no_mem	Reached k_mdl_max_hits; the rest were skipped but anchors_seen still counts them.

Precondition

html, base_url and out are non-NULL.

out points to writable mdl_hit_list_t storage.

Postcondition

out->count hits are unique by URL and each has a non-empty title.

out->anchors_seen >= out->count.

Note

Not thread-safe: writes caller storage.

See also

mdl_extract_anchors

mdl_search_classify

Since

0.1.0

Definition at line 372 of file mdl_extract_hits.c.

References mdl_hit_list_t::anchors_seen, mdl_hit_list_t::count, internal_emit_hit(), internal_find_ci(), internal_is_name_end(), k_mdl_max_hits, k_ra8_err_invalid_arg, k_ra8_err_no_mem, k_ra8_ok, memchr(), and memset().

Referenced by mdl_discover_run().

mdl_extract_images()

ra8_err_t mdl_extract_images	(	const char *	html,
		size_t	html_len,
		const char *	base_url,
		const char *	prefer_attr,
		const char *	url_contains,
		mdl_url_list_t *	out )

Scan html for <img> image URLs, resolved to absolute form.

Parameters

[in]	html	HTML bytes (need not be NUL-terminated).
[in]	html_len	Length of html in bytes.
[in]	base_url	Absolute URL of the page (for relative resolution).
[in]	prefer_attr	"data-src" or "src"; the other is tried as fallback.
[in]	url_contains	If non-NULL and non-empty, keep only URLs that contain this substring (drops loaders/ads/nav icons).
[out]	out	List to fill; out->count is reset first.

Return values

k_ra8_ok	Scan complete (count may be 0).
k_ra8_err_invalid_arg	NULL argument.
k_ra8_err_no_mem	Reached k_mdl_max_urls; remainder skipped.

Note

Duplicate URLs are dropped. Values longer than k_mdl_url_max are skipped rather than truncated.

Uses caller-owned fixed-capacity results and performs no allocation. Inputs stay borrowed and appended values are complete and terminated.

Returns

Operation status.

Precondition

Every required pointer is non-null and remains valid for the call.

Lengths and capacities describe complete referenced objects without overflow.

Postcondition

Documented outputs and the return value describe the same outcome.

A rejected or failed operation is never reported as successful.

Since

Version 0.1.0

Definition at line 330 of file mdl_extract.c.

References internal_scan_tags(), and strcmp().

Referenced by internal_extract_page_images(), and internal_mdl_fetch_chapter_html().

mdl_extract_resolve_url()

bool mdl_extract_resolve_url	(	const char *	base,
		const char *	raw,
		char *	out,
		size_t	out_cap )

Resolve a possibly relative HTTP(S) URL against an absolute base URL.

Accepts absolute, scheme-relative, root-relative, and path-relative URLs. Fragment-only and data: values are rejected so descriptor-derived cover links follow the same URL rules as chapter and page-image extraction.

Parameters

[in]	base	Absolute HTTP(S) page URL.
[in]	raw	Raw attribute value to resolve.
[out]	out	Destination for the absolute URL.
[in]	out_cap	Capacity of out, including the terminator.

Returns

Whether a complete absolute URL was written.

Return values

true	raw was supported and the result fit.
false	An argument was NULL, the URL kind was rejected, or it did not fit.

Precondition

out points to out_cap writable bytes when non-NULL.

base and raw are NUL-terminated when non-NULL.

Postcondition

On true, out contains one NUL-terminated absolute URL.

On false, callers do not consume out.

Note

Thread-safe: uses caller-owned storage only.

Since

0.1.0

Resolve a possibly relative HTTP(S) URL against an absolute base URL.

Definition at line 164 of file mdl_extract.c.

References internal_authority_of(), internal_copy_fits(), internal_resolve_path_rel(), internal_resolve_root_rel(), internal_resolve_scheme_rel(), k_authority_max, and strncmp().

Referenced by internal_emit_hit(), internal_emit_tag_url(), and internal_extract_cover().

mdl_extract_selector()

ra8_err_t mdl_extract_selector	(	const char *	html,
		size_t	html_len,
		const char *	selector,
		char *	out,
		size_t	out_cap )

Extract one bounded metadata value using a site-descriptor selector.

The deliberately small selector grammar is data-driven and portable: meta:og:title reads a matching <meta property|name> content value, class:post-title cleans the matching element's visible text, label:Author(s): cleans the next anchor's text, and literal:en copies a descriptor-provided constant. Markup is scanned without allocation.

Parameters

[in]	html	HTML bytes; need not be NUL-terminated.
[in]	html_len	Number of readable bytes at html.
[in]	selector	Selector in the grammar above.
[out]	out	Destination for cleaned UTF-8 bytes.
[in]	out_cap	Capacity of out, including the terminator.

Returns

An ra8_err_t extraction result.

Return values

k_ra8_ok	A non-empty value was written.
k_ra8_err_not_found	No matching non-empty value exists.
k_ra8_err_invalid_arg	A pointer was NULL, capacity was zero, or the selector was invalid.
k_ra8_err_invalid_size	The extracted value did not fit completely.

Precondition

html, selector, and out are non-NULL.

out points to out_cap writable bytes.

Postcondition

On success, out is NUL-terminated and non-empty.

On failure, out[0] is the NUL byte when out is writable.

Note

Thread-safe: uses caller-owned storage only.

Since

0.1.0

Definition at line 627 of file mdl_extract_hits.c.

References internal_extract_class(), internal_extract_label(), internal_extract_meta(), internal_selector_copy(), k_ra8_err_invalid_arg, strchr(), and strncmp().

Referenced by internal_extract_cover(), internal_extract_optional_metadata(), internal_extract_series_metadata(), internal_mdl_fetch_select_chapter_number(), and internal_mdl_fetch_select_chapter_title().