robots.txt parser, path matcher, and per-host cache. More...

#include <stddef.h>
#include <stdint.h>

Include dependency graph for mdl_robots.h:

This graph shows which files directly or indirectly include this file:

Data Structures
struct	mdl_robots_rule_t
	One Allow/Disallow rule from the selected group. More...
struct	mdl_robots_t
	Rules and crawl-delay for the group matching our user-agent. More...
struct	mdl_robots_cache_entry_t
	One host's cached robots.txt outcome. More...
struct	mdl_robots_cache_t
	Fixed-size per-host robots.txt cache for one run. More...

Typedefs
typedef mdl_robots_fetch_result_t(*	mdl_robots_fetch_fn) (void ctx, const char robots_url, char buf, size_t cap, size_t out_len)
	Fetch-callback type: retrieve robots_url into buf.

Enumerations
enum	mdl_robots_limits_t : uint16_t { k_mdl_robots_max_rules = 64 , k_mdl_robots_path_max = 256 }
	Fixed capacities for one parsed robots.txt group. More...
enum	mdl_robots_cache_limits_t : uint16_t { k_mdl_robots_max_hosts = 16 , k_mdl_robots_host_max = 128 }
	Fixed capacities for the per-host robots cache. More...
enum	mdl_robots_rule_kind_t : uint8_t { k_mdl_rule_disallow = 0 , k_mdl_rule_allow = 1 }
	Whether a rule permits or forbids a matching path. More...
enum	mdl_robots_fetch_result_t : uint8_t { k_mdl_robots_fetch_ok = 0 , k_mdl_robots_fetch_absent = 1 , k_mdl_robots_fetch_denied = 2 }
	Outcome class of a robots.txt fetch, per RFC 9309 convention. More...

Functions
void	mdl_robots_parse (const char text, size_t len, const char ua_token, mdl_robots_t *out)
	Parse robots.txt text for the group matching ua_token.
bool	mdl_robots_allows (const mdl_robots_t robots, const char path)
	Decide whether path is allowed by a parsed robots group.
const char *	mdl_robots_disallow_reason (const mdl_robots_t robots, const char path)
	Return the Disallow pattern that forbids path, or NULL if allowed.
const mdl_robots_t *	mdl_robots_cache_consult (mdl_robots_cache_t cache, const char scheme, const char host, const char ua_token, mdl_robots_fetch_fn fetch, void ctx, char scratch, size_t scratch_cap)
	Return the cached robots rules for host, fetching on first contact.

Detailed Description

robots.txt parser, path matcher, and per-host cache.

Honouring robots.txt is, for this tool, purely about not getting banned: a site's Crawl-delay is free authoritative guidance on the rate that will not draw a block, and a Disallow is a request this tool should respect rather than trip a bot trap. The parser selects the most specific matching User-agent group, applies longest-match Allow/Disallow (with Allow winning ties), and extracts Crawl-delay. Everything is fixed-size and allocation-free so the same model can port to the device later.

The parser and matcher are pure and unit-tested directly. The per-host cache takes a fetch callback (dependency injection) so the network stays out of this translation unit and a fake fetcher drives the cache in tests.

Copyright: Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Definition in file mdl_robots.h.

Typedef Documentation

◆ mdl_robots_fetch_fn

typedef mdl_robots_fetch_result_t( * mdl_robots_fetch_fn) (void *ctx, const char *robots_url, char *buf, size_t cap, size_t *out_len)

Fetch-callback type: retrieve robots_url into buf.

Parameters

[in]	ctx	Opaque caller context (the network interface).
[in]	robots_url	Absolute URL of the host's /robots.txt.
[out]	buf	Destination buffer for the body.
[in]	cap	Capacity of buf.
[out]	out_len	Bytes written to buf.

Returns: The fetch outcome class.

Since: 0.1.0

Definition at line 200 of file mdl_robots.h.

Enumeration Type Documentation

◆ mdl_robots_cache_limits_t

enum mdl_robots_cache_limits_t : uint16_t

Fixed capacities for the per-host robots cache.

Enumerator
k_mdl_robots_max_hosts	Distinct hosts cached per run.
k_mdl_robots_host_max	Max host string bytes.

Definition at line 32 of file mdl_robots.h.

◆ mdl_robots_fetch_result_t

enum mdl_robots_fetch_result_t : uint8_t

Outcome class of a robots.txt fetch, per RFC 9309 convention.

Enumerator
k_mdl_robots_fetch_ok	A body was retrieved to parse.
k_mdl_robots_fetch_absent	Absent / transport error: allow all.
k_mdl_robots_fetch_denied	5xx status: disallow all.

Definition at line 44 of file mdl_robots.h.

◆ mdl_robots_limits_t

enum mdl_robots_limits_t : uint16_t

Fixed capacities for one parsed robots.txt group.

Enumerator
k_mdl_robots_max_rules	Max Allow/Disallow rules retained.
k_mdl_robots_path_max	Max bytes per rule path pattern.

Definition at line 26 of file mdl_robots.h.

◆ mdl_robots_rule_kind_t

enum mdl_robots_rule_kind_t : uint8_t

Whether a rule permits or forbids a matching path.

Enumerator
k_mdl_rule_disallow	A Disallow rule.
k_mdl_rule_allow	An Allow rule.

Definition at line 38 of file mdl_robots.h.

Function Documentation

◆ mdl_robots_allows()

bool mdl_robots_allows	(	const mdl_robots_t *	robots,
		const char *	path )

Decide whether path is allowed by a parsed robots group.

Applies the longest-match rule; when an Allow and a Disallow match at the same pattern length, the Allow wins (RFC 9309). A group with no rules permits everything. Patterns honour * and a trailing $.

Parameters

[in]	robots	Parsed rules from mdl_robots_parse.
[in]	path	URL path to test (with leading /).

Returns: Whether a request for path is permitted.

Return values

true	No rule matches, or the winning rule is an Allow.
false	The winning (longest) matching rule is a Disallow.

Precondition: robots and path are non-NULL.; path begins with / (a URL path, not a full URL).

Postcondition: Neither argument is modified.; A tie at the longest match is resolved in favour of Allow.

Note: Thread-safe: depends only on its arguments.

Since: 0.1.0

Definition at line 578 of file mdl_robots.c.

References internal_best_matching_rule(), k_mdl_rule_allow, mdl_robots_rule_t::kind, and mdl_robots_t::valid.

◆ mdl_robots_cache_consult()

const mdl_robots_t * mdl_robots_cache_consult	(	mdl_robots_cache_t *	cache,
		const char *	scheme,
		const char *	host,
		const char *	ua_token,
		mdl_robots_fetch_fn	fetch,
		void *	ctx,
		char *	scratch,
		size_t	scratch_cap )

Return the cached robots rules for host, fetching on first contact.

On a cache miss, builds <scheme>://<host>/robots.txt, calls fetch, applies the RFC 9309 convention (absent/transport error = allow all; 5xx = disallow all), parses a retrieved body for ua_token, and stores the outcome. Subsequent calls for the same host return the stored entry without a network round-trip. When the cache is full, the fetched result is returned uncached rather than evicting a live entry.

Parameters

[in]	cache	Per-run cache (zero-initialised before first use).
[in]	scheme	URL scheme ("http" or "https").
[in]	host	Host to consult.
[in]	ua_token	Our user-agent product token.
[in]	fetch	Fetch callback (dependency injection seam).
[in]	ctx	Opaque context passed to fetch.
[out]	scratch	Working buffer the callback fills.
[in]	scratch_cap	Capacity of scratch.

Returns: Borrowed pointer to the host's parsed rules (owned by cache), or NULL when disallow_all applies (caller refuses every path).

Return values

non-NULL	Rules to test paths against.
NULL	The host said "disallow all" (5xx); refuse every path.

Precondition: Every pointer argument is non-NULL; scratch_cap > 0.; scheme is "http" or "https".

Postcondition: A cache entry for host exists unless the cache was full.; scratch may be overwritten.

Note: Not thread-safe: mutates cache.

Since: 0.1.0

Definition at line 620 of file mdl_robots.c.

References mdl_robots_cache_entry_t::disallow_all, mdl_robots_cache_entry_t::host, internal_cache_find(), internal_cache_slot(), k_mdl_robots_fetch_denied, k_mdl_robots_fetch_ok, k_mdl_robots_host_max, k_robots_url_max, mdl_robots_parse(), nullptr, mdl_robots_cache_entry_t::rules, mdl_robots_cache_entry_t::scheme, strcmp(), strnlen(), mdl_robots_cache_entry_t::used, and mdl_robots_t::valid.

Referenced by mdl_session_url_allowed().

◆ mdl_robots_disallow_reason()

const char * mdl_robots_disallow_reason	(	const mdl_robots_t *	robots,
		const char *	path )

Return the Disallow pattern that forbids path, or NULL if allowed.

Names the exact rule responsible for a refusal so the caller can log a clear, actionable message rather than a bare "disallowed". Uses the same longest-match, Allow-wins-ties resolution as mdl_robots_allows.

Parameters

[in]	robots	Parsed rules from mdl_robots_parse.
[in]	path	URL path to test (with leading /).

Returns: The winning Disallow pattern, or NULL when path is allowed.

Return values

non-NULL	Borrowed pointer to the blocking rule's pattern.
NULL	No rule blocks path (the winning rule allows, or none).

Precondition: robots and path are non-NULL.; path begins with /.

Postcondition: Neither argument is modified; the result is owned by robots.

Note: Thread-safe: depends only on its arguments.

Since: 0.1.0

Definition at line 587 of file mdl_robots.c.

References internal_best_matching_rule(), k_mdl_rule_disallow, mdl_robots_rule_t::kind, and mdl_robots_rule_t::path.

Referenced by mdl_session_url_allowed().

◆ mdl_robots_parse()

void mdl_robots_parse	(	const char *	text,
		size_t	len,
		const char *	ua_token,
		mdl_robots_t *	out )

Parse robots.txt text for the group matching ua_token.

Selects the most specific matching User-agent group (a case-insensitive User-agent value that is a prefix of ua_token; * is the fallback), merging groups that match at the same specificity, and collects that group's Allow/Disallow rules and the strictest Crawl-delay. Empty Disallow values (which impose no restriction) are skipped. Malformed lines are ignored. An input with no matching group leaves out empty (allow all).

Parameters

[in]	text	robots.txt bytes (need not be NUL-terminated).
[in]	len	Length of text in bytes.
[in]	ua_token	Our user-agent product token (e.g. "mdl").
[out]	out	Result; fully overwritten (zeroed first).

Returns: Nothing.

Precondition: ua_token and out are non-NULL; text is non-NULL when len > 0.; out points to writable storage of sizeof(mdl_robots_t).

Postcondition: out->count <= k_mdl_robots_max_rules.; out is fully initialised even when no group matches.

Note: Thread-safe: writes only out.

Since: 0.1.0

Definition at line 450 of file mdl_robots.c.

References internal_harvest(), internal_scan_spec(), k_spec_none, and mdl_robots_t::valid.

Referenced by mdl_robots_cache_consult().

Data Structures

Typedefs

Enumerations

Functions

Detailed Description

Typedef Documentation

◆ mdl_robots_fetch_fn

Enumeration Type Documentation

◆ mdl_robots_cache_limits_t

◆ mdl_robots_fetch_result_t

◆ mdl_robots_limits_t

◆ mdl_robots_rule_kind_t

Function Documentation

◆ mdl_robots_allows()

◆ mdl_robots_cache_consult()

◆ mdl_robots_disallow_reason()

◆ mdl_robots_parse()