Implementation of the session identity + robots.txt gating. More...

#include "mdl_session.h"
#include <stdio.h>
#include <string.h>
#include "mdl_stream_internal.h"
#include "mdl_url_guard.h"
#include "ra8_attributes.h"

Include dependency graph for mdl_session.c:

Enumerations
enum	mdl_session_net_t : uint32_t { k_robots_timeout_ms = 10000U }
	robots.txt fetch tunables. More...
enum	mdl_session_status_t : uint16_t { k_http_too_many = 429 , k_http_client_err_min = 400 , k_http_client_err_max = 499 , k_http_server_err_min = 500 , k_http_server_err_max = 599 }
	HTTP server-error range treated as "disallow all". More...

Functions
const char *	mdl_session_ua_token (void)
	The robots.txt product token this tool matches on ("mdl").
bool	mdl_session_build_ua (const char contact, char out, size_t cap)
	Build the truthful default User-Agent into out.
void	mdl_session_init (mdl_session_t session, mdl_net_iface_t net, const char user_agent, ra8_io_stream_t diagnostic, bool honor_robots)
	Initialise a session over a network backend.
static bool	internal_url_prefix (const char url, const char prefix)
	ASCII case-insensitive comparison of one URL prefix.
static const char *	internal_url_scheme (const char *url)
	Return the validated lower-case HTTP(S) scheme, or NULL.
static bool	internal_robots_target (const char url, char out, size_t cap)
	Extract path plus query, excluding fragment, for RFC 9309 matching.
static mdl_robots_fetch_result_t	internal_session_fetch (void ctx, const char robots_url, char buf, size_t cap, size_t out_len)
	robots.txt fetch callback: map an mdl_net GET to a fetch result.
static void	internal_report_robots_unavailable (mdl_session_t session, const char host, const char *path)
	Report a failed robots consultation for one origin and path.
static void	internal_report_robots_disallow (mdl_session_t session, const char path, const char host, const char reason)
	Report the robots rule that rejected one path.
bool	mdl_session_url_allowed (mdl_session_t session, const char url, uint32_t *crawl_delay_ms)
	Decide whether url may be fetched under robots.txt, and its delay.

Variables
static const char *const	s_ua_token = "mdl"
	Product token used to match robots.txt User-agent groups.
static const char *const	s_ua_version = "0.1.0"
	Tool version reported in the User-Agent.
static const char *const	s_ua_url = "https://github.com/bsikar/ra8-firmware"
	Project URL reported in the User-Agent for a contact channel.

Detailed Description

Implementation of the session identity + robots.txt gating.

Builds the truthful tool identity, derives each request's robots origin and match target, and connects the bounded robots cache to the abstract network interface. Policy remains fail-closed for throttling and server failures without introducing backend-specific dependencies.

Copyright: Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Definition in file mdl_session.c.

Enumeration Type Documentation

◆ mdl_session_net_t

enum mdl_session_net_t : uint32_t

robots.txt fetch tunables.

Enumerator
k_robots_timeout_ms	Per-robots.txt request budget, ms.

Definition at line 22 of file mdl_session.c.

◆ mdl_session_status_t

enum mdl_session_status_t : uint16_t

HTTP server-error range treated as "disallow all".

Enumerator
k_http_too_many	Rate limiting: do not bypass robots.
k_http_client_err_min	First 4xx unavailable status.
k_http_client_err_max	Last 4xx unavailable status.
k_http_server_err_min	First 5xx status.
k_http_server_err_max	Last 5xx status.

Definition at line 27 of file mdl_session.c.

Function Documentation

◆ internal_report_robots_disallow()

void internal_report_robots_disallow	(	mdl_session_t *	session,
		const char *	path,
		const char *	host,
		const char *	reason )

static

Report the robots rule that rejected one path.

Writes a bounded robots-policy diagnostic through the injected stream. Any sink error is retained in the session for the caller to observe.

Parameters

[in,out]	session	Caller-owned session state read and updated.
[in]	path	Validated URL or filesystem path.
[in]	host	Validated host name.
[in]	reason	Human-readable policy reason.

Precondition: Every required pointer is non-null and remains valid for the call.; Lengths and capacities describe complete referenced objects without overflow.

Postcondition: Documented outputs and the return value describe the same outcome.; A rejected or failed operation is never reported as successful.

Note: Thread safety follows ownership of the supplied context; no synchronization is added.

Since: Version 0.1.0

Definition at line 261 of file mdl_session.c.

References mdl_session_t::diagnostic, k_ra8_ok, priv_mdl_stream_text(), and RA8_INTERNAL.

Referenced by mdl_session_url_allowed().

◆ internal_report_robots_unavailable()

void internal_report_robots_unavailable	(	mdl_session_t *	session,
		const char *	host,
		const char *	path )

static

Report a failed robots consultation for one origin and path.

Writes a bounded robots-policy diagnostic through the injected stream. Any sink error is retained in the session for the caller to observe.

Parameters

[in,out]	session	Caller-owned session state read and updated.
[in]	host	Validated host name.
[in]	path	Validated URL or filesystem path.

Precondition: Every required pointer is non-null and remains valid for the call.; Lengths and capacities describe complete referenced objects without overflow.

Postcondition: Documented outputs and the return value describe the same outcome.; A rejected or failed operation is never reported as successful.

Note: Thread safety follows ownership of the supplied context; no synchronization is added.

Since: Version 0.1.0

Definition at line 235 of file mdl_session.c.

References mdl_session_t::diagnostic, k_ra8_ok, and priv_mdl_stream_text().

Referenced by mdl_session_url_allowed().

◆ internal_robots_target()

bool internal_robots_target	(	const char *	url,
		char *	out,
		size_t	cap )

static

Extract path plus query, excluding fragment, for RFC 9309 matching.

Skips the scheme and authority, preserves a query as part of the robots match target, and substitutes / when the URL has no explicit path. The fragment is never copied.

Parameters

[in]	url	Absolute HTTP(S) URL to inspect.
[out]	out	Destination for the NUL-terminated robots match target.
[in]	cap	Capacity of `out` in bytes.

Returns: Whether the complete target fit in out.

Return values

true	A path/query target was written.
false	Arguments were invalid, no scheme separator exists, or output is too small.

Precondition: url is NUL-terminated when non-NULL.; out is writable for cap bytes when non-NULL.

Postcondition: On true, out begins with / and contains no fragment.; On a size failure, a non-empty out is cleared.

Note: This helper extracts syntax only; scheme and host policy are checked separately.

Since: 0.1.0

Definition at line 144 of file mdl_session.c.

References RA8_INTERNAL, and strstr().

Referenced by mdl_session_url_allowed().

◆ internal_session_fetch()

mdl_robots_fetch_result_t internal_session_fetch	(	void *	ctx,
		const char *	robots_url,
		char *	buf,
		size_t	cap,
		size_t *	out_len )

static

robots.txt fetch callback: map an mdl_net GET to a fetch result.

Performs a bounded GET with the session identity and maps transport and HTTP status classes onto the robots cache's allow/deny convention.

Parameters

[in]	ctx	Initialised mdl_session_t.
[in]	robots_url	Absolute robots.txt URL.
[out]	buf	Caller buffer receiving a successful response body.
[in]	cap	Capacity of `buf` in bytes.
[out]	out_len	Successful body length.

Returns: Robots fetch outcome class.

Return values

k_mdl_robots_fetch_ok	A body was retrieved.
k_mdl_robots_fetch_absent	A client absence or transport failure permits access.
k_mdl_robots_fetch_denied	Throttle or server failure closes access.

Precondition: Pointer arguments are non-NULL and buf is writable for cap bytes.; ctx contains a usable network interface and User-Agent.

Postcondition: On success, out_len describes the retrieved body.; The session's borrowed identity and backend pointers are unchanged.

Note: Thread safety follows the network backend.

Since: 0.1.0

Definition at line 200 of file mdl_session.c.

References k_http_client_err_max, k_http_client_err_min, k_http_server_err_max, k_http_server_err_min, k_http_too_many, k_mdl_robots_fetch_absent, k_mdl_robots_fetch_denied, k_mdl_robots_fetch_ok, k_ra8_ok, k_robots_timeout_ms, mdl_net_get_buf(), mdl_session_t::net, mdl_net_resp_t::status, and mdl_session_t::user_agent.

Referenced by mdl_session_url_allowed().

◆ internal_url_prefix()

bool internal_url_prefix	(	const char *	url,
		const char *	prefix )

static

ASCII case-insensitive comparison of one URL prefix.

Lower-cases only bytes read from url and compares them with the already lower-case literal prefix. The comparison stops at the end of the prefix, so trailing URL bytes are intentionally ignored.

Parameters

[in]	url	NUL-terminated URL text to inspect.
[in]	prefix	NUL-terminated lower-case ASCII prefix.

Returns: Whether url begins with prefix ignoring ASCII case.

Return values

true	Every prefix byte matched.
false	At least one byte differed.

Precondition: url and prefix are non-NULL and NUL-terminated.; prefix contains lower-case ASCII comparison text.

Postcondition: Neither input string is modified.; The result depends only on the first strlen(prefix) URL bytes.

Note: This is deliberately locale-independent and intended for URL schemes.

Since: 0.1.0

Definition at line 96 of file mdl_session.c.

References RA8_INTERNAL.

Referenced by internal_url_scheme().

◆ internal_url_scheme()

const char * internal_url_scheme ( const char * url )

static

Return the validated lower-case HTTP(S) scheme, or NULL.

Definition at line 113 of file mdl_session.c.

References internal_url_prefix(), and RA8_INTERNAL.

Referenced by mdl_session_url_allowed().

◆ mdl_session_build_ua()

bool mdl_session_build_ua	(	const char *	contact,
		char *	out,
		size_t	cap )

Build the truthful default User-Agent into out.

Emits mdl/<version> (+<project-url>; <contact>), or the same without the contact clause when none is supplied. No browser-impersonation string is ever produced.

Parameters

[in]	contact	Operator contact (email/URL), or NULL/empty for none.
[out]	out	Destination buffer for the NUL-terminated UA string.
[in]	cap	Capacity of out in bytes.

Returns: Whether a contact clause was included.

Return values

true	contact was non-empty and appears in the UA.
false	No contact was configured (caller should warn once).

Precondition: out, when non-NULL, has room for at least cap bytes.; cap is large enough for the fixed portion (>= 64 recommended).

Postcondition: out is NUL-terminated whenever cap > 0.; The generated identity always contains the tool name, version, and project URL.

Note: Thread-safe: writes only caller-provided storage.

Since: 0.1.0

Definition at line 47 of file mdl_session.c.

References s_ua_token, s_ua_url, and s_ua_version.

Referenced by priv_mdl_app_start_session().

◆ mdl_session_init()

void mdl_session_init	(	mdl_session_t *	session,
		mdl_net_iface_t *	net,
		const char *	user_agent,
		ra8_io_stream_t *	diagnostic,
		bool	honor_robots )

Initialise a session over a network backend.

Stores the borrowed backend and User-Agent pointers, selects the robots policy, and clears the fixed per-host cache. No network request is made until mdl_session_url_allowed examines a host for the first time.

Parameters

[out]	session	Session to initialise (non-NULL).
[in]	net	Backend used for robots.txt fetches.
[in]	user_agent	Full UA header string (must outlive the session).
[in,out]	diagnostic	Borrowed stream receiving robots policy diagnostics.
[in]	honor_robots	Whether robots.txt is consulted (false = ignore).

Returns: Nothing.

Precondition: session, net, and user_agent are non-NULL.; user_agent outlives every call using session.

Postcondition: The per-host cache is empty.; session->honor_robots equals honor_robots.

Note: Not thread-safe: initialises caller storage.

Since: 0.1.0

Definition at line 61 of file mdl_session.c.

References mdl_session_t::cache, mdl_session_t::diagnostic, mdl_session_t::honor_robots, mdl_session_t::net, and mdl_session_t::user_agent.

Referenced by priv_mdl_app_start_session().

◆ mdl_session_ua_token()

const char * mdl_session_ua_token ( void )

The robots.txt product token this tool matches on ("mdl").

Returns: Borrowed pointer to a static token string.

Return values

non-NULL Always: the constant product token.

Precondition: None.

Postcondition: No state is modified.

Note: Thread-safe: returns a constant.

Since: 0.1.0

Definition at line 42 of file mdl_session.c.

References s_ua_token.

◆ mdl_session_url_allowed()

bool mdl_session_url_allowed	(	mdl_session_t *	session,
		const char *	url,
		uint32_t *	crawl_delay_ms )

Decide whether url may be fetched under robots.txt, and its delay.

Consults the per-host robots cache (fetching /robots.txt on first contact), and refuses – printing a message that names the blocking rule – a URL our user-agent is disallowed from. When robots gating is off (--ignore-robots) every URL is permitted. The host's Crawl-delay is reported so the caller can raise its per-host politeness floor.

Parameters

[in]	session	Initialised session.
[in]	url	Absolute http(s) URL to test.
[out]	crawl_delay_ms	Receives the host's Crawl-delay in ms (0 if none).

Returns: Whether the fetch is permitted.

Return values

true	Robots allows it, or gating was explicitly disabled.
false	A rule forbids it, robots is inaccessible/oversized, or the URL cannot be parsed safely; a message was printed to stderr.

Precondition: session and url are non-NULL.; url uses an http(s) scheme.

Postcondition: *crawl_delay_ms is set when crawl_delay_ms is non-NULL.; The session cache may gain an entry for url's host.

Note: Not thread-safe: mutates the session cache.

Since: 0.1.0

Definition at line 276 of file mdl_session.c.

References mdl_session_t::cache, mdl_robots_t::crawl_delay_ms, mdl_session_t::diagnostic, mdl_robots_t::have_crawl_delay, mdl_session_t::honor_robots, internal_report_robots_disallow(), internal_report_robots_unavailable(), internal_robots_target(), internal_session_fetch(), internal_url_scheme(), k_mdl_robots_host_max, k_mdl_robots_path_max, k_ra8_ok, mdl_robots_cache_consult(), mdl_robots_disallow_reason(), mdl_url_host(), priv_mdl_stream_text(), s_ua_token, and mdl_session_t::scratch.

Referenced by internal_discover_fetch(), internal_download_page_image(), internal_fetch_artifact(), internal_mdl_fetch_chapter_html(), internal_mdl_fetch_prepare_page(), mdl_app_run_page(), mdl_fetch_asset(), priv_mdl_app_ensure_series_cover(), and priv_mdl_app_prepare_chapters().

Variable Documentation

◆ s_ua_token

const char* const s_ua_token = "mdl"

static

Product token used to match robots.txt User-agent groups.

Definition at line 36 of file mdl_session.c.

Referenced by mdl_session_build_ua(), mdl_session_ua_token(), and mdl_session_url_allowed().

◆ s_ua_url

const char* const s_ua_url = "https://github.com/bsikar/ra8-firmware"

static

Project URL reported in the User-Agent for a contact channel.

Definition at line 40 of file mdl_session.c.

Referenced by mdl_session_build_ua().

◆ s_ua_version

const char* const s_ua_version = "0.1.0"

static

Tool version reported in the User-Agent.

Definition at line 38 of file mdl_session.c.

Referenced by mdl_session_build_ua().

Enumerations

Functions

Variables

Detailed Description

Enumeration Type Documentation

◆ mdl_session_net_t

◆ mdl_session_status_t

Function Documentation

◆ internal_report_robots_disallow()

◆ internal_report_robots_unavailable()

◆ internal_robots_target()

◆ internal_session_fetch()

◆ internal_url_prefix()

◆ internal_url_scheme()

◆ mdl_session_build_ua()

◆ mdl_session_init()

◆ mdl_session_ua_token()

◆ mdl_session_url_allowed()

Variable Documentation

◆ s_ua_token

◆ s_ua_url

◆ s_ua_version