Module-private fetch-loop decisions promoted for host unit tests and the CLI's end-of-run reporting. More...

#include <stddef.h>
#include <stdint.h>
#include "mdl_fetch.h"
#include "mdl_net.h"
#include "ra8_attributes.h"
#include "ra8_err.h"

Include dependency graph for mdl_fetch_internal.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	mdl_fetch_cache_request
	Governed retry context for one cache buffer callback. More...
struct	mdl_run_pos_t
	Stable chapter position carried into per-page progress events. More...

Typedefs
typedef struct mdl_fetch_cache_request	mdl_fetch_cache_request_t
	Caller-owned governor parameters for one cached buffer fetch.

Enumerations
enum	mdl_fetch_reason_size_t : uint16_t { k_mdl_reason_max = 96 }
	Buffer size the priv_mdl_fetch_reason renderer never overruns. More...

Functions
ra8_err_t	priv_mdl_fetch_cache_get_buf (void context, const char url, const mdl_net_req_t request, char buffer, size_t capacity, size_t out_length, mdl_net_resp_t response)
	Fetch one cache buffer through governor and bounded retry policy.
bool	priv_mdl_fetch_is_retryable (ra8_err_t rc)
	Whether a failed transfer result is worth retrying.
bool	priv_mdl_fetch_run_incomplete (const mdl_fetch_stats_t *stats)
	Whether a finished run left anything unfetched (chapter or page).
void	priv_mdl_fetch_reason (ra8_err_t err, long status, char *buf, size_t cap)
	Render a human-readable reason for a transfer result and HTTP status.
void	priv_mdl_fetch_record_fail (const mdl_fetch_ctx_t ctx, const char url, long status, ra8_err_t err)
	Record one classified failure in the caller's bounded log.
ra8_err_t	priv_mdl_fetch_checkpoint (const mdl_fetch_ctx_t *ctx)
	Persist the current fetch state when a checkpoint path is configured.
ra8_err_t	priv_mdl_fetch_with_retry (mdl_fetch_ctx_t ctx, const char host, const char url, const mdl_net_req_t req, mdl_net_body_sink_t sink, uint32_t jmin, uint32_t jmax, mdl_net_resp_t out_resp, size_t *out_bytes)
	Perform one bounded governed file transfer with retry classification.
ra8_err_t	priv_mdl_fetch_chapter_pages (mdl_fetch_ctx_t ctx, const char chapter_url, const char dest_abs, const char dest_rel, size_t base, mdl_chapter_rec_t rec, mdl_fetch_stats_t stats, const mdl_run_pos_t *pos)
	Fetch and checkpoint every extracted page in one chapter.

Detailed Description

Module-private fetch-loop decisions promoted for host unit tests and the CLI's end-of-run reporting.

These are pure decisions with no network dependency, factored out of the download loop so the host tests can drive each branch directly (per the "Test access to internal symbols" rule in CLAUDE.md) and so main.c can reuse the same retryability / incompleteness / reason logic the loop uses – rather than re-deriving it and drifting. Nothing here is part of the orchestrator's public API in mdl_fetch.h; production callers inside the tool reach them by name, and no code outside the mdl tool ever does.

Copyright: Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Since: 0.1.0

Definition in file mdl_fetch_internal.h.

Typedef Documentation

◆ mdl_fetch_cache_request_t

typedef struct mdl_fetch_cache_request mdl_fetch_cache_request_t

Caller-owned governor parameters for one cached buffer fetch.

Definition at line 31 of file mdl_fetch_internal.h.

Enumeration Type Documentation

◆ mdl_fetch_reason_size_t

enum mdl_fetch_reason_size_t : uint16_t

Buffer size the priv_mdl_fetch_reason renderer never overruns.

Enumerator
k_mdl_reason_max	Failure-reason string buffer bytes.

Definition at line 88 of file mdl_fetch_internal.h.

Function Documentation

◆ priv_mdl_fetch_cache_get_buf()

ra8_err_t priv_mdl_fetch_cache_get_buf	(	void *	context,
		const char *	url,
		const mdl_net_req_t *	request,
		char *	buffer,
		size_t	capacity,
		size_t *	out_length,
		mdl_net_resp_t *	response )

Fetch one cache buffer through governor and bounded retry policy.

Acquires and observes the host governor around each network attempt, retrying only results accepted by priv_mdl_fetch_is_retryable.

Parameters

[in,out]	context	mdl_fetch_cache_request_t state.
[in]	url	Exact request URL.
[in]	request	Conditional request metadata.
[out]	buffer	Bounded body destination.
[in]	capacity	Writable destination capacity.
[out]	out_length	Exact received bytes.
[out]	response	Finished response metadata.

Returns: Canonical governed network status.

Return values

k_ra8_ok	A complete response is available.
other	Governor acquisition or every bounded network attempt failed.

Precondition: Every pointer is non-NULL and context is initialized.; buffer spans capacity writable bytes.

Postcondition: Every attempted request is observed and released exactly once.; At most the configured retry count is attempted.

Note: Conforms directly to mdl_cache_fetch_fn.

Since: 0.1.0

Fetch one cache buffer through governor and bounded retry policy.

Parameters

[in,out]	context	mdl_fetch_cache_request_t state.
[in]	url	Exact chapter URL.
[in]	request	Conditional request metadata.
[out]	buffer	Bounded HTML destination.
[in]	capacity	Writable destination capacity.
[out]	out_length	Exact received bytes.
[out]	response	Finished response metadata.

Returns: Canonical governed network status.

Precondition: Every pointer is non-NULL and the context is fully initialized.; buffer spans capacity writable bytes.

Postcondition: Every attempt is acquired, observed, and released exactly once.; Retryable results make at most k_fetch_max_attempts attempts.

Note: No callback argument is retained.

Since: 0.1.0

Definition at line 421 of file mdl_fetch.c.

References mdl_fetch_cache_request::ctx, mdl_fetch_ctx_t::gov, mdl_fetch_cache_request::host, mdl_fetch_cache_request::jmax, mdl_fetch_cache_request::jmin, k_fetch_max_attempts, k_ra8_fail, k_ra8_ok, mdl_governor_acquire(), mdl_governor_observe(), mdl_governor_release(), mdl_net_get_buf(), mdl_session_t::net, priv_mdl_fetch_is_retryable(), RA8_PRIV, mdl_net_resp_t::retry_after, mdl_fetch_ctx_t::session, and mdl_net_resp_t::status.

Referenced by internal_cover_cached(), and internal_mdl_fetch_chapter_html().

◆ priv_mdl_fetch_chapter_pages()

ra8_err_t priv_mdl_fetch_chapter_pages	(	mdl_fetch_ctx_t *	ctx,
		const char *	chapter_url,
		const char *	dest_abs,
		const char *	dest_rel,
		size_t	base,
		mdl_chapter_rec_t *	rec,
		mdl_fetch_stats_t *	stats,
		const mdl_run_pos_t *	pos )

Fetch and checkpoint every extracted page in one chapter.

Performs chapter pages under the injected network, governor, and storage contracts; dependency failures are propagated before incomplete bytes are published.

Parameters

[in,out]	ctx	Fetch dependencies and extracted image list.
[in]	chapter_url	Chapter URL used as Referer.
[in]	dest_abs	Absolute output directory.
[in]	dest_rel	Series-relative output directory.
[in]	base	Page-number base for combined layout.
[in,out]	rec	Persistent chapter record.
[in,out]	stats	Run counters.
[in]	pos	Chapter progress coordinates.

Returns: k_ra8_ok after every page is durable; otherwise first failure.

Return values

k_ra8_ok	The operation completed.
other	Validation, capacity, network, or storage failed.

Precondition: Required pointer arguments remain valid for the call duration.; Supplied capacities cover their referenced bounded buffers.

Postcondition: No ownership of caller-provided storage is transferred.; Result status and outputs describe one completed synchronous attempt.

Note: The function performs no dynamic allocation and retains no caller pointer.

Since: 0.1.0

Fetch and checkpoint every extracted page in one chapter.

Definition at line 680 of file mdl_fetch_page.c.

References mdl_url_list_t::count, mdl_fetch_ctx_t::images, internal_mdl_fetch_emit_progress(), internal_mdl_fetch_one_page(), k_ra8_ok, mdl_fetch_ctx_t::progress_error, and RA8_PRIV.

Referenced by internal_mdl_fetch_chapter_pages_and_checkpoint().

◆ priv_mdl_fetch_checkpoint()

ra8_err_t priv_mdl_fetch_checkpoint ( const mdl_fetch_ctx_t * ctx )

Persist the current fetch state when a checkpoint path is configured.

Performs checkpoint under the injected network, governor, and storage contracts; dependency failures are propagated before incomplete bytes are published.

Parameters

[in] ctx Fetch context and state.

Returns: State-save result, or k_ra8_ok when checkpointing is disabled.

Return values

k_ra8_ok	The operation completed.
other	Validation, capacity, network, or storage failed.

Precondition: Required pointer arguments remain valid for the call duration.; Supplied capacities cover their referenced bounded buffers.

Postcondition: No ownership of caller-provided storage is transferred.; Result status and outputs describe one completed synchronous attempt.

Note: The function performs no dynamic allocation and retains no caller pointer.

Since: 0.1.0

Persist the current fetch state when a checkpoint path is configured.

Definition at line 193 of file mdl_fetch.c.

References k_ra8_ok, mdl_state_save(), priv_mdl_fetch_record_fail(), RA8_PRIV, mdl_fetch_ctx_t::state, mdl_fetch_ctx_t::state_path, and mdl_fetch_ctx_t::storage.

Referenced by internal_mdl_fetch_chapter_pages_and_checkpoint(), and internal_mdl_fetch_one_page().

◆ priv_mdl_fetch_is_retryable()

bool priv_mdl_fetch_is_retryable ( ra8_err_t rc )

Whether a failed transfer result is worth retrying.

The retry classifier the bounded per-request loop consults. A transport error (k_ra8_fail), a timeout (k_ra8_err_timeout) and a throttle (k_ra8_err_busy, i.e. HTTP 429/503) are transient and retried; every other outcome – success, an absent resource (k_ra8_err_not_found / 404), an over-cap body (k_ra8_err_no_mem), a refused argument or a governor decline – is terminal and breaks the loop. Retryability is decided on the real classified status (priv_mdl_net_curl_classify), never on a collapsed generic failure, so a 404 is never retried while a 503 is.

Parameters

[in] rc The transfer's classified ra8_err_t result.

Returns: Whether the request should be retried.

Return values

true	`rc` is k_ra8_err_busy, k_ra8_err_timeout or k_ra8_fail.
false	Any other value (success or a permanent error).

Precondition: rc is a value from the ra8_err_t contract.; The caller bounds the retry count independently (this makes no loop).

Postcondition: No state is modified.; k_ra8_ok always yields false, so a success ends the loop.

Note: Thread-safe: depends only on its argument.

MC/DC:

Decision: (rc == k_ra8_err_busy) || (rc == k_ra8_err_timeout) || (rc == k_ra8_fail) (3 conditions). Cited as apps/shared_libs/mdl/src/mdl_fetch.c@priv_mdl_fetch_is_retryable.

Vector 1: rc=k_ra8_err_not_found -> false (control: all three false)
Vector 2: rc=k_ra8_err_busy -> true (varies condition 1)
Vector 3: rc=k_ra8_err_timeout -> true (varies condition 2)
Vector 4: rc=k_ra8_fail -> true (varies condition 3) Vectors 1+2 prove condition 1's independent influence, 1+3 condition 2's, 1+4 condition 3's. N+1 = 4 vectors for N=3 conditions: minimal MC/DC.

Since: 0.1.0

Definition at line 110 of file mdl_fetch.c.

References k_ra8_err_busy, k_ra8_err_timeout, k_ra8_fail, and RA8_PRIV.

Referenced by internal_discover_fetch(), priv_mdl_fetch_cache_get_buf(), and priv_mdl_fetch_with_retry().

◆ priv_mdl_fetch_reason()

void priv_mdl_fetch_reason	(	ra8_err_t	err,
		long	status,
		char *	buf,
		size_t	cap )

Render a human-readable reason for a transfer result and HTTP status.

Maps the classified ra8_err_t (plus the observed HTTP status, where one applies) to a short prose reason, so a failure is reported as "rate limited (HTTP 503)" rather than a bare err 0x109. The HTTP status is appended in parentheses when it is non-zero. Used by the CLI for both the per-failure summary and any inline error line.

Parameters

[in]	err	The classified transfer result.
[in]	status	HTTP status observed, or 0 when none applies.
[out]	buf	Destination buffer for the NUL-terminated reason.
[in]	cap	Capacity of `buf` in bytes.

Returns: Nothing.

Precondition: buf is non-NULL and cap > 0 for output to be written.; err comes from a finished transfer.

Postcondition: buf is NUL-terminated when cap > 0.; A NULL buf or zero cap is a tolerated no-op.

Note: Thread-safe: writes only the caller-provided buffer.

Since: 0.1.0

Definition at line 130 of file mdl_fetch.c.

References internal_mdl_fetch_fail_reason(), k_ra8_err_busy, k_ra8_err_no_mem, k_ra8_err_not_found, k_ra8_err_retry_limit, k_ra8_err_timeout, k_ra8_fail, k_ra8_ok, and RA8_PRIV.

Referenced by internal_download_page_image(), internal_extract_page_images(), internal_finish_artifact_fetch(), mdl_discover_run(), and mdl_report_failures().

◆ priv_mdl_fetch_record_fail()

void priv_mdl_fetch_record_fail	(	const mdl_fetch_ctx_t *	ctx,
		const char *	url,
		long	status,
		ra8_err_t	err )

Record one classified failure in the caller's bounded log.

Performs record fail under the injected network, governor, and storage contracts; dependency failures are propagated before incomplete bytes are published.

Parameters

[in]	ctx	Fetch context carrying the optional log.
[in]	url	Failed URL or path.
[in]	status	Observed HTTP status, or zero.
[in]	err	Classified failure.

Precondition: ctx is non-NULL.; Supplied capacities cover their referenced bounded buffers.

Postcondition: A configured log tallies the failure without exceeding capacity.; Result status and outputs describe one completed synchronous attempt.

Note: The function performs no dynamic allocation and retains no caller pointer.

Since: 0.1.0

Record one classified failure in the caller's bounded log.

Definition at line 175 of file mdl_fetch.c.

References mdl_fetch_fail_t::err, mdl_fetch_ctx_t::faillog, k_mdl_fetch_fail_max, mdl_fetch_fail_t::status, and mdl_fetch_fail_t::url.

Referenced by internal_cover_cached(), internal_cover_stream(), internal_fetch_asset_execute(), internal_mdl_fetch_chapter_html(), internal_mdl_fetch_prepare_page(), internal_mdl_fetch_publish_page(), internal_mdl_fetch_resolve_not_modified(), internal_mdl_fetch_select_chapter_number(), internal_mdl_fetch_select_chapter_title(), mdl_fetch_asset(), and priv_mdl_fetch_checkpoint().

◆ priv_mdl_fetch_run_incomplete()

bool priv_mdl_fetch_run_incomplete ( const mdl_fetch_stats_t * stats )

Whether a finished run left anything unfetched (chapter or page).

The single honesty predicate the CLI's export gate reads: a run is incomplete when any chapter was left partial OR any individual page failed. It exists so the "do not package an incomplete archive" decision is one tested function rather than a compound test re-derived at each export site.

Parameters

[in] stats The run tallies (may be NULL).

Returns: Whether the run has an unfetched chapter or page.

Return values

true	chapters_failed > 0 or pages_failed > 0.
false	A NULL `stats`, or a run in which nothing failed.

Precondition: stats, when non-NULL, was filled by mdl_fetch_run.; The caller treats NULL as "cannot prove incomplete" (false).

Postcondition: No state is modified.; A fully clean run yields false.

Note: Thread-safe: reads only its argument.

MC/DC:

Decision: (stats->chapters_failed > 0) || (stats->pages_failed > 0) (2 conditions). Cited as apps/shared_libs/mdl/src/mdl_fetch.c@priv_mdl_fetch_run_incomplete.

Vector 1: chapters_failed=0, pages_failed=0 -> false (both false)
Vector 2: chapters_failed=1, pages_failed=0 -> true (varies chapters)
Vector 3: chapters_failed=0, pages_failed=1 -> true (varies pages) Vectors 1+2 prove the chapter condition's influence, 1+3 the page condition's. N+1 = 3 vectors for N=2 conditions: minimal MC/DC.

Since: 0.1.0

Definition at line 115 of file mdl_fetch.c.

References mdl_fetch_stats_t::chapters_failed, mdl_fetch_stats_t::pages_failed, and RA8_PRIV.

Referenced by mdl_pack_combined_meta().

◆ priv_mdl_fetch_with_retry()

ra8_err_t priv_mdl_fetch_with_retry	(	mdl_fetch_ctx_t *	ctx,
		const char *	host,
		const char *	url,
		const mdl_net_req_t *	req,
		mdl_net_body_sink_t *	sink,
		uint32_t	jmin,
		uint32_t	jmax,
		mdl_net_resp_t *	out_resp,
		size_t *	out_bytes )

Perform one bounded governed file transfer with retry classification.

Performs with retry under the injected network, governor, and storage contracts; dependency failures are propagated before incomplete bytes are published.

Parameters

[in,out]	ctx	Fetch dependencies.
[in]	host	Governor host key.
[in]	url	Absolute source URL.
[in]	req	Request headers and timeout.
[in,out]	sink	Reset/write body destination.
[in]	jmin	Minimum delay in milliseconds.
[in]	jmax	Maximum delay in milliseconds.
[out]	out_resp	Final response metadata.
[out]	out_bytes	Final response byte count.

Returns: Final classified transfer result.

Return values

k_ra8_ok	The operation completed.
other	Validation, capacity, network, or storage failed.

Precondition: Required pointer arguments remain valid for the call duration.; Supplied capacities cover their referenced bounded buffers.

Postcondition: No ownership of caller-provided storage is transferred.; Result status and outputs describe one completed synchronous attempt.

Note: The function performs no dynamic allocation and retains no caller pointer.

Since: 0.1.0

Perform one bounded governed file transfer with retry classification.

Definition at line 264 of file mdl_fetch.c.

References internal_mdl_fetch_governed_get_body(), k_fetch_max_attempts, k_ra8_fail, priv_mdl_fetch_is_retryable(), and RA8_PRIV.

Referenced by internal_cover_stream(), internal_fetch_asset_execute(), internal_mdl_fetch_prepare_page(), and internal_mdl_fetch_resolve_not_modified().

Data Structures

Typedefs

Enumerations

Functions

Detailed Description

Typedef Documentation

◆ mdl_fetch_cache_request_t

Enumeration Type Documentation

◆ mdl_fetch_reason_size_t

Function Documentation

◆ priv_mdl_fetch_cache_get_buf()

◆ priv_mdl_fetch_chapter_pages()

◆ priv_mdl_fetch_checkpoint()

◆ priv_mdl_fetch_is_retryable()

◆ priv_mdl_fetch_reason()

◆ priv_mdl_fetch_record_fail()

◆ priv_mdl_fetch_run_incomplete()

◆ priv_mdl_fetch_with_retry()