State-aware chapter/page download orchestrator for the media downloader. More...

#include <stddef.h>
#include <stdint.h>
#include "mdl_cache.h"
#include "mdl_config.h"
#include "mdl_extract.h"
#include "mdl_politeness.h"
#include "mdl_session.h"
#include "mdl_state.h"
#include "mdl_storage.h"
#include "ra8_err.h"
#include "ra8_io_stream.h"

Include dependency graph for mdl_fetch.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	mdl_fetch_stats_t
	Tallies a fetch run reports back (all zero-initialised by the run). More...
struct	mdl_fetch_fail_t
	One recorded page/chapter failure: what failed, and with what status. More...
struct	mdl_fetch_faillog_t
	Bounded record of every failure in a run (declare at file scope). More...
struct	mdl_fetch_progress_t
	One per-page progress event the run emits through mdl_progress_fn. More...
struct	mdl_fetch_ctx_t
	Everything mdl_fetch_run needs, injected for testability. More...

Typedefs
typedef ra8_err_t(*	mdl_progress_fn) (void ctx, const mdl_fetch_progress_t ev)
	Injected per-page progress sink; NULL disables all progress output.

Enumerations
enum	mdl_fetch_layout_t : uint8_t { k_mdl_layout_combined = 0 , k_mdl_layout_separate = 1 }
	Output directory layout the orchestrator writes. More...
enum	mdl_fetch_faillog_size_t : uint16_t { k_mdl_fetch_fail_max = 256 }
	Bounded run-failure log capacity (zero dynamic allocation). More...

Functions
ra8_err_t	mdl_fetch_asset (mdl_fetch_ctx_t ctx, const char url, const char target_abs, const char referer, mdl_net_resp_t out_resp, size_t out_bytes)
	Fetch one policy-governed asset to an absolute file path.
ra8_err_t	mdl_fetch_run (mdl_fetch_ctx_t ctx, const mdl_url_list_t chapters, mdl_fetch_layout_t layout, const char combined_dir_rel, mdl_fetch_stats_t stats)
	Download a series' chapters incrementally, resuming and deduping.

Detailed Description

State-aware chapter/page download orchestrator for the media downloader.

The incremental, resumable download loop, lifted out of main.c so it can be driven end to end through the mdl_net vtable mock with no network: the host tests script a fake backend and assert that a first run fetches N pages, a second run fetches only what is new, and an interrupted run resumes to a byte-identical result. Storage namespace, hashing, and dedup publication are injected through mdl_fetch_ctx_t and are portable across fw_if_fs backends. Response bodies publish through the same portable transaction contract, while persistent state checkpoints remain the residual host-path boundary before the complete orchestrator can run unchanged on a device.

What it does that the old index-based loop did not:

Addresses chapters by identity, not position. Each chapter's stable id (mdl_urlname_last_segment) keys a record in mdl_state_t, so a rerun recognises what it already has even as the site adds or reorders chapters.
Resumes. A chapter is marked complete only after every page is fetched AND its bytes verify against the recorded content hash; an interrupted chapter is resumed page-wise on the next run, never packaged half-done.
Dedups by content. A page whose source URL is already held (a rerun, or an image shared across chapters) is reused from the existing verified file instead of re-fetched.
Numbers combined pages from recorded counts, so a resumed combined download reproduces the numbering an uninterrupted one would have.

The loop only puts page bytes on disk and maintains state; packaging (archive export) is left to the caller, which reads the resulting directories.

Copyright: Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Definition in file mdl_fetch.h.

Typedef Documentation

◆ mdl_progress_fn

typedef ra8_err_t(* mdl_progress_fn) (void *ctx, const mdl_fetch_progress_t *ev)

Injected per-page progress sink; NULL disables all progress output.

The dependency-injection seam for run progress. Production wires a sink that prints one redirect-safe line per page; the host tests leave it NULL so the loop is silent and deterministic (and no wall clock is read). It is called once per page after the page is present, fetched or reused.

Parameters

[in]	ctx	Opaque context supplied in mdl_fetch_ctx_t::progress_ctx.
[in]	ev	The just-completed page's progress event (never NULL).

Returns: Nothing.

Since: 0.1.0

Definition at line 150 of file mdl_fetch.h.

Enumeration Type Documentation

◆ mdl_fetch_faillog_size_t

enum mdl_fetch_faillog_size_t : uint16_t

Bounded run-failure log capacity (zero dynamic allocation).

Enumerator
k_mdl_fetch_fail_max	Failures stored before the log saturates.

Definition at line 75 of file mdl_fetch.h.

◆ mdl_fetch_layout_t

enum mdl_fetch_layout_t : uint8_t

Output directory layout the orchestrator writes.

Enumerator
k_mdl_layout_combined	All chapters -> one dir, continuous numbering.
k_mdl_layout_separate	Each chapter -> own dir, per-chapter numbering.

Definition at line 51 of file mdl_fetch.h.

Function Documentation

◆ mdl_fetch_asset()

ra8_err_t mdl_fetch_asset	(	mdl_fetch_ctx_t *	ctx,
		const char *	url,
		const char *	target_abs,
		const char *	referer,
		mdl_net_resp_t *	out_resp,
		size_t *	out_bytes )

Fetch one policy-governed asset to an absolute file path.

Provides non-page callers (notably series-cover acquisition) the same robots gate, backend SSRF policy, per-host governor, bounded retry loop, and atomic publication used by chapter images. The transfer is staged beside the target; an existing target survives every failed or zero-byte response unchanged.

Parameters

[in,out]	ctx	Injected session/site/governor dependencies.
[in]	url	Absolute HTTP(S) asset URL (never NULL).
[in]	target_abs	Absolute destination path (never NULL).
[in]	referer	Referer header, or NULL to omit it.
[out]	out_resp	Finished response metadata, or NULL.
[out]	out_bytes	Committed byte count, or NULL.

Returns: An ra8_err_t transfer result.

Return values

k_ra8_ok	A non-empty asset was atomically committed.
k_ra8_err_invalid_arg	Required context/path/URL input was invalid.
k_ra8_err_invalid_size	The server returned a successful empty body.
k_ra8_fail	Robots refused the URL or publication failed.

Precondition: ctx has a session, site descriptor, and configured network backend.; target_abs names a writable absolute path whose parent exists.

Postcondition: On success, target_abs holds exactly *out_bytes nonzero bytes.; On failure, a pre-existing target_abs is unchanged and no temp remains.

Note: Not thread-safe: mutates the session cache and governor.

Since: 0.1.0

Definition at line 356 of file mdl_fetch.c.

References mdl_site_t::img_delay_max, mdl_site_t::img_delay_min, internal_fetch_asset_execute(), internal_mdl_fetch_max_u32(), internal_mdl_fetch_page_host(), k_mdl_gov_host_max, k_ra8_err_invalid_arg, k_ra8_fail, mdl_session_url_allowed(), mdl_session_t::net, priv_mdl_fetch_record_fail(), mdl_fetch_ctx_t::session, mdl_fetch_ctx_t::site, mdl_fetch_ctx_t::timeout_ms, and mdl_session_t::user_agent.

◆ mdl_fetch_run()

ra8_err_t mdl_fetch_run	(	mdl_fetch_ctx_t *	ctx,
		const mdl_url_list_t *	chapters,
		mdl_fetch_layout_t	layout,
		const char *	combined_dir_rel,
		mdl_fetch_stats_t *	stats )

Download a series' chapters incrementally, resuming and deduping.

Walks chapters in order. For each chapter it derives a stable identifier, finds-or-adds its state record, and – unless update_only is set and the record is already complete – fetches the chapter page, extracts its image URLs, and writes each page under ctx->series_abs_dir. A page already held (matched by source-URL hash and verified by content hash) is reused from disk rather than re-fetched, including across chapters. Each request is retried up to a small bounded number of times on a retryable outcome (a transport error, timeout, 429 or 5xx), always paced through the politeness governor so a retry never becomes an unpaced hammer; a 404 is never retried. A chapter is marked complete and its record checkpointed only once every page is present and verified; a page failure leaves the chapter partial for the next run to resume, and is appended to ctx->faillog (when set) with its URL and status. Per-page progress is emitted through ctx->progress_fn (when set). In k_mdl_layout_combined the pages of all chapters share one directory with numbering continued across chapters (derived from recorded per-chapter page counts, so a resume reproduces it); k_mdl_layout_separate gives each chapter its own directory numbered from one.

Parameters

[in,out]	ctx	Injected dependencies and scratch (never NULL).
[in]	chapters	Live, ordered chapter URL list (never NULL).
[in]	layout	Output directory layout.
[in]	combined_dir_rel	Directory name (relative to the series dir) for k_mdl_layout_combined; ignored otherwise.
[out]	stats	Receives the run tallies (never NULL).

Returns: An ra8_err_t summarising the run.

Return values

k_ra8_ok	Every attempted chapter completed.
k_ra8_err_invalid_arg	A required pointer argument was NULL.
k_ra8_fail	At least one chapter was left partial/failed.

Precondition: ctx, chapters, stats are non-NULL and ctx is fully populated.; ctx->series_abs_dir exists and is an absolute resolved path.

Postcondition: stats reflects the run; pages_fetched + pages_reused accounts for every page of every completed chapter.; ctx->state is updated and, when ctx->state_path is set, checkpointed atomically after each page and each completed chapter.