mdl_politeness.h File Reference

Jittered inter-request delay plus the per-host politeness governor. More...

#include <stdint.h>
#include "ra8_err.h"

Include dependency graph for mdl_politeness.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	mdl_politeness_t
	Deterministic jitter source plus its (optional) injected clock. More...
struct	mdl_gov_cfg_t
	Per-site politeness tunables the governor is initialised with. More...
struct	mdl_host_rec_t
	One host's live governor state (a slot in the fixed per-host table). More...
struct	mdl_governor_t
	Closed-loop per-host politeness governor (rate + backoff + concurrency). More...

Typedefs
typedef void(*	mdl_sleep_fn) (void *ctx, uint32_t ms)
	Injected blocking sleep: pause the caller for ms milliseconds.
typedef int64_t(*	mdl_now_fn) (void *ctx)
	Injected monotonic clock: milliseconds since an arbitrary fixed epoch.

Enumerations
enum	mdl_gov_limits_t : uint16_t {   k_mdl_gov_max_hosts = 16U ,   k_mdl_gov_host_max = 128U ,   k_mdl_gov_level_max = 16U ,   k_mdl_gov_ms_per_req = 60000U }
	Fixed sizes and bounds for the governor's per-host table. More...

Functions
void	mdl_politeness_init (mdl_politeness_t *p, uint64_t seed)
	Seed the jitter source, using the real host clock for sleeps.
void	mdl_politeness_init_clock (mdl_politeness_t *p, uint64_t seed, mdl_sleep_fn sleep_fn, void *sleep_ctx)
	Seed the jitter source and inject a clock for the blocking sleep.
uint32_t	mdl_politeness_wait (mdl_politeness_t *p, uint32_t min_ms, uint32_t max_ms)
	Sleep a jittered delay in [min_ms, max_ms] and return it.
mdl_gov_cfg_t	mdl_gov_cfg_default (void)
	Conservative default tunables for a site that configures none.
void	mdl_governor_init (mdl_governor_t *g, const mdl_gov_cfg_t *cfg, uint64_t seed)
	Initialise a governor on the real host clock and blocking sleep.
void	mdl_governor_init_clock (mdl_governor_t *g, const mdl_gov_cfg_t *cfg, uint64_t seed, mdl_now_fn now_fn, void *now_ctx, mdl_sleep_fn sleep_fn, void *sleep_ctx)
	Initialise a governor with injected clock and sleep seams (DI).
ra8_err_t	mdl_governor_acquire (mdl_governor_t *g, const char *host, uint32_t jitter_min_ms, uint32_t jitter_max_ms)
	Reserve an in-flight slot for a request to host, pacing as required.
void	mdl_governor_release (mdl_governor_t *g, const char *host)
	Release the in-flight slot reserved by a matching mdl_governor_acquire.
void	mdl_governor_observe (mdl_governor_t *g, const char *host, long status, const char *retry_after)
	Feed a completed request's outcome back into the host's governor state.
void	mdl_governor_observe_at_wall (mdl_governor_t *g, const char *host, long status, const char *retry_after, int64_t now_wall_s)
	Observe a response using an explicit wall-clock timestamp.
bool	mdl_governor_peek (const mdl_governor_t *g, const char *host, uint16_t *backoff_level, int64_t *earliest_next_ms)
	Read a host's current backoff level and earliest-next gate.
bool	mdl_retry_after_parse (const char *value, int64_t now_wall_s, uint32_t *out_ms)
	Parse an HTTP Retry-After header value into a delay in milliseconds.

Detailed Description

Jittered inter-request delay plus the per-host politeness governor.

Two layers live here. The lower is the jitter primitive kept from the Kotlin original: mdl_politeness_wait sleeps a seeded, deterministic random delay so fixed-interval requests do not fingerprint us. The upper is the closed-loop governor (mdl_governor_t) the issue calls for: a per-host token bucket that bounds the sustained request rate, exponential backoff with full jitter on a 429/503, Retry-After honouring (both delta-seconds and HTTP-date forms), and a per-host in-flight cap. The governor reacts to what the server tells us; open-loop jitter never did.

Both layers reach the outside world only through injectable seams so their timing is unit-testable with no real sleeps and no real clock: production wires the host clock (nanosleep / CLOCK_MONOTONIC), tests wire fakes that record the requested delay and advance a virtual clock. The maths is identical either way, so a test asserts spacing, backoff growth/decay and Retry-After precedence deterministically.

Copyright

Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Definition in file mdl_politeness.h.

Typedef Documentation

mdl_now_fn

typedef int64_t(* mdl_now_fn) (void *ctx)

Injected monotonic clock: milliseconds since an arbitrary fixed epoch.

The governor reads time only through this seam. Production binds CLOCK_MONOTONIC in ms; a unit test binds a fake it can advance by hand, so token refill and backoff scheduling are deterministic. Only DIFFERENCES are ever taken, and the clock must never run backward: a steppable wall clock jumped forward makes the governor believe the request spacing has elapsed and hammer the remote host (#509). The Retry-After HTTP-date form is therefore compared against a wall-clock second count passed explicitly to mdl_retry_after_parse, not derived from this seam.

Parameters

[in]

ctx

Opaque context supplied at mdl_governor_init_clock.

Returns

Current time in milliseconds.

Since

0.1.0

Definition at line 161 of file mdl_politeness.h.

mdl_sleep_fn

typedef void(* mdl_sleep_fn) (void *ctx, uint32_t ms)

Injected blocking sleep: pause the caller for ms milliseconds.

The dependency-injection seam for mdl_politeness_wait. Production binds the host clock; a unit test binds a fake that records ms and returns at once, so a backoff/spacing policy is verified without wall-clock time passing.

Parameters

[in]	ctx	Opaque context supplied at mdl_politeness_init_clock.
[in]	ms	Requested sleep duration in milliseconds.

Returns

Nothing.

Since

0.1.0

Definition at line 43 of file mdl_politeness.h.

Enumeration Type Documentation

mdl_gov_limits_t

enum mdl_gov_limits_t : uint16_t

Fixed sizes and bounds for the governor's per-host table.

Enumerator
k_mdl_gov_max_hosts	Per-host record slots (origin + CDNs).
k_mdl_gov_host_max	Host-key buffer bytes (matches config).
k_mdl_gov_level_max	Backoff-exponent ceiling (overflow guard).
k_mdl_gov_ms_per_req	Milliseconds per minute (rate -> interval).

Definition at line 164 of file mdl_politeness.h.

Function Documentation

mdl_gov_cfg_default()

mdl_gov_cfg_t mdl_gov_cfg_default

(

void

)

Conservative default tunables for a site that configures none.

A cautious ceiling that a careful human reader would not exceed: roughly one request per second sustained with a small burst, one-second base backoff doubling to a one-minute cap, and strictly serial per host. A site descriptor may raise or lower any field.

Returns

The default mdl_gov_cfg_t (60 req/min, burst 4, 1s..60s backoff, decay after 4 successes, 1 in-flight).

Return values

(by	value) Never fails; always returns the conservative defaults.

Precondition

None.

The caller accepts the documented conservative constants.

Postcondition

burst >= 1 and max_inflight >= 1 in the returned config.

rate_per_min > 0, so rate limiting is on by default.

Note

Thread-safe: returns a value computed from constants.

Since

0.1.0

Definition at line 484 of file mdl_politeness.c.

References k_def_backoff_base_ms, k_def_backoff_max_ms, k_def_burst, k_def_decay_after, k_def_max_inflight, and k_def_rate_per_min.

Referenced by internal_config_set_defaults(), mdl_config_gov_cfg(), and mdl_governor_init_clock().

mdl_governor_acquire()

ra8_err_t mdl_governor_acquire	(	mdl_governor_t *	g,
		const char *	host,
		uint32_t	jitter_min_ms,
		uint32_t	jitter_max_ms )

Reserve an in-flight slot for a request to host, pacing as required.

The gate every request passes before hitting the network. In order it (1) refuses – without sleeping – when host already has max_inflight requests in flight, so a future parallel loop is bounded per host; (2) computes the token-bucket wait so the sustained rate stays under rate_per_min with up to burst back-to-back; (3) raises that to the host's backoff / Retry-After gate if it is later; (4) adds a jittered spacing delay drawn from [jitter_min_ms, jitter_max_ms]; then sleeps the total through the injected clock and marks one request in flight. On success the caller MUST later call mdl_governor_release. A NULL governor is a no-op that succeeds (politeness disabled), matching a NULL jitter source.

Parameters

[in,out]	g	Governor, or NULL to disable pacing.
[in]	host	Host key (e.g. from mdl_url_host); may be NULL.
[in]	jitter_min_ms	Baseline spacing floor (e.g. the site img delay).
[in]	jitter_max_ms	Baseline spacing ceiling (clamped up to the floor).

Returns

Whether the slot was reserved.

Return values

k_ra8_ok	Reserved; the request may proceed (release later).
k_ra8_err_would_block	host is at its in-flight cap; try again later.

Precondition

g, when non-NULL, was initialised by a governor init function.

The caller pairs a k_ra8_ok with exactly one mdl_governor_release.

Postcondition

On k_ra8_ok the host's in-flight count is one higher.

On k_ra8_err_would_block no time was slept and no state changed.

Note

Not thread-safe: mutates the per-host table and advances the clock.

See also

mdl_governor_release

mdl_governor_observe

Since

0.1.0

Definition at line 527 of file mdl_politeness.c.

References mdl_governor_t::cfg, mdl_host_rec_t::inflight, internal_draw_range(), internal_gov_get(), internal_gov_now(), internal_gov_schedule(), internal_gov_sleep(), k_ra8_err_would_block, k_ra8_ok, mdl_gov_cfg_t::max_inflight, and mdl_governor_t::rng.

Referenced by internal_discover_fetch(), internal_mdl_fetch_governed_get_body(), and priv_mdl_fetch_cache_get_buf().

mdl_governor_init()

void mdl_governor_init	(	mdl_governor_t *	g,
		const mdl_gov_cfg_t *	cfg,
		uint64_t	seed )

Initialise a governor on the real host clock and blocking sleep.

Equivalent to mdl_governor_init_clock with NULL clock seams: time comes from the host monotonic clock and mdl_governor_acquire blocks on nanosleep. Clamps cfg->burst and cfg->max_inflight up to 1 and seeds the jitter PRNG.

Parameters

[out]	g	Governor to initialise (non-NULL).
[in]	cfg	Tunables to copy, or NULL for mdl_gov_cfg_default.
[in]	seed	Jitter seed; 0 is remapped to a non-zero constant.

Returns

Nothing.

Precondition

g, when non-NULL, points to writable mdl_governor_t storage.

A NULL g is a tolerated no-op.

Postcondition

The per-host table is empty and g->rng is non-zero.

g->cfg.burst >= 1 and g->cfg.max_inflight >= 1.

Note

Not thread-safe: initialises caller storage.

See also

mdl_governor_init_clock

Since

0.1.0

Definition at line 496 of file mdl_politeness.c.

References mdl_governor_init_clock().

Referenced by internal_run_prepared(), and mdl_app_run_discover().

mdl_governor_init_clock()

void mdl_governor_init_clock	(	mdl_governor_t *	g,
		const mdl_gov_cfg_t *	cfg,
		uint64_t	seed,
		mdl_now_fn	now_fn,
		void *	now_ctx,
		mdl_sleep_fn	sleep_fn,
		void *	sleep_ctx )

Initialise a governor with injected clock and sleep seams (DI).

The dependency-injection entry point. Wires now_fn as the time source and sleep_fn as the sleeper, so a test drives the whole governor against a virtual clock with no real delay. NULL seams select the host clock/sleep, making this a strict superset of mdl_governor_init.

Parameters

[out]	g	Governor to initialise (non-NULL).
[in]	cfg	Tunables to copy, or NULL for mdl_gov_cfg_default.
[in]	seed	Jitter seed; 0 is remapped to a non-zero constant.
[in]	now_fn	Injected monotonic clock, or NULL for the host clock.
[in]	now_ctx	Context forwarded to now_fn (may be NULL).
[in]	sleep_fn	Injected sleeper, or NULL for the host clock.
[in]	sleep_ctx	Context forwarded to sleep_fn (may be NULL).

Returns

Nothing.

Precondition

g, when non-NULL, points to writable mdl_governor_t storage.

now_ctx / sleep_ctx outlive every governor call on g.

Postcondition

The per-host table is empty and g->rng is non-zero.

g->now_fn/g->sleep_fn equal the arguments (NULL = host default).

Note

Not thread-safe: initialises caller storage.

Since

0.1.0

Definition at line 501 of file mdl_politeness.c.

References mdl_gov_cfg_t::burst, mdl_governor_t::cfg, k_seed_fallback, mdl_gov_cfg_t::max_inflight, mdl_gov_cfg_default(), memset(), mdl_governor_t::now_ctx, mdl_governor_t::now_fn, mdl_governor_t::rng, mdl_governor_t::sleep_ctx, and mdl_governor_t::sleep_fn.

Referenced by mdl_governor_init().

mdl_governor_observe()

void mdl_governor_observe	(	mdl_governor_t *	g,
		const char *	host,
		long	status,
		const char *	retry_after )

Feed a completed request's outcome back into the host's governor state.

Closes the loop. On a throttle (status 429 or 503) it raises backoff_level by one (capped), computes an exponential window min(base << (level-1), ceil), draws a full-jitter delay in [0, window], and sets the host's earliest-next gate to now + max(jittered_backoff, Retry-After) – so Retry-After wins whenever it is longer, in both delta-seconds and HTTP-date forms. On any non-throttle outcome it counts a success and, after decay_after of them, drops one backoff level, easing back toward the base rate; a non-throttle response that still carries Retry-After is honoured as a floor. Call once per request, after the fetch, before the next acquire to the same host observes the new gate.

Parameters

[in,out]	g	Governor, or NULL (no-op).
[in]	host	Host key of the completed request; may be NULL.
[in]	status	HTTP status observed (0 if none / transport error).
[in]	retry_after	Raw Retry-After header, or NULL/"" if absent.

Returns

Nothing.

Precondition

g, when non-NULL, was initialised by a governor init function.

status and retry_after come from the just-finished request.

Postcondition

A 429/503 raises backoff_level and pushes earliest_next_ms forward.

A non-throttle outcome advances the success streak and may decay a level.

Note

Not thread-safe: mutates the per-host table and reads the clock.

See also

mdl_governor_acquire

mdl_retry_after_parse

Since

0.1.0

Definition at line 678 of file mdl_politeness.c.

References mdl_governor_observe_at_wall().

Referenced by internal_discover_fetch(), internal_mdl_fetch_governed_get_body(), and priv_mdl_fetch_cache_get_buf().

mdl_governor_observe_at_wall()

void mdl_governor_observe_at_wall	(	mdl_governor_t *	g,
		const char *	host,
		long	status,
		const char *	retry_after,
		int64_t	now_wall_s )

Observe a response using an explicit wall-clock timestamp.

This is the deterministic-clock form of mdl_governor_observe. The governor still schedules gates against its monotonic clock, but parses an HTTP-date Retry-After relative to now_wall_s. Production callers should use mdl_governor_observe; tests and platforms with an injected wall clock may use this entry point.

Parameters

[in,out]	g	Governor, or NULL (no-op).
[in]	host	Host key of the completed request; may be NULL.
[in]	status	HTTP status observed.
[in]	retry_after	Raw Retry-After header, or NULL/empty.
[in]	now_wall_s	Current Unix wall-clock time in seconds.

Returns

Nothing.

Precondition

g, when non-NULL, was initialised by a governor init function.

status, retry_after, and now_wall_s describe the same completed request.

Postcondition

A 429/503 raises the host backoff and advances its earliest-next gate.

Other outcomes advance the success streak and may decay one backoff level.

Note

Not thread-safe: mutates the per-host table and reads the monotonic clock.

Since

0.1.0

Definition at line 652 of file mdl_politeness.c.

References internal_gov_get(), internal_gov_now(), internal_gov_on_success(), internal_gov_on_throttle(), k_http_too_many_req, k_http_unavailable, and mdl_retry_after_parse().

Referenced by mdl_governor_observe().

mdl_governor_peek()

bool mdl_governor_peek	(	const mdl_governor_t *	g,
		const char *	host,
		uint16_t *	backoff_level,
		int64_t *	earliest_next_ms )

Read a host's current backoff level and earliest-next gate.

Read-only introspection into the per-host table, used by the unit tests to assert backoff growth/decay and gate scheduling deterministically, and usable by a caller that wants to log a host's throttle state. Does not create a record: an unknown host reports false and leaves the outputs untouched.

Parameters

[in]	g	Governor, or NULL.
[in]	host	Host key to look up; may be NULL.
[out]	backoff_level	Receives the consecutive-throttle level. May be NULL.
[out]	earliest_next_ms	Receives the earliest-next gate (mono-ms). May be NULL.

Returns

Whether a record exists for host.

Return values

true	A record exists; the requested outputs were written.
false	No record for host (or NULL argument); outputs untouched.

Precondition

g, when non-NULL, was initialised by a governor init function.

The caller treats a false return as "host not yet seen".

Postcondition

No governor state is modified.

Output pointers are written only when the function returns true.

Note

Not thread-safe: reads the per-host table.

Since

0.1.0

Definition at line 683 of file mdl_politeness.c.

References mdl_host_rec_t::backoff_level, mdl_host_rec_t::earliest_next_ms, mdl_host_rec_t::host, mdl_governor_t::hosts, k_mdl_gov_max_hosts, strcmp(), and mdl_host_rec_t::used.

mdl_governor_release()

void mdl_governor_release	(	mdl_governor_t *	g,
		const char *	host )

Release the in-flight slot reserved by a matching mdl_governor_acquire.

Decrements the matching host record's in-flight count when it is non-zero. NULL arguments, unknown hosts, and unmatched releases are tolerated no-ops so cleanup paths can remain unconditional.

Parameters

[in,out]	g	Governor, or NULL (no-op).
[in]	host	Host key passed to the paired acquire; may be NULL.

Returns

Nothing.

Precondition

Called exactly once per k_ra8_ok from mdl_governor_acquire.

g, when non-NULL, was initialised by a governor init function.

Postcondition

The host's in-flight count is one lower (never below zero).

No time is slept.

Note

Not thread-safe: mutates the per-host table.

See also

mdl_governor_acquire

Since

0.1.0

Definition at line 552 of file mdl_politeness.c.

References mdl_host_rec_t::inflight, and internal_gov_find().

Referenced by internal_discover_fetch(), internal_mdl_fetch_governed_get_body(), and priv_mdl_fetch_cache_get_buf().

mdl_politeness_init()

void mdl_politeness_init	(	mdl_politeness_t *	p,
		uint64_t	seed )

Seed the jitter source, using the real host clock for sleeps.

Equivalent to mdl_politeness_init_clock with a NULL sleeper: the jitter is seeded deterministically and mdl_politeness_wait blocks on nanosleep.

Parameters

[in,out]	p	State to initialise (must be non-NULL).
[in]	seed	Any value; 0 is remapped to a non-zero constant.

Returns

Nothing.

Precondition

p, when non-NULL, points to writable mdl_politeness_t storage.

A NULL p is a tolerated no-op.

Postcondition

p->state is non-zero and p->sleep_fn is NULL.

The same seed yields the same delay sequence.

Note

Not thread-safe: initialises caller storage.

Since

0.1.0

Definition at line 48 of file mdl_politeness.c.

References mdl_politeness_init_clock().

Referenced by internal_download_page_images().

mdl_politeness_init_clock()

void mdl_politeness_init_clock	(	mdl_politeness_t *	p,
		uint64_t	seed,
		mdl_sleep_fn	sleep_fn,
		void *	sleep_ctx )

Seed the jitter source and inject a clock for the blocking sleep.

The dependency-injection entry point. Wires sleep_fn / sleep_ctx as the sleeper mdl_politeness_wait calls, so a test drives spacing/backoff timing through a fake clock with no real delay. A NULL sleep_fn selects the host clock, making this a strict superset of mdl_politeness_init.

Parameters

[in,out]	p	State to initialise (must be non-NULL).
[in]	seed	Any value; 0 is remapped to a non-zero constant.
[in]	sleep_fn	Injected sleeper, or NULL for the host clock.
[in]	sleep_ctx	Context forwarded to sleep_fn (may be NULL).

Returns

Nothing.

Precondition

p, when non-NULL, points to writable mdl_politeness_t storage.

sleep_ctx outlives every mdl_politeness_wait call on p.

Postcondition

p->state is non-zero; p->sleep_fn/p->sleep_ctx equal the args.

The same seed yields the same delay sequence regardless of clock.

Note

Not thread-safe: initialises caller storage.

Since

0.1.0

Definition at line 53 of file mdl_politeness.c.

References k_seed_fallback, mdl_politeness_t::sleep_ctx, mdl_politeness_t::sleep_fn, and mdl_politeness_t::state.

Referenced by mdl_politeness_init().

mdl_politeness_wait()

uint32_t mdl_politeness_wait	(	mdl_politeness_t *	p,
		uint32_t	min_ms,
		uint32_t	max_ms )

Sleep a jittered delay in [min_ms, max_ms] and return it.

Draws the next xorshift64 value, maps it into [min_ms, max_ms], then blocks for that long – through the injected mdl_sleep_fn when one was wired, else on the host nanosleep. The returned delay is exactly the one requested of the sleeper, so a test with a recording fake can assert the timing without any wall-clock time elapsing.

Parameters

[in,out]	p	Jitter source (may be NULL).
[in]	min_ms	Lower bound, milliseconds.
[in]	max_ms	Upper bound, milliseconds (clamped up to min_ms).

Returns

The delay actually requested, in milliseconds.

Return values

0	p is NULL (no sleep is performed).
other	A value in [min_ms, max(min_ms, max_ms)].

Precondition

p, when non-NULL, was seeded by an init function.

A NULL p returns 0 without sleeping.

Postcondition

p->state has advanced by exactly one PRNG step (non-NULL p).

The sleeper (real or injected) was asked for the returned duration.

Note

Not thread-safe: advances p->state.

Since

0.1.0

Definition at line 134 of file mdl_politeness.c.

References internal_draw_range(), internal_host_sleep_ms(), mdl_politeness_t::sleep_ctx, mdl_politeness_t::sleep_fn, and mdl_politeness_t::state.

Referenced by internal_download_page_image().

mdl_retry_after_parse()

bool mdl_retry_after_parse	(	const char *	value,
		int64_t	now_wall_s,
		uint32_t *	out_ms )

Parse an HTTP Retry-After header value into a delay in milliseconds.

Accepts both forms RFC 7231 defines: a non-negative delta-seconds integer ("120"), returned as 120000; and an HTTP-date, whose delay is the date minus now_wall_s (clamped at zero, so a past date yields 0). Both the preferred IMF-fixdate ("Sun, 06 Nov 1994 08:49:37 GMT") and the obsolete RFC 850 form are recognised. A pure function with no clock of its own, so a test drives both forms deterministically by supplying now_wall_s.

Parameters

[in]	value	Raw header value (leading spaces tolerated), or NULL.
[in]	now_wall_s	Current wall-clock time in seconds (for the date form).
[out]	out_ms	Receives the delay in milliseconds (non-NULL).

Returns

Whether value parsed as a valid Retry-After.

Return values

true	Parsed; *out_ms holds the delay (saturated at UINT32_MAX ms).
false	NULL/empty/unparseable input; *out_ms is left unchanged.

Precondition

out_ms is non-NULL for a result to be written.

now_wall_s is the seconds the date form is measured against.

Postcondition

On false the output is not modified.

On true *out_ms is a non-negative millisecond delay.

Note

Thread-safe: depends only on its arguments (uses timegm, not mktime).

Since

0.1.0

Definition at line 262 of file mdl_politeness.c.

References internal_all_digits(), internal_ms_from_secs(), internal_parse_http_date(), and k_dec_base.

Referenced by mdl_governor_observe_at_wall().