mdl_site_t Struct Reference

One site's scraping rules (all data, no logic). More...

#include <mdl_config.h>

Data Fields
char	name [k_mdl_name_max]
	Human-readable site name.
char	host [k_mdl_host_max]
	Site host, e.g.
char	kind [k_mdl_kind_max]
	Content kind hint (reader mode).
char	contact [k_mdl_contact_max]
	Operator contact for the UA.
char	chapter_url_contains [k_mdl_match_max]
	href must contain this.
char	chapter_url_prefix [k_mdl_search_url_max]
	Absolute series-chapter prefix.
mdl_chapter_order_t	chapter_order
	Ordering to apply.
char	page_img_attr [k_mdl_attr_max]
	Preferred attr (fallback other).
char	page_img_url_contains [k_mdl_match_max]
	Keep only URLs with this.
char	search_url [k_mdl_search_url_max]
	Query template with a {q} slot.
char	search_result_contains [k_mdl_match_max]
	Series-link substring on results.
char	browse_url [k_mdl_search_url_max]
	Latest-updates page (no {q}).
char	series_title_selector [k_mdl_match_max]
	Series display-title selector.
char	series_summary_selector [k_mdl_match_max]
	Series synopsis selector.
char	series_author_selector [k_mdl_match_max]
	Series writer selector.
char	series_artist_selector [k_mdl_match_max]
	Series artist selector.
char	series_cover_selector [k_mdl_match_max]
	Series cover-URL selector.
char	chapter_title_selector [k_mdl_match_max]
	Per-chapter title selector.
char	chapter_number_selector [k_mdl_match_max]
	Per-chapter number selector.
char	language [k_mdl_kind_max]
	BCP-47 language tag.
char	reading_direction [k_mdl_kind_max]
	ltr or rtl.
uint32_t	img_delay_min
	Per-image spacing floor.
uint32_t	img_delay_max
	Per-image spacing ceiling.
uint32_t	chapter_delay_min
	Inter-chapter spacing floor.
uint32_t	chapter_delay_max
	Inter-chapter spacing ceiling.
uint32_t	rate_per_min
	Sustained per-host request ceiling (0 = off).
uint32_t	burst
	Token-bucket capacity, in requests.
uint32_t	backoff_base_ms
	First backoff window on a 429/503.
uint32_t	backoff_max_ms
	Backoff-window ceiling.
uint32_t	max_inflight
	Per-host in-flight request cap.

Detailed Description

One site's scraping rules (all data, no logic).

Invariant

Every string field is NUL-terminated.

Definition at line 46 of file mdl_config.h.

Field Documentation

backoff_base_ms

uint32_t mdl_site_t::backoff_base_ms

First backoff window on a 429/503.

Definition at line 96 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), and mdl_config_gov_cfg().

backoff_max_ms

uint32_t mdl_site_t::backoff_max_ms

Backoff-window ceiling.

Definition at line 97 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), and mdl_config_gov_cfg().

browse_url

char mdl_site_t::browse_url[k_mdl_search_url_max]

Latest-updates page (no {q}).

Definition at line 67 of file mdl_config.h.

Referenced by internal_config_apply_string(), and internal_discover_url().

burst

uint32_t mdl_site_t::burst

Token-bucket capacity, in requests.

Definition at line 95 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), and mdl_config_gov_cfg().

chapter_delay_max

uint32_t mdl_site_t::chapter_delay_max

Inter-chapter spacing ceiling.

Definition at line 90 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), internal_discover_fetch(), internal_mdl_fetch_chapter_html(), and mdl_config_apply_polite().

chapter_delay_min

uint32_t mdl_site_t::chapter_delay_min

Inter-chapter spacing floor.

Definition at line 89 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), internal_discover_fetch(), internal_mdl_fetch_chapter_html(), and mdl_config_apply_polite().

chapter_number_selector

char mdl_site_t::chapter_number_selector[k_mdl_match_max]

Per-chapter number selector.

Definition at line 80 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_mdl_fetch_select_chapter_number().

chapter_order

mdl_chapter_order_t mdl_site_t::chapter_order

Ordering to apply.

Definition at line 56 of file mdl_config.h.

Referenced by internal_config_apply_pair(), internal_config_set_defaults(), and internal_prepare_filter_chapters().

chapter_title_selector

char mdl_site_t::chapter_title_selector[k_mdl_match_max]

Per-chapter title selector.

Definition at line 78 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_mdl_fetch_select_chapter_title().

chapter_url_contains

char mdl_site_t::chapter_url_contains[k_mdl_match_max]

href must contain this.

Definition at line 53 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_set_defaults(), mdl_discover_run(), and priv_mdl_app_prepare_chapters().

chapter_url_prefix

char mdl_site_t::chapter_url_prefix[k_mdl_search_url_max]

Absolute series-chapter prefix.

Definition at line 54 of file mdl_config.h.

Referenced by internal_config_apply_string(), and internal_prepare_filter_chapters().

contact

char mdl_site_t::contact[k_mdl_contact_max]

Operator contact for the UA.

Definition at line 50 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_run_series_network(), and mdl_app_run_discover().

host

char mdl_site_t::host[k_mdl_host_max]

Site host, e.g.

"manhwaus.net".

Definition at line 48 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), internal_print_header(), internal_print_markup_changed(), internal_print_zero(), internal_reconcile_series_state(), and mdl_app_run_series().

img_delay_max

uint32_t mdl_site_t::img_delay_max

Per-image spacing ceiling.

Definition at line 88 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), internal_mdl_fetch_prepare_page(), mdl_config_apply_polite(), mdl_fetch_asset(), and priv_mdl_app_ensure_series_cover().

img_delay_min

uint32_t mdl_site_t::img_delay_min

Per-image spacing floor.

Definition at line 87 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), internal_mdl_fetch_prepare_page(), mdl_config_apply_polite(), mdl_fetch_asset(), and priv_mdl_app_ensure_series_cover().

kind

char mdl_site_t::kind[k_mdl_kind_max]

Content kind hint (reader mode).

Definition at line 49 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_set_defaults(), and mdl_app_run_series().

language

char mdl_site_t::language[k_mdl_kind_max]

BCP-47 language tag.

Definition at line 82 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_set_defaults(), internal_config_valid(), and internal_extract_series_metadata().

max_inflight

uint32_t mdl_site_t::max_inflight

Per-host in-flight request cap.

Definition at line 98 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), internal_config_valid(), and mdl_config_gov_cfg().

name

char mdl_site_t::name[k_mdl_name_max]

Human-readable site name.

Definition at line 47 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_set_defaults(), internal_reconcile_series_state(), and mdl_app_run_series().

page_img_attr

char mdl_site_t::page_img_attr[k_mdl_attr_max]

Preferred attr (fallback other).

Definition at line 59 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_set_defaults(), and internal_mdl_fetch_chapter_html().

page_img_url_contains

char mdl_site_t::page_img_url_contains[k_mdl_match_max]

Keep only URLs with this.

Definition at line 60 of file mdl_config.h.

Referenced by internal_config_apply_string(), and internal_mdl_fetch_chapter_html().

rate_per_min

uint32_t mdl_site_t::rate_per_min

Sustained per-host request ceiling (0 = off).

Definition at line 94 of file mdl_config.h.

Referenced by internal_config_apply_u32(), internal_config_set_defaults(), and mdl_config_gov_cfg().

reading_direction

char mdl_site_t::reading_direction[k_mdl_kind_max]

ltr or rtl.

Definition at line 83 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_set_defaults(), internal_config_valid(), and internal_extract_series_metadata().

search_result_contains

char mdl_site_t::search_result_contains[k_mdl_match_max]

Series-link substring on results.

Definition at line 65 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_print_zero(), and mdl_discover_run().

search_url

char mdl_site_t::search_url[k_mdl_search_url_max]

Query template with a {q} slot.

Definition at line 63 of file mdl_config.h.

Referenced by internal_config_apply_string(), and internal_discover_url().

series_artist_selector

char mdl_site_t::series_artist_selector[k_mdl_match_max]

Series artist selector.

Definition at line 76 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_extract_optional_metadata().

series_author_selector

char mdl_site_t::series_author_selector[k_mdl_match_max]

Series writer selector.

Definition at line 75 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_extract_optional_metadata().

series_cover_selector

char mdl_site_t::series_cover_selector[k_mdl_match_max]

Series cover-URL selector.

Definition at line 77 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_extract_cover().

series_summary_selector

char mdl_site_t::series_summary_selector[k_mdl_match_max]

Series synopsis selector.

Definition at line 73 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_extract_optional_metadata().

series_title_selector

char mdl_site_t::series_title_selector[k_mdl_match_max]

Series display-title selector.

Definition at line 71 of file mdl_config.h.

Referenced by internal_config_apply_string(), internal_config_valid(), and internal_extract_series_metadata().

---

The documentation for this struct was generated from the following file:

• apps/shared_libs/mdl/inc/mdl_config.h