- Generated by
1.16.1
|
ra8-firmware 0.1.0
Bare-metal firmware for the Renesas RA8 family (RA8D2 / RA8P1)
|
Session identity (honest User-Agent) and robots.txt gating. More...
#include <stddef.h>#include <stdint.h>#include "mdl_net.h"#include "mdl_robots.h"#include "ra8_io_stream.h"Go to the source code of this file.
Data Structures | |
| struct | mdl_session_t |
| One download session's identity and robots.txt state. More... | |
Enumerations | |
| enum | mdl_session_size_t : uint32_t { k_mdl_session_scratch = 65536U , k_mdl_ua_max = 256U } |
| Fixed session buffer sizes. More... | |
Functions | |
| bool | mdl_session_build_ua (const char *contact, char *out, size_t cap) |
| Build the truthful default User-Agent into out. | |
| const char * | mdl_session_ua_token (void) |
| The robots.txt product token this tool matches on ("mdl"). | |
| void | mdl_session_init (mdl_session_t *session, mdl_net_iface_t *net, const char *user_agent, ra8_io_stream_t *diagnostic, bool honor_robots) |
| Initialise a session over a network backend. | |
| bool | mdl_session_url_allowed (mdl_session_t *session, const char *url, uint32_t *crawl_delay_ms) |
| Decide whether url may be fetched under robots.txt, and its delay. | |
Session identity (honest User-Agent) and robots.txt gating.
Ties the network backend, a truthful configurable User-Agent, and the per-host robots.txt cache into one object the download loop consults. The User-Agent identifies the tool by name, version and a project URL rather than impersonating a browser – a lying UA is what gets a well-behaved bot hard-banned instead of soft-throttled. Robots gating is on by default and fetches /robots.txt per host on first contact.
Definition in file mdl_session.h.
| enum mdl_session_size_t : uint32_t |
Fixed session buffer sizes.
| Enumerator | |
|---|---|
| k_mdl_session_scratch | robots.txt fetch scratch bytes. |
| k_mdl_ua_max | Built User-Agent buffer bytes. |
Definition at line 25 of file mdl_session.h.
| bool mdl_session_build_ua | ( | const char * | contact, |
| char * | out, | ||
| size_t | cap ) |
Build the truthful default User-Agent into out.
Emits mdl/<version> (+<project-url>; <contact>), or the same without the contact clause when none is supplied. No browser-impersonation string is ever produced.
| [in] | contact | Operator contact (email/URL), or NULL/empty for none. |
| [out] | out | Destination buffer for the NUL-terminated UA string. |
| [in] | cap | Capacity of out in bytes. |
| true | contact was non-empty and appears in the UA. |
| false | No contact was configured (caller should warn once). |
Definition at line 47 of file mdl_session.c.
References s_ua_token, s_ua_url, and s_ua_version.
Referenced by priv_mdl_app_start_session().
| void mdl_session_init | ( | mdl_session_t * | session, |
| mdl_net_iface_t * | net, | ||
| const char * | user_agent, | ||
| ra8_io_stream_t * | diagnostic, | ||
| bool | honor_robots ) |
Initialise a session over a network backend.
Stores the borrowed backend and User-Agent pointers, selects the robots policy, and clears the fixed per-host cache. No network request is made until mdl_session_url_allowed examines a host for the first time.
| [out] | session | Session to initialise (non-NULL). |
| [in] | net | Backend used for robots.txt fetches. |
| [in] | user_agent | Full UA header string (must outlive the session). |
| [in,out] | diagnostic | Borrowed stream receiving robots policy diagnostics. |
| [in] | honor_robots | Whether robots.txt is consulted (false = ignore). |
honor_robots.Definition at line 61 of file mdl_session.c.
References mdl_session_t::cache, mdl_session_t::diagnostic, mdl_session_t::honor_robots, mdl_session_t::net, and mdl_session_t::user_agent.
Referenced by priv_mdl_app_start_session().
| const char * mdl_session_ua_token | ( | void | ) |
The robots.txt product token this tool matches on ("mdl").
| non-NULL | Always: the constant product token. |
Definition at line 42 of file mdl_session.c.
References s_ua_token.
| bool mdl_session_url_allowed | ( | mdl_session_t * | session, |
| const char * | url, | ||
| uint32_t * | crawl_delay_ms ) |
Decide whether url may be fetched under robots.txt, and its delay.
Consults the per-host robots cache (fetching /robots.txt on first contact), and refuses – printing a message that names the blocking rule – a URL our user-agent is disallowed from. When robots gating is off (--ignore-robots) every URL is permitted. The host's Crawl-delay is reported so the caller can raise its per-host politeness floor.
| [in] | session | Initialised session. |
| [in] | url | Absolute http(s) URL to test. |
| [out] | crawl_delay_ms | Receives the host's Crawl-delay in ms (0 if none). |
| true | Robots allows it, or gating was explicitly disabled. |
| false | A rule forbids it, robots is inaccessible/oversized, or the URL cannot be parsed safely; a message was printed to stderr. |
Definition at line 276 of file mdl_session.c.
References mdl_session_t::cache, mdl_robots_t::crawl_delay_ms, mdl_session_t::diagnostic, mdl_robots_t::have_crawl_delay, mdl_session_t::honor_robots, internal_report_robots_disallow(), internal_report_robots_unavailable(), internal_robots_target(), internal_session_fetch(), internal_url_scheme(), k_mdl_robots_host_max, k_mdl_robots_path_max, k_ra8_ok, mdl_robots_cache_consult(), mdl_robots_disallow_reason(), mdl_url_host(), priv_mdl_stream_text(), s_ua_token, and mdl_session_t::scratch.
Referenced by internal_discover_fetch(), internal_download_page_image(), internal_fetch_artifact(), internal_mdl_fetch_chapter_html(), internal_mdl_fetch_prepare_page(), mdl_app_run_page(), mdl_fetch_asset(), priv_mdl_app_ensure_series_cover(), and priv_mdl_app_prepare_chapters().
1.16.1