mdl_robots_t Struct Reference

Rules and crawl-delay for the group matching our user-agent. More...

#include <mdl_robots.h>

Collaboration diagram for mdl_robots_t:

Data Fields
mdl_robots_rule_t	rules [k_mdl_robots_max_rules]
	Selected group's rules.
size_t	count
	Number of valid rules.
bool	valid
	False if input exceeded a bound.
bool	have_crawl_delay
	A Crawl-delay was seen.
uint32_t	crawl_delay_ms
	Crawl-delay in ms.

Detailed Description

Rules and crawl-delay for the group matching our user-agent.

The result of parsing one robots.txt for one user-agent token; an all-zero value (no rules) permits every path.

Invariant

count <= k_mdl_robots_max_rules.

valid is false if any line, rule, or rule table exceeded a bound.

See also

mdl_robots_parse()

Since

0.1.0

Definition at line 74 of file mdl_robots.h.

Field Documentation

count

size_t mdl_robots_t::count

Number of valid rules.

Definition at line 76 of file mdl_robots.h.

Referenced by internal_add_rule(), and internal_best_matching_rule().

crawl_delay_ms

uint32_t mdl_robots_t::crawl_delay_ms

Crawl-delay in ms.

Definition at line 79 of file mdl_robots.h.

Referenced by internal_set_crawl(), and mdl_session_url_allowed().

have_crawl_delay

bool mdl_robots_t::have_crawl_delay

A Crawl-delay was seen.

Definition at line 78 of file mdl_robots.h.

Referenced by internal_set_crawl(), and mdl_session_url_allowed().

rules

mdl_robots_rule_t mdl_robots_t::rules[k_mdl_robots_max_rules]

Selected group's rules.

Definition at line 75 of file mdl_robots.h.

Referenced by internal_add_rule(), and internal_best_matching_rule().

valid

bool mdl_robots_t::valid

False if input exceeded a bound.

Definition at line 77 of file mdl_robots.h.

Referenced by internal_add_rule(), internal_harvest(), mdl_robots_allows(), mdl_robots_cache_consult(), and mdl_robots_parse().

---

The documentation for this struct was generated from the following file:

• apps/shared_libs/mdl/inc/mdl_robots.h