reflow_tokenize_internal.h File Reference

Test-access surface for reflow_tokenize.c internal helpers. More...

#include <stddef.h>
#include <stdint.h>
#include "ra8_attributes.h"
#include "ra8_err.h"
#include "reflow.h"

Include dependency graph for reflow_tokenize_internal.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Enumerations
enum	priv_tok_consts_t : uint32_t {   k_priv_max_depth = 64U ,   k_priv_tag_name_cap = 16U ,   k_priv_entity_window = 12U ,   k_priv_entity_min = 4U ,   k_priv_uc_2byte = 0x80U ,   k_priv_uc_3byte = 0x800U ,   k_priv_uc_4byte = 0x10000U ,   k_priv_uc_max = 0x10FFFFU ,   k_priv_utf8_lead2 = 0xC0U ,   k_priv_utf8_lead3 = 0xE0U ,   k_priv_utf8_lead4 = 0xF0U ,   k_priv_utf8_cont = 0x80U ,   k_priv_utf8_mask = 0x3FU ,   k_priv_utf8_sh6 = 6U ,   k_priv_utf8_sh12 = 12U ,   k_priv_utf8_sh18 = 18U ,   k_priv_base_dec = 10U ,   k_priv_base_hex = 16U ,   k_priv_hex_offset = 10U ,   k_priv_style_mask }
	Tokenizer bounds and UTF-8 / numeric-entity encoding constants. More...

Functions
bool	priv_reflow_tok_is_xml_whitespace (char c)
	True for the ASCII characters XHTML treats as whitespace.
reflow_html_tag_t	priv_reflow_tok_classify (const char *name, size_t len)
	Map a (possibly namespace-prefixed) tag name to its enum.
bool	priv_reflow_tok_decode_entity (const char *src, size_t avail, uint32_t *out_cp, size_t *out_used)
	Decode one XML entity reference beginning at &.
size_t	priv_reflow_tok_utf8_encode (uint32_t cp, uint8_t *dst)
	UTF-8 encode a code point into dst (up to 4 bytes).
bool	priv_reflow_tok_is_block (reflow_html_tag_t tag)
	Test whether a tag is a block-flow container in the v1 subset.
uint8_t	priv_reflow_tok_style_for (reflow_html_tag_t tag)
	Return the style bit an inline tag contributes to its subtree.
bool	priv_reflow_tok_emit (reflow_t *engine, reflow_token_kind_t kind, reflow_html_tag_t tag, uint8_t style, uint32_t off, uint32_t len)
	Append one token to the engine's token pool.
bool	priv_reflow_tok_feed (reflow_t *engine, char ch, bool *last_ws)
	Write one byte through the whitespace-collapse state machine.
bool	priv_reflow_tok_stash_run (reflow_t *engine, const uint8_t *buf, size_t start, size_t end, uint32_t *out_off, uint32_t *out_len)
	Entity-decode and whitespace-collapse a text run into the pool.
bool	priv_reflow_tok_starts_with (const uint8_t *buf, size_t i, size_t len, const char *lit)
	Test whether buf at i begins with the literal lit.
size_t	priv_reflow_tok_skip_past (const uint8_t *buf, size_t i, size_t len, const char *lit)
	Return the index just past the first occurrence of lit.
size_t	priv_reflow_tok_find_lit (const uint8_t *buf, size_t i, size_t len, const char *lit)
	Return the index of the first occurrence of lit (its start), or len.
bool	priv_reflow_tok_find_attr (const uint8_t *tag, size_t tag_len, const char *name, size_t name_len, size_t *out_voff, size_t *out_vlen)
	Locate a named attribute's quoted value span within a tag (no copy).
void	priv_reflow_tok_capture_attr (reflow_t *engine, const uint8_t *tag, size_t tag_len, const char *name, size_t name_len, uint32_t *out_off, uint32_t *out_len)
	Copy a named attribute's quoted value from a tag span into the pool.
ra8_css_element_t	priv_reflow_tok_css_element (reflow_html_tag_t kind, const uint8_t *tag, size_t span)
	Build a CSS element identity from a just-opened tag's attributes.
ra8_css_style_t	priv_reflow_tok_css_inline (const uint8_t *tag, size_t span)
	Parse a just-opened tag's inline style="..." into a CSS declaration.
uint8_t	priv_reflow_tok_intern_link (reflow_t *engine, uint32_t href_off, uint32_t href_len)
	Intern an <a> href slice into the link-target table.
bool	priv_reflow_tok_rel_is_stylesheet (const uint8_t *rel, size_t len)
	True if a rel attribute value contains the stylesheet token.
uint8_t	priv_reflow_tok_resolve_face_slot (const reflow_t *engine, const ra8_css_style_t *comp)
	Map a cascaded run's (font-family + emphasis) to an embedded face slot.
ra8_err_t	priv_reflow_xml_walk (reflow_t *engine, const uint8_t *xhtml_buf, size_t xhtml_len)
	Tokenize the XHTML buffer and populate engine->tokens[].

Detailed Description

Test-access surface for reflow_tokenize.c internal helpers.

Exposes the small, individually-MC/DC-able helpers of the no-heap XHTML tokenizer so tests/src/test_reflow_tokenize.c can drive both arms of each decision directly (the tokenizer's main loop is otherwise reached only through priv_reflow_xml_walk). Not part of the public API.

[Ring 4 / Reflow] {World: NS}

Copyright

Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Since

0.1.0

Definition in file reflow_tokenize_internal.h.

Enumeration Type Documentation

priv_tok_consts_t

enum priv_tok_consts_t : uint32_t

Tokenizer bounds and UTF-8 / numeric-entity encoding constants.

Shared across the tokenizer translation units (reflow_tokenize.c, reflow_tokenize_lex.c): the lexical primitives use the UTF-8 / numeric-entity / tag-name constants while the markup engine uses the nesting-depth bound and the run-style mask. Kept here so a single definition is visible to every TU.

Invariant

k_priv_uc_2byte < k_priv_uc_3byte < k_priv_uc_4byte < k_priv_uc_max.

See also

priv_reflow_tok_utf8_encode

Since

0.1.0

Enumerator
k_priv_max_depth	Max element nesting (matches old shim).
k_priv_tag_name_cap	Lower-cased tag-name buffer size.
k_priv_entity_window	Max bytes scanned for one '&...;'.
k_priv_entity_min	Shortest valid reference ("<").
k_priv_uc_2byte	Code points >= need >= 2 UTF-8 bytes.
k_priv_uc_3byte	Code points >= need >= 3 UTF-8 bytes.
k_priv_uc_4byte	Code points >= need 4 UTF-8 bytes.
k_priv_uc_max	Highest valid Unicode code point.
k_priv_utf8_lead2	2-byte sequence lead-byte prefix.
k_priv_utf8_lead3	3-byte sequence lead-byte prefix.
k_priv_utf8_lead4	4-byte sequence lead-byte prefix.
k_priv_utf8_cont	Continuation-byte prefix.
k_priv_utf8_mask	Low 6 bits per continuation byte.
k_priv_utf8_sh6	Shift for one continuation byte.
k_priv_utf8_sh12	Shift for two continuation bytes.
k_priv_utf8_sh18	Shift for three continuation bytes.
k_priv_base_dec	Decimal numeric-entity base.
k_priv_base_hex	Hexadecimal numeric-entity base.
k_priv_hex_offset	Value of hex 'a'/'A' minus the letter.
k_priv_style_mask	Run-style bits.

Definition at line 42 of file reflow_tokenize_internal.h.

Function Documentation

priv_reflow_tok_capture_attr()

void priv_reflow_tok_capture_attr	(	reflow_t *	engine,
		const uint8_t *	tag,
		size_t	tag_len,
		const char *	name,
		size_t	name_len,
		uint32_t *	out_off,
		uint32_t *	out_len )

Copy a named attribute's quoted value from a tag span into the pool.

Scans the raw <...> span for attribute name and copies its quoted value verbatim into the engine text pool. On no match / malformed attribute / empty value / full pool the outputs are zeroed. Defined in reflow_tokenize_attr.c.

Parameters

[in,out]	engine	Engine whose text pool receives the value bytes.
[in]	tag	Raw tag span starting at '<'.
[in]	tag_len	Length of tag, bytes.
[in]	name	Lower-case attribute name to find.
[in]	name_len	Length of name, bytes.
[out]	out_off	Receives the text-pool offset of the value (0 if none).
[out]	out_len	Receives the value byte length (0 if none).

Returns

Nothing.

Precondition

engine, tag, name, out_off, out_len are non-null.

engine->text_pool_used <= k_reflow_text_pool_bytes.

Postcondition

*out_len > 0 iff a quoted name value was stored in the pool.

The pool grows by *out_len bytes when a value is stored.

Note

Not thread-safe.

Since

0.1.0

Copy a named attribute's quoted value from a tag span into the pool.

Scans the raw <...> span for attribute name (case-insensitive, not preceded by another name character) and copies its quoted value verbatim into the engine text pool. Used for <img src> / <a href> / element id. On no match, a malformed attribute, an empty value, or a full pool the outputs are zeroed, which callers treat as "attribute absent".

Parameters

[in,out]	engine	Engine whose text pool receives the value bytes.
[in]	tag	Raw tag span starting at '<'.
[in]	tag_len	Length of tag, bytes.
[in]	name	Lower-case attribute name to find.
[in]	name_len	Length of name, bytes.
[out]	out_off	Receives the text-pool offset of the value (0 if none).
[out]	out_len	Receives the value byte length (0 if none).

Returns

Nothing.

Precondition

engine, tag, name, out_off, out_len are non-null.

engine->text_pool_used <= k_reflow_text_pool_bytes.

Postcondition

*out_len > 0 iff a quoted name value was stored in the pool.

The pool grows by *out_len bytes when a value is stored.

Note

Not thread-safe.

Since

0.1.0

Definition at line 196 of file reflow_tokenize_attr.c.

References k_reflow_text_pool_bytes, memcpy(), priv_reflow_tok_find_attr(), reflow_t::text_pool, and reflow_t::text_pool_used.

Referenced by internal_handle_void(), and internal_open_attrs().

priv_reflow_tok_classify()

reflow_html_tag_t priv_reflow_tok_classify	(	const char *	name,
		size_t	len )

Map a (possibly namespace-prefixed) tag name to its enum.

Strips any prefix: namespace, lower-cases the local name, and matches it against the v1 subset's tag table.

Parameters

[in]	name	NUL-terminated local-or-prefixed tag name.
[in]	len	Length of name in bytes (excluding any NUL).

Returns

The matching reflow_html_tag_t, or k_reflow_tag_unknown.

Return values

k_reflow_tag_unknown

Name is null or unrecognised.

Precondition

None.

None.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function; case-insensitive on ASCII.

Since

0.1.0

< Word.

< Tag.

Definition at line 118 of file reflow_tokenize_lex.c.

References internal_local_lower(), k_priv_tag_name_cap, k_reflow_tag_a, k_reflow_tag_b, k_reflow_tag_blockquote, k_reflow_tag_br, k_reflow_tag_em, k_reflow_tag_h1, k_reflow_tag_h2, k_reflow_tag_h3, k_reflow_tag_h4, k_reflow_tag_h5, k_reflow_tag_h6, k_reflow_tag_hr, k_reflow_tag_i, k_reflow_tag_img, k_reflow_tag_li, k_reflow_tag_link, k_reflow_tag_ol, k_reflow_tag_p, k_reflow_tag_strong, k_reflow_tag_table, k_reflow_tag_td, k_reflow_tag_th, k_reflow_tag_tr, k_reflow_tag_ul, k_reflow_tag_unknown, and strcmp().

Referenced by internal_parse_sel_type(), and internal_parse_start().

priv_reflow_tok_css_element()

ra8_css_element_t priv_reflow_tok_css_element	(	reflow_html_tag_t	kind,
		const uint8_t *	tag,
		size_t	span )

Build a CSS element identity from a just-opened tag's attributes.

Locates the id and class attribute value spans in place (no copy) so the cascade can match #id / .class selectors; the returned pointers alias tag. Defined in reflow_tokenize_attr.c.

Parameters

[in]	kind	Classified tag.
[in]	tag	Raw tag span starting at '<'.
[in]	span	Length of tag, bytes.

Returns

The element identity for ra8_css_cascade().

Return values

ra8_css_element_t

Populated struct; absent attributes have NULL pointer and zero length.

Precondition

tag is non-null and holds span bytes.

kind is a valid reflow_html_tag_t value.

Postcondition

No state mutated; returned pointers alias tag.

el.tag == (uint8_t)kind.

Note

Pure read of tag.

Since

0.1.0

Locates the id and class attribute value spans in place (no copy) so the cascade can match #id / .class selectors; the returned pointers alias tag. A missing attribute leaves its pointer NULL and its length zero. The element's tag enum is stored so type selectors also work.

Parameters

[in]	kind	Classified tag.
[in]	tag	Raw tag span starting at '<'.
[in]	span	Length of tag, bytes.

Returns

The element identity for ra8_css_cascade().

Return values

ra8_css_element_t

Fully populated struct; absent attributes have NULL pointer and zero length.

Precondition

tag is non-null and holds span bytes.

kind is a valid reflow_html_tag_t value.

Postcondition

No state mutated; returned pointers alias tag.

el.tag == (uint8_t)kind.

Note

Pure read of tag.

Since

0.1.0

Definition at line 243 of file reflow_tokenize_attr.c.

References ra8_css_element_t::class_len, ra8_css_element_t::class_str, ra8_css_element_t::id, ra8_css_element_t::id_len, priv_reflow_tok_find_attr(), and ra8_css_element_t::tag.

Referenced by internal_handle_start().

priv_reflow_tok_css_inline()

ra8_css_style_t priv_reflow_tok_css_inline	(	const uint8_t *	tag,
		size_t	span )

Parse a just-opened tag's inline style="..." into a CSS declaration.

Locates the style attribute and delegates to ra8_css_parse_inline(). The returned struct has set == 0 when no inline declarations override the cascade. Defined in reflow_tokenize_attr.c.

Parameters

[in]	tag	Raw tag span starting at '<'.
[in]	span	Length of tag, bytes.

Returns

The inline CSS declaration for this element.

Return values

ra8_css_style_t

Struct with set == 0 if no style attribute is present.

Precondition

tag is non-null and holds span bytes.

span > 0.

Postcondition

No state mutated.

Returned struct set field is 0 when no style attribute was found.

Note

Pure read of tag.

Since

0.1.0

Locates the style attribute within the raw tag span and delegates to ra8_css_parse_inline() to parse its value into a declaration struct. When the attribute is absent or the parser fails the returned struct has set == 0, indicating no inline declarations override the cascade.

Parameters

[in]	tag	Raw tag span starting at '<'.
[in]	span	Length of tag, bytes.

Returns

The inline CSS declaration for this element.

Return values

ra8_css_style_t

Struct with set == 0 if no style attribute is present.

Precondition

tag is non-null and holds span bytes.

span > 0 (a zero-span tag cannot carry attributes).

Postcondition

No state mutated.

Returned struct set field is 0 when no style attribute was found.

Note

Pure read of tag.

Since

0.1.0

Definition at line 279 of file reflow_tokenize_attr.c.

References priv_reflow_tok_find_attr(), and ra8_css_parse_inline().

Referenced by internal_open_styled().

priv_reflow_tok_decode_entity()

bool priv_reflow_tok_decode_entity	(	const char *	src,
		size_t	avail,
		uint32_t *	out_cp,
		size_t *	out_used )

Decode one XML entity reference beginning at &.

Recognises the named entities amp/lt/gt/quot/apos and numeric &#dec; / &#xhex; references. Unrecognised sequences are reported as "not an entity" so the caller emits the literal &.

Parameters

[in]	src	Buffer positioned so src[0] == '&'.
[in]	avail	Bytes available from src (>= 1).
[out]	out_cp	Decoded Unicode code point on success.
[out]	out_used	Bytes consumed from src on success.

Returns

true if a complete recognised entity was decoded.

Return values

false

Not a recognised/complete entity (caller emits '&').

Precondition

src, out_cp, out_used are non-null and avail >= 1.

src[0] == '&'.

Postcondition

On false, out params are unspecified and no input is consumed.

On true, *out_used is in [3, avail].

Note

Pure function.

Since

0.1.0

< Word.

< Cp.

Definition at line 220 of file reflow_tokenize_lex.c.

References internal_decode_numeric(), k_priv_entity_min, k_priv_entity_window, strlen(), and strncmp().

Referenced by internal_stash_one().

priv_reflow_tok_emit()

bool priv_reflow_tok_emit	(	reflow_t *	engine,
		reflow_token_kind_t	kind,
		reflow_html_tag_t	tag,
		uint8_t	style,
		uint32_t	off,
		uint32_t	len )

Append one token to the engine's token pool.

Fills the next reflow_token_t and advances token_count, or fails if the fixed pool is full. Defined in reflow_tokenize_lex.c.

Parameters

[in,out]	engine	Engine whose token pool is appended to.
[in]	kind	Token kind.
[in]	tag	Associated tag (or k_reflow_tag_unknown).
[in]	style	Inline-style bits for the token.
[in]	off	Text-pool offset (text tokens only).
[in]	len	Text-pool length (text tokens only).

Returns

true on success, false on pool overflow.

Return values

false

The token pool is full.

Precondition

engine is non-null.

engine->token_count <= k_reflow_max_tokens.

Postcondition

On success token_count is incremented by one.

On failure the pool is unchanged.

Note

Not thread-safe.

Since

0.1.0

Fills the next reflow_token_t and advances token_count, or fails if the fixed pool is full.

Parameters

[in,out]	engine	Engine whose token pool is appended to.
[in]	kind	Token kind.
[in]	tag	Associated tag (or k_reflow_tag_unknown).
[in]	style	Inline-style bits for the token.
[in]	off	Text-pool offset (text tokens only).
[in]	len	Text-pool length (text tokens only).

Returns

true on success, false on pool overflow.

Return values

false

The token pool is full.

Precondition

engine is non-null.

engine->token_count <= k_reflow_max_tokens.

Postcondition

On success token_count is incremented by one.

On failure the pool is unchanged.

Note

Not thread-safe.

Since

0.1.0

Definition at line 358 of file reflow_tokenize_lex.c.

References reflow_token_t::color, reflow_token_t::css_font_px, k_reflow_color_inherit, k_reflow_max_tokens, reflow_token_t::kind, reflow_token_t::reserved, reflow_token_t::reserved16, reflow_token_t::style, reflow_token_t::tag, reflow_token_t::text_len, reflow_token_t::text_off, reflow_t::token_count, and reflow_t::tokens.

Referenced by internal_emit_text_run(), internal_handle_cdata(), internal_handle_end(), internal_handle_start(), internal_handle_void(), and internal_open_attrs().

priv_reflow_tok_feed()

bool priv_reflow_tok_feed	(	reflow_t *	engine,
		char	ch,
		bool *	last_ws )

Write one byte through the whitespace-collapse state machine.

Runs of ASCII whitespace collapse to a single space; a run that begins with *last_ws true (start of node) drops its leading whitespace. Defined in reflow_tokenize_lex.c.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	ch	Byte to write.
[in,out]	last_ws	Collapse state (true if the prior byte was space).

Returns

true on success, false on text-pool overflow.

Return values

false

The text pool is full.

Precondition

engine and last_ws are non-null.

engine->text_pool_used <= k_reflow_text_pool_bytes.

Postcondition

On a written byte text_pool_used is incremented.

Collapsed (skipped) whitespace leaves the pool unchanged.

Note

Not thread-safe.

Since

0.1.0

Runs of ASCII whitespace collapse to a single space; a run that begins with *last_ws true (start of node) drops its leading whitespace.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	ch	Byte to write.
[in,out]	last_ws	Collapse state (true if the prior byte was space).

Returns

true on success, false on text-pool overflow.

Return values

false

The text pool is full.

Precondition

engine and last_ws are non-null.

engine->text_pool_used <= k_reflow_text_pool_bytes.

Postcondition

On a written byte text_pool_used is incremented.

Collapsed (skipped) whitespace leaves the pool unchanged.

Note

Not thread-safe.

Since

0.1.0

Definition at line 400 of file reflow_tokenize_lex.c.

References k_reflow_text_pool_bytes, priv_reflow_tok_is_xml_whitespace(), reflow_t::text_pool, and reflow_t::text_pool_used.

Referenced by internal_feed_utf8(), internal_handle_cdata(), and internal_stash_one().

priv_reflow_tok_find_attr()

bool priv_reflow_tok_find_attr	(	const uint8_t *	tag,
		size_t	tag_len,
		const char *	name,
		size_t	name_len,
		size_t *	out_voff,
		size_t *	out_vlen )

Locate a named attribute's quoted value span within a tag (no copy).

Returns the value's offset + length within tag so callers can copy it to the pool or scan it in place. Defined in reflow_tokenize_attr.c.

Parameters

[in]	tag	Raw tag span starting at '<'.
[in]	tag_len	Length of tag, bytes.
[in]	name	Lower-case attribute name to find.
[in]	name_len	Length of name, bytes.
[out]	out_voff	Receives the value offset within tag.
[out]	out_vlen	Receives the value length, bytes.

Returns

true iff a quoted name value was found.

Return values

true	*out_voff / *out_vlen describe the value within tag.
false	Attribute absent or malformed.

Precondition

tag, name, out_voff, out_vlen are non-null.

tag_len >= name_len.

Postcondition

No state mutated.

On false, *out_voff and *out_vlen are unspecified.

Note

Pure read of tag.

Since

0.1.0

The non-copying core of priv_reflow_tok_capture_attr(): returns the value's offset + length *within tag* so callers can either copy it to the pool or scan it in place (e.g. an inline style).

Parameters

[in]	tag	Raw tag span starting at '<'.
[in]	tag_len	Length of tag, bytes.
[in]	name	Lower-case attribute name to find.
[in]	name_len	Length of name, bytes.
[out]	out_voff	Receives the value offset within tag.
[out]	out_vlen	Receives the value length, bytes.

Returns

true iff a quoted name value was found.

Return values

true	*out_voff / *out_vlen describe the value within tag.
false	Attribute absent or malformed.

Precondition

tag, name, out_voff, out_vlen are non-null.

tag_len >= name_len (zero-length searches return false immediately).

Postcondition

No state mutated.

On false, *out_voff and *out_vlen are unspecified.

Note

Pure read of tag.

Since

0.1.0

Definition at line 153 of file reflow_tokenize_attr.c.

References internal_attr_name_at(), and internal_attr_quoted_value().

Referenced by internal_handle_link(), priv_reflow_tok_capture_attr(), priv_reflow_tok_css_element(), and priv_reflow_tok_css_inline().

priv_reflow_tok_find_lit()

size_t priv_reflow_tok_find_lit	(	const uint8_t *	buf,
		size_t	i,
		size_t	len,
		const char *	lit )

Return the index of the first occurrence of lit (its start), or len.

Like priv_reflow_tok_skip_past but returns the literal's START offset. Defined in reflow_tokenize_lex.c.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to search from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to find.

Returns

Start index of lit, or len if absent.

Return values

len	The literal does not occur in [i, len).

Precondition

buf and lit are non-null.

i <= len.

Postcondition

No state is modified (pure).

Return value is in [i, len].

Note

Pure function.

Since

0.1.0

Like priv_reflow_tok_skip_past but returns the literal's START offset, so the caller can both bound the preceding content and resume past the literal.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to search from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to find.

Returns

Start index of lit, or len if absent.

Return values

len	The literal does not occur in [i, len).

Precondition

buf and lit are non-null.

i <= len.

Postcondition

No state is modified (pure).

Return value is in [i, len].

Note

Pure function.

Since

0.1.0

Definition at line 623 of file reflow_tokenize_lex.c.

References internal_bytes_equal(), and strlen().

Referenced by internal_handle_raw_text().

priv_reflow_tok_intern_link()

uint8_t priv_reflow_tok_intern_link	(	reflow_t *	engine,
		uint32_t	href_off,
		uint32_t	href_len )

Intern an <a> href slice into the link-target table.

Appends a {href_off, href_len} entry and returns its 1-based id. An empty href or a full table returns 0. Defined in reflow_tokenize_attr.c.

Parameters

[in,out]	engine	Engine whose link-target table grows.
[in]	href_off	Href text-pool offset.
[in]	href_len	Href byte length.

Returns

1-based link id, or 0 if not interned.

Return values

0	Empty href or the table is full.
1..k_reflow_max_links	The newly assigned 1-based link id.

Precondition

engine is non-null.

engine->link_target_count <= k_reflow_max_links.

Postcondition

On a non-zero return the table grew by one entry.

engine->link_target_count is unchanged on a zero return.

Note

Not thread-safe.

Since

0.1.0

Appends a {href_off, href_len} entry and returns its 1-based id (stored in subsequent text tokens' reserved byte). An empty href or a full table returns 0 (the run renders as plain underlined text – not tappable).

Parameters

[in,out]	engine	Engine whose link-target table grows.
[in]	href_off	Href text-pool offset.
[in]	href_len	Href byte length.

Returns

1-based link id, or 0 if not interned.

Return values

0	Empty href or the table is full.
1..k_reflow_max_links	The newly assigned 1-based link id.

Precondition

engine is non-null.

engine->link_target_count <= k_reflow_max_links.

Postcondition

On a non-zero return the table grew by one entry.

engine->link_target_count is unchanged on a zero return.

Note

Not thread-safe.

Since

0.1.0

Definition at line 315 of file reflow_tokenize_attr.c.

References reflow_link_target_t::href_len, reflow_link_target_t::href_off, k_reflow_max_links, reflow_t::link_target_count, and reflow_t::link_targets.

Referenced by internal_open_attrs().

priv_reflow_tok_is_block()

bool priv_reflow_tok_is_block

(

reflow_html_tag_t

tag

)

Test whether a tag is a block-flow container in the v1 subset.

Block tags emit block-start / block-end tokens; all others do not introduce a block box. Defined in reflow_tokenize_lex.c.

Parameters

[in]

tag

Classified tag.

Returns

true for p / h1-h6 / ul / ol / li / blockquote / table family.

Return values

false

Any inline, void, or unknown tag.

Precondition

None.

None.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Block tags emit block-start / block-end tokens; all others do not introduce a block box.

Parameters

[in]

tag

Classified tag.

Returns

true for p / h1-h6 / ul / ol / li / blockquote.

Return values

false

Any inline, void, or unknown tag.

Precondition

None.

None.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Definition at line 276 of file reflow_tokenize_lex.c.

References k_reflow_tag_blockquote, k_reflow_tag_h1, k_reflow_tag_h2, k_reflow_tag_h3, k_reflow_tag_h4, k_reflow_tag_h5, k_reflow_tag_h6, k_reflow_tag_li, k_reflow_tag_ol, k_reflow_tag_p, k_reflow_tag_table, k_reflow_tag_td, k_reflow_tag_th, k_reflow_tag_tr, and k_reflow_tag_ul.

Referenced by internal_handle_end(), and internal_handle_start().

priv_reflow_tok_is_xml_whitespace()

bool priv_reflow_tok_is_xml_whitespace

(

char

c

)

True for the ASCII characters XHTML treats as whitespace.

Matches space, tab, newline, carriage-return, form-feed and vertical-tab – the set collapsed by the tokenizer's text handling.

Parameters

[in]

c

Byte to classify.

Returns

true if c is space/tab/newline/carriage-return/form-feed/vtab.

Return values

false

c is any non-whitespace byte.

Precondition

None.

None.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Definition at line 41 of file reflow_tokenize_lex.c.

Referenced by internal_attr_quoted_value(), internal_parse_start(), internal_tag_is(), and priv_reflow_tok_feed().

priv_reflow_tok_rel_is_stylesheet()

bool priv_reflow_tok_rel_is_stylesheet	(	const uint8_t *	rel,
		size_t	len )

True if a rel attribute value contains the stylesheet token.

Case-sensitive substring scan, bounded by len. Defined in reflow_tokenize_attr.c.

Parameters

[in]	rel	Attribute-value bytes (not NUL-terminated).
[in]	len	Length of rel, bytes.

Returns

true if stylesheet occurs in rel.

Return values

true	The substring "stylesheet" was found in rel.
false	The substring is absent or len is too short.

Precondition

rel addresses len readable bytes.

rel is non-null.

Postcondition

No state mutated.

Return value depends solely on the inputs.

Note

Pure.

Since

0.1.0

Case-sensitive substring scan (EPUB <link> rels are authored lowercase; this also accepts the alternate stylesheet form). Bounded by the value length. The inner loop iterates at most len times so the scan is O(len) with no dynamic allocation.

Parameters

[in]	rel	Attribute-value bytes (not NUL-terminated).
[in]	len	Length of rel, bytes.

Returns

true if stylesheet occurs in rel.

Return values

true	The substring "stylesheet" was found in rel.
false	The substring is absent or len is too short to contain it.

Precondition

rel addresses len readable bytes.

rel is non-null (a zero len is a valid empty-value case).

Postcondition

No state mutated.

Return value depends solely on the inputs.

Note

Pure.

Since

0.1.0

Definition at line 347 of file reflow_tokenize_attr.c.

Referenced by internal_handle_link().

priv_reflow_tok_resolve_face_slot()

uint8_t priv_reflow_tok_resolve_face_slot	(	const reflow_t *	engine,
		const ra8_css_style_t *	comp )

Map a cascaded run's (font-family + emphasis) to an embedded face slot.

Returns 0 (the engine's default bound face) when no embedded face matches, else 1 + the registry index. Defined in reflow_tokenize_attr.c.

Parameters

[in]	engine	Engine whose face registry + parsed sheet are consulted.
[in]	comp	The element's cascaded style (family + emphasis bits).

Returns

0 for the default face, else 1 + engine->faces[] index.

Return values

0	No registered face matches the cascaded family + emphasis combination.

Precondition

engine and comp are non-null.

engine->face_count <= k_reflow_max_faces.

Postcondition

No state is modified.

Return value is in [0, engine->face_count].

Note

Pure; reads engine state only.

Since

0.1.0

Returns 0 (the engine's default bound face) when no embedded face is registered, the run has no resolved font-family, no @font-face matches, or the matched @font-face was never registered (its src had no manifest font). Otherwise returns 1 + the registry index, which the render pass uses to pick the matching stbtt_fontinfo.

Parameters

[in]	engine	Engine whose face registry + parsed sheet are consulted.
[in]	comp	The element's cascaded style (family + emphasis bits).

Returns

0 for the default face, else 1 + engine->faces[] index.

Return values

0	No registered face matches the cascaded family + emphasis combination.

Precondition

engine and comp are non-null.

engine->face_count <= k_reflow_max_faces.

Postcondition

No state is modified.

Return value is in [0, engine->face_count].

Note

Pure; not thread-safe only insofar as it reads engine state.

Since

0.1.0

Definition at line 382 of file reflow_tokenize_attr.c.

References reflow_t::css, reflow_face_t::css_face_idx, reflow_t::face_count, reflow_t::faces, ra8_css_style_t::family_len, ra8_css_style_t::family_off, k_ra8_css_set_family, k_reflow_style_bold, k_reflow_style_italic, ra8_css_sheet_t::names, ra8_css_match_face(), ra8_css_style_t::set, and ra8_css_style_t::style.

Referenced by internal_open_styled().

priv_reflow_tok_skip_past()

size_t priv_reflow_tok_skip_past	(	const uint8_t *	buf,
		size_t	i,
		size_t	len,
		const char *	lit )

Return the index just past the first occurrence of lit.

Linear bounded scan; used to skip comments / PIs / declarations. Defined in reflow_tokenize_lex.c.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to search from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to find.

Returns

Index just past lit, or len if not found.

Return values

len	The literal does not occur in [i, len).

Precondition

buf and lit are non-null.

i <= len.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Linear bounded scan; used to skip comments / PIs / declarations.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to search from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to find.

Returns

Index just past lit, or len if not found.

Return values

len	The literal does not occur in [i, len).

Precondition

buf and lit are non-null.

i <= len.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Definition at line 591 of file reflow_tokenize_lex.c.

References internal_bytes_equal(), and strlen().

Referenced by internal_handle_lt(), and internal_handle_raw_text().

priv_reflow_tok_starts_with()

bool priv_reflow_tok_starts_with	(	const uint8_t *	buf,
		size_t	i,
		size_t	len,
		const char *	lit )

Test whether buf at i begins with the literal lit.

Bounds-checks that lit fits within [i, len) before comparing. Defined in reflow_tokenize_lex.c.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to test from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to match.

Returns

true if the literal fits and matches at i.

Return values

false

Out of range or mismatch.

Precondition

buf and lit are non-null.

i <= len.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Definition at line 564 of file reflow_tokenize_lex.c.

References internal_bytes_equal(), and strlen().

Referenced by internal_handle_lt(), and internal_tag_is().

priv_reflow_tok_stash_run()

bool priv_reflow_tok_stash_run	(	reflow_t *	engine,
		const uint8_t *	buf,
		size_t	start,
		size_t	end,
		uint32_t *	out_off,
		uint32_t *	out_len )

Entity-decode and whitespace-collapse a text run into the pool.

Walks the run one source position at a time, decoding a recognised entity to UTF-8 (else passing '&' through) and collapsing whitespace. Defined in reflow_tokenize_lex.c.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	buf	Source buffer.
[in]	start	Start offset of the run (inclusive).
[in]	end	End offset of the run (exclusive).
[out]	out_off	Text-pool offset of the stored run.
[out]	out_len	Number of bytes stored.

Returns

true on success, false on text-pool overflow.

Return values

false

The text pool is full.

Precondition

engine, buf, out_off, out_len are non-null.

start <= end.

Postcondition

On success *out_off / *out_len describe the stored slice.

On failure the pool may be partially written and is abandoned.

Note

Not thread-safe.

Since

0.1.0

Walks the run one source position at a time via internal_stash_one(), which decodes a recognised entity to UTF-8 (else passes '&' through) and collapses whitespace.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	buf	Source buffer.
[in]	start	Start offset of the run (inclusive).
[in]	end	End offset of the run (exclusive).
[out]	out_off	Text-pool offset of the stored run.
[out]	out_len	Number of bytes stored.

Returns

true on success, false on text-pool overflow.

Return values

false

The text pool is full.

Precondition

engine, buf, out_off, out_len are non-null.

start <= end.

Postcondition

On success *out_off / *out_len describe the stored slice.

On failure the pool may be partially written and is abandoned.

Note

Not thread-safe.

Since

0.1.0

Definition at line 514 of file reflow_tokenize_lex.c.

References internal_stash_one(), and reflow_t::text_pool_used.

Referenced by internal_emit_text_run().

priv_reflow_tok_style_for()

uint8_t priv_reflow_tok_style_for

(

reflow_html_tag_t

tag

)

Return the style bit an inline tag contributes to its subtree.

b/strong add bold, i/em add italic, a adds underline; every other tag contributes nothing. Defined in reflow_tokenize_lex.c.

Parameters

[in]

tag

Classified tag.

Returns

The reflow_style bit to OR in, or 0.

Return values

0	The tag carries no inline style.

Precondition

None.

None.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

b/strong add bold, i/em add italic, a adds underline; every other tag contributes nothing.

Parameters

[in]

tag

Classified tag.

Returns

The reflow_style bit to OR in, or 0.

Return values

0	The tag carries no inline style.

Precondition

None.

None.

Postcondition

No state is modified (pure).

No state is modified (pure).

Note

Pure function.

Since

0.1.0

Definition at line 316 of file reflow_tokenize_lex.c.

References k_reflow_style_bold, k_reflow_style_italic, k_reflow_style_underline, k_reflow_tag_a, k_reflow_tag_b, k_reflow_tag_em, k_reflow_tag_i, and k_reflow_tag_strong.

Referenced by internal_open_styled().

priv_reflow_tok_utf8_encode()

size_t priv_reflow_tok_utf8_encode	(	uint32_t	cp,
		uint8_t *	dst )

UTF-8 encode a code point into dst (up to 4 bytes).

Selects the 1/2/3/4-byte form by code-point range; values above U+10FFFF are clamped to the maximum valid code point.

Parameters

[in]	cp	Unicode code point (clamped to the valid range).
[out]	dst	Destination of at least 4 bytes.

Returns

Number of bytes written (1..4).

Return values

1	ASCII code point.

Precondition

dst is non-null with room for 4 bytes.

None.

Postcondition

Exactly the returned number of bytes in dst are written.

No other state is modified.

Note

Pure function.

Since

0.1.0

Definition at line 46 of file reflow_tokenize_lex.c.

References k_priv_uc_2byte, k_priv_uc_3byte, k_priv_uc_4byte, k_priv_uc_max, k_priv_utf8_cont, k_priv_utf8_lead2, k_priv_utf8_lead3, k_priv_utf8_lead4, k_priv_utf8_mask, k_priv_utf8_sh12, k_priv_utf8_sh18, and k_priv_utf8_sh6.

Referenced by internal_feed_utf8().

priv_reflow_xml_walk()

ra8_err_t priv_reflow_xml_walk	(	reflow_t *	engine,
		const uint8_t *	xhtml_buf,
		size_t	xhtml_len )

Tokenize the XHTML buffer and populate engine->tokens[].

The tokenizer entry point. A single forward pass over the XHTML source with no heap and no DOM. Defined in reflow_tokenize.c and consumed by the layout driver in reflow_parse.c; the per-decision helpers above are the individually-MC/DC-able primitives it drives.

Parameters

[in,out]	engine	Engine whose token / text pools are populated.
[in]	xhtml_buf	XHTML source bytes.
[in]	xhtml_len	Length of xhtml_buf, in bytes.

Returns

ra8_err_t Error code.

Return values

k_ra8_ok	Tokenization succeeded.
k_ra8_err_null_ptr	engine or xhtml_buf is null.
k_ra8_err_invalid_size	xhtml_len is zero.

Precondition

engine and xhtml_buf are non-null.

xhtml_len is non-zero.

Postcondition

On success engine->tokens[] is populated.

On failure the engine token / text pools are left unmodified.

Note

Not thread-safe; single-threaded, non-recursive.

Since

0.1.0

Definition at line 827 of file reflow_tokenize.c.

References tok_ctx_t::color, reflow_t::css, tok_ctx_t::engine, internal_emit_text_run(), internal_handle_lt(), k_ra8_err_invalid_size, k_ra8_err_no_mem, k_ra8_err_null_ptr, k_ra8_err_validation_failed, k_ra8_ok, k_reflow_color_inherit, k_reflow_style_normal, ra8_css_sheet_reset(), tok_ctx_t::saw_element, tok_ctx_t::sp, and tok_ctx_t::style.

Referenced by reflow_parse_xhtml().