Lexical primitives for the no-heap streaming XHTML tokenizer: whitespace/entity/UTF-8 scanning, tag classification, and the text-pool / token-pool emit helpers. More...

#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include "ra8_attributes.h"
#include "ra8_err.h"
#include "reflow.h"
#include "reflow_tokenize_internal.h"

Include dependency graph for reflow_tokenize_lex.c:

Functions
bool	priv_reflow_tok_is_xml_whitespace (char c)
	True for the ASCII characters XHTML treats as whitespace.
size_t	priv_reflow_tok_utf8_encode (uint32_t cp, uint8_t *dst)
	UTF-8 encode a code point into dst (up to 4 bytes).
static size_t	internal_local_lower (const char name, size_t len, char dst)
	Lower-case ASCII and copy the local part of a tag name.
reflow_html_tag_t	priv_reflow_tok_classify (const char *name, size_t len)
	Map a (possibly namespace-prefixed) tag name to its enum.
static bool	internal_decode_numeric (const char src, size_t avail, uint32_t out_cp, size_t *out_used)
	Decode a &#dec; / &#xhex; numeric character reference.
bool	priv_reflow_tok_decode_entity (const char src, size_t avail, uint32_t out_cp, size_t *out_used)
	Decode one XML entity reference beginning at &.
bool	priv_reflow_tok_is_block (reflow_html_tag_t tag)
	Test whether a tag is a block-flow container in the v1 subset.
uint8_t	priv_reflow_tok_style_for (reflow_html_tag_t tag)
	Return the style bit an inline tag contributes to its subtree.
bool	priv_reflow_tok_emit (reflow_t *engine, reflow_token_kind_t kind, reflow_html_tag_t tag, uint8_t style, uint32_t off, uint32_t len)
	Append one token to the engine's token pool.
bool	priv_reflow_tok_feed (reflow_t engine, char ch, bool last_ws)
	Write one byte through the whitespace-collapse state machine.
static bool	internal_feed_utf8 (reflow_t engine, uint32_t cp, bool last_ws)
	Feed a code point's UTF-8 bytes through the whitespace-collapse.
static bool	internal_stash_one (reflow_t engine, const uint8_t buf, size_t i, size_t end, bool last_ws, size_t consumed)
	Stash one source position of a text run (entity or literal byte).
bool	priv_reflow_tok_stash_run (reflow_t engine, const uint8_t buf, size_t start, size_t end, uint32_t out_off, uint32_t out_len)
	Entity-decode and whitespace-collapse a text run into the pool.
static bool	internal_bytes_equal (const uint8_t buf, const char lit, size_t len)
	Test whether buf equals the first len bytes of lit.
bool	priv_reflow_tok_starts_with (const uint8_t buf, size_t i, size_t len, const char lit)
	Test whether buf at i begins with the literal lit.
size_t	priv_reflow_tok_skip_past (const uint8_t buf, size_t i, size_t len, const char lit)
	Return the index just past the first occurrence of lit.
size_t	priv_reflow_tok_find_lit (const uint8_t buf, size_t i, size_t len, const char lit)
	Return the index of the first occurrence of lit (its start), or len.

Detailed Description

Lexical primitives for the no-heap streaming XHTML tokenizer: whitespace/entity/UTF-8 scanning, tag classification, and the text-pool / token-pool emit helpers.

Companion translation unit to reflow_tokenize.c. It collects the leaf scanning routines that do not touch the tokenizer's mutable element-stack context (tok_ctx_t): XHTML-whitespace classification, UTF-8 encoding, named/numeric entity decoding, tag-name classification, the whitespace- collapse text-pool feed, the entity-decoding run stash, the token-pool emit, and the literal-search markup scanners. The markup handlers and the cascade engine live in reflow_tokenize.c and call the cross-TU helpers promoted into reflow_tokenize_internal.h. Pure forward-pass scanning – no recursion (NASA P10 Rule 1) and bounded loops (Rule 2).

[Ring 4 / Reflow] {World: NS}

Copyright: Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Since: 0.1.0

Definition in file reflow_tokenize_lex.c.

Function Documentation

◆ internal_bytes_equal()

bool internal_bytes_equal	(	const uint8_t *	buf,
		const char *	lit,
		size_t	len )

static

Test whether buf equals the first len bytes of lit.

Compares exactly the caller-bounded extent without reading beyond either extent established by the caller.

Parameters

[in]	buf	Source buffer.
[in]	lit	NUL-terminated literal to match.
[in]	len	Number of bytes to compare.

Returns: Whether all len bytes match.

Return values

true	Every compared byte matches.
false	At least one compared byte differs.

Precondition: buf and lit are non-null.; Both inputs address at least len readable bytes.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function.

Since: 0.1.0

Definition at line 555 of file reflow_tokenize_lex.c.

Referenced by priv_reflow_tok_find_lit(), priv_reflow_tok_skip_past(), and priv_reflow_tok_starts_with().

◆ internal_decode_numeric()

bool internal_decode_numeric	(	const char *	src,
		size_t	avail,
		uint32_t *	out_cp,
		size_t *	out_used )

static

Decode a &#dec; / &#xhex; numeric character reference.

Assumes src begins with "&#". Reads an optional x/X for hexadecimal, then base-appropriate digits up to a terminating ';'.

Parameters

[in]	src	Buffer positioned at the '&' of "&#...".
[in]	avail	Bytes available from src.
[out]	out_cp	Decoded code point on success.
[out]	out_used	Bytes consumed (through ';') on success.

Returns: true if a complete numeric reference was parsed.

Return values

false No digits, bad digit, or missing terminator.

Precondition: src, out_cp, out_used are non-null.; src[0..1] are "&#".

Postcondition: On false the output params are unspecified.; On true *out_used is the index just past ';'.

Note: Pure aside from writing the output params.

Since: 0.1.0

Definition at line 184 of file reflow_tokenize_lex.c.

References k_priv_base_dec, k_priv_base_hex, and k_priv_hex_offset.

Referenced by priv_reflow_tok_decode_entity().

◆ internal_feed_utf8()

bool internal_feed_utf8	(	reflow_t *	engine,
		uint32_t	cp,
		bool *	last_ws )

static

Feed a code point's UTF-8 bytes through the whitespace-collapse.

Encodes cp to 1-4 bytes and feeds each via priv_reflow_tok_feed().

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	cp	Code point to encode.
[in,out]	last_ws	Whitespace-collapse state.

Returns: true on success, false on text-pool overflow.

Return values

false The text pool is full.

Precondition: engine and last_ws are non-null.; cp is a Unicode code point (clamped by the encoder).

Postcondition: On success 1-4 bytes are appended to the pool.; On failure the pool may be partially written.

Note: Not thread-safe.

Since: 0.1.0

Definition at line 438 of file reflow_tokenize_lex.c.

References priv_reflow_tok_feed(), and priv_reflow_tok_utf8_encode().

Referenced by internal_stash_one().

◆ internal_local_lower()

size_t internal_local_lower	(	const char *	name,
		size_t	len,
		char *	dst )

static

Lower-case ASCII and copy the local part of a tag name.

Scans for the last ':' to drop any namespace prefix, then copies the remaining local name, folding A-Z to a-z, truncating at the buffer capacity. Always NUL-terminates.

Parameters

[in]	name	Source name (may carry a prefix: namespace).
[in]	len	Bytes in name.
[out]	dst	NUL-terminated lower-cased local name buffer.

Returns: Length written to dst (excluding the NUL).

Return values

0	Empty local name.

Precondition: name and dst are non-null.; dst has room for k_priv_tag_name_cap bytes.

Postcondition: dst is NUL-terminated.; No input bytes are modified.

Note: Pure aside from writing dst.

Since: 0.1.0

Definition at line 97 of file reflow_tokenize_lex.c.

References k_priv_tag_name_cap.

Referenced by priv_reflow_tok_classify().

◆ internal_stash_one()

bool internal_stash_one	(	reflow_t *	engine,
		const uint8_t *	buf,
		size_t	i,
		size_t	end,
		bool *	last_ws,
		size_t *	consumed )

static

Stash one source position of a text run (entity or literal byte).

If buf[i] starts a recognised entity it is decoded and emitted as UTF-8; otherwise the single byte (including an unrecognised '&') is fed through the whitespace-collapse.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	buf	Source buffer.
[in]	i	Current offset within the run.
[in]	end	End offset of the run (exclusive).
[in,out]	last_ws	Whitespace-collapse state.
[out]	consumed	Source bytes consumed (1, or the entity length).

Returns: true on success, false on text-pool overflow.

Return values

false The text pool is full.

Precondition: engine, buf, last_ws, consumed are non-null.; i < end.

Postcondition: On success *consumed is in [1, end - i].; On failure the pool may be partially written.

Note: Not thread-safe.

Since: 0.1.0

Definition at line 473 of file reflow_tokenize_lex.c.

References internal_feed_utf8(), priv_reflow_tok_decode_entity(), and priv_reflow_tok_feed().

Referenced by priv_reflow_tok_stash_run().

◆ priv_reflow_tok_classify()

reflow_html_tag_t priv_reflow_tok_classify	(	const char *	name,
		size_t	len )

Map a (possibly namespace-prefixed) tag name to its enum.

Strips any prefix: namespace, lower-cases the local name, and matches it against the v1 subset's tag table.

Parameters

[in]	name	NUL-terminated local-or-prefixed tag name.
[in]	len	Length of name in bytes (excluding any NUL).

Returns: The matching reflow_html_tag_t, or k_reflow_tag_unknown.

Return values

k_reflow_tag_unknown Name is null or unrecognised.

Precondition: None.; None.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function; case-insensitive on ASCII.

Since: 0.1.0

< Word.

< Tag.

Definition at line 118 of file reflow_tokenize_lex.c.

References internal_local_lower(), k_priv_tag_name_cap, k_reflow_tag_a, k_reflow_tag_b, k_reflow_tag_blockquote, k_reflow_tag_br, k_reflow_tag_em, k_reflow_tag_h1, k_reflow_tag_h2, k_reflow_tag_h3, k_reflow_tag_h4, k_reflow_tag_h5, k_reflow_tag_h6, k_reflow_tag_hr, k_reflow_tag_i, k_reflow_tag_img, k_reflow_tag_li, k_reflow_tag_link, k_reflow_tag_ol, k_reflow_tag_p, k_reflow_tag_strong, k_reflow_tag_table, k_reflow_tag_td, k_reflow_tag_th, k_reflow_tag_tr, k_reflow_tag_ul, k_reflow_tag_unknown, and strcmp().

Referenced by internal_parse_sel_type(), and internal_parse_start().

◆ priv_reflow_tok_decode_entity()

bool priv_reflow_tok_decode_entity	(	const char *	src,
		size_t	avail,
		uint32_t *	out_cp,
		size_t *	out_used )

Decode one XML entity reference beginning at &.

Recognises the named entities amp/lt/gt/quot/apos and numeric &#dec; / &#xhex; references. Unrecognised sequences are reported as "not an entity" so the caller emits the literal &.

Parameters

[in]	src	Buffer positioned so src[0] == '&'.
[in]	avail	Bytes available from src (>= 1).
[out]	out_cp	Decoded Unicode code point on success.
[out]	out_used	Bytes consumed from src on success.

Returns: true if a complete recognised entity was decoded.

Return values

false Not a recognised/complete entity (caller emits '&').

Precondition: src, out_cp, out_used are non-null and avail >= 1.; src[0] == '&'.

Postcondition: On false, out params are unspecified and no input is consumed.; On true, *out_used is in [3, avail].

Note: Pure function.

Since: 0.1.0

< Word.

< Cp.

Definition at line 220 of file reflow_tokenize_lex.c.

References internal_decode_numeric(), k_priv_entity_min, k_priv_entity_window, strlen(), and strncmp().

Referenced by internal_stash_one().

◆ priv_reflow_tok_emit()

bool priv_reflow_tok_emit	(	reflow_t *	engine,
		reflow_token_kind_t	kind,
		reflow_html_tag_t	tag,
		uint8_t	style,
		uint32_t	off,
		uint32_t	len )

Append one token to the engine's token pool.

Fills the next reflow_token_t and advances token_count, or fails if the fixed pool is full.

Parameters

[in,out]	engine	Engine whose token pool is appended to.
[in]	kind	Token kind.
[in]	tag	Associated tag (or k_reflow_tag_unknown).
[in]	style	Inline-style bits for the token.
[in]	off	Text-pool offset (text tokens only).
[in]	len	Text-pool length (text tokens only).

Returns: true on success, false on pool overflow.

Return values

false The token pool is full.

Precondition: engine is non-null.; engine->token_count <= k_reflow_max_tokens.

Postcondition: On success token_count is incremented by one.; On failure the pool is unchanged.

Note: Not thread-safe.

Since: 0.1.0

Definition at line 358 of file reflow_tokenize_lex.c.

References reflow_token_t::color, reflow_token_t::css_font_px, k_reflow_color_inherit, k_reflow_max_tokens, reflow_token_t::kind, reflow_token_t::reserved, reflow_token_t::reserved16, reflow_token_t::style, reflow_token_t::tag, reflow_token_t::text_len, reflow_token_t::text_off, reflow_t::token_count, and reflow_t::tokens.

Referenced by internal_emit_text_run(), internal_handle_cdata(), internal_handle_end(), internal_handle_start(), internal_handle_void(), and internal_open_attrs().

◆ priv_reflow_tok_feed()

bool priv_reflow_tok_feed	(	reflow_t *	engine,
		char	ch,
		bool *	last_ws )

Write one byte through the whitespace-collapse state machine.

Runs of ASCII whitespace collapse to a single space; a run that begins with *last_ws true (start of node) drops its leading whitespace.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	ch	Byte to write.
[in,out]	last_ws	Collapse state (true if the prior byte was space).

Returns: true on success, false on text-pool overflow.

Return values

false The text pool is full.

Precondition: engine and last_ws are non-null.; engine->text_pool_used <= k_reflow_text_pool_bytes.

Postcondition: On a written byte text_pool_used is incremented.; Collapsed (skipped) whitespace leaves the pool unchanged.

Note: Not thread-safe.

Since: 0.1.0

Definition at line 400 of file reflow_tokenize_lex.c.

References k_reflow_text_pool_bytes, priv_reflow_tok_is_xml_whitespace(), reflow_t::text_pool, and reflow_t::text_pool_used.

Referenced by internal_feed_utf8(), internal_handle_cdata(), and internal_stash_one().

◆ priv_reflow_tok_find_lit()

size_t priv_reflow_tok_find_lit	(	const uint8_t *	buf,
		size_t	i,
		size_t	len,
		const char *	lit )

Return the index of the first occurrence of lit (its start), or len.

Like priv_reflow_tok_skip_past but returns the literal's START offset, so the caller can both bound the preceding content and resume past the literal.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to search from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to find.

Returns: Start index of lit, or len if absent.

Return values

len	The literal does not occur in [i, len).

Precondition: buf and lit are non-null.; i <= len.

Postcondition: No state is modified (pure).; Return value is in [i, len].

Note: Pure function.

Since: 0.1.0

Definition at line 623 of file reflow_tokenize_lex.c.

References internal_bytes_equal(), and strlen().

Referenced by internal_handle_raw_text().

◆ priv_reflow_tok_is_block()

bool priv_reflow_tok_is_block ( reflow_html_tag_t tag )

Test whether a tag is a block-flow container in the v1 subset.

Block tags emit block-start / block-end tokens; all others do not introduce a block box.

Parameters

[in] tag Classified tag.

Returns: true for p / h1-h6 / ul / ol / li / blockquote.

Return values

false Any inline, void, or unknown tag.

Precondition: None.; None.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function.

Since: 0.1.0

Definition at line 276 of file reflow_tokenize_lex.c.

References k_reflow_tag_blockquote, k_reflow_tag_h1, k_reflow_tag_h2, k_reflow_tag_h3, k_reflow_tag_h4, k_reflow_tag_h5, k_reflow_tag_h6, k_reflow_tag_li, k_reflow_tag_ol, k_reflow_tag_p, k_reflow_tag_table, k_reflow_tag_td, k_reflow_tag_th, k_reflow_tag_tr, and k_reflow_tag_ul.

Referenced by internal_handle_end(), and internal_handle_start().

◆ priv_reflow_tok_is_xml_whitespace()

bool priv_reflow_tok_is_xml_whitespace ( char c )

True for the ASCII characters XHTML treats as whitespace.

Matches space, tab, newline, carriage-return, form-feed and vertical-tab – the set collapsed by the tokenizer's text handling.

Parameters

[in] c Byte to classify.

Returns: true if c is space/tab/newline/carriage-return/form-feed/vtab.

Return values

false c is any non-whitespace byte.

Precondition: None.; None.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function.

Since: 0.1.0

Definition at line 41 of file reflow_tokenize_lex.c.

Referenced by internal_attr_quoted_value(), internal_parse_start(), internal_tag_is(), and priv_reflow_tok_feed().

◆ priv_reflow_tok_skip_past()

size_t priv_reflow_tok_skip_past	(	const uint8_t *	buf,
		size_t	i,
		size_t	len,
		const char *	lit )

Return the index just past the first occurrence of lit.

Linear bounded scan; used to skip comments / PIs / declarations.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to search from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to find.

Returns: Index just past lit, or len if not found.

Return values

len	The literal does not occur in [i, len).

Precondition: buf and lit are non-null.; i <= len.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function.

Since: 0.1.0

Definition at line 591 of file reflow_tokenize_lex.c.

References internal_bytes_equal(), and strlen().

Referenced by internal_handle_lt(), and internal_handle_raw_text().

◆ priv_reflow_tok_starts_with()

bool priv_reflow_tok_starts_with	(	const uint8_t *	buf,
		size_t	i,
		size_t	len,
		const char *	lit )

Test whether buf at i begins with the literal lit.

Bounds-checks that lit fits within [i, len) before comparing. Defined in reflow_tokenize_lex.c.

Parameters

[in]	buf	Source buffer.
[in]	i	Offset to test from.
[in]	len	Total buffer length.
[in]	lit	NUL-terminated literal to match.

Returns: true if the literal fits and matches at i.

Return values

false Out of range or mismatch.

Precondition: buf and lit are non-null.; i <= len.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function.

Since: 0.1.0

Definition at line 564 of file reflow_tokenize_lex.c.

References internal_bytes_equal(), and strlen().

Referenced by internal_handle_lt(), and internal_tag_is().

◆ priv_reflow_tok_stash_run()

bool priv_reflow_tok_stash_run	(	reflow_t *	engine,
		const uint8_t *	buf,
		size_t	start,
		size_t	end,
		uint32_t *	out_off,
		uint32_t *	out_len )

Entity-decode and whitespace-collapse a text run into the pool.

Walks the run one source position at a time via internal_stash_one(), which decodes a recognised entity to UTF-8 (else passes '&' through) and collapses whitespace.

Parameters

[in,out]	engine	Engine whose text pool is appended to.
[in]	buf	Source buffer.
[in]	start	Start offset of the run (inclusive).
[in]	end	End offset of the run (exclusive).
[out]	out_off	Text-pool offset of the stored run.
[out]	out_len	Number of bytes stored.

Returns: true on success, false on text-pool overflow.

Return values

false The text pool is full.

Precondition: engine, buf, out_off, out_len are non-null.; start <= end.

Postcondition: On success *out_off / *out_len describe the stored slice.; On failure the pool may be partially written and is abandoned.

Note: Not thread-safe.

Since: 0.1.0

Definition at line 514 of file reflow_tokenize_lex.c.

References internal_stash_one(), and reflow_t::text_pool_used.

Referenced by internal_emit_text_run().

◆ priv_reflow_tok_style_for()

uint8_t priv_reflow_tok_style_for ( reflow_html_tag_t tag )

Return the style bit an inline tag contributes to its subtree.

b/strong add bold, i/em add italic, a adds underline; every other tag contributes nothing.

Parameters

[in] tag Classified tag.

Returns: The reflow_style bit to OR in, or 0.

Return values

0	The tag carries no inline style.

Precondition: None.; None.

Postcondition: No state is modified (pure).; No state is modified (pure).

Note: Pure function.

Since: 0.1.0

Definition at line 316 of file reflow_tokenize_lex.c.

References k_reflow_style_bold, k_reflow_style_italic, k_reflow_style_underline, k_reflow_tag_a, k_reflow_tag_b, k_reflow_tag_em, k_reflow_tag_i, and k_reflow_tag_strong.

Referenced by internal_open_styled().

◆ priv_reflow_tok_utf8_encode()

size_t priv_reflow_tok_utf8_encode	(	uint32_t	cp,
		uint8_t *	dst )

UTF-8 encode a code point into dst (up to 4 bytes).

Selects the 1/2/3/4-byte form by code-point range; values above U+10FFFF are clamped to the maximum valid code point.

Parameters

[in]	cp	Unicode code point (clamped to the valid range).
[out]	dst	Destination of at least 4 bytes.

Returns: Number of bytes written (1..4).

Return values

1	ASCII code point.

Precondition: dst is non-null with room for 4 bytes.; None.

Postcondition: Exactly the returned number of bytes in dst are written.; No other state is modified.

Note: Pure function.

Since: 0.1.0

Definition at line 46 of file reflow_tokenize_lex.c.

References k_priv_uc_2byte, k_priv_uc_3byte, k_priv_uc_4byte, k_priv_uc_max, k_priv_utf8_cont, k_priv_utf8_lead2, k_priv_utf8_lead3, k_priv_utf8_lead4, k_priv_utf8_mask, k_priv_utf8_sh12, k_priv_utf8_sh18, and k_priv_utf8_sh6.

Referenced by internal_feed_utf8().

Functions

Detailed Description

Function Documentation

◆ internal_bytes_equal()

◆ internal_decode_numeric()

◆ internal_feed_utf8()

◆ internal_local_lower()

◆ internal_stash_one()

◆ priv_reflow_tok_classify()

◆ priv_reflow_tok_decode_entity()

◆ priv_reflow_tok_emit()

◆ priv_reflow_tok_feed()

◆ priv_reflow_tok_find_lit()

◆ priv_reflow_tok_is_block()

◆ priv_reflow_tok_is_xml_whitespace()

◆ priv_reflow_tok_skip_past()

◆ priv_reflow_tok_starts_with()

◆ priv_reflow_tok_stash_run()

◆ priv_reflow_tok_style_for()

◆ priv_reflow_tok_utf8_encode()