The one UTF-8 <-> UTF-16LE seam between ra8_fs's API and its disks. More...

#include <stddef.h>
#include <stdint.h>
#include "ra8_attributes.h"
#include "ra8_err.h"
#include "ra8_fs_fat_internal.h"
#include "ra8_fs_utf_internal.h"

Include dependency graph for ra8_fs_utf.c:

Functions
static uint8_t	internal_utf8_lead (uint8_t b, uint32_t out_len, uint32_t out_cp)
	Classify a UTF-8 lead byte into a sequence length and its payload bits.
static uint8_t	internal_utf8_tail (const char in, uint32_t lead, uint32_t len, uint32_t io_cp)
	Fold a sequence's continuation bytes into the code point under assembly.
static uint8_t	internal_utf8_wellformed (uint32_t cp, uint32_t len)
	Is `cp` a code point that a `len` byte sequence may legally encode?
static ra8_err_t	internal_utf8_next (const char in, uint32_t io_pos, uint32_t *out_cp)
	Decode the sequence at *io_pos, advancing the cursor past it.
static ra8_err_t	internal_utf16_put (uint32_t cp, uint16_t out, uint32_t cap, uint32_t io_n)
	Append `cp` to a UTF-16 buffer as one unit or as a surrogate pair.
ra8_err_t	priv_utf8_to_utf16 (const char in, uint16_t out, uint32_t cap, uint32_t *out_units)
	Convert a NUL-terminated UTF-8 name into UTF-16LE code units.
static ra8_err_t	internal_utf16_take (const uint16_t in, uint32_t units, uint32_t io_i, uint32_t *out_cp)
	Take the character at *io_i, consuming a surrogate pair as one.
static uint32_t	internal_utf8_len_of (uint32_t cp)
	How many UTF-8 bytes does `cp` occupy?
static void	internal_utf8_put_tail (char *out, uint32_t from, uint32_t len, uint32_t cp)
	Write the continuation bytes of `cp` after its lead byte.
static ra8_err_t	internal_utf8_put (uint32_t cp, char out, uint32_t cap, uint32_t io_n)
	Append `cp` to a UTF-8 buffer, reserving room for the terminator.
ra8_err_t	priv_utf16_to_utf8 (const uint16_t in, uint32_t units, char out, uint32_t cap)
	Convert UTF-16LE code units into a NUL-terminated UTF-8 name.
uint8_t	priv_utf16_ieq (const uint16_t a, uint32_t an, const uint16_t b, uint32_t bn)
	Compare two UTF-16 names for case-insensitive equality.
uint8_t	priv_utf16_all_ascii (const uint16_t *in, uint32_t units)
	Is every unit of `in` inside the ASCII range?

Detailed Description

The one UTF-8 <-> UTF-16LE seam between ra8_fs's API and its disks.

Implements the contracts in ra8_fs_utf_internal.h. The decoder is split into three deliberately dull steps – classify the lead byte, fold in the continuation bytes, then judge the assembled code point – because that last step is the one every naive UTF-8 reader skips, and skipping it is what lets an over-long encoding or a raw surrogate through into a file name.

Nothing here substitutes a character for a byte it did not understand. A conversion either produces the caller's name or reports why it could not.

Copyright: Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Since: 0.1.0

Definition in file ra8_fs_utf.c.

Function Documentation

◆ internal_utf16_put()

ra8_err_t internal_utf16_put	(	uint32_t	cp,
		uint16_t *	out,
		uint32_t	cap,
		uint32_t *	io_n )

static

Append cp to a UTF-16 buffer as one unit or as a surrogate pair.

The capacity test covers the WHOLE character: a supplementary code point with one unit of room left is refused rather than half-written, so a truncated buffer never ends in a lone high surrogate.

Parameters

[in]	cp	Code point to append.
[out]	out	Destination unit buffer.
[in]	cap	Capacity of `out` in units.
[in,out]	io_n	Units already written; advanced by 1 or 2.

Returns: Error code.

Return values

k_ra8_ok	Appended.
k_ra8_err_no_mem	`cap` has no room for the whole character.

Precondition: out addresses cap writable units; io_n is non-NULL.; cp is a Unicode scalar value (never a surrogate).

Postcondition: On success *io_n grew by exactly the units the character needs.; On failure out and *io_n are unchanged.

Note: Pure apart from the outputs; trivially thread-safe.

Since: 0.1.0

Definition at line 259 of file ra8_fs_utf.c.

References k_ra8_err_no_mem, k_ra8_ok, k_utf_min_4byte, k_utf_sur_hi_first, k_utf_sur_lo_first, k_utf_sur_mask, and k_utf_sur_shift.

Referenced by priv_utf8_to_utf16().

◆ internal_utf16_take()

ra8_err_t internal_utf16_take	(	const uint16_t *	in,
		uint32_t	units,
		uint32_t *	io_i,
		uint32_t *	out_cp )

static

Take the character at *io_i, consuming a surrogate pair as one.

A high surrogate is only a character together with the low surrogate that follows it. Either half on its own is refused here rather than replaced, which is what keeps priv_utf16_to_utf8()'s promise that every name it returns can be handed straight back to priv_utf8_to_utf16().

Parameters

[in]	in	Code units.
[in]	units	Number of units in `in`.
[in,out]	io_i	Unit index to read at; advanced by 1 or 2.
[out]	out_cp	Receives the code point.

Returns: Error code.

Return values

k_ra8_ok	One character taken; cursor advanced.
k_ra8_err_invalid_arg	An unpaired surrogate sits at *io_i.

Precondition: All pointers are non-NULL and *io_i is below units.; in addresses at least units readable units.

Postcondition: On success *out_cp is a Unicode scalar value.; On failure *io_i and *out_cp are unspecified.

Note: Pure apart from the outputs; trivially thread-safe.

Since: 0.1.0

Definition at line 347 of file ra8_fs_utf.c.

References k_ra8_err_invalid_arg, k_ra8_ok, k_utf_min_4byte, k_utf_sur_hi_first, k_utf_sur_last, k_utf_sur_lo_first, and k_utf_sur_shift.

Referenced by priv_utf16_to_utf8().

◆ internal_utf8_lead()

uint8_t internal_utf8_lead	(	uint8_t	b,
		uint32_t *	out_len,
		uint32_t *	out_cp )

static

Classify a UTF-8 lead byte into a sequence length and its payload bits.

The four legal lead-byte shapes, tested longest-tag first so that a continuation byte (10xxxxxx) cannot be mistaken for anything: it matches none of them and is reported as illegal, which is what makes a sequence starting mid-character an error rather than a resync.

Parameters

[in]	b	Candidate lead byte.
[out]	out_len	Receives the total sequence length in bytes (1..4).
[out]	out_cp	Receives the lead byte's payload bits.

Returns: Legality flag.

Return values

1U	`b` is a legal lead byte; both outputs are written.
0U	`b` is a continuation byte or announces five or more bytes.

Precondition: out_len and out_cp are non-NULL.; b is the first byte of a candidate sequence.

Postcondition: On 1 the outputs describe b; on 0 they are untouched.; No state outside the outputs is modified.

Note: Pure function; trivially thread-safe.

Since: 0.1.0

Definition at line 72 of file ra8_fs_utf.c.

References k_utf_ascii_max, k_utf_lead2_mask, k_utf_lead2_payload, k_utf_lead2_tag, k_utf_lead3_mask, k_utf_lead3_payload, k_utf_lead3_tag, k_utf_lead4_mask, k_utf_lead4_payload, k_utf_lead4_tag, k_utf_len_1, k_utf_len_2, k_utf_len_3, and k_utf_len_4.

Referenced by internal_utf8_next().

◆ internal_utf8_len_of()

uint32_t internal_utf8_len_of ( uint32_t cp )

static

How many UTF-8 bytes does cp occupy?

The thresholds are the same ra8_fs_utf_t constants the decoder tests over-long forms against, which is the point of naming them: the encoder and the validator cannot drift into disagreeing about which length a code point belongs to.

Parameters

[in] cp Unicode scalar value.

Returns: Byte count.

Return values

1..4	The shortest UTF-8 form's length.

Precondition: cp is at most k_utf_code_max.; cp is not a surrogate.

Postcondition: No state is modified.; The result is the SHORTEST form's length, never a longer legal one.

Note: Pure function; trivially thread-safe.

Since: 0.1.0

Definition at line 396 of file ra8_fs_utf.c.

References k_utf_len_1, k_utf_len_2, k_utf_len_3, k_utf_len_4, k_utf_min_2byte, k_utf_min_3byte, and k_utf_min_4byte.

Referenced by internal_utf8_put().

◆ internal_utf8_next()

ra8_err_t internal_utf8_next	(	const char *	in,
		uint32_t *	io_pos,
		uint32_t *	out_cp )

static

Decode the sequence at *io_pos, advancing the cursor past it.

The three steps in order: classify, gather, judge. Every failure is the same answer to the caller – this is not UTF-8 – because a filesystem has nothing useful to do with the distinction and a caller that could tell them apart would be tempted to recover from one of them.

Parameters

[in]	in	NUL-terminated UTF-8 name.
[in,out]	io_pos	Byte index to decode at; advanced past the sequence.
[out]	out_cp	Receives the decoded code point.

Returns: Error code.

Return values

k_ra8_ok	One code point decoded; cursor advanced.
k_ra8_err_invalid_arg	The bytes at *io_pos are not well-formed.

Precondition: All pointers are non-NULL and in[*io_pos] is not the NUL terminator.; *io_pos indexes a lead byte, not the middle of a sequence.

Postcondition: On success *io_pos advanced by 1..4 and *out_cp is a scalar value.; On failure *io_pos and *out_cp are unspecified.

Note: Pure apart from the outputs; trivially thread-safe.

Since: 0.1.0

Definition at line 214 of file ra8_fs_utf.c.

References internal_utf8_lead(), internal_utf8_tail(), internal_utf8_wellformed(), k_ra8_err_invalid_arg, and k_ra8_ok.

Referenced by priv_utf8_to_utf16().

◆ internal_utf8_put()

ra8_err_t internal_utf8_put	(	uint32_t	cp,
		char *	out,
		uint32_t	cap,
		uint32_t *	io_n )

static

Append cp to a UTF-8 buffer, reserving room for the terminator.

The capacity test keeps one byte back for the NUL, so a caller never has to remember to. As with ::priv_utf16_put(), a character that does not fit whole is refused whole.

Parameters

[in]	cp	Unicode scalar value to append.
[out]	out	Destination byte buffer.
[in]	cap	Capacity of `out` in bytes, including the terminator.
[in,out]	io_n	Bytes already written; advanced by 1..4.

Returns: Error code.

Return values

k_ra8_ok	Appended.
k_ra8_err_no_mem	`cap` has no room for the character and a NUL.

Precondition: out addresses cap writable bytes; io_n is non-NULL.; cp is a Unicode scalar value (never a surrogate).

Postcondition: On success *io_n grew by ::priv_utf8_len_of(cp).; On failure out and *io_n are unchanged.

Note: Pure apart from the outputs; trivially thread-safe.

Since: 0.1.0

Definition at line 471 of file ra8_fs_utf.c.

References internal_utf8_len_of(), internal_utf8_put_tail(), k_ra8_err_no_mem, k_ra8_ok, k_utf_cont_shift, k_utf_lead2_tag, k_utf_lead3_tag, k_utf_lead4_tag, k_utf_len_1, k_utf_len_2, and k_utf_len_3.

Referenced by priv_utf16_to_utf8().

◆ internal_utf8_put_tail()

void internal_utf8_put_tail	(	char *	out,
		uint32_t	from,
		uint32_t	len,
		uint32_t	cp )

static

Write the continuation bytes of cp after its lead byte.

Emitted from the LAST byte backwards, because each one carries the low k_utf_cont_shift bits of what is left. Splitting this out of ::priv_utf8_put() is what keeps that function's four length cases from becoming four copies of the same shift loop.

Parameters

[out]	out	Destination byte buffer.
[in]	from	Index of the sequence's lead byte.
[in]	len	Total sequence length (2..4).
[in]	cp	Code point being written.

Returns: Nothing.

Precondition: out has at least from + len writable bytes.; len is ::priv_utf8_len_of(cp) and is at least 2.

Postcondition: Bytes from + 1 .. from + len - 1 are continuation bytes.; The lead byte at from is NOT written here.

Note: Pure apart from out; trivially thread-safe.

Since: 0.1.0

Definition at line 435 of file ra8_fs_utf.c.

References k_utf_cont_payload, k_utf_cont_shift, and k_utf_cont_tag.

Referenced by internal_utf8_put().

◆ internal_utf8_tail()

uint8_t internal_utf8_tail	(	const char *	in,
		uint32_t	lead,
		uint32_t	len,
		uint32_t *	io_cp )

static

Fold a sequence's continuation bytes into the code point under assembly.

Each continuation byte contributes k_utf_cont_shift payload bits. A NUL ends the string, so it fails the continuation test like any other non-continuation byte and a truncated sequence at the end of a name is rejected without reading past the terminator.

Parameters

[in]	in	NUL-terminated UTF-8 name.
[in]	lead	Index of the sequence's lead byte within `in`.
[in]	len	Total sequence length from ::priv_utf8_lead().
[in,out]	io_cp	Code point under assembly; extended in place.

Returns: Legality flag.

Return values

1U	All `len` - 1 continuation bytes were present and well-formed.
0U	A byte was missing or was not a continuation byte.

Precondition: in and io_cp are non-NULL; len is 1..4.; in[lead] is the lead byte len came from.

Postcondition: On 1 *io_cp holds every payload bit of the sequence.; On 0 *io_cp is unspecified and the caller must discard it.

Note: Pure apart from io_cp; trivially thread-safe.

Since: 0.1.0

Definition at line 125 of file ra8_fs_utf.c.

References k_utf_cont_mask, k_utf_cont_payload, k_utf_cont_shift, and k_utf_cont_tag.

Referenced by internal_utf8_next().

◆ internal_utf8_wellformed()

uint8_t internal_utf8_wellformed	(	uint32_t	cp,
		uint32_t	len )

static

Is cp a code point that a len byte sequence may legally encode?

Three refusals, all of which a decoder that only assembles bits will let through: an over-long form (the code point has a shorter encoding, so this byte string is a second spelling of it), a surrogate code point (UTF-8 does not encode them; UTF-16 uses them as machinery), and anything past U+10FFFF.

Parameters

[in]	cp	Assembled code point.
[in]	len	Sequence length it was assembled from.

Returns: Legality flag.

Return values

1U	`cp` is well-formed for `len`.
0U	Over-long, a surrogate, or out of range.

Precondition: len is 1..4 and came from ::priv_utf8_lead().; cp was assembled by ::priv_utf8_tail() from that sequence.

Postcondition: No state is modified.; The verdict depends only on the inputs.

Note: Pure function; trivially thread-safe.

Since: 0.1.0

Definition at line 163 of file ra8_fs_utf.c.

References k_utf_code_max, k_utf_len_2, k_utf_len_3, k_utf_len_4, k_utf_min_2byte, k_utf_min_3byte, k_utf_min_4byte, k_utf_sur_hi_first, and k_utf_sur_last.

Referenced by internal_utf8_next().

◆ priv_utf16_all_ascii()

uint8_t priv_utf16_all_ascii	(	const uint16_t *	in,
		uint32_t	units )

Is every unit of in inside the ASCII range?

The question a caller asks before it relies on the volume's up-case table: an ASCII-only name folds identically under every conforming table, so it is safe even when the volume carries one this build cannot reproduce, while a name with any other unit is not.

Parameters

[in]	in	Code units to inspect.
[in]	units	Number of units in `in`.

Returns: ASCII-only flag.

Return values

1U	Every unit is at most k_utf_ascii_max (or `units` is 0).
0U	At least one unit is above it.

Precondition: in addresses at least units readable units, or units is 0.; The caller treats a 0 result as "needs the volume's own table".

Postcondition: in is not modified.; The verdict depends only on the inputs.

Note: Pure function; trivially thread-safe.

Since: 0.1.0

Definition at line 543 of file ra8_fs_utf.c.

References k_utf_ascii_max.

Referenced by priv_exfat_name_to_units().

◆ priv_utf16_ieq()

uint8_t priv_utf16_ieq	(	const uint16_t *	a,
		uint32_t	an,
		const uint16_t *	b,
		uint32_t	bn )

Compare two UTF-16 names for case-insensitive equality.

Folds every unit through priv_exfat_upcase_unit() – the canonical up-case table – and compares unit by unit. Lengths must match first: the table is a simple one-to-one map, so folding never changes a name's length and a length difference is a difference.

Surrogate units fold to themselves, because the table covers the BMP and a supplementary code point has no simple case mapping inside it. Two supplementary characters therefore compare exactly, which is what a host does with the same table.

Parameters

[in]	a	First name's units.
[in]	an	Number of units in `a`.
[in]	b	Second name's units.
[in]	bn	Number of units in `b`.

Returns: Equality flag.

Return values

1U	The names are equal after folding.
0U	They differ in length or in at least one folded unit.

Precondition: a addresses an units and b addresses bn units.; Neither pointer is NULL unless its count is 0.

Postcondition: Neither input is modified.; The verdict depends only on the inputs and the fixed up-case table.

Note: Pure function; trivially thread-safe.

Since: 0.1.0

Definition at line 529 of file ra8_fs_utf.c.

References priv_exfat_upcase_unit().

Referenced by internal_dir_find_long_sector().

◆ priv_utf16_to_utf8()

ra8_err_t priv_utf16_to_utf8	(	const uint16_t *	in,
		uint32_t	units,
		char *	out,
		uint32_t	cap )

Convert UTF-16LE code units into a NUL-terminated UTF-8 name.

The inverse of priv_utf8_to_utf16(): a high surrogate followed by a low one becomes the four-byte form of the supplementary code point, every other unit becomes its one-, two- or three-byte form.

An unpaired surrogate – a high one not followed by a low one, or a low one on its own – is a name no UTF-8 string can express, so it is reported rather than substituted. The caller's fallback (the 8.3 alias on FAT) is a name that still opens the same file, which a replacement character would not be.

Parameters

[in]	in	Code units to convert.
[in]	units	Number of units in `in`.
[out]	out	Receives the NUL-terminated UTF-8 name.
[in]	cap	Capacity of `out` in bytes, including the terminator.

Returns: Error code.

Return values

k_ra8_ok	Converted; `out` is NUL-terminated.
k_ra8_err_null_ptr	`in` or `out` is NULL, or `cap` is 0.
k_ra8_err_invalid_arg	`in` holds an unpaired surrogate.
k_ra8_err_no_mem	The name plus its terminator exceeds `cap`.

Precondition: out addresses at least cap writable bytes.; in addresses at least units readable code units.

Postcondition: On success out holds well-formed UTF-8 and a NUL terminator.; On failure out[0] is NUL, so a caller that ignores the code sees an empty name rather than a partial one.

Note: Pure apart from out; trivially thread-safe against distinct buffers.

Since: 0.1.0

Definition at line 499 of file ra8_fs_utf.c.

References internal_utf16_take(), internal_utf8_put(), k_ra8_err_null_ptr, and k_ra8_ok.

Referenced by internal_exfat_gather_name(), internal_fat_dir_scan_sector(), and internal_listdir_visit_sector().

◆ priv_utf8_to_utf16()

ra8_err_t priv_utf8_to_utf16	(	const char *	in,
		uint16_t *	out,
		uint32_t	cap,
		uint32_t *	out_units )

Convert a NUL-terminated UTF-8 name into UTF-16LE code units.

Decodes each UTF-8 sequence to a code point, rejecting every malformed and non-shortest form (see the file header), then emits it as one BMP unit or as a high/low surrogate pair. The output is NOT NUL-terminated: a UTF-16 name on either format carries an explicit length, and appending a terminator here would invite a caller to use it as one.

The whole conversion is transactional in the sense that matters: on any failure out_units is set to zero, so a caller that ignores the return code writes an empty name rather than a truncated one.

Parameters

[in]	in	NUL-terminated UTF-8 name.
[out]	out	Receives the code units.
[in]	cap	Capacity of `out` in UTF-16 units.
[out]	out_units	Receives the number of units written.

Returns: Error code.

Return values

k_ra8_ok	Converted; `out` holds *out_units units.
k_ra8_err_null_ptr	`in`, `out` or `out_units` is NULL.
k_ra8_err_invalid_arg	`in` is not well-formed UTF-8.
k_ra8_err_no_mem	The name needs more than `cap` units.

Precondition: out addresses at least cap writable units.; in is NUL-terminated within the caller's buffer.

Postcondition: On success *out_units is at most cap.; On failure *out_units is 0 and out holds nothing meaningful.

Note: Pure apart from out; trivially thread-safe against distinct buffers.

Since: 0.1.0

Definition at line 283 of file ra8_fs_utf.c.

References internal_utf16_put(), internal_utf8_next(), k_ra8_err_no_mem, k_ra8_err_null_ptr, and k_ra8_ok.

Referenced by priv_dir_find_long(), priv_exfat_name_to_units(), and priv_name_classify().

Functions

Detailed Description

Function Documentation

◆ internal_utf16_put()

◆ internal_utf16_take()

◆ internal_utf8_lead()

◆ internal_utf8_len_of()

◆ internal_utf8_next()

◆ internal_utf8_put()

◆ internal_utf8_put_tail()

◆ internal_utf8_tail()

◆ internal_utf8_wellformed()

◆ priv_utf16_all_ascii()

◆ priv_utf16_ieq()

◆ priv_utf16_to_utf8()

◆ priv_utf8_to_utf16()