ra8_fs_utf_internal.h File Reference

UTF-8 <-> UTF-16LE conversion and case folding for ra8_fs names. More...

#include <stdint.h>
#include "ra8_attributes.h"
#include "ra8_err.h"

Include dependency graph for ra8_fs_utf_internal.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Enumerations
enum	ra8_fs_utf_t : uint32_t {   k_utf_ascii_max = 0x7FU ,   k_utf_cont_mask = 0xC0U ,   k_utf_cont_tag = 0x80U ,   k_utf_cont_payload = 0x3FU ,   k_utf_cont_shift = 6U ,   k_utf_lead2_mask = 0xE0U ,   k_utf_lead2_tag = 0xC0U ,   k_utf_lead2_payload = 0x1FU ,   k_utf_lead3_mask = 0xF0U ,   k_utf_lead3_tag = 0xE0U ,   k_utf_lead3_payload = 0x0FU ,   k_utf_lead4_mask = 0xF8U ,   k_utf_lead4_tag = 0xF0U ,   k_utf_lead4_payload = 0x07U ,   k_utf_len_1 = 1U ,   k_utf_len_2 = 2U ,   k_utf_len_3 = 3U ,   k_utf_len_4 = 4U ,   k_utf_min_2byte = 0x80U ,   k_utf_min_3byte = 0x800U ,   k_utf_min_4byte = 0x10000U ,   k_utf_sur_hi_first = 0xD800U ,   k_utf_sur_lo_first = 0xDC00U ,   k_utf_sur_last = 0xDFFFU ,   k_utf_sur_shift = 10U ,   k_utf_sur_mask = 0x3FFU ,   k_utf_unit_max = 0xFFFFU ,   k_utf_code_max = 0x10FFFFU ,   k_utf_byte_mask = 0xFFU ,   k_utf_byte_shift = 8U ,   k_utf8_max_per_unit = 3U }
	Bit patterns, thresholds and shifts of the UTF-8 / UTF-16 encodings. More...

Functions
ra8_err_t	priv_utf8_to_utf16 (const char *in, uint16_t *out, uint32_t cap, uint32_t *out_units)
	Convert a NUL-terminated UTF-8 name into UTF-16LE code units.
ra8_err_t	priv_utf16_to_utf8 (const uint16_t *in, uint32_t units, char *out, uint32_t cap)
	Convert UTF-16LE code units into a NUL-terminated UTF-8 name.
uint8_t	priv_utf16_ieq (const uint16_t *a, uint32_t an, const uint16_t *b, uint32_t bn)
	Compare two UTF-16 names for case-insensitive equality.
uint8_t	priv_utf16_all_ascii (const uint16_t *in, uint32_t units)
	Is every unit of in inside the ASCII range?

Detailed Description

UTF-8 <-> UTF-16LE conversion and case folding for ra8_fs names.

Both on-disk name formats this adapter speaks store UTF-16LE: a VFAT long-name chain carries thirteen units per slot, and an exFAT file-name entry carries fifteen. The public API is UTF-8 char* and stays that way, so exactly one seam has to exist between the two – this one. It is a seam and not a scattering of casts on purpose: the three separate places that used to do their own byte-to-unit arithmetic each got it wrong in a different direction (#606), and a single conversion cannot disagree with itself.

What is representable

The whole of Unicode that UTF-16 can express: the Basic Multilingual Plane in one unit, and the supplementary planes as a surrogate PAIR. A four-byte UTF-8 sequence therefore costs two UTF-16 units on disk and comes back as the same four bytes – NameLength on exFAT and the VFAT group count are unit counts, which is what the formats mean by "length" and what a host counts.

What is refused, loudly

Everything a decoder may not silently accept, because accepting it is how a name stops round-tripping:

• an over-long encoding (C0 80 for NUL, E0 80 80, ...), which is a different byte string for a code point that already has one;
• a surrogate code point encoded directly in UTF-8 (ED A0 80), which is CESU-8 / WTF-8 and not UTF-8;
• a truncated sequence, a continuation byte where a lead byte belongs, or a lead byte announcing five or more bytes;
• anything above U+10FFFF.

Each yields k_ra8_err_invalid_arg from the conversion, which every caller turns into a failed create / open rather than a mangled name. There is no substitution character anywhere in this module: ? in place of a code point is exactly the defect this file exists to remove.

The one asymmetry is deliberate. priv_utf16_to_utf8() can be handed an UNPAIRED surrogate, because that is a sequence of units that already exists on someone else's volume, and no UTF-8 string encodes it. It reports k_ra8_err_invalid_arg and the caller falls back to the entry's 8.3 alias, so the invariant "a name this library reports is a name it can re-open" holds even for a volume written by something that did not check.

Case folding

priv_utf16_ieq() folds through priv_exfat_upcase_unit(), the canonical Microsoft up-case table this tree already embeds and writes at format time. That is the table the exFAT specification's name hash is defined against, so using it for FAT long names too means one fold serves both formats – and Resume.txt matching RESUME.TXT is the same mechanism as its accented sibling matching, rather than a second rule that only covers ASCII.

Copyright

Copyright (c) 2026 Brighton Sikarskie SPDX-License-Identifier: MIT

Since

0.1.0

Definition in file ra8_fs_utf_internal.h.

Enumeration Type Documentation

ra8_fs_utf_t

enum ra8_fs_utf_t : uint32_t

Bit patterns, thresholds and shifts of the UTF-8 / UTF-16 encodings.

Named rather than inlined because every one of them appears in more than one direction of the codec: the mask that recognises a three-byte lead byte is the same mask that builds one. Unicode 15.0 Table 3-6 ("UTF-8 Bit Distribution") and Table 3-5 ("UTF-16 Bit Distribution") are the source for the layouts.

Invariant

k_utf_min_2byte, k_utf_min_3byte and k_utf_min_4byte are the SMALLEST code point each sequence length may encode, so a value below its length's minimum is over-long by definition.

k_utf8_max_per_unit is 3: a BMP code point is one UTF-16 unit and at most three UTF-8 bytes, and a supplementary one is two units and four bytes, which is less per unit.

See also

priv_utf8_to_utf16()

Since

0.1.0

Enumerator
k_utf_ascii_max	Highest code point that is one UTF-8 byte.
k_utf_cont_mask	Mask isolating a continuation byte's tag.
k_utf_cont_tag	Tag every continuation byte carries.
k_utf_cont_payload	Payload bits in a continuation byte.
k_utf_cont_shift	Payload width of one continuation byte.
k_utf_lead2_mask	Mask isolating a two-byte lead's tag.
k_utf_lead2_tag	Tag of a two-byte lead byte.
k_utf_lead2_payload	Payload bits in a two-byte lead.
k_utf_lead3_mask	Mask isolating a three-byte lead's tag.
k_utf_lead3_tag	Tag of a three-byte lead byte.
k_utf_lead3_payload	Payload bits in a three-byte lead.
k_utf_lead4_mask	Mask isolating a four-byte lead's tag.
k_utf_lead4_tag	Tag of a four-byte lead byte.
k_utf_lead4_payload	Payload bits in a four-byte lead.
k_utf_len_1	Sequence length: plain ASCII.
k_utf_len_2	Sequence length: two bytes.
k_utf_len_3	Sequence length: three bytes.
k_utf_len_4	Sequence length: four bytes.
k_utf_min_2byte	Smallest code point a 2-byte form may hold.
k_utf_min_3byte	Smallest code point a 3-byte form may hold.
k_utf_min_4byte	Smallest code point a 4-byte form may hold.
k_utf_sur_hi_first	First high (leading) surrogate unit.
k_utf_sur_lo_first	First low (trailing) surrogate unit.
k_utf_sur_last	Last surrogate unit of either half.
k_utf_sur_shift	Payload width of one surrogate unit.
k_utf_sur_mask	Payload bits in one surrogate unit.
k_utf_unit_max	Largest UTF-16 code unit.
k_utf_code_max	Largest code point Unicode defines.
k_utf_byte_mask	Mask isolating a unit's low byte.
k_utf_byte_shift	Shift bringing a unit's high byte down.
k_utf8_max_per_unit	Worst-case UTF-8 bytes per UTF-16 unit.

Definition at line 88 of file ra8_fs_utf_internal.h.

Function Documentation

priv_utf16_all_ascii()

uint8_t priv_utf16_all_ascii	(	const uint16_t *	in,
		uint32_t	units )

Is every unit of in inside the ASCII range?

The question a caller asks before it relies on the volume's up-case table: an ASCII-only name folds identically under every conforming table, so it is safe even when the volume carries one this build cannot reproduce, while a name with any other unit is not.

Parameters

[in]	in	Code units to inspect.
[in]	units	Number of units in in.

Returns

ASCII-only flag.

Return values

1U	Every unit is at most k_utf_ascii_max (or units is 0).
0U	At least one unit is above it.

Precondition

in addresses at least units readable units, or units is 0.

The caller treats a 0 result as "needs the volume's own table".

Postcondition

in is not modified.

The verdict depends only on the inputs.

Note

Pure function; trivially thread-safe.

Since

0.1.0

Definition at line 543 of file ra8_fs_utf.c.

References k_utf_ascii_max.

Referenced by priv_exfat_name_to_units().

priv_utf16_ieq()

uint8_t priv_utf16_ieq	(	const uint16_t *	a,
		uint32_t	an,
		const uint16_t *	b,
		uint32_t	bn )

Compare two UTF-16 names for case-insensitive equality.

Folds every unit through priv_exfat_upcase_unit() – the canonical up-case table – and compares unit by unit. Lengths must match first: the table is a simple one-to-one map, so folding never changes a name's length and a length difference is a difference.

Surrogate units fold to themselves, because the table covers the BMP and a supplementary code point has no simple case mapping inside it. Two supplementary characters therefore compare exactly, which is what a host does with the same table.

Parameters

[in]	a	First name's units.
[in]	an	Number of units in a.
[in]	b	Second name's units.
[in]	bn	Number of units in b.

Returns

Equality flag.

Return values

1U	The names are equal after folding.
0U	They differ in length or in at least one folded unit.

Precondition

a addresses an units and b addresses bn units.

Neither pointer is NULL unless its count is 0.

Postcondition

Neither input is modified.

The verdict depends only on the inputs and the fixed up-case table.

Note

Pure function; trivially thread-safe.

Since

0.1.0

Definition at line 529 of file ra8_fs_utf.c.

References priv_exfat_upcase_unit().

Referenced by internal_dir_find_long_sector().

priv_utf16_to_utf8()

ra8_err_t priv_utf16_to_utf8	(	const uint16_t *	in,
		uint32_t	units,
		char *	out,
		uint32_t	cap )

Convert UTF-16LE code units into a NUL-terminated UTF-8 name.

The inverse of priv_utf8_to_utf16(): a high surrogate followed by a low one becomes the four-byte form of the supplementary code point, every other unit becomes its one-, two- or three-byte form.

An unpaired surrogate – a high one not followed by a low one, or a low one on its own – is a name no UTF-8 string can express, so it is reported rather than substituted. The caller's fallback (the 8.3 alias on FAT) is a name that still opens the same file, which a replacement character would not be.

Parameters

[in]	in	Code units to convert.
[in]	units	Number of units in in.
[out]	out	Receives the NUL-terminated UTF-8 name.
[in]	cap	Capacity of out in bytes, including the terminator.

Returns

Error code.

Return values

k_ra8_ok	Converted; out is NUL-terminated.
k_ra8_err_null_ptr	in or out is NULL, or cap is 0.
k_ra8_err_invalid_arg	in holds an unpaired surrogate.
k_ra8_err_no_mem	The name plus its terminator exceeds cap.

Precondition

out addresses at least cap writable bytes.

in addresses at least units readable code units.

Postcondition

On success out holds well-formed UTF-8 and a NUL terminator.

On failure out[0] is NUL, so a caller that ignores the code sees an empty name rather than a partial one.

Note

Pure apart from out; trivially thread-safe against distinct buffers.

Since

0.1.0

Definition at line 499 of file ra8_fs_utf.c.

References internal_utf16_take(), internal_utf8_put(), k_ra8_err_null_ptr, and k_ra8_ok.

Referenced by internal_exfat_gather_name(), internal_fat_dir_scan_sector(), and internal_listdir_visit_sector().

priv_utf8_to_utf16()

ra8_err_t priv_utf8_to_utf16	(	const char *	in,
		uint16_t *	out,
		uint32_t	cap,
		uint32_t *	out_units )

Convert a NUL-terminated UTF-8 name into UTF-16LE code units.

Decodes each UTF-8 sequence to a code point, rejecting every malformed and non-shortest form (see the file header), then emits it as one BMP unit or as a high/low surrogate pair. The output is NOT NUL-terminated: a UTF-16 name on either format carries an explicit length, and appending a terminator here would invite a caller to use it as one.

The whole conversion is transactional in the sense that matters: on any failure out_units is set to zero, so a caller that ignores the return code writes an empty name rather than a truncated one.

Parameters

[in]	in	NUL-terminated UTF-8 name.
[out]	out	Receives the code units.
[in]	cap	Capacity of out in UTF-16 units.
[out]	out_units	Receives the number of units written.

Returns

Error code.

Return values

k_ra8_ok	Converted; out holds *out_units units.
k_ra8_err_null_ptr	in, out or out_units is NULL.
k_ra8_err_invalid_arg	in is not well-formed UTF-8.
k_ra8_err_no_mem	The name needs more than cap units.

Precondition

out addresses at least cap writable units.

in is NUL-terminated within the caller's buffer.

Postcondition

On success *out_units is at most cap.

On failure *out_units is 0 and out holds nothing meaningful.

Note

Pure apart from out; trivially thread-safe against distinct buffers.

Since

0.1.0

Definition at line 283 of file ra8_fs_utf.c.

References internal_utf16_put(), internal_utf8_next(), k_ra8_err_no_mem, k_ra8_err_null_ptr, and k_ra8_ok.

Referenced by priv_dir_find_long(), priv_exfat_name_to_units(), and priv_name_classify().