SIMD-accelerated UTF-8 to UTF-32/UTF-16 conversion - #127
Merged
Conversation
Add SIMD fast paths for convert_to<char32_t> and convert_to<char16_t> from UTF-8 input. ASCII bytes are bulk-widened using SIMD zero-extend instructions (SSE4.1/AVX2/AVX-512 on x86_64, NEON on ARM64), with automatic fallback to scalar decoding for multi-byte sequences. Architecture support: - x86_64: SSE4.1 (128-bit), AVX2 (256-bit), AVX-512 (512-bit) with runtime dispatch via CPUID - ARM64: NEON (128-bit), always available on AArch64 - Scalar fallback for SIMD_IMPLEMENTATION=none (e.g. emscripten) - std::simd / std::experimental::simd when available New files: - convert_simd_impl.h: template SIMD kernels parameterized on bit width - convert.cpp: runtime dispatcher selecting 128/256/512-bit path - convert256.cpp / convert512.cpp: arch-specific instantiations from_utf8() in utf8.h now delegates to the same SIMD-accelerated path. Closes #20 Signed-off-by: Christian Parpart <christian@parpart.family>
Add convert.cpp, convert256.cpp, convert512.cpp to the library build and convert_simd_impl.h to private headers. Fix AVX2/AVX-512 compile flags to use /arch:AVX2 and /arch:AVX512 on MSVC instead of the GCC/Clang -mavx2 / -mavx512f flags. This also fixes the pre-existing flag issue for the scan256/scan512 files. Signed-off-by: Christian Parpart <christian@parpart.family>
Add 11 test cases (88 assertions) covering SIMD boundary sizes, ASCII-only, multi-byte-only, mixed, interleaved, roundtrip, and from_utf8 consistency. All pass with both SIMD and scalar fallback. Add UTF-8->UTF-32 and UTF-8->UTF-16 benchmarks at various input sizes (16B to 1MB) for pure ASCII and mixed content. Signed-off-by: Christian Parpart <christian@parpart.family>
Add ubuntu_arm64 job using GitHub's native ubuntu-24.04-arm runner to exercise the NEON SIMD path in CI alongside the existing x86_64 and macOS ARM64 jobs. Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
force-pushed
the
feature/issue-20-simd-utf8-conversion
branch
from
April 8, 2026 18:51
8e55ca3 to
2bfd2cb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
convert_to<char32_t>andconvert_to<char16_t>from UTF-8 input. ASCII bytes are bulk-widened using SIMD zero-extend instructions, with automatic fallback to scalar decoding for multi-byte sequences. Supports SSE4.1/AVX2/AVX-512 on x86_64, NEON on ARM64, std::simd when available, and pure scalar for emscripten. Runtime dispatch via CPUID on x86_64;from_utf8()now shares the same accelerated path./arch:AVX2,/arch:AVX512) for both new convert and existing scan compilation units. Add 11 SIMD-specific test cases (88 assertions) covering boundary sizes, mixed content, roundtrips, and consistency — all pass with both SIMD and scalar fallback.ubuntu-24.04-arm) to exercise the NEON path alongside existing x86_64 and macOS ARM64 builds. Add conversion benchmarks showing 6–51x speedup over scalar for ASCII-dominant text.Closes #20