Skip to content

feat(ext/node): add a byteLength/length parameter to Buffer.indexOf/lastIndexOf/includes - #35872

Merged
bartlomieju merged 6 commits into
denoland:mainfrom
Medo-ID:parallel/test-buffer-indexof
Jul 22, 2026
Merged

bartlomieju merged 6 commits into
denoland:mainfrom
Medo-ID:parallel/test-buffer-indexof

Conversation

@Medo-ID

@Medo-ID Medo-ID commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes part of #35293 (Buffer#indexOf / lastIndexOf / includes).

This PR updates the Node Buffer search implementation to support the new end parameter introduced in newer Node.js versions.

The changes:

  • Propagate the end argument through the internal Buffer search APIs.
  • Update indexOfBuffer, indexOfNumber, indexOfNeedle, and findLastIndex to accept and forward the new parameter.
  • Re-enable the corresponding Node compatibility test.

Testing

  • ./x test-compat parallel/test-buffer-indexof.js

AI Disclosure

I used Cloude to help understand the Node.js implementation, discuss the design, and review the proposed changes + Write this description for me. The implementation, testing, and final code changes were completed and verified by me.

@Medo-ID Medo-ID changed the title fix(node/buffer): add a byteLength/length parameter to Buffer.indexOf/lastIndexOf/includes feat(node/buffer): add a byteLength/length parameter to Buffer.indexOf/lastIndexOf/includes Jul 8, 2026
@Medo-ID
Medo-ID force-pushed the parallel/test-buffer-indexof branch from af1c6d2 to e99bf1f Compare July 8, 2026 16:11
@Medo-ID

Medo-ID commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Hi! When you have some time, could you please review this PR?

Notes:

  • The changes in ext/node/polyfills/internal/buffer.mjs are copied directly from Node's implementation.
  • The changes in ext/node/polyfills/internal_binding/buffer.ts are my implementation and would especially benefit from verification to ensure they match the intended behavior.
  • My first attempt failed CI. In the second commit, I fixed the formatting/lint issues and corrected the loop condition from:
// buffer.ts - line 50
  for (let i = start; i < lastStart; i += step) {

to :

// buffer.ts - line 50
  for (let i = start; i <= lastStart; i += step) {

After those changes, the checks passed.

@bartlomieju

Copy link
Copy Markdown
Member

Nice work, this tracks Node's lib/buffer.js wrappers and the C++ search semantics closely. I traced every assertion in the re-enabled test through the port and they all check out.

Two small divergences from Node's C++ (src/node_buffer.cc) that the test doesn't currently exercise, so they're latent rather than failing. Both are in indexOfBuffer in ext/node/polyfills/internal_binding/buffer.ts right around:

end = end < targetBufferLength ? end : targetBufferLength;
  1. Missing max(end, 0) clamp. Node computes search_end = min(max(end, 0), haystack_length) (node_buffer.cc, the search_end line). We only do the min side. It only surfaces for the empty-needle + negative-end case: Buffer.from('abcabc').indexOf('', 0, -1) returns -1 here vs Node's 0 (Node clamps min(opt_offset, search_end) with search_end floored at 0). Obscure, but a real parity gap.

  2. Missing UCS2 even-rounding. Node does if (enc == UCS2) search_end &= ~1 so the search range is a whole number of 16-bit units. We don't. In practice it's masked by the step = 2 alignment from an even byteOffset, so I couldn't build a failing case with even offsets, only a theoretical odd-offset ucs2 case the test never passes.

Both are cheap to close if we want byte-exact parity:

end = end < targetBufferLength ? end : targetBufferLength;
if (end < 0) end = 0;
if (isUcs2) end &= ~1;

Neither blocks the PR since the test passes as written; up to you whether to fold them in now or leave a follow-up.

@bartlomieju bartlomieju changed the title feat(node/buffer): add a byteLength/length parameter to Buffer.indexOf/lastIndexOf/includes feat(ext/node): add a byteLength/length parameter to Buffer.indexOf/lastIndexOf/includes Jul 9, 2026
@Medo-ID
Medo-ID force-pushed the parallel/test-buffer-indexof branch from e99bf1f to ae43740 Compare July 10, 2026 09:51
@Medo-ID

Medo-ID commented Jul 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks! I'll add unit tests to cover these additional cases that were missing from the upstream test-buffer-indexof.js:

  • Empty needle with a negative end should clamp to 0, matching Node's search_end = min(max(end, 0), haystack_length).
  • ucs2/utf16le searches should round an odd end down to the nearest code unit boundary (search_end &= ~1).

Notes

I'll also update the TypeScript declarations for indexOf, includes, and lastIndexOf in ext/node/polyfills/internal/buffer.d.ts to include the new end parameter.

While investigating the type errors, I also found that cli/tsc/dts/node/buffer.d.cts needs to be updated as well; otherwise calls such as buf.indexOf("", 0, -1) and buf.indexOf("", 0, 3, "ucs2") fail type checking.

One question: for the new overloads, should the signature be

byteOffset: number | undefined,
end: number,

or should both parameters remain optional?

byteOffset?: number,
end?: number,

I believe the first option is more appropriate because end is only meaningful when byteOffset is provided, but I'd appreciate your confirmation before finalizing the typings.

@Medo-ID
Medo-ID force-pushed the parallel/test-buffer-indexof branch 2 times, most recently from d5b58e3 to 6d17e27 Compare July 10, 2026 17:11
@Medo-ID
Medo-ID force-pushed the parallel/test-buffer-indexof branch from 6d17e27 to 2ae9bdf Compare July 10, 2026 17:20
Reimplement indexOfBuffer/indexOfNumber to mirror node/src/node_buffer.cc
(IndexOfOffset + search_end clamp + guard sequence + SearchString) instead
of the ad-hoc end clamping. Fixes three divergences from Node:

- indexOf(val, 0, NaN) searched the whole buffer instead of returning -1
  (Node reads end as an int64, so NaN floors to 0 via max(end, 0)).
- Empty needle with a negative byteOffset returned the raw negative offset
  instead of the normalized position.
- UCS2 empty needle returned -1 instead of the clamped offset.

Verified against all 718 assertions of Node's test/parallel/test-buffer-indexof.js.
The two runtime/ sites gated malloc_trim on target_os = linux only, but
it is a glibc extension absent on musl. Match the existing gnu-gated
pattern in cli/lib/worker.rs.
…ffset in Buffer search

Mirror three details of Node's node_buffer.cc that the byte-space port
missed:

- UCS2 forward search now aligns an odd byteOffset down to the code-unit
  grid (Node's offset / 2); previously it scanned odd byte positions.
- A UCS2 needle with a trailing odd byte is truncated to whole code units
  (Node's needle_length / 2).
- A fractional byteOffset is truncated toward zero, matching Node reading
  it as an int64 at the binding boundary.

Add unit_node tests for each case (none are covered by the re-enabled
upstream test-buffer-indexof.js, whose ucs2 cases use even offsets).
Move the end/encoding argument normalization into bidirectionalIndexOf
instead of repeating it in indexOf/lastIndexOf/includes, and factor the
byteOffset/end int64 coercion into a toInt64 helper.
@bartlomieju
bartlomieju merged commit e2f15f2 into denoland:main Jul 22, 2026
136 checks passed
bartlomieju added a commit that referenced this pull request Jul 23, 2026
…astIndexOf/includes (#35872)

Fixes part of #35293 (`Buffer#indexOf` / `lastIndexOf` / `includes`).

---------

Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants