📺 Info Siaran: keras team/keras/pull/

LinZiyuu · 2026-06-06T05:01:09Z

Description

keras.datasets.imdb.load_data and keras.datasets.reuters.load_data load
their .npz files with np.load(allow_pickle=True). Loading a crafted .npz
therefore unpickles arbitrary objects, which can execute code through a pickle
__reduce__ gadget (insecure deserialization, CWE-502).

#23026 and #23034 removed allow_pickle=True from the numeric datasets
(mnist, boston_housing, california_housing) and NpzIOStore, but IMDB and
Reuters were left unchanged. These two datasets store ragged sequences as
dtype=object arrays, which numpy can only persist via pickle, so
allow_pickle=False cannot be used here without breaking the dataset format.

This PR adds keras.src.datasets.npz_utils.load_npz, used by both loaders:

numeric members are read with pickling fully disabled;
object members are read with a restricted unpickler that only permits numpy
array reconstruction (numpy.ndarray, numpy.dtype, and the
numpy(._core).multiarray _reconstruct/scalar globals).

This keeps the existing dataset format working while preventing a crafted
.npz from executing arbitrary code. A new npz_utils_test.py covers loading
ragged object arrays, loading numeric arrays, and rejecting a pickle gadget
without executing its payload.

Contributor Agreement

I am a human, and not a bot.
I will be responsible for responding to review comments in a timely manner.
I will work with the maintainers to push this PR forward until submission.

`keras.datasets.imdb.load_data` and `keras.datasets.reuters.load_data` loaded their `.npz` files with `np.load(allow_pickle=True)`. These two datasets store ragged sequences as `dtype=object` arrays, so pickling is required to read them and `allow_pickle=False` (as applied to the numeric datasets in keras-team#23026 and keras-team#23034) cannot be used here. Add `keras.src.datasets.npz_utils.load_npz`, which reads each archive member with pickling disabled for numeric arrays and, for object arrays, uses a restricted unpickler that only permits numpy array reconstruction. This keeps the existing dataset format working while preventing a crafted `.npz` from executing arbitrary code through a pickle `__reduce__` gadget.

gemini-code-assist

Code Review

This pull request introduces a secure utility npz_utils to safely load .npz files containing ragged object arrays (used in IMDB and Reuters datasets) without allowing arbitrary code execution via pickling. It replaces the unsafe np.load(allow_pickle=True) calls with a restricted unpickler that only permits NumPy array reconstruction. The review feedback suggests extending the .npy header parsing in _load_npy_member to support version 3.0 headers (using read_array_header_3_0) to ensure compatibility with modern NumPy files.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

codecov-commenter · 2026-06-06T05:13:04Z

Codecov Report

❌ Patch coverage is 71.79487% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.82%. Comparing base (4725b68) to head (290b22e).
⚠️ Report is 38 commits behind head on master.

Files with missing lines	Patch %	Lines
keras/src/datasets/npz_utils.py	81.25%	4 Missing and 2 partials ⚠️
keras/src/datasets/imdb.py	25.00%	3 Missing ⚠️
keras/src/datasets/reuters.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #23047      +/-   ##
==========================================
+ Coverage   84.74%   84.82%   +0.07%     
==========================================
  Files         464      465       +1     
  Lines       68883    69041     +158     
  Branches    11315    11346      +31     
==========================================
+ Hits        58378    58562     +184     
+ Misses       7562     7546      -16     
+ Partials     2943     2933      -10

Flag	Coverage Δ
keras	`84.62% <71.79%> (+0.07%)`	⬆️
keras-cpu	`83.89% <71.79%> (+0.07%)`	⬆️
keras-gpu	`69.50% <71.79%> (+0.07%)`	⬆️
keras-jax	`58.32% <71.79%> (+0.03%)`	⬆️
keras-numpy	`53.74% <71.79%> (+0.05%)`	⬆️
keras-openvino	`59.52% <71.79%> (+0.06%)`	⬆️
keras-tensorflow	`59.84% <71.79%> (+0.10%)`	⬆️
keras-torch	`59.07% <71.79%> (+0.07%)`	⬆️
keras-tpu	`57.13% <71.79%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

`_load_npy_member` only advanced past version 1.0 and 2.0 `.npy` headers before handing the stream to the restricted unpickler, and raised `ValueError` for anything else. numpy also supports version 3.0, so handle it too. numpy exposes no public `read_array_header_3_0`; the 3.0 format only differs from 2.0 by encoding the header string as UTF-8 instead of latin1, while the 4-byte header-length layout we skip past is identical, so 2.0's header reader handles both.

- Drop the leading underscore from `RestrictedUnpickler` and `load_npy_member`, matching the keras/src convention for module-level helpers. - Trim `_ALLOWED_PICKLE_GLOBALS` to the globals IMDB and Reuters actually use. `np.load(allow_pickle=False)` refuses object arrays entirely, and rebuilding the stored `dtype=object` arrays only needs `numpy.ndarray`, `numpy.dtype` and `multiarray._reconstruct` (both the numpy < 2 and numpy >= 2 spellings). The `multiarray.scalar` globals were never exercised by either dataset, verified against the real IMDB and Reuters `.npz` files, so they are removed.

LinZiyuu · 2026-06-24T02:22:09Z

Thanks for the review @hertschuh! Addressed in 290b22e.

Renamed _RestrictedUnpickler → RestrictedUnpickler and _load_npy_member → load_npy_member to match the naming convention.

Just for my understanding, what in this list is used by Reuters and IMDB that is not allowed by np.load(allow_pickle=False)?

Good question. The short answer is that none of them are allowed by allow_pickle=False. That flag refuses object arrays entirely (ValueError: Object arrays cannot be loaded when allow_pickle=False), so it can't load these datasets at all. IMDB and Reuters store their ragged sequences as dtype=object arrays, which numpy can only persist as a pickle stream.

I traced find_class against the actual imdb.npz / reuters.npz, and each object member only ever needs three globals:

numpy.ndarray
numpy.dtype
numpy.core.multiarray._reconstruct

The hosted files were written with numpy < 2, hence the numpy.core spelling; a numpy ≥ 2 re-save emits numpy._core.multiarray._reconstruct instead, which is why both spellings are listed. multiarray.scalar was never actually exercised by either dataset (the elements are plain Python lists of ints, not numpy scalars), so I've dropped those two entries — the allowlist is now exactly the set the datasets need, and anything else (os.system, builtins.eval, …) still raises UnpicklingError.

hertschuh

Thanks for implementing this!

* Restrict unpickling when loading IMDB and Reuters datasets (#23047) * Restrict unpickling when loading IMDB and Reuters datasets `keras.datasets.imdb.load_data` and `keras.datasets.reuters.load_data` loaded their `.npz` files with `np.load(allow_pickle=True)`. These two datasets store ragged sequences as `dtype=object` arrays, so pickling is required to read them and `allow_pickle=False` (as applied to the numeric datasets in #23026 and #23034) cannot be used here. Add `keras.src.datasets.npz_utils.load_npz`, which reads each archive member with pickling disabled for numeric arrays and, for object arrays, uses a restricted unpickler that only permits numpy array reconstruction. This keeps the existing dataset format working while preventing a crafted `.npz` from executing arbitrary code through a pickle `__reduce__` gadget. * Support npy format version 3.0 in restricted npz loader `_load_npy_member` only advanced past version 1.0 and 2.0 `.npy` headers before handing the stream to the restricted unpickler, and raised `ValueError` for anything else. numpy also supports version 3.0, so handle it too. numpy exposes no public `read_array_header_3_0`; the 3.0 format only differs from 2.0 by encoding the header string as UTF-8 instead of latin1, while the 4-byte header-length layout we skip past is identical, so 2.0's header reader handles both. * Apply review feedback to the restricted npz loader - Drop the leading underscore from `RestrictedUnpickler` and `load_npy_member`, matching the keras/src convention for module-level helpers. - Trim `_ALLOWED_PICKLE_GLOBALS` to the globals IMDB and Reuters actually use. `np.load(allow_pickle=False)` refuses object arrays entirely, and rebuilding the stored `dtype=object` arrays only needs `numpy.ndarray`, `numpy.dtype` and `multiarray._reconstruct` (both the numpy < 2 and numpy >= 2 spellings). The `multiarray.scalar` globals were never exercised by either dataset, verified against the real IMDB and Reuters `.npz` files, so they are removed. * Verify all intermediary H5 groups when navigating H5 files. (#23168) The H5 library is able to resolve nested group addressed using slashes in the name. Instead, resolve these manually to verify the group type each step of the way. * [Fix] Refactor _load_state to capture weight store while preserving Keras test passing (#23226) * Fixes for Python 3.14. (#23259) Python 3.14 no longer allows `NotImplementedError`s to be treated as booleans, instead it actually throws. We were using things like `(-1).__ne__` to filter out dimensions, but this didn't handle `None` dimensions correctly. Replaced with list comprehension. * Reject decompression-bomb members on the .keras asset extraction path (#23101) * Reject decompression-bomb archive members on the .keras asset extract path The decompression-bomb guard `_reject_zip_bomb` is only applied to the in-memory reads (config.json, model.weights.h5, the shard map). When a `.keras` archive has more than three members, the loader builds a disk-backed asset store whose constructor extracts the whole archive with `ZipFile.extractall`, decompressing every member to disk with no ratio check. An attacker-added member that decompresses far beyond its stored size therefore lands on disk in full, exhausting it from a tiny file under the default safe mode. Add `_reject_zip_extract_bomb`, which applies the same per-member declared-vs-stored ratio to every member before extraction. The per-member ratio is used (not the aggregate) so a bomb member cannot be diluted by genuine weights stored alongside it; genuine `.keras` members are stored uncompressed, so the ratio is false-positive-free. * Address review: trim comments and drop in-test keras import - Shorten the extract-bomb floor comment to a single line - Remove the self-explanatory call-site and test-method comments - Use the module-level keras import in the test instead of a local one * Restrict unpickling when loading CIFAR datasets (#23252) * restrict unpickling when loading CIFAR datasets read data_batch_*/test_batch through the numpy-only RestrictedUnpickler so a tampered batch file cannot run a __reduce__ gadget * Use platform-independent gadget in cifar unpickling test * Drop cifar unpickling test and inline comment * Fix TraceContext error for NNX backend reported in #23289 (#23326) * fix trace context error bug in keras * cleanup * fix * fix * clean up PR * code reformat * Bump version to 3.15.1 --------- Co-authored-by: Ziyu Lin <104151270+LinZiyuu@users.noreply.github.com> Co-authored-by: hertschuh <1091026+hertschuh@users.noreply.github.com> Co-authored-by: Suhana <suhanaaa@google.com> Co-authored-by: SABITHSAHEB <shabi7204192361@gmail.com> Co-authored-by: Divyashree Sreepathihalli <divyashreepathihalli@gmail.com>

* Restrict unpickling when loading IMDB and Reuters datasets (#23047) * Restrict unpickling when loading IMDB and Reuters datasets `keras.datasets.imdb.load_data` and `keras.datasets.reuters.load_data` loaded their `.npz` files with `np.load(allow_pickle=True)`. These two datasets store ragged sequences as `dtype=object` arrays, so pickling is required to read them and `allow_pickle=False` (as applied to the numeric datasets in #23026 and #23034) cannot be used here. Add `keras.src.datasets.npz_utils.load_npz`, which reads each archive member with pickling disabled for numeric arrays and, for object arrays, uses a restricted unpickler that only permits numpy array reconstruction. This keeps the existing dataset format working while preventing a crafted `.npz` from executing arbitrary code through a pickle `__reduce__` gadget. * Support npy format version 3.0 in restricted npz loader `_load_npy_member` only advanced past version 1.0 and 2.0 `.npy` headers before handing the stream to the restricted unpickler, and raised `ValueError` for anything else. numpy also supports version 3.0, so handle it too. numpy exposes no public `read_array_header_3_0`; the 3.0 format only differs from 2.0 by encoding the header string as UTF-8 instead of latin1, while the 4-byte header-length layout we skip past is identical, so 2.0's header reader handles both. * Apply review feedback to the restricted npz loader - Drop the leading underscore from `RestrictedUnpickler` and `load_npy_member`, matching the keras/src convention for module-level helpers. - Trim `_ALLOWED_PICKLE_GLOBALS` to the globals IMDB and Reuters actually use. `np.load(allow_pickle=False)` refuses object arrays entirely, and rebuilding the stored `dtype=object` arrays only needs `numpy.ndarray`, `numpy.dtype` and `multiarray._reconstruct` (both the numpy < 2 and numpy >= 2 spellings). The `multiarray.scalar` globals were never exercised by either dataset, verified against the real IMDB and Reuters `.npz` files, so they are removed. * Verify all intermediary H5 groups when navigating H5 files. (#23168) The H5 library is able to resolve nested group addressed using slashes in the name. Instead, resolve these manually to verify the group type each step of the way. * Reject decompression-bomb members on the .keras asset extraction path (#23101) * Reject decompression-bomb archive members on the .keras asset extract path The decompression-bomb guard `_reject_zip_bomb` is only applied to the in-memory reads (config.json, model.weights.h5, the shard map). When a `.keras` archive has more than three members, the loader builds a disk-backed asset store whose constructor extracts the whole archive with `ZipFile.extractall`, decompressing every member to disk with no ratio check. An attacker-added member that decompresses far beyond its stored size therefore lands on disk in full, exhausting it from a tiny file under the default safe mode. Add `_reject_zip_extract_bomb`, which applies the same per-member declared-vs-stored ratio to every member before extraction. The per-member ratio is used (not the aggregate) so a bomb member cannot be diluted by genuine weights stored alongside it; genuine `.keras` members are stored uncompressed, so the ratio is false-positive-free. * Address review: trim comments and drop in-test keras import - Shorten the extract-bomb floor comment to a single line - Remove the self-explanatory call-site and test-method comments - Use the module-level keras import in the test instead of a local one * Restrict unpickling when loading CIFAR datasets (#23252) * restrict unpickling when loading CIFAR datasets read data_batch_*/test_batch through the numpy-only RestrictedUnpickler so a tampered batch file cannot run a __reduce__ gadget * Use platform-independent gadget in cifar unpickling test * Drop cifar unpickling test and inline comment * Bump version to 3.12.4 --------- Co-authored-by: Ziyu Lin <104151270+LinZiyuu@users.noreply.github.com> Co-authored-by: hertschuh <1091026+hertschuh@users.noreply.github.com> Co-authored-by: SABITHSAHEB <shabi7204192361@gmail.com>

…m#23047) * Restrict unpickling when loading IMDB and Reuters datasets `keras.datasets.imdb.load_data` and `keras.datasets.reuters.load_data` loaded their `.npz` files with `np.load(allow_pickle=True)`. These two datasets store ragged sequences as `dtype=object` arrays, so pickling is required to read them and `allow_pickle=False` (as applied to the numeric datasets in keras-team#23026 and keras-team#23034) cannot be used here. Add `keras.src.datasets.npz_utils.load_npz`, which reads each archive member with pickling disabled for numeric arrays and, for object arrays, uses a restricted unpickler that only permits numpy array reconstruction. This keeps the existing dataset format working while preventing a crafted `.npz` from executing arbitrary code through a pickle `__reduce__` gadget. * Support npy format version 3.0 in restricted npz loader `_load_npy_member` only advanced past version 1.0 and 2.0 `.npy` headers before handing the stream to the restricted unpickler, and raised `ValueError` for anything else. numpy also supports version 3.0, so handle it too. numpy exposes no public `read_array_header_3_0`; the 3.0 format only differs from 2.0 by encoding the header string as UTF-8 instead of latin1, while the 4-byte header-length layout we skip past is identical, so 2.0's header reader handles both. * Apply review feedback to the restricted npz loader - Drop the leading underscore from `RestrictedUnpickler` and `load_npy_member`, matching the keras/src convention for module-level helpers. - Trim `_ALLOWED_PICKLE_GLOBALS` to the globals IMDB and Reuters actually use. `np.load(allow_pickle=False)` refuses object arrays entirely, and rebuilding the stored `dtype=object` arrays only needs `numpy.ndarray`, `numpy.dtype` and `multiarray._reconstruct` (both the numpy < 2 and numpy >= 2 spellings). The `multiarray.scalar` globals were never exercised by either dataset, verified against the real IMDB and Reuters `.npz` files, so they are removed.

google-ml-butler Bot added the size:M label Jun 6, 2026

google-ml-butler Bot assigned gbaned Jun 6, 2026

gemini-code-assist Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread keras/src/datasets/npz_utils.py

keerthanakadiri added the stat:awaiting response from contributor label Jun 8, 2026

hertschuh requested changes Jun 23, 2026

View reviewed changes

Comment thread keras/src/datasets/npz_utils.py Outdated

Comment thread keras/src/datasets/npz_utils.py Outdated

Comment thread keras/src/datasets/npz_utils.py Outdated

google-ml-butler Bot removed the stat:awaiting response from contributor label Jun 24, 2026

LinZiyuu requested a review from hertschuh June 25, 2026 07:00

google-ml-butler Bot added the awaiting review label Jun 25, 2026

hertschuh approved these changes Jun 26, 2026

View reviewed changes

google-ml-butler Bot added kokoro:force-run ready to pull Ready to be merged into the codebase labels Jun 26, 2026

hertschuh merged commit d0fea16 into keras-team:master Jun 26, 2026
16 checks passed

google-ml-butler Bot removed awaiting review ready to pull Ready to be merged into the codebase kokoro:force-run labels Jun 26, 2026

SABITHSAHEB mentioned this pull request Jul 14, 2026

Restrict unpickling when loading CIFAR datasets #23252

Merged

3 tasks

bernhardkaindl mentioned this pull request Aug 16, 2026

py-keras: Deprecate versions with security issues(CWE-502, CWE-73) spack/spack-packages#6107

Merged

Navigation Menu

Uh oh!

Restrict unpickling when loading IMDB and Reuters datasets - #23047

Restrict unpickling when loading IMDB and Reuters datasets#23047
hertschuh merged 3 commits into
keras-team:masterfrom
LinZiyuu:harden-imdb-reuters-npz-pickle

LinZiyuu commented Jun 6, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

codecov-commenter commented Jun 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LinZiyuu commented Jun 24, 2026

Uh oh!

hertschuh left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

LinZiyuu commented Jun 6, 2026

Description

Contributor Agreement

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

codecov-commenter commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LinZiyuu commented Jun 24, 2026

Uh oh!

hertschuh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jun 6, 2026 •

edited

Loading