Skip to content

_pickle.Unpickler crashes after assigning an empty memo dictionary #162334

Description

@hamdyaea

Bug description

Assigning an empty dictionary to the C-accelerated _pickle.Unpickler.memo causes a subsequent protocol-4 MEMOIZE opcode to crash the interpreter instead of completing safely or raising a Python exception.

This is a deterministic, single-threaded reproduction. An AddressSanitizer-enabled non-debug CPython build reports a heap-buffer-overflow (8-byte out-of-bounds read) in _Unpickler_MemoPut().

This report is being opened as a regular bug report at the request of the CPython security triage team. It does not claim that deserializing untrusted pickle data is safe, nor does it claim remote exploitability.

Minimal reproducer

import io
import _pickle

u = _pickle.Unpickler(io.BytesIO(b"\x80\x04N\x94."))
u.memo = {}
u.load()

The stream contains PROTO 4, NONE, MEMOIZE, and STOP.

Actual behavior

  • CPython 3.16.0a2+dev (commit b09a7e92fb0ab958a52ad94e376402cdd3da09b2), Clang 23.1.1, non-debug, --without-pymalloc, -O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined: AddressSanitizer reports heap-buffer-overflow, READ of size 8, at Modules/_pickle.c:1742 (_Unpickler_MemoPut), immediately after a zero-sized allocation represented as a 1-byte region.
  • CPython 3.16 debug build: assertion failure in _Unpickler_ResizeMemoList() (Modules/_pickle.c:1661), SIGABRT.
  • CPython 3.14.7 release build: SIGSEGV.

Relevant symbolized frames on the tested 3.16 build:

_Unpickler_MemoPut      Modules/_pickle.c:1742
_Unpickler_MemoPut      Modules/_pickle.c:1664 (resize path)
load                    Modules/_pickle.c:6975
_pickle_Unpickler_load  Modules/_pickle.c:7592

Expected behavior

Assigning an empty memo dictionary should not cause an out-of-bounds native memory access. Subsequent load() should either work or raise a controlled Python exception.

Suspected root cause

After u.memo = {}, the internal memo has zero capacity (memo_size == 0). The first MEMOIZE insertion uses index zero. The resize request calculated from idx * 2 is therefore zero, leaving no element available at index zero. The subsequent old_item = self->memo[idx] reads outside the allocated memo array.

A possible fix is to ensure memo capacity is strictly greater than the insertion index, including when idx == 0, with appropriate overflow checks. This suggestion has not yet been validated as a patch.

Related issue

  • Assigning a dict to the memo of an Unpickler object fails #111066 describes incorrect behavior when assigning a dictionary to Unpickler.memo. This report adds a deterministic native crash and sanitizer-confirmed out-of-bounds read after assigning an empty dictionary.
  • This does not require concurrent access and is distinct from free-threading races.

CPython versions tested on

3.14.7; 3.16.0a2+dev

Operating systems tested on

Linux x86-64

Reporter

Hamdy Abou El Anein

I can provide the full ASan log, symbolized trace, and test a regression fix if useful.

Linked PRs

Activity

  1. hamarshehmhmd commented on Nov 22, 2026

    @hamarshehmhmd

    Reproduced on main (1822ff5) with a --with-pydebug build. Your 4-byte stream hits Assertion 'new_size > self->memo_size' failed in _Unpickler_ResizeMemoList().

    The cause is what you suspected. memo = {} leaves memo_size == 0, and the first MEMOIZE uses index 0, so _Unpickler_MemoPut() asks for idx * 2 == 0 slots. My fix is in _Unpickler_MemoPut(), so it covers every way of ending up with an empty memo:

    --- a/Modules/_pickle.c
    +++ b/Modules/_pickle.c
    @@ -1735,7 +1735,9 @@ _Unpickler_MemoPut(UnpicklerObject *self, size_t idx, PyObject *value)
                 Py_DECREF(key);
                 return 0;
             }
    -        if (_Unpickler_ResizeMemoList(self, idx * 2) < 0)
    +        /* idx is 0 if the memo is empty, e.g. after "memo = {}" (gh-162334);
    +           start again from the initial memo size then. */
    +        if (_Unpickler_ResizeMemoList(self, idx ? idx * 2 : 32) < 0)
                 return -1;
             assert(idx < self->memo_size);
         }

    32 is the size a new Unpickler starts with. I first tried (idx + 1) * 2, but that changes the growth sequence, and SizeofTests.test_unpickler (which expects 32 → 64 → 128) catches it. With this version, growth for idx > 0 stays exactly the same.

    This is separate from gh-111066 and its PR gh-162329. That PR still sets memo_size to len(dict), so memo = {} still gives a zero-size array, and this check is still needed alongside it. Reassigning {} a second time on the same unpickler also runs into gh-111066 (memo_len isn't reset), so I left that case out of the test.

    Test, added to AbstractPicklerUnpicklerObjectTests in Lib/test/pickletester.py, so it runs for both the C and Python unpicklers:

        def test_empty_unpickler_memo(self):
            # gh-162334: assigning an empty memo used to crash the C unpickler
            # on the next memoized object.
            for proto in protocols:
                with self.subTest(proto=proto):
                    value = [b'repeated'] * 2
                    data = pickle.dumps(value, protocol=proto)
                    unpickler = self.unpickler_class(io.BytesIO(data))
                    unpickler.memo = {}
                    result = unpickler.load()
                    self.assertEqual(result, value)
                    self.assertIs(result[0], result[1])
            unpickler = self.unpickler_class(io.BytesIO(b'\x80\x04N\x94.'))
            unpickler.memo = {}
            self.assertIsNone(unpickler.load())
    before after
    your reproducer (debug build) assertion failure, abort returns None
    test_empty_unpickler_memo Fatal Python error: Aborted passes
    ./python -m test test_pickle test_pickletools test_copy test_shelve n/a 1,592 run, all pass
    ./python -m test test_pickle -R 3:3 -m '*memo*' n/a no leaks

    I also added a NEWS entry (Misc/NEWS.d/next/Library/...gh-issue-162334...rst). Happy to open the PR if nobody else has one going.

  2. hamdyaea commented on Nov 22, 2026

    @hamdyaea
    Author

    Thank you, @hamarshehmhmd, for independently reproducing this and for the carefully tested fix. I especially appreciate your investigation of the 32 → 64 → 128 growth behavior, the regression tests, and the NEWS entry.

    I originally reported the crash in gh-162334 and would like to prepare the PR from my fork, with explicit credit to you for the proposed patch and regression tests. I would be grateful for your review and any further collaboration. I also plan to validate the proposed fix using an ASan-enabled build.

    Thanks again for contributing to the resolution of this issue!

    Hamdy Abou El Anein

  3. added 3 commits that reference this issue on Nov 22, 2026
  4. added
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions