Repository navigation
Description
Bug description
Assigning an empty dictionary to the C-accelerated _pickle.Unpickler.memo causes a subsequent protocol-4 MEMOIZE opcode to crash the interpreter instead of completing safely or raising a Python exception.
This is a deterministic, single-threaded reproduction. An AddressSanitizer-enabled non-debug CPython build reports a heap-buffer-overflow (8-byte out-of-bounds read) in _Unpickler_MemoPut().
This report is being opened as a regular bug report at the request of the CPython security triage team. It does not claim that deserializing untrusted pickle data is safe, nor does it claim remote exploitability.
Minimal reproducer
import io
import _pickle
u = _pickle.Unpickler(io.BytesIO(b"\x80\x04N\x94."))
u.memo = {}
u.load()The stream contains PROTO 4, NONE, MEMOIZE, and STOP.
Actual behavior
- CPython 3.16.0a2+dev (commit
b09a7e92fb0ab958a52ad94e376402cdd3da09b2), Clang 23.1.1, non-debug,--without-pymalloc,-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined: AddressSanitizer reportsheap-buffer-overflow,READ of size 8, atModules/_pickle.c:1742(_Unpickler_MemoPut), immediately after a zero-sized allocation represented as a 1-byte region. - CPython 3.16 debug build: assertion failure in
_Unpickler_ResizeMemoList()(Modules/_pickle.c:1661), SIGABRT. - CPython 3.14.7 release build: SIGSEGV.
Relevant symbolized frames on the tested 3.16 build:
_Unpickler_MemoPut Modules/_pickle.c:1742
_Unpickler_MemoPut Modules/_pickle.c:1664 (resize path)
load Modules/_pickle.c:6975
_pickle_Unpickler_load Modules/_pickle.c:7592
Expected behavior
Assigning an empty memo dictionary should not cause an out-of-bounds native memory access. Subsequent load() should either work or raise a controlled Python exception.
Suspected root cause
After u.memo = {}, the internal memo has zero capacity (memo_size == 0). The first MEMOIZE insertion uses index zero. The resize request calculated from idx * 2 is therefore zero, leaving no element available at index zero. The subsequent old_item = self->memo[idx] reads outside the allocated memo array.
A possible fix is to ensure memo capacity is strictly greater than the insertion index, including when idx == 0, with appropriate overflow checks. This suggestion has not yet been validated as a patch.
Related issue
- Assigning a dict to the memo of an Unpickler object fails #111066 describes incorrect behavior when assigning a dictionary to
Unpickler.memo. This report adds a deterministic native crash and sanitizer-confirmed out-of-bounds read after assigning an empty dictionary. - This does not require concurrent access and is distinct from free-threading races.
CPython versions tested on
3.14.7; 3.16.0a2+dev
Operating systems tested on
Linux x86-64
Reporter
Hamdy Abou El Anein
I can provide the full ASan log, symbolized trace, and test a regression fix if useful.
Linked PRs
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsNo status
Activity
hamarshehmhmd commentedon Nov 22, 2026
Reproduced on
main(1822ff5) with a--with-pydebugbuild. Your 4-byte stream hitsAssertion 'new_size > self->memo_size' failedin_Unpickler_ResizeMemoList().The cause is what you suspected.
memo = {}leavesmemo_size == 0, and the firstMEMOIZEuses index 0, so_Unpickler_MemoPut()asks foridx * 2 == 0slots. My fix is in_Unpickler_MemoPut(), so it covers every way of ending up with an empty memo:32 is the size a new
Unpicklerstarts with. I first tried(idx + 1) * 2, but that changes the growth sequence, andSizeofTests.test_unpickler(which expects 32 → 64 → 128) catches it. With this version, growth foridx > 0stays exactly the same.This is separate from gh-111066 and its PR gh-162329. That PR still sets
memo_sizetolen(dict), somemo = {}still gives a zero-size array, and this check is still needed alongside it. Reassigning{}a second time on the same unpickler also runs into gh-111066 (memo_lenisn't reset), so I left that case out of the test.Test, added to
AbstractPicklerUnpicklerObjectTestsinLib/test/pickletester.py, so it runs for both the C and Python unpicklers:Nonetest_empty_unpickler_memoFatal Python error: Aborted./python -m test test_pickle test_pickletools test_copy test_shelve./python -m test test_pickle -R 3:3 -m '*memo*'I also added a NEWS entry (
Misc/NEWS.d/next/Library/...gh-issue-162334...rst). Happy to open the PR if nobody else has one going.hamdyaea commentedon Nov 22, 2026
Thank you, @hamarshehmhmd, for independently reproducing this and for the carefully tested fix. I especially appreciate your investigation of the 32 → 64 → 128 growth behavior, the regression tests, and the NEWS entry.
I originally reported the crash in gh-162334 and would like to prepare the PR from my fork, with explicit credit to you for the proposed patch and regression tests. I would be grateful for your review and any further collaboration. I also plan to validate the proposed fix using an ASan-enabled build.
Thanks again for contributing to the resolution of this issue!
Hamdy Abou El Anein
pythongh-162334: Restore memo capacity when inserting at index zero
pythongh-162334: Add regression tests for empty unpickler memo
pythongh-162334: Document fix for empty Unpickler memo