You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gh-162334: Fix crash after setting an empty Unpickler memo - #162425
Fix a reproducible crash in the C-accelerated _pickle.Unpickler after setting unpickler.memo = {}. The first MEMOIZE operation with index zero can request a zero-capacity resize, causing an out-of-bounds read in non-debug builds or an assertion failure in debug builds.
This patch restores the normal initial capacity of 32 entries for an empty memo, while retaining the existing memo growth behavior for other indices.
Guard the zero-sized memo resize in _Unpickler_MemoPut().
Add a regression test for all supported pickle protocols and the minimal reproducer, through AbstractPicklerUnpicklerObjectTests.
Add a library NEWS entry.
Testing
In the original issue discussion, @hamarshehmhmd reported that this change fixes the debug assertion and passes 1,592 tests from test_pickle, test_pickletools, test_copy, and test_shelve, as well as focused memo reference-leak checks.
These results were reported by @hamarshehmhmd. I have not independently run the test suite on this PR branch. CI can validate this branch.
Proposed C fix, regression test, compatibility analysis, and reported test results:@hamarshehmhmd, in their detailed issue comment. The patch and test in this PR are based on that work and submitted with explicit credit. I welcome their review and collaboration.
CI triage for run 45007844848: the only substantive failing job is Sanitizers / TSan (free-threading) (job 137163398200). It aborts in test_free_threading.test_generators.TestFTGenerators.test_concurrent_write with a mimalloc assertion in Objects/mimalloc/arena-abandon.c:162 (_mi_arena_segment_mark_abandoned). The test_free_threading.test_pickle suite passes in the same job, as do the ASan, UBSan, MSan, and other platform jobs. The All required checks pass failure is downstream of this TSan failure.
This appears unrelated to the changes in Modules/_pickle.c and Lib/test/pickletester.py, though a rerun is needed to assess whether it is intermittent. I attempted to rerun the individual failed job, but the GitHub Actions API returned 403 (Resource not accessible by integration). Could someone with the necessary Actions permissions rerun that TSan job? Thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix a reproducible crash in the C-accelerated
_pickle.Unpicklerafter settingunpickler.memo = {}. The firstMEMOIZEoperation with index zero can request a zero-capacity resize, causing an out-of-bounds read in non-debug builds or an assertion failure in debug builds.This patch restores the normal initial capacity of 32 entries for an empty memo, while retaining the existing memo growth behavior for other indices.
Minimal reproducer
Changes
_Unpickler_MemoPut().AbstractPicklerUnpicklerObjectTests.Testing
In the original issue discussion, @hamarshehmhmd reported that this change fixes the debug assertion and passes 1,592 tests from
test_pickle,test_pickletools,test_copy, andtest_shelve, as well as focused memo reference-leak checks.These results were reported by @hamarshehmhmd. I have not independently run the test suite on this PR branch. CI can validate this branch.
Attribution
Fixes #162334.