weaseljson.py's WeaselJsonParser allocates the underlying C parser in __init__
(weaseljson.py:109-114, WeaselJsonParser_create), but the only path that ever frees it is close() (weaseljson.py:138-141, WeaselJsonParser_destroy). The class has __enter__/__exit__ (weaseljson.py:135-144) but no __del__ and no weakref.finalize, so a
parser that is garbage-collected without an explicit close()/with block silently leaks its
whole allocation for the life of the process. gc.collect() cannot help: self.p is a plain
integer (ctypes.c_void_p), so nothing in the Python object graph knows that the address has to
be handed to WeaselJsonParser_destroy.
Size of the leak
One parser is Parser3::allocationSize(stackSize) (src/parser3.h:110-113)
for the binding's default stackSize=1024, i.e. ~1.1 KiB (plus allocator overhead) per dropped
parser.
Reproduction
Build the shared library (e.g. g++ -std=c++20 -O2 -fPIC -shared -Iinclude -o build/libweaseljson.so src/lib.cpp, which is where the binding looks), then run from the repo
root. VmSize is used instead of VmRSS because the leaked pages are only partly touched:
# leak_demo.py -- python3 leak_demo.py <repo-root>importgc,syssys.path.insert(0,sys.argv[1])importweaseljsondefvm_kb():forlineinopen("/proc/self/status"):ifline.startswith("VmSize"):returnint(line.split()[1])classNoop(weaseljson.WeaselJsonCallbacksBase):passN=20000base=vm_kb()print("baseline VmSize = %6d KiB"%base)for_inrange(N):p=weaseljson.WeaselJsonParser(Noop())# createp.parse(b'{"a":1}')# usedelp# drop without close()gc.collect()leak=vm_kb()-baseprint("after %d unclosed +%d KiB (%.2f KiB per dropped parser)"%(N,leak,leak/N))base=vm_kb()for_inrange(N):p=weaseljson.WeaselJsonParser(Noop())p.parse(b'{"a":1}')p.close()# released explicitlygc.collect()print("after %d closed +%d KiB"%(N,vm_kb()-base))
Observed (aarch64, GCC 16.1, CPython 3.14):
baseline VmSize = 246112 KiB
after 20000 unclosed +25600 KiB (1.28 KiB per dropped parser)
after 20000 closed +0 KiB
The first loop grows by 20000 * 1118 bytes (malloc rounds each block up), and those parsers can
never be recovered; the second loop, which calls close(), does not grow at all, which isolates
the cause to the missing finalizer.
Expected
Releasing the last Python reference to a WeaselJsonParser should release the parser, the same
way the C++ builder that contrib/schemagen generates does it
(contrib/schemagen/weaseljson_schemagen.py:982):
include/weaseljson.h requires that every parser returned by WeaselJsonParser_create is
eventually passed to WeaselJsonParser_destroy. For the Python wrapper the only alternative to with/close() is a leak, because Python has no other deterministic scope exit.
Impact
Any long-running process that parses documents in a loop without wrapping every parser in a with block (e.g. parser = WeaselJsonParser(cb); parser.parse(chunk); ..., or a per-request
helper that forgets close()) leaks ~1.1 KiB per document, unbounded and silently, until the
process runs out of memory. Nothing in the binding documents that close() is mandatory, and
forgetting it is silent.
Suggested fix
def__del__(self):# May run during interpreter shutdown, when the ctypes library object can# already be gone, hence the guard.try:self.close()exceptException:pass
or, more robustly, create a weakref.finalize(self, self._lib.WeaselJsonParser_destroy, self.p)
in __init__.
(For reference, the other Python-binding issues filed here were about the missing key callback, a
misaligned callback struct, the missing null check on WeaselJsonParser_create and the library
load path; none of them covers object lifetime.)
`weaseljson.py`'s `WeaselJsonParser` allocates the underlying C parser in `__init__`
(`weaseljson.py:109-114`, `WeaselJsonParser_create`), but the only path that ever frees it is
`close()` (`weaseljson.py:138-141`, `WeaselJsonParser_destroy`). The class has
`__enter__`/`__exit__` (`weaseljson.py:135-144`) but no `__del__` and no `weakref.finalize`, so a
parser that is garbage-collected without an explicit `close()`/`with` block silently leaks its
whole allocation for the life of the process. `gc.collect()` cannot help: `self.p` is a plain
integer (`ctypes.c_void_p`), so nothing in the Python object graph knows that the address has to
be handed to `WeaselJsonParser_destroy`.
### Size of the leak
One parser is `Parser3::allocationSize(stackSize)` (`src/parser3.h:110-113`)
```
sizeof(Parser3) + (stackSize + kTransientSymbols) * sizeof(Symbol) = 88 + 1030 = 1118 bytes
```
for the binding's default `stackSize=1024`, i.e. ~1.1 KiB (plus allocator overhead) per dropped
parser.
### Reproduction
Build the shared library (e.g. `g++ -std=c++20 -O2 -fPIC -shared -Iinclude -o
build/libweaseljson.so src/lib.cpp`, which is where the binding looks), then run from the repo
root. `VmSize` is used instead of `VmRSS` because the leaked pages are only partly touched:
```python
# leak_demo.py -- python3 leak_demo.py <repo-root>
import gc, sys
sys.path.insert(0, sys.argv[1])
import weaseljson
def vm_kb():
for line in open("/proc/self/status"):
if line.startswith("VmSize"):
return int(line.split()[1])
class Noop(weaseljson.WeaselJsonCallbacksBase):
pass
N = 20000
base = vm_kb()
print("baseline VmSize = %6d KiB" % base)
for _ in range(N):
p = weaseljson.WeaselJsonParser(Noop()) # create
p.parse(b'{"a":1}') # use
del p # drop without close()
gc.collect()
leak = vm_kb() - base
print("after %d unclosed +%d KiB (%.2f KiB per dropped parser)" % (N, leak, leak / N))
base = vm_kb()
for _ in range(N):
p = weaseljson.WeaselJsonParser(Noop())
p.parse(b'{"a":1}')
p.close() # released explicitly
gc.collect()
print("after %d closed +%d KiB" % (N, vm_kb() - base))
```
Observed (aarch64, GCC 16.1, CPython 3.14):
```
baseline VmSize = 246112 KiB
after 20000 unclosed +25600 KiB (1.28 KiB per dropped parser)
after 20000 closed +0 KiB
```
The first loop grows by `20000 * 1118` bytes (malloc rounds each block up), and those parsers can
never be recovered; the second loop, which calls `close()`, does not grow at all, which isolates
the cause to the missing finalizer.
### Expected
Releasing the last Python reference to a `WeaselJsonParser` should release the parser, the same
way the C++ builder that `contrib/schemagen` generates does it
(`contrib/schemagen/weaseljson_schemagen.py:982`):
```cpp
~RootBuilder() { if (parser_) WeaselJsonParser_destroy(parser_); }
```
`include/weaseljson.h` requires that every parser returned by `WeaselJsonParser_create` is
eventually passed to `WeaselJsonParser_destroy`. For the Python wrapper the only alternative to
`with`/`close()` is a leak, because Python has no other deterministic scope exit.
### Impact
Any long-running process that parses documents in a loop without wrapping every parser in a
`with` block (e.g. `parser = WeaselJsonParser(cb); parser.parse(chunk); ...`, or a per-request
helper that forgets `close()`) leaks ~1.1 KiB per document, unbounded and silently, until the
process runs out of memory. Nothing in the binding documents that `close()` is mandatory, and
forgetting it is silent.
### Suggested fix
```python
def __del__(self):
# May run during interpreter shutdown, when the ctypes library object can
# already be gone, hence the guard.
try:
self.close()
except Exception:
pass
```
or, more robustly, create a `weakref.finalize(self, self._lib.WeaselJsonParser_destroy, self.p)`
in `__init__`.
(For reference, the other Python-binding issues filed here were about the missing key callback, a
misaligned callback struct, the missing null check on `WeaselJsonParser_create` and the library
load path; none of them covers object lifetime.)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
weaseljson.py'sWeaselJsonParserallocates the underlying C parser in__init__(
weaseljson.py:109-114,WeaselJsonParser_create), but the only path that ever frees it isclose()(weaseljson.py:138-141,WeaselJsonParser_destroy). The class has__enter__/__exit__(weaseljson.py:135-144) but no__del__and noweakref.finalize, so aparser that is garbage-collected without an explicit
close()/withblock silently leaks itswhole allocation for the life of the process.
gc.collect()cannot help:self.pis a plaininteger (
ctypes.c_void_p), so nothing in the Python object graph knows that the address has tobe handed to
WeaselJsonParser_destroy.Size of the leak
One parser is
Parser3::allocationSize(stackSize)(src/parser3.h:110-113)for the binding's default
stackSize=1024, i.e. ~1.1 KiB (plus allocator overhead) per droppedparser.
Reproduction
Build the shared library (e.g.
g++ -std=c++20 -O2 -fPIC -shared -Iinclude -o build/libweaseljson.so src/lib.cpp, which is where the binding looks), then run from the reporoot.
VmSizeis used instead ofVmRSSbecause the leaked pages are only partly touched:Observed (aarch64, GCC 16.1, CPython 3.14):
The first loop grows by
20000 * 1118bytes (malloc rounds each block up), and those parsers cannever be recovered; the second loop, which calls
close(), does not grow at all, which isolatesthe cause to the missing finalizer.
Expected
Releasing the last Python reference to a
WeaselJsonParsershould release the parser, the sameway the C++ builder that
contrib/schemagengenerates does it(
contrib/schemagen/weaseljson_schemagen.py:982):include/weaseljson.hrequires that every parser returned byWeaselJsonParser_createiseventually passed to
WeaselJsonParser_destroy. For the Python wrapper the only alternative towith/close()is a leak, because Python has no other deterministic scope exit.Impact
Any long-running process that parses documents in a loop without wrapping every parser in a
withblock (e.g.parser = WeaselJsonParser(cb); parser.parse(chunk); ..., or a per-requesthelper that forgets
close()) leaks ~1.1 KiB per document, unbounded and silently, until theprocess runs out of memory. Nothing in the binding documents that
close()is mandatory, andforgetting it is silent.
Suggested fix
or, more robustly, create a
weakref.finalize(self, self._lib.WeaselJsonParser_destroy, self.p)in
__init__.(For reference, the other Python-binding issues filed here were about the missing key callback, a
misaligned callback struct, the missing null check on
WeaselJsonParser_createand the libraryload path; none of them covers object lifetime.)