Setting `__dict__` breaks PEP 412

[psippola]

In pydantic v1, replacing each instantiated object's __dict__ with a dict of validated values used to broke
PEP 412 (use of shared dictionary keys), which probably explains a part of the excessive memory usage of V1 when a lot of objects of same type were created.

This may be a concern still in v2, if I read rows like this correctly:

force_setattr(py, model, intern!(py, DUNDER_DICT), validated_dict)?;

I have no adequate rust skills to test this, but it could be beneficial to study how shared keys affect memory consumption of objects validated by pydantic, and could updating __dict__ instead of replacing result in a smaller memory footprint.

More about shared keys, I hope that this is still relevant:

Internals of sets and dicts

[Viicos]

Hi @psippola,

it is hard to say whether Pydantic leverages PEP 412 or not. To investigate more, I did some memory profiling using Memray. First, by attesting the improvement from PEP 412:

The following script:

# Profile using `memray run -o pep412.bin script.py && memray flamegraph pep412.bin -o pep412.html && xdg-open pep412.html`
dict_list = []

for _ in range(1_000_000):
    dict_list.append(
        {
            'a': 1,
            'b': '2',
            'c': 1,
            'd': 1,
            'e': 2,
            'f': 1,
            'g': 1,
        }
    )

Ends up consuming 350MB of memory on my end.

Then when populating an instance's __dict__:

class A: ...

a_list = []

for _ in range(1_000_000):
    a = A()
    dct = {}
    dct['a'] = 1
    dct['b'] = '2'
    dct['c'] = 1
    dct['d'] = 1
    dct['e'] = 2
    dct['f'] = 1
    dct['g'] = 1
    a.__dict__ = dct
    a_list.append(a)

It goes down to 150MB, confirming the optimization from PEP 412 (note that I've arranged the __dict__ creation in distinct steps, to reproduce what pydantic-core is doing under the hood).

Edit: this is strange, and maybe I ran the wrong tests on my end when writing this comment, but the PEP 412 optimization doesn't seem to be used when assigning __dict__ manually (I still get ~400MB). It only does when assigning attributes to a directly (e.g. a.a = 1; a.b = '2'; ...).

But if I then try to do the same using Pydantic models:

from pydantic import BaseModel

class Model(BaseModel):
    a: int
    b: str
    c: int
    d: int
    e: int
    f: int
    g: int

model_list: list[Model] = []

for _ in range(1_000_000):
    model = Model(
        a=1,
        b='2',
        c=1,
        d=1,
        e=2,
        f=1,
        g=1,
    )
    model_list.append(model)

It goes up to 1.2GB. The reason for this is that we populate model_fields_set on each instance (a set of field names that were explicitly provided at instantiation). If I tweak the pydantic-core logic to not populate this attribute (along with extra and private fields), it goes down to 400MB.

It is hard to tell what part of the code is using this memory (whether it is pydantic-core or pydantic). However, it seems that the memory usage is going to be way higher by default using Pydantic because of the model_fields_set, no matter if PEP 412 is being used or not.

[davidhewitt]

It goes up to 1.2GB. The reason for this is that we populate model_fields_set on each instance (a set of field names that were explicitly provided at instantiation). If I tweak the pydantic-core logic to not populate this attribute (along with extra and private fields), it goes down to 400MB.

This seems like a great find and something that we should consider caching if we can get away with using a frozenset for this.

But maybe it's too breaking and the proposed "RustModel" is the way out of both of these problems.

[psippo]

Interestingly, I wasn't able to reproduce the memory savings with the following example (Python 3.12.3, pydantic 2.11.7), but it peaks at 400 + MB. This may be due to conversion to combined-table forms on dict resizing: https://peps.python.org/pep-0412/#split-table-dictionaries

class A: ...

a_list = []

for _ in range(1_000_000):
    a = A()
    dct = {}
    dct['a'] = 1
    dct['b'] = '2'
    dct['c'] = 1
    dct['d'] = 1
    dct['e'] = 2
    dct['f'] = 1
    dct['g'] = 1
    a.__dict__ = dct
    a_list.append(a)

Instead, setting attributes one by one results in 150 + MB:

class A: ...

a_list = []

for _ in range(1_000_000):
    a = A()
    a.a = 1
    a.b = '2'
    a.c = 1
    a.d = 1
    a.e = 2
    a.f = 1
    a.g = 1
    a_list.append(a)

150 MB+ is also achieved by using dataclasses and initialising the values from a temporary dict:

from dataclasses import dataclass

@dataclass
class A:
    a: int
    b: str
    c: int
    d: int
    e: int
    f: int
    g: int

a_list = []

for _ in range(1_000_000):
    dct = {}
    dct['a'] = 1
    dct['b'] = '2'
    dct['c'] = 1
    dct['d'] = 1
    dct['e'] = 2
    dct['f'] = 1
    dct['g'] = 1
    a = A(**dct)
    a_list.append(a)

The problem seems to arise when the dict is dynamically updated after the __init__of the first class instance.

What it comes to model_fields_set, I think that the it could benefit from a similar system of shared hashes / keys as key-sharing dictionaries, as the possible field names are known beforehand.