Skip to content

Share pointer children#1284

Draft
frenchy64 wants to merge 56 commits into
metosin:masterfrom
frenchy64:eagerly-parse-registry
Draft

Share pointer children#1284
frenchy64 wants to merge 56 commits into
metosin:masterfrom
frenchy64:eagerly-parse-registry

Conversation

@frenchy64

@frenchy64 frenchy64 commented May 1, 2026

Copy link
Copy Markdown
Collaborator

Schemas like [:schema {:registry {::FOO :bar}} ::FOO] now create :bar once (before, twice). This is regardless of how many times ::FOO is mentioned in the registry or body

For example it also holds for

[:schema {:registry {::BAZ ::FOO ::FOO ::BAR ::BAR :bar}} [:tuple ::BAZ ::BAZ ::BAZ]]

which would previously create 3 copies of :bar via -property-registry, then 3 additional copies via [:tuple ::BAZ ::BAZ ::BAZ] for a total of 6 copies (now 1).

The situation is more subtle for nested registries, see the tests.

This tackles exponential duplication of schemas. Previously, (m/schema [:schema {:registry exponential-schema}} ::creates-4194304-validators]) would create at least 4,194,304 copies of ::counting, now only 1. I'm not sure of the exact number because of OOM errors, but it could be closer to 20 million.

(def exponential-registry
  {::creates-1-validator ::counting
   ::creates-2-validators [:tuple ::creates-1-validator ::creates-1-validator ::creates-1-validator ::creates-1-validator]
   ::creates-16-validators [:tuple ::creates-2-validators ::creates-2-validators ::creates-2-validators ::creates-2-validators]
   ::creates-64-validators [:tuple ::creates-16-validators ::creates-16-validators ::creates-16-validators ::creates-16-validators]
   ::creates-256-validators [:tuple ::creates-64-validators ::creates-64-validators ::creates-64-validators ::creates-64-validators]
   ::creates-1024-validators [:tuple ::creates-256-validators ::creates-256-validators ::creates-256-validators ::creates-256-validators]
   ::creates-4096-validators [:tuple ::creates-1024-validators ::creates-1024-validators ::creates-1024-validators ::creates-1024-validators]
   ::creates-16384-validators [:tuple ::creates-4096-validators ::creates-4096-validators ::creates-4096-validators ::creates-4096-validators]
   ::creates-65536-validators [:tuple ::creates-16384-validators ::creates-16384-validators ::creates-16384-validators ::creates-16384-validators]
   ::creates-262144-validators [:tuple ::creates-65536-validators ::creates-65536-validators ::creates-65536-validators ::creates-65536-validators]
   ::creates-1048576-validators [:tuple ::creates-262144-validators ::creates-262144-validators ::creates-262144-validators ::creates-262144-validators]
   ::creates-4194304-validators [:tuple ::creates-1048576-validators ::creates-1048576-validators ::creates-1048576-validators ::creates-1048576-validators]})

Note: (m/validator [:schema {:registry exponential-schema}} ::creates-4194304-validators]) still calls m/-validator 4,194,304 times, as noted in the TODO at the bottom of malli.core-test in this PR. This can be tackled separately.

Metabase works around this a similar issue here by creating their own :ref implementation. We don't handle :ref here yet.

@frenchy64

Copy link
Copy Markdown
Collaborator Author

After some reflection, I think the essence of this change is an integrated mechanism to detect recursive schemas. We endlessly work around this missing feature in order to detect cycles in each operation. If malli intrinsically handled recursive schemas to always be equal, then these cycle checks become simply equality checks.

This needs to be a user-level expectation, with documentation explaining the scenarios where the knot is tied at the schema level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ⌛Waiting

Development

Successfully merging this pull request may close these issues.

3 participants