Upgrading DataDescription which lacks values that modern schema requires

[Sun-flow]

How do we want to handle upgrades for data descriptions that lack relevant data which we now require?

I'm using data_description_upgrade and seeing that, in the new PydanticV2 implementation of data_description, the investigators and funding_source fields have a minimum size. When testing the upgrader though, multiple of the test examples on CO-test lack investigators/funders to be upgraded. This throws an error, which I think might also bin the entire upgrade, and is probably worse functionality than upgrading the description with a hole in the data.

Just wanted to open some discussion on how we think this sort of thing should be handled. Here is a screencap of the error thrown by the upgrader at runtime:

[Sun-flow]

Interested in thoughts from @dyf, @saskiad, and @jtyoung84

[jtyoung84]

The three options from what I see are:

• Enforce people to provide that information
• If the validation fails, then allow people to use model_constuct to create a model without validation
• Loosen the restrictions on the fields, i.e., make more fields optional, etc.

[dyf]

I like middle bullet. The upgrader should fail by default when the new model is not valid, but there should be a default-off flag to construct a model anyway.

[saskiad]

I don't understand - doesn't creating a back door for people to avoid requirements?

[saskiad]

These aren't fields that can be optional. If the examples (or real files) are missing this information, we need to get that information from the people who's data it is

[alejoe91]

Let me chime in here. I agree with Saskia: as we stabilize the schema, input data that do not conform to the updated schema will need to be updated accordingly.

Maybe we could have an action that goes through all the data in the bucket and check that they conform to the schema? We could also run this kind of job manually, but having an action to do that (say weekly) will ensure that we spot any discrepancy in a timely manner. Thoughts?

[dyf]

This is a good idea, and we should do this, but it doesn't solve the problem of missing metadata that is now required. That will need to be pulled from primary sources again.

[saskiad]

I think there's a different answer for general cases where we decide now our devices should have a version number that we didn't require in the past, and then this specific case where there needs to be, and absolutely is, an investigator and funding source for each data asset. This was supposed to be required from the start (I have learned some subtle things about how "Optional" works), and not knowing what these are is a fundamental problem.