Pruning data results

[hacheigriega]

Explanation of Changes

The challenge here is that there is no mapping to data result object from batch number or data result ID, which is only data available to us when we prune batches. Therefore, we simply go through MaxDataResultsToCheckForPrune (a module parameter) items in the store and delete those whose associated batches have been pruned.

To prevent checking similar set of data results over multiple blocks, we use the hash of last commit hash as the starting point of this iteration. The iteration direction is also switched at every block.

Note this PR also includes some logic change of batching EndBlock to ensure that pruning is executed at every block and to catch errors from pruning functions without halting the chain.

Testing

An integration test.

Planning to test with actual data exported from testnet or mainnet.

Related PRs and Issues

Closes #663

[Thomasvdam]

I still need to take a good look at the PR, but maybe good to already create a follow up issue for building a mapping that allows us to do this more efficiently in the future. Feels like the pruning isn't complete without that :)

[hacheigriega]

This strategy has two problems:

1. Most data results we check would not be prune-able until batch pruning has made significant progress. This issue can be avoided by starting data result pruning after batch pruning catches up.
2. Data result-related collections (dataResults and batchAssignments) would become mixed with items that support the future simple pruning strategy (PR Legacy data result pruning and basic pruning #673). Therefore, this pruning strategy will leave dangling items, or at least we won't be able to know when this pruning strategy is completed.

An obvious solution would be to add new collections to store incoming data results separately. I'll see if there is a better alternative.

[Thomasvdam]

This strategy has two problems:
[...]
An obvious solution would be to add new collections to store incoming data results separately. I'll see if there is a better alternative.

While not ideal I do agree that this might make the most sense. This could also open up new pruning strategies. One stupid idea I just had:

• Create new data storage that allows for writing, reading, and pruning according to the specifications we have now.
• We store the height at which the new store became active
• When reading we check both stores (or do something smart to determine correct store based on the activation height)
• When the current_height - activation_height > retention_window we know all results in the old store are prunable and can simply remove the first X every block

Not a great solution since we have some temporary code for the transition period (although I think this is required for all workable solutions) and it doesn't address the size problem until some time beyond the upgrade. But maybe this sparks a good idea on your side :)

[hacheigriega]

Closing as PR #673 now supersedes this PR.