Pruning batches

[hacheigriega]

Explanation of Changes

Prune batches and their associated data at every block based on two new module parameters NumBatchesToKeep and MaxBatchPrunePerBlock.

See #667 for pruning data results.

See the branch hy/batch-prune-opt for further optimization attempts.

Testing

Added an integration test

Related PRs and Issues

Related #663

[hacheigriega]

Using Clear instead of Remove for deleting batch tree entries and signatures gives significant performance improvement because it doesn't involve unmarshalling every key.

However, I have not been able to find a way to use Clear to delete batches themselves due to lack of corresponding API for the IndexedMap type ..

Benchmark of pruning 10000 batches (Remove batch data vs Clear batch data):

BenchmarkBatchPruning-10               1          2880832958 ns/op  1590882176 B/op  31168956 allocs/op
BenchmarkBatchPruning-10               1        136149743375 ns/op  6618547616 B/op  33323126 allocs/op

Pruning 100 batches out of 2000 batches takes over 1 second.

BenchmarkBatchPruning-10               1        1104534583 ns/op        620941184 B/op  10755396 allocs/op

Tally end block of 100 tallies is 0.1 seconds according to the benchmark.

BenchmarkDataRequestFlow-10            9         122426375 ns/op        60334072 B/op     584187 allocs/op

[Thomasvdam]

If we clear the data tree entries how will we be able to figure out which data results can be pruned?

Oh wait we store results under the DR ID and height, it's difficult regardless :P

[hacheigriega]

I have tried a few different strategies, including one where I kept the first batch number as a state to completely remove the need for Iterate and only use Clear. Unfortunately there were no performance improvements. It seems Clear only brings significant performance gains when we are dealing with composite keys like validator tree entries and batch signatures, where we have a lot of entries saved under one batch number prefix.

Will do a little more digging on how exactly Clear can be efficient with composite keys, but for now the first commit a143e6a seems to be the best performance Clear can give us.