feat: Add migration backfill for mptoken_issuance_history

[BryanJ1ang]

Note: this PR depends on previous ETL indexing PR here

Summary

Adds MPTTransactionHistoryMigrator, a Cassandra migration that backfills historical MPT issuance transaction history into the new MPT index tables:

• mptoken_issuance_transactions
• account_mptoken_issuance_transactions

The migrator full-scans the transactions table, reuses the live ETL MPT extractor, writes the same deterministic index rows produced by live indexing, and waits for queued async writes before marking the migration as complete.

Details

• Registers MPTTransactionHistoryMigrator with the Cassandra migration manager.
• Uses migration scan settings for full-table scanning:
  • migration.full_scan_threads
  • migration.full_scan_jobs
  • migration.cursors_per_job
• Adds Cassandra driver paging support so token-range scans read all result pages.
• Makes migration scans select explicit columns instead of relying on table column order.
• Propagates full-table scanner worker failures so failed scans do not get marked migrated.
• Keeps the migration non-blocking for Clio startup.

Testing

Added coverage for:

• Migrator status before and after running.
• Backfilling MPT issuance create paths and multi-issuance transaction fan-out.
• Idempotent reruns.
• Empty transaction tables.
• Parity between migration backfill rows and live ETL rows.
• Failed transactions with explicit MPT issuance references.
• Cassandra paging across more than one driver page.
• Full-table scanner validation and worker error propagation.

[codecov]

Codecov Report

❌ Patch coverage is 38.09524% with 65 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../migration/cassandra/CassandraMigrationBackend.hpp	0.00%	34 Missing ⚠️
...ration/cassandra/MPTTransactionHistoryMigrator.cpp	0.00%	25 Missing ⚠️
...c/migration/cassandra/impl/TransactionsAdapter.cpp	72.72%	1 Missing and 2 partials ⚠️
src/migration/MigrationApplication.cpp	60.00%	1 Missing and 1 partial ⚠️
src/main/Main.cpp	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[BryanJ1ang]

/ai-reviewer

[Kassaking7]

You may also want to add missing unit tests based on the codecov report

[BryanJ1ang]

You may also want to add missing unit tests based on the codecov report

Added some unit tests but for CassandraMigrationBackend.hpp MPTTransactionHistoryMigrator.cpp and MigrationApplication.cpp these can't be unit tested since they need a Cassandra db. They are covered by integration tests.

[BryanJ1ang]

@PeterChen13579 @yinyiqian1 I can't add you guys as reviewers but please take a look when you guys can!

[Kassaking7]

LGTM

[Copilot]

Pull request overview

Adds a Cassandra migration to backfill historical MPToken issuance transaction history into the new MPT index tables, reusing the existing live ETL extractor and improving migration full-table scan robustness (paging + error propagation).

Changes:

• Introduces MPTTransactionHistoryMigrator and registers it with the Cassandra migration manager.
• Enhances full-table scanning: explicit column selection via kSelectColumns, Cassandra driver paging support, and propagation of worker/read failures.
• Adds/extends unit + integration coverage around scanner behavior, adapter decoding failures, migrator status semantics, and parity with live ETL indexing.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/util/BatchingTests.cpp	Adds unit coverage for util::BatchBuffer batching behavior and concurrency.
tests/unit/migration/MigratorRegisterTests.cpp	Verifies failed migrators don’t write “Migrated” status.
tests/unit/migration/MigrationApplicationTests.cpp	Adds unit tests for migrator CLI/application behavior via a mock manager.
tests/unit/migration/cassandra/TransactionsAdapterTests.cpp	Tests transaction row deserialization and decode-failure signaling.
tests/unit/migration/cassandra/SpecTests.cpp	Updates table spec test to include explicit select-column lists.
tests/unit/migration/cassandra/FullTableScannerTests.cpp	Updates tests for new scanner wait/error behavior and adds new validations.
tests/unit/CMakeLists.txt	Registers new unit test sources.
tests/integration/migration/cassandra/MPTTransactionHistoryMigratorTests.cpp	End-to-end integration coverage for MPT backfill, idempotency, paging, and ETL parity.
tests/integration/migration/cassandra/ExampleTransactionsMigrator.cpp	Updates example migrator to new adapter + scanner wait API.
tests/integration/migration/cassandra/ExampleObjectsMigrator.cpp	Updates example migrator to new scanner wait API.
tests/integration/migration/cassandra/CassandraMigrationManagerTests.cpp	Adds integration test to ensure migration scans read all driver pages.
tests/integration/data/cassandra/BaseTests.cpp	Adds base Cassandra test for paging-state iteration across pages.
tests/integration/CMakeLists.txt	Registers new integration test source.
tests/common/util/MockMigrationManager.hpp	Adds a mock MigrationManagerInterface for application-layer tests.
tests/common/migration/TestMigrators.hpp	Adds a throwing migrator used to validate failure semantics.
src/util/Batching.hpp	Adds util::BatchBuffer for streaming batching + flush semantics.
src/migration/MigrationApplication.hpp	Adds constructor overload allowing injection of a prebuilt migration manager (tests).
src/migration/MigrationApplication.cpp	Avoids double Prometheus init and implements injected-manager constructor.
src/migration/CMakeLists.txt	Links migration library against clio_etl to reuse extractor for backfill.
src/migration/cassandra/MPTTransactionHistoryMigrator.hpp	Declares new backfill migrator for historical MPT issuance tx indexing.
src/migration/cassandra/MPTTransactionHistoryMigrator.cpp	Implements full-table scan + batching + decode-failure handling + durable completion.
src/migration/cassandra/impl/TransactionsAdapter.hpp	Adds explicit select columns + decode-failure callback + logging support.
src/migration/cassandra/impl/TransactionsAdapter.cpp	Implements deserialization with controlled handling of decode failures.
src/migration/cassandra/impl/Spec.hpp	Requires kSelectColumns and enforces tuple/column-count alignment.
src/migration/cassandra/impl/ObjectsAdapter.hpp	Adds explicit kSelectColumns matching row tuple order.
src/migration/cassandra/impl/FullTableScanner.hpp	Adds settings validation and replaces silent waits with error-propagating join.
src/migration/cassandra/impl/CassandraMigrationSchema.hpp	Prepares full-scan statements using explicit column lists.
src/migration/cassandra/CassandraMigrationManager.hpp	Registers MPTTransactionHistoryMigrator for Cassandra migrations.
src/migration/cassandra/CassandraMigrationBackend.hpp	Adds prepared-statement caching, paging-size/state handling, and fail-closed scan errors.
src/main/Main.cpp	Initializes Prometheus in the migrate CLI path before creating the migrator app.
src/etl/impl/ext/MPT.cpp	Simplifies issuance-index row counting (removes per-tx warning threshold logic).
src/data/cassandra/impl/Statement.hpp	Adds paging-size/state helpers and centralizes error throwing.
src/data/cassandra/impl/Result.hpp	Adds hasMorePages() API.
src/data/cassandra/impl/Result.cpp	Implements hasMorePages().
src/data/cassandra/impl/Collection.hpp	Switches to shared Cassandra error helper (removes local throw helper).
src/data/cassandra/Error.hpp	Adds shared throwErrorIfNeeded helper used by statement/binding/paging APIs.

Comments suppressed due to low confidence (1)

src/util/Batching.hpp:111

• Same capacity issue as in add(): std::exchange(buffer_, {}) drops buffer_ capacity, so the next accumulation may reallocate. Reserving batchSize_ after clearing helps keep allocation behavior stable across flush cycles.

            std::scoped_lock const lock{mutex_};
            batch = std::exchange(buffer_, {});
        }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[PeterChen13579]

Just realized you have to merge with latest develop which updates to the newest libxrpl. Make sure to do a clean build. I will continue to review after this as merging with latest develop will change a lot

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/migration/MigrationApplication.cpp:48

• The injecting constructor accepts a potentially null migrationManager; run()/printStatus()/migrate() dereference migrationManager_ unconditionally, so constructing with nullptr would crash. Add a defensive check (throw/ASSERT) here to fail fast with a clear error.

MigratorApplication::MigratorApplication(
    MigrateSubCmd command,
    std::shared_ptr<migration::MigrationManagerInterface> migrationManager
)
    : migrationManager_(std::move(migrationManager)), cmd_(std::move(command))
{
}