Skip to content

Make the pipeline results repeatable across runs and thread counts - #2226

Open
servantftransperfect wants to merge 14 commits into
developfrom
dev/determinism
Open

servantftransperfect wants to merge 14 commits into
developfrom
dev/determinism

Conversation

@servantftransperfect

Copy link
Copy Markdown
Contributor

Description

Several nodes gave different results from one run to the next on the same inputs. The main cause was random generators shared by all the threads of a parallel loop (a data race, and RANSAC results that depend on thread scheduling). Other causes were results collected in thread completion order, sorts without a tie-break, and files read in directory order.

This PR makes these nodes repeatable for a given thread count, and in most cases independent of it.

Features list

  • [numeric] Add createSeededGenerator(baseSeed, ids...): a generator for one work item (image pair, track), seeded from a base seed drawn once outside the parallel loop.
  • FeatureExtraction (sift, akaze): features are kept in keypoint order instead of thread completion order. dspsift already was. Adds a features_sift test comparing 1 and 4 threads.
  • FeatureMatching, GeometricFilterEstimating: one generator per image pair in robustModelEstimation.
  • RelativePoseEstimating: one generator per pair (the result no longer depends on thread count or chunking). Pairs are processed in bounded batches and written in pair order; the output format is unchanged.
  • SfmBootstrapping, NodalSfM: pairs are sorted after loading, and best pair / best view picks break ties on the view ids.
  • GlobalRotationEstimating, PanoramaRefining: pairs are sorted after loading (directory order is unspecified).
  • PanoramaSeams: tiles are ordered by their number, so imageCounter and the new view ids are repeatable.
  • StructureFromMotion, SfMTriangulation: one generator per track in the multi-view triangulation; next best views break ties on the view id.
  • SfmExpanding: one generator per track in the triangulation; both expansion policies break ties on the view id; resections are applied in view id order (with rigs, the last finished thread used to set the shared pose).
  • GlobalSfM: one generator per pose pair for the relative rotations, sorted before the rotation averaging.
  • LdrToHdrSampling, LdrToHdrCalibration: fixed seed instead of std::random_device.

Implementation remarks

  • RANSAC now draws different numbers than before, so results are not bit-identical to previous versions. They were not repeatable before either.
  • Still not repeatable:
    • Meshing / LidarMeshing: the default seed of 0 means std::random_device, the helper points noise uses a shared generator, and the graph weights are float sums accumulated from several threads.
    • GlobalSfM translation triplets: the edge coverage is greedy across threads (and reads the shared set without a lock), so which triplets are solved depends on scheduling.
    • KeyframeSelection: writes a random Exif:ImageUniqueID on purpose.
  • Across machines with different core counts, multithreaded Ceres (bundle adjustment, rotation and position averaging) may still differ in the last bits. Using the same maxCores on every machine avoids it.

Tests

  • All 109 ctest tests pass (OpenMP enabled), including the new features_sift.
  • Not done yet: two-run output comparisons on a real dataset with different --maxCores.

🤖 Generated with Claude Code

Fabien Servant and others added 14 commits October 27, 2026 15:21
Creates a generator dedicated to one work item (an image pair, a track...)
from a base seed and the item ids, so that parallel loops can draw random
numbers without depending on thread scheduling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Descriptors were appended under a critical section in thread completion
order, then sorted by scale with a non-stable sort. Features with the same
scale (one keypoint with several orientations) ended up in a different order
on each run, and the grid filtering could keep different ones.

Store the results per keypoint and append them in keypoint order. Add a test
checking that the output is the same with 1 and 4 threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The subpixel refinement appended the stable keypoints in thread completion
order. Flag them in parallel and keep them in input order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All the threads shared the same generator (a data race), so the RANSAC of a
pair depended on thread scheduling. Draw a base seed once and seed one
generator per image pair. Makes FeatureMatching and GeometricFilterEstimating
repeatable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eads

Each thread had its own generator, all seeded identically, so a pair result
depended on the thread count and on the pairs processed before by the same
thread. The pairs were also written in thread completion order.

Seed one generator per pair, and process the pairs by bounded batches which
are written in pair order. The output format is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pairs files were read in directory order, and findBestPair kept the first
pair reaching the best score in thread completion order. Sort the loaded
pairs and break score ties on the view ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The next view with the most landmarks was picked in thread completion order
on equal counts, and the pairs (read in directory order) were sorted by score
only. Break both ties on the view ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
globalRotationEstimating and panoramaRefining read the pairs files in
directory order, which is unspecified, and the pairs order changes the solver
input. Sort them on the view ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tiles were numbered (imageCounter, used in the new view ids) in directory
order. Key them by the tile number parsed from the file name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atable

The multi-view LO-RANSAC triangulation used the engine generator from all the
threads. Seed one generator per track instead.

The candidate next views were collected in thread completion order and sorted
on the score only. Break ties on the view id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All the threads shared the caller generator. Draw a base seed once and seed
one generator per track.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expansion policies collected the view scores in thread completion order
and did not break ties. ExpansionChunk added the resected poses in thread
completion order, so with rigs the last finished view set the shared pose.
Break ties on the view id and sort the resections by view id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relative poses were estimated with the engine generator shared by all the
threads, and collected in thread completion order. Seed one generator per pose
pair and sort the rotations before the rotation averaging.

The translation triplets estimation is still scheduling dependent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The samples subsets were selected with a generator seeded from
std::random_device. Use a fixed seed, and sort the merged per-thread lists
before shuffling them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant