(prototype) One SimCLR example, one benchmark, and the gate between them - #2038
(prototype) One SimCLR example, one benchmark, and the gate between them#2038gabrielfruet wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da85567260
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| with no_grad(): | ||
| features = self.backbone.embed(sample.views[0].data) |
There was a problem hiding this comment.
For both shipped ResNet settings, the backbone remains in training mode here, and no_grad() disables autograd but not BatchNorm buffer updates. Every training batch therefore updates the encoder's running statistics once with the fused views and a second time with only view 0; validation, kNN scores, and saved checkpoints use these skewed statistics, so the online probe changes the encoder being benchmarked. Reuse detached features from the fused forward or perform this probe pass without updating BatchNorm state.
Useful? React with 👍 / 👎.
| num_workers=args.num_workers, | ||
| ), | ||
| # Resume is one argument. | ||
| ckpt_path="last" if not args.fast_dev_run else None, |
There was a problem hiding this comment.
When the documented benchmark command is restarted after preemption, "last" refers to the current Trainer callback's recorded last_model_path, which is empty in a fresh process. In addition, the unversioned TensorBoardLogger selects a new version_N directory, so the existing version_(N-1)/checkpoints/last.ckpt is not discovered and training starts again from epoch zero. Use a stable checkpoint directory/version or explicitly resolve the existing last.ckpt before calling fit.
Useful? React with 👍 / 👎.
gabrielfruet
changed the title
1 participant
6 of 7 in a stack. Base: #2037.
SimCLR exists four times in this tree and nothing compares the copies. The two shipped heads differ by 4,922,112 parameters, so
load_state_dictraises. The split and fused forwards give gradients at cosine similarity 0.0837. Both examples takeNTXentLoss()'s default 0.5 while the benchmark passes 0.1.examples/simclr.pyis plain torch with the loop in view, small enough to run: ResNet-18, CIFAR-10, batch 256.benchmarks/simclr/is the same method on Lightning, parameterised by one row ofdatasets.py, where theimagenetrow is the paper's numbers. Every row states every field, so two rows side by side show every difference.tests/test_simclr_agrees.pyis what makes them one method. It compares the method, not the run: block classes, the view contract, the fused forward, the head's width against the backbone's, and that each file's criterion carries the temperature that file declares. Batch size, backbone, learning rate and temperature values are free to differ.The old four are deleted in PR 7, not here.
Testing: 14 gate cases; mutating either side's forward to run one view at a time fails the gate and names the side.