Our entry to the DrivenData DaT Parkinson's Challenge: a registration-based imaging pipeline, a small ensemble of 2D networks, and about sixty experiments — most of which didn't work.
A DaT SPECT scan shows where a radioactive tracer binds to dopamine transporters. In a healthy brain it collects in the striatum as two bright, comma-shaped structures. In Parkinsonian syndromes the tail of that comma (the putamen) fades first.
The challenge supplied 1,362 labelled scans from ten French hospitals, each read by experts as normal or abnormal. Submissions were code that ran offline on hidden test scans and returned a probability for each; the score was log loss. Scans came from many cameras, with voxel sizes from 1.4 mm to 4.4 mm and more than 90 combinations of matrix size and spacing.
We finished 10th on the private leaderboard, up from 15th on the public one, with a best private log loss of 0.2760. Every submission scored worse on the public test scans than in local cross-validation, and about 0.04 worse again on the private scans; other teams saw the same. The order of our submissions barely changed from one board to the other, so the improvements were real: 0.2851 to 0.2760 on the private board from the first submission to the last.
Log loss, lower is better. The dashed line is the private-leaderboard winner.
| Submission | What changed | Local CV | Public LB | Private LB |
|---|
The two biggest leaderboard moves came from strong augmentation (v3) and the slice-sequence network (v7). Changes that only averaged more copies of existing models barely moved the public score. The last upload, final A, gave the slice-sequence model twice the weight and was the best submission on both boards.
Places 4 to 10 were separated by less than 0.004. Our AUROC was the second highest in the top ten: the models ranked scans well, but on the harder private scans they were too confident on the cases they got wrong, and log loss punishes that heavily.
| Rank | Team | Private log loss | AUROC |
|---|
One shared library does the preprocessing for both training and inference, so the two can't drift apart. Every test scan is processed on its own, as the rules require.
Nine 4 mm axial slices, from 16 mm below to 16 mm above the striatal peak, each pass through a shared ImageNet EfficientNet-B0. A bidirectional GRU reads them in order and gated attention decides which slices matter. Each slice carries three channels: background-normalised, self-normalised, and a 12 mm local context.
The final model averages 135 CNN checkpoints and 90 tabular models, bagged over two fold splits and 10-fold runs. One explicit weight table produces both the stacker's training inputs and the deployed model, and an assertion checks that the two agree.
Five changes improved the model by more than run-to-run noise. Each fixed a specific weakness rather than adding capacity.
0.2319 → 0.2153 in local CV. Aligning every scan to a common template lets us measure the caudate and putamen separately, the same quantities expert readers judge. The single best feature, worst-side posterior putamen uptake, reached an AUROC of 0.92 alone.
0.3529 → 0.2716 for CNNs on scanner types they never saw. Following Buddenkotte and Buchert (J Nucl Med, 2024), blurring and adding noise far beyond realistic levels taught the networks to ignore camera resolution. It was the change that moved the leaderboard most.
0.2372 → 0.2249 for a single CNN. Extra channels divided by each scan's own 99th percentile give a second view of intensity. A mathematically "cleaner" version that removed all dependence on the background estimate did worse, so the original design stayed.
0.2014 → 0.1988 for the ensemble, and −0.0039 on the public leaderboard. Averaging three thick slabs throws away how uptake changes from bottom to top; reading thin slices in order keeps it. It is also less correlated with the other CNNs (0.94), so it adds new information.
about −0.002 each time. Training the whole ensemble again on a different partition and averaging reduced variance, with a small but real leaderboard gain.
Log loss with random folds and with whole scanner types held out. Lower is better.
| Model | Random folds | Held-out scanners |
|---|
Each idea was tested against the current ensemble on the same folds and seeds. The chart shows the change in ensemble log loss; bars to the left helped, bars to the right hurt.
Change versus the ensemble at the time it was tested. Negative is better.
| Change | Δ log loss | Verdict |
|---|
| Idea | Result | Why it failed |
|---|---|---|
| Temperature, affine and beta calibration | ±0.0001 | The logistic stacker is already calibrated. A calibration submission also scored 0.2408 against 0.2405. |
| Probability floors (0.002–0.01) | +0.0005 to +0.0030 | Confident predictions were almost always right, so capping them only cost points. |
| Averaging probabilities instead of logits | +0.0057 | The stacker learns better weights than a fixed blend. |
| 3D CNN, tri-plane views, ConvNeXt, 256 px inputs | +0.0003 to +0.0019 | At SPECT resolution (~8–10 mm) more pixels or views add noise, not information. |
| Hemisphere MIL and learned bilateral comparison | +0.0000 / +0.0005 | Splitting at the midline removes the context the network uses to compare sides. |
| Image + measurement fusion network | +0.0010 | The best single model (0.2234), but it duplicates what the tabular models already give the stacker. |
| Regional auxiliary targets | +0.0006 | The extra loss pulled capacity away from the main task. |
| Rigid-only registration, wider field of view | +0.0009 | Affine scaling to the template carries real signal; extra cortex is mostly noise. |
| Multi-template registration | worse | The template a scan picks already depends on its label, which distorts the regions for exactly the borderline cases. |
| Frozen DINOv2 features | −0.0002 | A different representation (AUROC 0.93 alone), but not complementary enough to matter. |
| External pretrained DaT networks | no gain | Their predictions correlated 0.93 with ours. PPMI-trained checkpoints were also barred from prizes. |
| SAM, frozen BatchNorm, per-scan blur, mixup, label smoothing | ±0.0000 to +0.0005 | Training tweaks on an already heavily regularised recipe. |
| Training for 50 epochs instead of 25 | +0.0014 | Even with strong augmentation the networks overfit past 25 epochs. |
| Model soup (weight averaging) | much worse | Independently seeded runs don't share a loss basin. |
| Disagreement-based shrinkage, label-noise weighting | +0.0031 / noise | Where the families disagree, the stacker's blend is already the best guess. |
Two identical training runs differed by 0.004 in log loss. Several "improvements" were smaller than that and vanished once three seeds were averaged. Comparisons were made paired, on the same folds and seeds, with a bootstrap of the per-scan differences.
Holding out whole scanner types reproduced one public score almost exactly (0.2444 against 0.2446) and correctly flagged strong augmentation as a big win. It over-predicted later gains, so it guided choices rather than replacing submissions.
An external review found the deployed ensemble weighted two runs 2:1 while validation weighted them 1:1. The effect was negligible (0.2015 against 0.2014), but it led to one weight table that drives both, with an assertion. An evaluation leak in a bagged comparison also once produced a fake 0.15; results that look too good get checked first.
A label-noise simulation predicted that calibration would gain 0.009 on the leaderboard. The real submission lost 0.0003 on the public board. From then on, a change needed a measured local gain before it earned a submission.
The same calibrated submission turned out better than its uncalibrated twin on the private board (0.2794 against 0.2806). Softer probabilities help when the test scans are harder than the training scans. With a small public set, it is worth keeping one mildly calibrated variant among the final submissions even when the public score doesn't reward it.
The repository has the full code, the nine training configurations behind the final ensemble, and one script that rebuilds and verifies it.
# 1. put the competition data in data/niftis/, data/train_labels.csv and data/smoke/
# 2. Python 3.12, PyTorch for your CUDA version, then:
pip install -r requirements.txt
# 3. preprocess, train, bag and verify (about 5 hours on one 8 GB GPU)
bash reproduce.shEvery submission was checked the same way before upload: a clean unzip, a run on the organisers' smoke data, a forced failure of every preprocessing worker, a parity check against the training pipeline, and an import under the competition's Python 3.12 package versions.
The repository and this page contain code and whole-dataset aggregate metrics only. No scans, labels, per-scan predictions, derived features or trained weights are published. The only pretrained weights are ImageNet checkpoints from timm.