AI
Transformers for EEG: What Actually Matters
A practical reading of recent papers — what the architectures get right, where the inductive bias still matters.
Why this post
Every week another paper puts a Transformer on EEG. Most report a small gain on one benchmark. As a student trying to learn what actually generalizes, I need a way to read these without being swept up.
What tends to help
- Patch-based tokenization over raw samples — attention over 500 Hz sequences is wasteful.
- Channel-aware attention. EEG channels are not interchangeable; ignoring topology throws away structure.
- Self-supervised pretraining on unlabelled recordings, when the downstream label budget is small.
Where inductive bias still wins
Small convolutional front-ends often match or beat pure Transformer stacks on subject-limited datasets. The bias toward local temporal structure is real physics, not a limitation to overcome.
# Rough sketch of the front-end I keep coming back to
x = TemporalConv(kernel=25)(x) # local temporal features
x = SpatialConv(channels=64)(x) # learn a spatial filter bank
x = Transformer(depth=4)(x) # long-range context on top
Honest baselines
I keep a shallow ConvNet baseline in every experiment. If the Transformer cannot clearly beat it under the same preprocessing and training budget, the result is not real yet.