Neuromorphic Computing · Live demo

Do spikes fail differently?

Damage a gesture recording from an event camera and watch two networks answer the same damaged input: one built from spiking neurons, one from ordinary ones.

An event camera does not take pictures. Each pixel reports only when the brightness in front of it changes, so a still scene is invisible and a moving hand is a cloud of dots. Two networks were trained on the same recordings to name eleven hand and arm gestures. One uses spiking neurons, which fire short pulses and stay silent otherwise. The other is the very same network with ordinary always-on units. Same layers, same number of weights, same data, same training.

Everything below runs live. Pick a gesture, choose how to damage the recording, and the page sends the damaged frames through both networks and reports what each one answered and how sure it was.

DVS128Gesture · 16 frames per recording · PyTorch + SpikingJelly · runs on Modal

Read the study Source Code

The study in four numbers

Headline results, spiking network against the ordinary network
result	spiking	ordinary
Right on undamaged recordings	93.1%	96.5%
Calibration error, lower is better	0.040	0.039
Right under heavy noise, λ = 0.2	91.7%	63.5%
Energy per recording, on paper	3.30 mJ	61.90 mJ

288 test recordings · one training seed · the study puts a 95% interval on every number

The input

Choose a gesture

Eleven gestures, each drawn from one of its recordings: every dot is a pixel that saw the brightness change. These recordings were held out from training, so neither network has seen them before.

where these recordings come from

DVS128Gesture is IBM’s event-camera dataset, published with Amir et al., CVPR 2017: 11 hand and arm gestures performed by 29 people under three lighting conditions, recorded by a 128 by 128 pixel sensor that reports only brightness changes. The standard split holds out whole people, so the 288 test recordings show people the networks never trained on.

SpikingJelly bins each recording’s events into 16 frames, one channel for brightness-up events and one for brightness-down. This bank is 33 of the test recordings, three per gesture: the first three in test-set order that both networks classify correctly when undamaged, so any disagreement you produce here comes from the damage. The thumbnails and the player are those frames shrunk to 64 by 64 for display; the networks always see the full 128 by 128.

The damage

Damage the recording

Four ways to damage it, each modelled on something that goes wrong with a real sensor. Pick one, then how much. Both networks always see the identical damaged frames.

how much · nothing to set, the recording is untouched

running both networks

The recording exactly as the camera saved it. Both networks get every recording on this page right when it is undamaged, so this is the baseline. Pick a kind of damage to break things.

exactly what each kind of damage does to the frames

Damage is applied to the 16 binned frames, identically for both networks, with a random seed fixed by the recording and the setting. The same choice always produces the same damaged frames until you re-roll.

Lose events: every event is kept or dropped by a coin flip, surviving with probability 1 − p.
Add noise: a Poisson draw with mean λ is added to every pixel in every frame, for both polarities, on top of the real events.
Block a patch: one square of s by s pixels, placed at random per recording, is set to zero in all 16 frames.
Shuffle time: the 16 frames are cut into windows of w frames and the frames inside each window are reordered at random. The frames themselves are untouched.

The four settings on the strip are the study’s four severities, so the published curves in step 04 used exactly this damage.

The answers

Watch both networks answer

The player shows the damaged recording the networks are looking at. The chart beside it shows each network making up its mind frame by frame. The two cards are their final answers.

waiting for the first run

waiting for the first run · dark = brightness went up · faint = went down · one dot per pixel that changed

three recordings of hand clapping · # = its number in the test set

How sure, frame by frame

right at that framewrong· line = how sure of its answer so far · hover or arrow keys to read a frame

how the frame-by-frame chart is built

Each network produces one response per gesture, eleven in all. The spiking network’s response is a firing rate: how often that output neuron fired. The chart replays the run one frame at a time. After frame t it averages the responses seen so far, takes the largest as the answer, and divides it by the sum of all eleven to get how sure. The ordinary network answers every frame on its own, so its point at frame t is the average of its first t answers. Frame 16 is the final answer on the cards.

At frame 1 the spiking network has fired only a handful of spikes, so whichever output fired first looks unanimous. That early 1.0 is small-sample noise, not confidence, and it settles within a few frames as spikes accumulate.

Waiting for the first answer.

Spiking network · SNN

Neurons fire short pulses and stay silent otherwise. Its answer is how often each output fired over the 16 frames.

its answer

—

—sure

each gesture's share of the total response · bold = the true gesture

hand clapping
right hand wave
left hand wave
right hand clockwise
right hand counter clockwise
left hand clockwise
left hand counter clockwise
forearm roll backward
drums
guitar
random other gestures

cost: —

Ordinary network · ANN

The same network with always-on units. It answers each frame on its own and averages the 16 answers.

its answer

—

—sure

each gesture's share of the total response · bold = the true gesture

hand clapping
right hand wave
left hand wave
right hand clockwise
right hand counter clockwise
left hand clockwise
left hand counter clockwise
forearm roll backward
drums
guitar
random other gestures

cost: —

energy is an accounting model, not measured power · on this CPU both networks cost the same watts

how “sure” and the bars are calculated

Neither network outputs probabilities. Both were trained with a squared-error loss that pushes the right gesture’s output toward 1 and the other ten toward 0, so each output is a response somewhere between 0 and about 1.3, not a share of a whole. The spiking network’s responses are firing rates, so they never exceed 1; the ordinary network’s can overshoot.

“Sure” is the winning response divided by the sum of all eleven. Each bar is that gesture’s share of the same total, so the bars add up to 100% and the top bar is the “sure” figure. The study deliberately does not use softmax here: on responses in this range it squashes everything toward one eleventh and makes both networks look far less sure than they are.

how the energy figure is calculated

Both figures come from a bookkeeping model, not a power meter. One multiply-accumulate is priced at 4.6 pJ and one plain accumulate at 0.9 pJ, Horowitz’s 2014 figures for 45 nm silicon and the standard prices in the spiking-network literature. The ordinary network does every multiply on every frame whatever the input, so its bill is fixed: multiplies per frame, times 16 frames, times 4.6 pJ, for every layer. A spiking layer only works when an input spike arrives, and a spike triggers an add rather than a multiply, so each layer is billed on how busy its input was in this run: multiplies per frame, times 16, times the fraction of input positions that spiked, times 0.9 pJ.

The first layer is the exception. It sees the analog event-count frames, not spikes, so it is charged at full multiply price for both networks. That one layer is most of the spiking bill, and the choice is deliberately conservative against the spiking network. “Silent” is one minus the average firing rate over every neuron and every frame, weighted by layer size. The table is this run’s bookkeeping, recomputed in the browser from the measured activity.

What the model leaves out, and why the number is “on paper”: it counts arithmetic only, while on a real chip memory traffic usually costs more than arithmetic. It does not bill the spiking network for updating every neuron’s membrane every frame, silent or not. It bills the ordinary network as fully dense even though ReLU outputs are often zero too, which a zero-skipping chip could exploit, although a multiply would still cost about five times an add. The prices are from a 2014 process, so the ratio matters more than the absolute. And it is sensitive to bookkeeping: measuring spike density before pooling instead of after changed the study’s per-layer numbers by half. The saving exists only on event-driven hardware. Live runs are on CPU, where spike timing can differ slightly from the GPU run behind the published numbers.

The whole test set

Compare with the whole test set

One live recording is an anecdote. These curves are all 288 test recordings under the kind of damage you picked, from the study's recorded run. Left: how often each network was right. Right: how far its confidence sits above or below that. Above zero means it is surer than it should be.

SNNANNbootstrap 95% CIn = 288 · single seed

lose events · how often right

add noise · how often right

block a patch · how often right

shuffle time · how often right

Undamaged test set: spiking network 93.1% right [89.9, 95.8], ordinary network 96.5% [94.4, 98.6]. Brackets are the range the true number likely sits in, given only 288 recordings. On undamaged data both are slightly less sure than they should be, by about the same amount.

how the curves and intervals are calculated

Every point is from one recorded run over the 288 test recordings. Each result is then bootstrapped: the 288 outcomes are resampled with replacement 3,000 times, accuracy is recomputed each time, and the shaded band is the range holding the middle 95% of those values. “Right N points more often” is a paired difference: each resample compares the two networks on the same recordings, so its interval reflects where they disagree rather than how hard the recordings are. Confidence is the same top-share figure as on the cards, averaged over the test set, and “surer than it should be” is that average minus accuracy.

Single seed means one training run per network. The intervals cover sampling noise on 288 recordings; they say nothing about what a different random initialisation would do, which is the study’s stated next step.

Back to the study