Take a bucket with a small hole in the bottom and put it under a tap. Water comes in from the top and leaks out from the hole. The more water in the bucket, the faster it leaks. After a while the level stops moving: what comes in equals what goes out.
That bucket is a leaky integrator. So is a neuron, more or less. It is the first equation I teach, and most models of the brain I have worked with over 25 years start from it.
The equation
dx/dt = −Ax + I
- x is the activity (the water level, or the voltage of a neuron).
- I is the input (the tap).
- A is the leak rate (the size of the hole). It is positive.
Read it from left to right: the activity changes by what comes in, minus a leak that grows with the activity itself.
What it does
- With a steady input, x climbs fast at first, then slower, and settles at I/A.
- With the input off, x decays toward zero, fast at first and then slower. It never falls off a cliff.
- The time constant is 1/A. After that much time, x has covered about 63% of the way to where it is going.
The full solution for a constant input, starting from x(0), is:
x(t) = I/A + (x(0) − I/A) e−At
A worked example
Take A = 2, I = 10 and start from zero. The activity will settle at I/A = 5. The time constant is 1/A = 0.5 seconds, so after half a second x is at about 3.16 (63% of 5). After three seconds it is at 4.99 and you can call it done.
In code
Six lines of Python, using the simplest method there is (Euler):
A, I = 2.0, 10.0
x, dt = 0.0, 0.01
for step in range(300): # 3 seconds
x += dt * (-A * x + I)
print(round(x, 2)) # 4.99Keep the time step dt small compared to 1/A, or the numbers will misbehave.
Why “leaky” and why “integrator”
Take the leak away (A = 0) and x just adds up everything that comes in, forever. That is an integrator. Put the leak back and old input fades away. I think of it as a memory with an expiry date: recent input counts, old input is forgiven.
Where you meet it
| Field | What it is called | What plays the role of x |
|---|---|---|
| Neuroscience | Membrane equation, leaky integrate-and-fire neuron | Membrane voltage |
| Electronics | RC circuit | Voltage on the capacitor (time constant = RC) |
| Signal processing | Low-pass filter, exponential moving average | The smoothed signal |
| Plumbing | A bucket with a hole | Water level |
In discrete time the same equation reads x[n+1] = (1 − A·dt)·x[n] + dt·I[n]. If that looks like exponential smoothing, it is because it is.
From leaky integrator to spiking neuron
Add two rules: when x crosses a threshold, emit a spike, then reset x. You now have the leaky integrate-and-fire neuron, the workhorse of spiking neural networks. The idea goes back to Louis Lapicque in 1907 (Abbott, 1999).
The next step up, and the one I grew up with scientifically, is Stephen Grossberg’s shunting equation, where the input is multiplied by how far the cell is from its ceiling and its floor: dx/dt = −Ax + (B − x)E − (x + D)J, with E the excitatory input and J the inhibitory one. The activity now stays between −D and B whatever you throw at it. Real neurons have that property too.
Why I care, and why robots should
A leaky integrator needs almost nothing: one number to remember and a little arithmetic. If nothing comes in, nothing happens. That is a very good deal for a machine that runs on a battery, and it is the starting point of neuromorphic computing and of a lot of Physical AI.
Questions I get
What is a leaky integrator?
A leaky integrator is a system that adds up its input over time while steadily losing part of what it has accumulated. It is described by the equation dx/dt = −Ax + I, where x is the activity, I is the input and A is the leak rate.
What is the time constant of a leaky integrator?
The time constant is 1/A. It is the time the system takes to cover about 63% of the distance to its new resting level after the input changes.
What is the difference between a leaky integrator and a leaky integrate-and-fire neuron?
A leaky integrate-and-fire neuron is a leaky integrator with two extra rules: when the activity crosses a threshold the neuron emits a spike, and the activity is reset.
Is a leaky integrator a low-pass filter?
Yes. It follows slow changes in its input and smooths out fast ones. In discrete time it is the same as exponential smoothing, also called an exponential moving average.