What problem does an LSTM solve?
A plain recurrent network passes a hidden state from step to step. When training adjusts the weights, the signal from early steps shrinks toward zero across many steps, the vanishing gradient problem, so the network forgets the start of long sequences. Hochreiter and Schmidhuber introduced the LSTM in 1997 to fix this (Neural Computation, doi 10.1162/neco.1997.9.8.1735). The fix is a separate memory lane, the cell state, that information can travel along almost unchanged. Learned gates, not fixed rules, control what enters and leaves it.
How do the three gates work?
The gates are dials, not switches. Each gate is a small neural network layer with a sigmoid activation function that outputs a number between 0 and 1, used as a dial.
| Gate | Question it answers | Effect |
|---|---|---|
| Forget gate | What in memory is no longer needed? | Scales old cell state toward 0 to erase it |
| Input gate | What new information is worth storing? | Scales the candidate update before adding it |
| Output gate | What should the next step see? | Scales the cell state that becomes the hidden state |
A worked intuition: reading “The invoices from March were late, so they …”, the forget gate keeps “invoices” and “late” in memory, and drops the filler words, so the verb that follows can agree with “invoices”, several words back.
How does an LSTM compare with an RNN and a transformer?
| Plain RNN | LSTM | Transformer | |
|---|---|---|---|
| Memory of long sequences | Weak | Better | Strong, by looking at every position |
| Training | Sequential | Sequential | Parallel |
| Parameters per layer | Fewest | About four times an RNN | Many |
| Typical use today | Rare | Small time series, sensors | Language, vision, most new models |
The transformer architecture replaced LSTMs for language because its attention mechanism reads all positions at once and trains in parallel on large hardware. An LSTM is still a reasonable choice when you have a modest sequence dataset and need a small model.
Where are LSTMs still used?
Small sequence datasets that need a compact model still suit LSTMs. Forecasting demand or energy load from a few years of history, detecting faults in streams of machine sensor readings, and small on-device models for speech or gesture recognition all fit, because the data is a sequence and there is not much of it. For a first baseline on such a problem, an LSTM is easy to train in any deep learning library. If a pretrained transformer fits the task, start there instead.