What Is Long Short-Term Memory (LSTM)? Gates and Use Cases

LLM foundationsModels and inferencePublished By Simon Budziak

Long short-term memory (LSTM) is a recurrent neural network that uses gates to remember patterns across long sequences. It reads a sequence one step at a time, and three gates decide what to keep, add and forget in its memory. Plain recurrent networks lose such patterns.

What problem does an LSTM solve?

A plain recurrent network passes a hidden state from step to step. When training adjusts the weights, the signal from early steps shrinks toward zero across many steps, the vanishing gradient problem, so the network forgets the start of long sequences. Hochreiter and Schmidhuber introduced the LSTM in 1997 to fix this (Neural Computation, doi 10.1162/neco.1997.9.8.1735). The fix is a separate memory lane, the cell state, that information can travel along almost unchanged. Learned gates, not fixed rules, control what enters and leaves it.

How do the three gates work?

The gates are dials, not switches. Each gate is a small neural network layer with a sigmoid activation function that outputs a number between 0 and 1, used as a dial.

Gate Question it answers Effect
Forget gate What in memory is no longer needed? Scales old cell state toward 0 to erase it
Input gate What new information is worth storing? Scales the candidate update before adding it
Output gate What should the next step see? Scales the cell state that becomes the hidden state

A worked intuition: reading “The invoices from March were late, so they …”, the forget gate keeps “invoices” and “late” in memory, and drops the filler words, so the verb that follows can agree with “invoices”, several words back.

How does an LSTM compare with an RNN and a transformer?

Plain RNN LSTM Transformer
Memory of long sequences Weak Better Strong, by looking at every position
Training Sequential Sequential Parallel
Parameters per layer Fewest About four times an RNN Many
Typical use today Rare Small time series, sensors Language, vision, most new models

The transformer architecture replaced LSTMs for language because its attention mechanism reads all positions at once and trains in parallel on large hardware. An LSTM is still a reasonable choice when you have a modest sequence dataset and need a small model.

Where are LSTMs still used?

Small sequence datasets that need a compact model still suit LSTMs. Forecasting demand or energy load from a few years of history, detecting faults in streams of machine sensor readings, and small on-device models for speech or gesture recognition all fit, because the data is a sequence and there is not much of it. For a first baseline on such a problem, an LSTM is easy to train in any deep learning library. If a pretrained transformer fits the task, start there instead.

This entry was drafted with AI assistance.

Frequently asked questions

Is LSTM outdated?

For language, largely yes: transformers replaced it because they train in parallel and handle long context better. LSTMs still fit small time-series, sensor and embedded jobs where data is scarce and the model must be tiny.

What is the difference between short-term and long-term memory in an LSTM?

The hidden state carries what the network is attending to right now, the short-term memory. The cell state is a protected channel that carries information across many steps, the long-term memory.

Summarize this page with

Train your team to build this