Bag of Words (BoW) in NLP: How It Works and Where It Fails

LLM foundationsRetrieval and dataPublished By Simon Budziak

Bag of words (BoW) turns text into numbers by counting how often each word appears. Word order and grammar are ignored, so each document becomes a vector of word counts over a fixed vocabulary, which simple classifiers and keyword search can use directly. It is fast and transparent but blind to meaning.

How does bag of words work?

Build a vocabulary of every word in the collection, after tokenization. Then represent each document as one count per vocabulary word. “The invoice is overdue” and “is the invoice overdue” produce the same vector, because order is thrown away. The bag keeps which words occur, never how they relate. In Python, scikit-learn’s CountVectorizer builds these vectors; in version 1.9.1 it lowercases text and counts single words by default, with no stop words removed (scikit-learn docs, read 9 November 2026).

Bag of words vs TF-IDF vs embeddings: what’s the difference?

Bag of words TF-IDF Embeddings
Represents Raw word counts Counts weighted by rarity Meaning as dense vectors
Knows synonyms No No Yes
Word order Ignored Ignored Partly captured
Cost Tiny Tiny Needs a model
Explainable Fully Fully Hardly

TF-IDF fixes the worst flaw of raw counts, that common words like “the” dominate, by down-weighting words that appear everywhere. Embeddings solve the meaning problem but lose the transparency. Move from counts to embeddings when synonyms matter more than exact terms.

Where does bag of words still beat a language model?

Wherever exact terms, speed and explainability matter more than nuance: routing emails by product code, spam filtering, first-pass sentiment analysis at high volume, and keyword search. Keyword search is bag of words with better weighting, which is why sparse retrieval still sits beside vector search in most production retrieval systems. A bag of words classifier you can explain in one table often outperforms a model nobody can audit on a narrow, stable task.

What are the limits of bag of words?

It cannot tell “not good” from “good” without extra features such as word pairs, it treats every synonym as a stranger, and its vectors grow with the vocabulary. These are exactly the gaps natural language processing moved past with embeddings and transformers, so use bag of words as a baseline to beat, not as the end state.

This entry was drafted with AI assistance.

Frequently asked questions

What is BoW in NLP?

BoW stands for bag of words, a text representation that records which words a document contains and how often, without their order. It is one of the oldest and simplest ways to prepare text for machine learning.

What is the main limitation of bag of words?

It ignores order and meaning. "Dog bites man" and "man bites dog" get identical vectors, synonyms count as unrelated words, and a word outside the vocabulary is simply dropped.

Is bag of words still used?

Yes, as a baseline and inside keyword search. BM25, a standard keyword ranking function, scores documents from word counts, and hybrid search pairs it with embeddings to catch exact terms that embeddings miss.

Summarize this page with
ChatGPTClaudePerplexity

Train your team to build this