NLP for sentiment analysis: Naive Bayes, negation and transformer fine-tuning

In this article

1. Before you start
2. Why interviewers ask this
3. NLP for sentiment analysis from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

The last part of this page uses the transformer architecture, and the whole page builds on how neural networks work.

Why interviewers ask this

A sentiment question is a compact way to see whether a candidate can frame a language problem as classification, pick a cheap baseline first, and say where that baseline breaks. It stays small enough to explain end to end in a few minutes, which is why it can appear in natural language processing and machine learning rounds.

A weak answer says "train a model on reviews." A strong answer names the label set, the features, the loss or scoring rule, and one failure such as negation. A Staff-level answer compares a counting baseline with a transformer adapted by fine-tuning on the same labelled data and says what the extra cost buys. Prepare for the follow-ups as well: the NLP interview questions and large language model (LLM) interview questions lists cover the neighbouring topics.

NLP for sentiment analysis from first principles

Sentiment analysis

Sentiment analysis is a text-classification task that extracts the sentiment, the positive or negative orientation a writer expresses toward some object such as a product, a film or a candidate, and outputs a label for the text.

Naive Bayes classifier

A Naive Bayes classifier is a text classifier that gives each class a score made of the class's prior probability and the probability of every word given that class, treating the words as independent, and returns the class with the highest score.

Negation marking

Negation marking is a preprocessing rule that adds a prefix such as NOT_ to the words after a negation like "not", so "not boring" produces a different token from "boring" and can carry the opposite polarity.

In plain words

Imagine sorting a pile of reviews into a "happy" tray and an "unhappy" tray. You notice that "fantastic" turns up mostly in the happy tray and "dull" mostly in the unhappy one. For a new review you check which tray its words belong to more often. That is the whole idea of a counting model. The analogy breaks on word order: a person reads "not boring" as one idea, while a pure counting model sees two separate words.

In one precise sentence, sentiment analysis maps a text to a label that records the writer's attitude.

It is one case of text categorization, which is the task of assigning a label or category to a whole document. The simplest version is binary classification between positive and negative. Three classes (positive, negative, neutral) are common, as are emotion labels. Uses range from marketing to politics, wherever reviews or opinion text appear.

Step 1: Tokenize the review and mark negation

Tokenization turns a review into word-like pieces, and negation marking rewrites the word after "not" so the model can tell "not boring" from "boring". A classic model lowercases the text and splits it into words; modern transformers use subword pieces, as in byte pair encoding. The review is then treated as a bag of words. A bag of words is a representation that keeps which words occur and drops their order, the same simplification behind the continuous bag of words model.

The textbook rule prefixes every word after a negation token (n't, not, no, never) until the next punctuation mark. The textbook gives the reason: after marking, new tokens such as undefined and undefined become cues for negative sentiment, while undefined and undefined acquire positive associations. The toy example on this page uses a simplified rule, prefixing only the next word, so "not boring" becomes the tokens not and NOT_boring. Input: a raw review. Output: a list of tokens. You now know why the same words can need different tokens; the next step counts them.

Step 2: Learn how likely each word is in each class

Training counts how often every token appears in the reviews of each class and turns the counts into probabilities. The prior of a class is its share of the training reviews. The likelihood of a word in a class is its count in that class divided by the total token count of the class, after add-one smoothing.

Add-one smoothing is a fix that adds one to every word count, and the vocabulary size to the denominator, so a word never seen in a class does not get probability zero and erase the whole score; the textbook calls it add-one (Laplace) smoothing. One tweak is specific to sentiment: whether a word occurs seems to matter more than how often, so clipping each word's count at one per review, binary counting, often helps.

Input: labelled reviews. Output: one table of word probabilities per class.

Step 3: Score a new review and pick the label

Scoring adds, for each class, the log of its prior and the log probability of every token in the new review, and the class with the higher log score becomes the label. The classifier is called naive because it multiplies the word probabilities as if the words were independent given the class, which is false for real language but works well enough as a baseline. Adding logarithms instead of multiplying the probabilities keeps long reviews from underflowing to zero.

Each class gets one number, the difference between the two numbers is the margin, and a margin near zero marks a review the model is unsure about. Input: tokens and the two tables. Output: a label. The figure shows the three steps.

1 · Mark negation2 · Word probabilities3 · ScoreReplay

Step-by-step description

1. The review is split into tokens and negation is marked.
2. Each token has a smoothed probability in each class.
3. Log scores are added and the higher class wins.

• Negation marking rewrites the word after not, so a negated word becomes a different token.
• Training counts how often every token appears in the reviews of each class.
• The class with the higher log score becomes the label.

Lexicon-based and logistic-regression baselines

A lexicon-based scorer is a sentiment method that looks each word up in a list of words with positive or negative scores, adds the scores, and adjusts them for intensifiers such as "very" and for negation words, with no training at all. It is the cheapest option and a sensible start when no labelled reviews exist, but it cannot learn the vocabulary of a new domain.

Logistic regression is a classifier that learns one weight per feature, for example per word count or per word's term frequency (TF) times inverse document frequency (IDF) score, called TF-IDF. Chapter 4 of the textbook is built on it. It can give "fantastic" a large weight and "the" a weight near zero, and unlike Naive Bayes it does not assume the words are independent. It is the usual stronger counting baseline.

How a transformer changes the recipe

A transformer approach replaces the hand-built counts with a pretrained network that reads the whole review and, after fine-tuning, outputs the label. A pretrained encoder such as Bidirectional Encoder Representations from Transformers (BERT) turns the tokens into context-dependent vectors using self-attention, in which each token's vector depends on the other tokens. Devlin et al. (2019) fine-tune BERT with a small classification layer on the vector of the special classification (CLS) token, written [CLS], and report results on the Stanford Sentiment Treebank (SST) binary task, SST-2, a two-class sentiment benchmark.

That layer turns the vector for the review into class scores, and training on labelled reviews adjusts the weights. Attention alone ignores word order; position embeddings supply it. Because each word's vector depends on its neighbours, the model can learn that "not" changes "boring" without a marking rule. Several attention heads run side by side, as in multi-head attention. The cost is more computation and a network that is harder to inspect than a table of word probabilities.

How do you evaluate a sentiment classifier?

A sentiment classifier is evaluated on held-out labelled reviews it never trained on, using precision, recall and F1 rather than accuracy alone. Precision is the share of reviews labelled positive that really are positive, and recall is the share of truly positive reviews the model finds. The F1 score balances the two in one number.

The textbook defines precision, recall, the F-measure and macro-averaging. For three classes, macro-averaged F1 computes the score for each class separately and averages them with equal weight, so a rare neutral class counts as much as a common one. Accuracy alone can hide poor results on that rare class.

Worked example

A toy model shows how marking negation changes a label. The training set is six illustrative reviews, three positive and three negative: "a fun and moving film", "great acting and a great story", "not boring and not slow" (positive) and "boring and predictable", "a dull film with no fun", "not good and boring" (negative). The test review is "not boring". Both priors are 0.5.¹

Without marking, the vocabulary has 16 words, the positive class has 16 tokens and the negative class 13. The smoothed likelihoods of "not" are 0.0938 for positive and 0.069 for negative, and of "boring" 0.0625 and 0.1034, so the log scores are -5.8329 positive and -5.636 negative: the review is labelled negative, which is wrong.¹

With marking, the vocabulary is 17 words and the test tokens are not and NOT_boring. The likelihood of NOT_boring is 0.0606 for positive and 0.0333 for negative, because it was seen once in a positive review and never in a negative one. The log scores become -5.8944 positive and -6.8024 negative, a margin of 0.908 in favour of positive, so the label is right.¹ With binary counts the margin is 0.6276 and the label stays positive.¹

Text description of the diagram

• Without negation marking the log scores are -5.8329 for positive and -5.636 for negative.
• With negation marking the log scores are -5.8944 for positive and -6.8024 for negative.
• The margin moves from -0.1969 to 0.908.

Six reviews is far too few to trust in practice; the point is that you can redo every number with a calculator.

Trade-offs and edge cases

Counting models are cheap and transparent, and transformers use context at a higher cost. The textbook notes that standard Naive Bayes can work well for sentiment analysis and that small changes such as binary counts generally improve it, so it is a fair first baseline. Compare the two on the same held-out labelled reviews before choosing.

Text description of the diagram

• Naive Bayes ignores word order and is cheap.
• A transformer uses context at a higher cost.

Edge cases to name in an answer:

• Sarcasm and tone. A model that sees only words can miss a sentence whose meaning depends on tone or context.
• Domain shift. A word's polarity depends on the domain: "unpredictable" can praise a thriller and condemn a car (illustrative), so a model trained on one domain can mislabel another.
• Mixed opinions. One review can praise the acting and criticize the plot, so a single label loses detail. Aspect-level sentiment analysis is the task that labels each aspect, such as acting or plot, separately.
• Label set. With a neutral class or emotion labels, the model needs more than a sign, and the margin between the top two classes matters.
• Negation scope. A marking rule is a baseline; a parser or a transformer follows how far a negation reaches more accurately.
• Prompting a large language model (LLM). Asking an instruction-following LLM to label each review, with no examples (zero-shot) or a few (few-shot), needs no labelled training set. It costs more computation per review and its labels are harder to calibrate, so check them against labelled held-out reviews.

How to answer it in an interview

The spoken structure: name the task and labels, give the baseline and its scoring rule, name its weakness, then say when you would move to a transformer.

Common mistakes

• "Sentiment analysis is positive versus negative." (wrong) Binary is the simplest form, but three classes and emotion labels are common. (right) "I would first fix the label set: binary, three classes or emotions."
• "More frequent words mean a stronger signal." (wrong) For sentiment, whether a word occurs matters more than how often. (right) "I would try binary counts, which clip each word at one per review."
• "A bag of words cannot touch negation." (wrong) Simple marking of the words after "not" handles many cases. (right) "I would mark negation, and treat scope and sarcasm as known limits."
• "We always need deep learning." (wrong) A counting baseline is a strong, cheap starting point. (right) "I would build the baseline first and move up only if the held-out results justify it."

Follow-up questions

How does Naive Bayes handle negation in sentiment analysis?

It does not understand negation by itself; negation marking gives the words after "not" a prefix such as NOT_, so they become separate tokens whose counts reflect the opposite polarity. In the worked example this moves the label of "not boring" from negative to positive.

Why use binary word counts instead of word frequencies for sentiment?

For sentiment, word occurrence tends to matter more than word frequency, so clipping each word's count at one per review often improves a Naive Bayes classifier. One long rave that repeats "great" then does not outweigh the rest of the review.

Is sentiment analysis the same as text classification?

Sentiment analysis is one kind of text classification: text categorization assigns a label to a text, and sentiment is the label. Topic detection and emotion detection are siblings with other label sets.

When should you use a transformer instead of Naive Bayes for sentiment analysis?

Use a transformer when the baseline's errors come from context, such as negation scope or sarcasm, and the extra computation is acceptable. Measure both on one held-out set first. The transformer overview explains the architecture behind that choice.

Is sentiment analysis only positive versus negative?

No. Three classes with a neutral label are common, and some tasks use emotion labels, so the label set is the first design decision.

Sources

1. Speech and Language Processing, Ch. 4: Logistic Regression and Text Classification (3rd ed. draft) — Stanford University (Jurafsky and Martin), 1 Jan 2026
2. Speech and Language Processing: Naive Bayes, Text Classification, and Sentiment (August 2024 draft) — Stanford University (Jurafsky and Martin), 1 Aug 2024
3. Speech and Language Processing: Naive Bayes and Sentiment Classification, section 4.4 (October 2019 draft) — Stanford University (Jurafsky and Martin), 1 Oct 2019
4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805v2) — arXiv (Devlin et al.), 24 May 2019