SVMs, kernels, naive Bayes and PCA: margins, probabilities and variance

In this article

1. Before you start
2. Why interviewers ask this
3. SVMs, kernels, naive Bayes and PCA from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the machine learning algorithms learning path, and the probability behind naive Bayes is covered in the statistics interview guide.

Why interviewers ask this

SVMs, kernels, naive Bayes and PCA are reported by candidates in machine learning interviews at large tech companies, because they test whether a candidate understands machine learning below the library call. A candidate who recites "an SVM maximizes the margin" stalls at the first follow-up: which points matter, what the kernel really computes, why the probabilities from naive Bayes are not trustworthy, and whether PCA looks at labels.

A weak answer names the algorithm and its library function. A strong answer says what each one optimizes, what it assumes, and what breaks when the assumption fails. A Staff-level answer adds the operating detail: scale features before an SVM, smooth counts in naive Bayes, center before PCA, and know why each choice is made. Expect the questions to arrive in a chain, so one gap in the first answer is found by the third follow-up.

SVMs, kernels, naive Bayes and PCA from first principles

Support vector

A support vector is a training point that lies on the margin or on its wrong side in a support vector machine and has a non-zero dual coefficient. It is one of the only points the fitted decision function uses; every other training point could be deleted without changing the boundary.

Laplace smoothing

Laplace smoothing is a correction for count-based naive Bayes that adds one pseudo-count to every word in every class. A word never seen with a class then gets a small positive probability instead of zero, so it cannot erase the whole product.

Principal component

A principal component is a direction in feature space, found as an eigenvector of the covariance matrix of the centered data, along which the data varies the most and that is perpendicular to all earlier components. PCA keeps the first few and drops the rest.

In plain words

Picture two neighborhoods separated by empty land. An SVM builds the widest possible road between them. Only the houses nearest the road decide where it runs. A kernel lets that road bend around hills. Naive Bayes is a clerk who sorts mail by counting each word in spam and in normal mail, then multiplies the counts. PCA is a photographer who finds the angle from which a crowd looks the most spread out and shoots from there. The analogy breaks in one place: the road is a rule learned from labeled houses, while the photographer never looks at labels at all.

In one precise sentence, an SVM and naive Bayes learn a rule from labeled examples, a kernel changes the geometry an SVM works in, and PCA compresses features without using labels.

Step 1: An SVM draws the widest-margin boundary

A support vector machine is a supervised classifier that separates two classes with the boundary that leaves the widest empty band, called the margin, between them. The margin is the empty band between the two lines that touch the nearest points of each class. Its width is 2/‖w‖, so each class sits 1/‖w‖ from the boundary. Cortes and Vapnik (1995) built a linear decision surface in a feature space and extended the method to non-separable data, which is the soft margin. The soft-margin training problem is:

Here w is the weight vector and b the offset, so the boundary is where $w·x+b=0$. The label $y_i$ is +1 or -1. The constraint asks every point to sit on its correct side with a score of at least 1. The slack ${\xi }_i$ is how far point i falls short of that. Minimizing $|w{|}^2$ widens the margin, and C prices the total slack: a large C punishes violations, a small C tolerates them. Because the smallest valid slack is $\max (0,1-yf(x))$, the problem is the same as minimizing hinge loss plus a penalty on $|w{|}^2$.

Input: labeled points. Output: a sign rule. Only support vectors define the SVM boundary: scikit-learn notes the sum runs over the support vectors because the dual coefficients are zero for every other sample, and the predicted class is the sign of that sum. The knob C sets regularization; scikit-learn says C is 1 by default and that noisy data calls for a smaller value. You now know the straight-line SVM. The next step bends the line.

Step 2: A kernel compares points instead of building features

A kernel gives an SVM a curved boundary by returning the dot product two points would have in a higher-dimensional space, without ever computing that space. The idea is credited by Cortes and Vapnik (1995) to Boser, Guyon and Vapnik (1992): compare two vectors in input space first, then transform that one number, instead of mapping each vector and taking the dot product afterwards. In the dual form, the decision function of the fitted SVM is the following sum over the support vectors, where K is the kernel:

The radial basis function (RBF) kernel, used in the worked example, is:

Here $\gamma$ sets how quickly the similarity falls as two points move apart.

Input: two points. Output: one similarity number. Common choices are the linear, polynomial and radial basis function (RBF) kernels; the RBF kernel falls toward zero as two points move apart. The trick needs the dual form of the problem. That is why scikit-learn says LinearSVC, which solves the hinge-loss primal form, cannot use the kernel trick. You now know why a curved boundary costs little: the high-dimensional features are never built.

Step 3: Naive Bayes multiplies per-feature probabilities

Naive Bayes is a classifier that applies Bayes' theorem with the assumption that features are independent of each other once the class is known. scikit-learn reports it has worked quite well in many real-world situations, famously document classification and spam filtering, despite that over-simplified assumption.

Input: word counts. Output: a score per class. The prediction is the class with the highest prior times the product of the per-word probabilities:

Here $P(y)$ is the class prior and $P(x_i|y)$ is the probability of feature i given the class. In practice the code adds logarithms instead of multiplying, because a long product of small numbers underflows to zero. The common error is the zero count. A word never seen with the spam class makes the spam product exactly zero, however strongly the other words point to spam. The fix is Laplace smoothing: in scikit-learn's formula, the smoothed estimate is:

where $N_{yi}$ is the count of word i in class y, $N_y$ is the class total and n is the vocabulary size. Setting alpha to one is Laplace smoothing. The worked example below computes it. You now have both classifiers; the last step compresses features.

Step 4: PCA keeps the directions of greatest variance

Principal component analysis is an unsupervised, linear method that rotates centered data onto orthogonal directions, ordered by how much variance each one explains. scikit-learn describes it as decomposing a dataset into successive orthogonal components that explain a maximum amount of the variance.

Input: a feature matrix. Output: fewer columns. PCA is usually computed with singular value decomposition (SVD) of the centered data, and the PCA reference states that the input is centered but not scaled. So features in different units need standardizing first. Each component mixes all the original features, so PCA extracts new features and does not select old ones. It never reads labels, so a low-variance direction that separates the classes can be thrown away. See feature engineering techniques for scaling and encoding.

1 · Widest margin2 · Kernel3 · Naive Bayes4 · PCAReplay

Step-by-step description

1. An SVM is defined by its support vectors.
2. A kernel compares two points, no features built.
3. Multiply probabilities, smooth the counts.
4. Center, then keep the top variance directions.

• Only support vectors define the SVM boundary.
• A kernel compares two points without building the high-dimensional features.
• Naive Bayes multiplies per-feature probabilities and needs Laplace smoothing.
• PCA centers the data and keeps the directions of greatest variance.

Worked example

Four small calculations make each step checkable with a calculator. All inputs are illustrative; the results follow from them by arithmetic.¹

SVM. Take three points on a line: x = -1 (class minus), x = 1 (class plus) and x = 3 (class plus). The two inner points are the support vectors with dual coefficients 0.5 each, and the point at 3 has a coefficient of 0, so it plays no part. The weight is 1, so each class sits 1 from the boundary, the margin width is 2.0, and the decision value at x = 0.3 is 0.3, so the sign says class plus.¹

Kernel. For x = (1, 2) and z = (3, 1), the degree-two polynomial kernel is the dot product squared: 5 squared is 25. Building the three explicit features of each point and taking their dot product also gives 25.¹ For the RBF kernel with gamma 0.5 and two points at squared distance 2, the similarity is 0.3679.¹ The saving grows with dimension: with 200 input features and degree four there are 68,685,050 distinct monomial features, or 1,600,000,000 ordered products, while one kernel call needs about 200 multiplications plus raising one number to the fourth power.¹

Naive Bayes. Two spam documents and two normal ones give a prior of 0.5 each. Over a four-word vocabulary, spam holds 3 "win", 2 "prize", 0 "meeting" and 1 "report"; normal mail holds 1, 0, 3 and 2. To score "win prize meeting" without smoothing, both classes multiply to exactly 0, so no answer exists. With alpha 1, the spam score is 0.006 against 0.004 for normal mail, a spam probability of 0.6.¹ Now count "win" twice, a duplicated feature with no new information: the spam probability rises to 0.75.¹ That is why the probabilities are not to be trusted.

PCA. On the six-point toy set from the scikit-learn PCA reference, the singular values are 6.30061 and 0.54980, so component 1 holds 99.24% of the variance and component 2 holds 0.76%.¹

The last row uses the bound of Cortes and Vapnik (1995): for training vectors separated without errors by the optimal hyperplane, the expected test error is at most the expected number of support vectors divided by the number of training vectors, and the bound does not explicitly contain the dimension of the space.¹

Text description of the diagram

• Component 1 holds 99.24% of the variance.
• Component 2 holds 0.76% of the variance.

Trade-offs and edge cases

Scale before an SVM. scikit-learn says SVMs are not scale invariant and recommends scaling, with the same scaling applied to the test vector. Fit the scaler on training data only, ideally inside a pipeline. Scaling is not just for kernels.

Size and shape of data. For the linear case, LinearSVC can scale almost linearly to millions of samples, and the kernel support vector classification (SVC) is much slower. When features far outnumber samples, the kernel and regularization choice is crucial to avoid over-fitting (same page). A higher C is not always better: it fits noise, and the docs link smaller C to more regularization; see L1 vs L2 regularization.

Naive Bayes probabilities. scikit-learn calls it a decent classifier but a bad estimator, so outputs from predict_proba should not be taken too seriously. For a thresholded decision such as credit risk, calibrate them or use a model built for probabilities, as with cross-entropy loss. The variant follows the features: multinomial for counts, Bernoulli for binary features.

PCA limits. It is linear and unsupervised. For sparse data that cannot be centered, TruncatedSVD is the tool; for data too big for memory, IncrementalPCA (same decomposition page). If the data shifts after training, all four degrade; see distribution shift.

Text description of the diagram

• SVM and naive Bayes use labels; PCA does not.
• Each method has one main setting: C and kernel, smoothing alpha, or the number of components.

How to answer it in an interview

The spoken structure: say what each one optimizes, name its key assumption, then give the one thing that breaks it.

Common mistakes

• "Every training point shapes an SVM." (wrong) Only support vectors have non-zero coefficients. (right) "Delete the other points and the boundary does not move."
• "A kernel builds the high-dimensional features." (wrong) It compares two points in input space. (right) "The kernel returns the dot product, so the mapping is never built."
• "Naive Bayes fails because features are not independent." (wrong) It often works well anyway. (right) "It classifies well, but correlated features make its probabilities too extreme."
• "PCA standardizes and selects features." (wrong) It centers only, and each component mixes all features. (right) "I standardize first, and PCA extracts new features."

Follow-up questions

Can you explain how SVM works?

An SVM picks the boundary with the widest margin between two classes, uses only the support vectors to define it, and allows some violations through C when classes overlap. Add a kernel when a straight line is not enough.

How are support vectors produced?

Support vectors come out of training: solving the dual problem gives each training point a coefficient, and the points with non-zero coefficients, which sit on the margin or on its wrong side, are the support vectors.

SVM vs logistic regression: which should you pick?

The difference is the loss. An SVM minimizes hinge loss, $\max (0,1-yf(x))$, which is exactly zero for points beyond the margin, so only support vectors shape the boundary and the model is sparse in them; it has no native probabilities. Logistic regression minimizes log loss (see cross-entropy loss), which never reaches zero, so every point contributes and it outputs probabilities. Accuracy is often similar: choose logistic regression when you need probabilities, and an SVM with a kernel when a curved boundary matters.

What is the bias and variance trade-off of k-nearest neighbors?

With a small k, k-nearest neighbors (KNN) follows every point, so it has low bias and high variance; a large k smooths the boundary and reverses that. See k-means vs KNN and bias vs variance.

Why does the curse of dimensionality make you reach for dimensionality reduction?

As features grow, points spread thin and distances lose meaning, so models need more data or fewer features. PCA is one way to cut features when the signal lives in a few high-variance directions.

Can PCA replace recursive feature elimination when modeling credit risk?

Not directly. Recursive elimination keeps original features chosen by a model that sees labels, so it stays interpretable. PCA ignores labels and mixes all features, so use it for compression and check the result against the target.

Sources

1. 1.4. Support Vector Machines (scikit-learn 1.9.1 documentation) — scikit-learn, 26 Oct 2026
2. Support-Vector Networks (Cortes and Vapnik, Machine Learning 20, 273-297) — Springer (Cortes and Vapnik), 15 Sep 1995
3. 1.9. Naive Bayes (scikit-learn 1.9.1 documentation) — scikit-learn, 26 Oct 2026
4. MultinomialNB (scikit-learn 1.9.1 API reference) — scikit-learn, 26 Oct 2026
5. 2.5. Decomposing signals in components (scikit-learn 1.9.1 documentation) — scikit-learn, 26 Oct 2026
6. PCA (scikit-learn 1.9.1 API reference) — scikit-learn, 26 Oct 2026