What are some differences between TensorFlow and PyTorch?

In this article

1. Before you start
2. Why interviewers ask this
3. Frameworks from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This question belongs to the PyTorch interview questions and machine learning (ML) coding round guide.

Why interviewers ask this

The question is reported by candidates in machine learning coding and applied science rounds, and it costs a candidate the point when the answer is a slogan. Interviewers use it to see whether you have used both frameworks enough to know what changed, and whether you can separate habits of thought from facts.

A weak answer repeats "PyTorch is dynamic, TensorFlow is static" or "TensorFlow is for production, PyTorch is for research". A strong answer says that both run eagerly by default, names how each reaches a compiled graph, and contrasts how gradients are recorded. A Staff-level answer adds a decision rule: pick by the team's existing code, the deployment target and the libraries you need, and say that you would check current documentation for the deployment story, because it moves.

Frameworks from first principles

Eager execution

Eager execution is a mode in which each tensor operation runs immediately when the Python line is reached and returns a real value, so you can print and debug intermediate results like ordinary Python.

Automatic differentiation

Automatic differentiation is a technique that computes exact gradients of a loss by recording the operations performed and applying the chain rule backwards through that record.

In plain words

Think of a cook following a recipe. In eager mode the cook reads one line, does it, and tastes the result. In graph mode the cook first writes the whole recipe on a card. A kitchen manager then reorders the steps for speed. The card is reused for every meal. The analogy breaks in one place. A framework can write the card by watching the cook once. Python control flow that changes later may then be missed.

In one precise sentence, TensorFlow and PyTorch compute the same maths on tensors, and differ mainly in how they expose graph compilation, gradient recording and tooling around the model.

Step 1: Both frameworks run eagerly by default

Current TensorFlow runs operations eagerly by default: the graphs guide says "TensorFlow operations are executed by Python, operation by operation, and return results back to Python", and the guide on TensorFlow version 1 (TF1) against TensorFlow version 2 (TF2) (TF1 vs TF2) covers the move to eager execution in TF2. PyTorch has always worked this way: each operation runs when its line runs, and the autograd notes explain that the graph for differentiation is built as operations execute.

Input: a Python line that adds or multiplies tensors. Output: a real tensor you can print. That is why debugging with print or a normal debugger works in both. You now know the first correction to the slogan: "static versus dynamic" no longer describes the default in either framework. The next step covers where they still differ.

Step 2: Both frameworks reach compiled graphs by different routes

TensorFlow reaches graph mode when you wrap a Python function in tf.function, which traces the function into a graph that can be optimised and reused; the graphs guide explains this. Tracing means running the Python once to record the operations. Python side effects such as print then run during tracing, not on every call, which surprises newcomers. AutoGraph, TensorFlow's converter, turns if and while statements that depend on tensors into graph operations, while a branch on a plain Python value is fixed at trace time. A graph may not be reusable for inputs with a different signature, so a new signature triggers a new trace, called retracing (see the graphs guide above).

PyTorch reaches compiled execution through torch.compile, which the PyTorch tutorial presents as a way to speed up eager code with a one-line wrapper. It captures the Python bytecode with a component called TorchDynamo. It records guards on the assumptions it made and recompiles when a guard fails. At code it cannot capture it inserts a graph break and runs that part eagerly instead of failing.

So the accurate statement is: both can run eagerly and both can compile, but TensorFlow's compile step is tf.function and PyTorch's is torch.compile. Input: an eager function. Output: an optimised version of it.

Step 3: Gradients are recorded in two different styles

In TensorFlow you open a recording scope, tf.GradientTape, run the forward pass inside it, and ask the tape for gradients with tape.gradient. In PyTorch you mark parameters with requires_grad=True, run the forward pass, and call loss.backward(), which fills each parameter's .grad field, as the autograd notes describe.

Input: a loss computed from parameters. Output: one gradient per parameter. The maths is identical, so a correct answer never claims one framework "computes better gradients". What differs is where the gradient lives: a value returned from the tape in TensorFlow, a field stored on the parameter in PyTorch. A common PyTorch bug follows from that design: gradients accumulate in .grad across calls unless you zero them.

Step 4: The same training loop, in both APIs

The same training loop differs in TensorFlow and PyTorch only in where the gradient is recorded and stored. The toy fit below learns y = 2x + 1 from four points in both APIs. The code is a sketch of the two idioms, not a benchmark.

# TensorFlow
import tensorflow as tf
x = tf.constant([0., 1., 2., 3.]); y = 2 * x + 1
w = tf.Variable(0.); b = tf.Variable(0.)
for step in range(4):
    with tf.GradientTape() as tape:
        loss = tf.reduce_mean((w * x + b - y) ** 2)
    dw, db = tape.gradient(loss, [w, b])
    w.assign_sub(0.1 * dw); b.assign_sub(0.1 * db)

# PyTorch
import torch
x = torch.tensor([0., 1., 2., 3.]); y = 2 * x + 1
w = torch.zeros((), requires_grad=True); b = torch.zeros((), requires_grad=True)
for step in range(4):
    loss = ((w * x + b - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad; b -= 0.1 * b.grad
    w.grad.zero_(); b.grad.zero_()

With a learning rate of 0.1 (illustrative) and both parameters starting at zero, the mean squared error is 21 at step 1, with gradients of -17 for the slope and -8 for the intercept, giving a slope of 1.7 and an intercept of 0.8.¹ At step 2 the loss is 0.535 and the update gives 1.97 and 0.93; at step 3 the loss is about 0.0144; at step 4 it is 0.001.¹ Both loops give these same values in exact arithmetic, because the maths is the same; single-precision rounding in either framework is below the precision shown.

1 · Eager by default2 · Graph route3 · Gradients4 · Same numbersReplay

Step-by-step description

1. Both run operations as the line runs.
2. Each compiles through its own wrapper.
3. A tape versus backward and .grad.
4. The toy fit gives identical losses.

• Both frameworks run eagerly by default.
• TensorFlow reaches graphs through tf.function and PyTorch through torch.compile.
• TensorFlow records gradients with a tape and PyTorch calls backward on the loss.

Text description of the diagram

• The toy fit loss is 21 at step 1 and 0.535 at step 2.
• The loss is about 0.0144 at step 3 and 0.001 at step 4.

Where the everyday APIs differ

The everyday APIs differ in names and defaults, and the names below are current as of October 2026. The static slogan has a historical reason: in TensorFlow 1 you first built a graph and then ran it in a session, a design the same migration guide contrasts with TensorFlow 2.

In the toy loops, TensorFlow keeps parameters in a tf.Variable updated with assign_sub, while PyTorch uses a plain tensor with requires_grad updated in place under no_grad. Model classes are tf.keras.Model or tf.Module against torch.nn.Module. TensorFlow places operations on an available graphics processing unit (GPU) automatically, while PyTorch needs an explicit .to(device). Input pipelines are tf.data against DataLoader, and export is a SavedModel against torch.export. Confirm any of these in the current documentation before quoting it.

The Staff-level answer

What makes it Staff-level: it corrects the slogan, separates maths from API, and ends with a decision rule instead of a loyalty.

The rubric

Three variants of this question

Is PyTorch dynamic and TensorFlow static?

Not any more: both run eagerly by default, and both offer a compile route (TensorFlow graphs guide, PyTorch compile tutorial). Say that the old split described the first major version, which built a graph and ran it in a session, before the second became eager (migration guide, linked above).

What does Keras have to do with TensorFlow and PyTorch?

Keras is a high-level model-building library. The Keras 3 announcement describes it as working with more than one backend, so "TensorFlow means Keras" is no longer a safe equation. Name the Keras version you used if asked.

Which one would you pick for a new project?

Pick by the existing code, the deployment target and the libraries you need, and confirm the current deployment documentation. A framework preference with no stated reason is the weak answer.

Common mistakes

• "PyTorch is dynamic and TensorFlow is static." (wrong) Both run eagerly by default. (right) "Both run eagerly; they reach compiled graphs through torch.compile and tf.function."
• "TensorFlow is for production and PyTorch is for research." (wrong) This is a slogan with no current source. (right) "Deployment tooling differs and changes, so I check the target's documentation."
• "One framework computes more accurate gradients." (wrong) The maths is the same. (right) "They differ in API: a tape versus backward and .grad."
• "PyTorch resets gradients after each backward call." (wrong) PyTorch accumulates in .grad. (right) "I clear gradients each step, or use an optimizer's zeroing call."

Follow-up questions

How do the two compute gradients?

Both use automatic differentiation: TensorFlow records operations on a tf.GradientTape and differentiates the tape, while PyTorch differentiates the graph built as operations ran, when you call loss.backward(). In the toy fit both give a slope gradient of -17 at step 1.¹

Which framework is used for production?

Both are used in production, and the difference is in the export path: TensorFlow exports a SavedModel and PyTorch offers torch.export (names as of October 2026). Serving and mobile runtimes change between releases, so name the deployment target first, then read that framework's current export documentation for it.

How does training across many GPUs differ?

Both offer data-parallel training and ways to shard large models, with different APIs: TensorFlow uses tf.distribute strategies, while PyTorch uses DistributedDataParallel and fully sharded data parallel (as of October 2026). For the PyTorch route to sharding, read what fully sharded data parallel (FSDP) is, and for model code in PyTorch see building generative pre-trained transformer (GPT) from scratch.

Why do you have to zero gradients in PyTorch?

Because backward adds to each parameter's .grad instead of replacing it, as the autograd notes describe, so an old gradient would mix into the next step. The toy loop clears them with grad.zero_(); the backward pass itself is worked through in multi-layer perceptron (MLP) forward and backward by hand.

What is retracing in tf.function?

Retracing is tf.function building a new graph because the inputs have a different signature from the ones it already traced. Frequent retracing makes a compiled function slow, so keep input shapes and types stable.

Is TensorFlow or PyTorch easier to learn?

Neither has a sourced answer, so say it depends on the code you already know and the tutorials your team uses. Writing the two short loops above from memory is a fair test, and neighbouring prompts appear in the AI engineer coding interview questions.

Sources

1. Introduction to graphs and tf.function — TensorFlow documentation (accessed 2026-10-26), 26 Oct 2026
2. TF1 vs TF2 behaviours — TensorFlow documentation (accessed 2026-10-26), 26 Oct 2026
3. Introduction to torch.compile — PyTorch tutorials (accessed 2026-10-26), 26 Oct 2026
4. Autograd mechanics — PyTorch documentation (accessed 2026-10-26), 26 Oct 2026
5. Keras 3 announcement — keras.io (accessed 2026-10-26), 26 Oct 2026