4.7 Physics-Informed Neural Networks

1. Introduction¶

Physics-Informed Neural Networks (PINNs) build physical laws, expressed as ordinary or partial differential equations (ODEs/PDEs), directly into the training of a neural network. The physics enters through the loss function: in addition to fitting data, the network is penalized when its predictions violate the governing equation. PINNs can:

• solve differential equations with or without data,
• work with few labeled samples, because the physics constrains the solution,
• fold prior knowledge into a machine learning workflow.

Geoscience is full of processes governed by well-known equations: heat diffusion in the crust, seismic wave propagation, groundwater flow (Darcy’s law), and fluid dynamics (Navier-Stokes). PINNs were introduced by Raissi et al. (2019) and reviewed for the broader physics community by Karniadakis et al. (2021); see the References at the end of this notebook. Example uses in geoscience:

1. Temperature diffusion in the crust: infer thermal structure and geothermal potential from sparse borehole temperature profiles.
2. Seismic wave propagation: solve wave equations in heterogeneous media without meshing the domain.
3. Groundwater flow: solve the Darcy and advection-diffusion equations to model aquifer response and pollutant transport.

In all three cases the appeal is the same: field data are sparse and noisy, but the governing PDE is known. The physics fills in where the data cannot.

🖥️ Lecture slides — Session 26 (Wed Dec 2, enrichment context)

2. Mathematical framework¶

A PINN has three components:

1. A neural network $u_\theta(x, t)$ with parameters θ that approximates the solution $u(x, t)$.
2. A governing equation written in residual form $\mathcal{L}(u, x, t) = 0$, where $\mathcal{L}$ is a differential operator. The derivatives of $u_\theta$ that appear in $\mathcal{L}$ are computed exactly with automatic differentiation, the same machinery that backpropagation uses.
3. A composite loss function

where

• the data loss measures misfit to observations,
• the physics loss is the mean squared PDE residual $\mathcal{L}(u_\theta)$ evaluated at collocation points sampled inside the domain,
• the initial/boundary condition loss pins the solution where it is known.

Training minimizes the combined loss with a gradient-based optimizer such as Adam. The relative weights of the three terms are hyperparameters, and balancing them is one of the practical difficulties of PINNs (more on this in Section 5).

3. Warm-up: Newton’s law of cooling¶

Before tackling a PDE, we start with an ODE, where automatic differentiation only needs first derivatives. Consider a rock sample cooling toward the ambient temperature $T_{\text{env}}$. Newton’s law of cooling states

where $R$ is a cooling rate constant. With initial temperature $T(0) = T_0$, the analytic solution is

We will run a three-way comparison on the same 10 noisy measurements, all taken early in the cooling history:

1. a plain neural network (baseline),
2. the same network with L2 weight regularization,
3. the same network with a physics loss (a PINN).

The question is which model extrapolates correctly beyond the time range covered by the data.

We generate synthetic noisy data: the true curve spans 1000 s, but the 10 training samples only cover the first 300 s.

3.1 A network with a pluggable second loss¶

The Net class below is a small fully connected network. Its fit method minimizes the data misfit plus an optional second loss loss2, weighted by loss2_weight. We will reuse the same class for all three experiments and only swap loss2: None for the baseline, an L2 penalty for the regularized network, and a physics residual for the PINN.

The grad helper computes the derivative of network outputs with respect to inputs using torch.autograd.grad with create_graph=True, so the result can itself be differentiated during backpropagation. This is the core trick of PINNs.

3.2 Baseline network¶

Train on the 10 noisy samples with no extra loss term.

The baseline fits the samples it saw and does whatever it wants after 300 s: nothing constrains the extrapolation.

3.3 L2-regularized network¶

A classic remedy for overfitting is to penalize large weights. We pass the L2 norm of the parameters as loss2.

Regularization smooths the prediction, but it has no reason to follow the exponential decay toward $T_{\text{env}}$: it just keeps the weights small.

3.4 PINN¶

Now replace the L2 penalty with a physics loss. We evaluate the network at 1000 collocation times spanning the full 1000 s window (far beyond the data), differentiate the output with respect to time using grad, and penalize the mean squared residual of the cooling law

Note that the physics loss uses no temperature observations at all, only the equation.

The PINN follows the true exponential decay well past the last data point, because the physics loss constrains the solution everywhere the collocation points reach. The data anchor the amplitude; the equation supplies the shape. That is the whole idea.

Training here is deliberately short. On your own machine you can raise the epoch counts for smoother fits.

4. A real PDE: 1-D heat diffusion in the shallow subsurface¶

The cooling law was an ODE. Now we solve an actual PDE with a PINN: heat diffusion,

where $T(x,t)$ is temperature and $k$ is thermal diffusivity.

Physical setting. Think of a temperature anomaly in the top $L = 10$ m of soil or rock, a permafrost active layer or a shallow geothermal profile. A warm anomaly peaks mid-column and relaxes by diffusion, while the surface ($x=0$) and the base ($x=L$) stay pinned at the mean annual temperature, which we take as the 0 °C reference. A typical rock/soil thermal diffusivity is $k \approx 10^{-6}\ \mathrm{m^2/s}$.

Nondimensionalization. Raw SI values ($L=10$, $k=10^{-6}$, $t$ in units of 10⁷ s) make ugly numbers for a neural network, so we rescale: $\hat{x} = x/L$ and $\hat{t} = k t / L^2$. One unit of $\hat{t}$ corresponds to $L^2/k = 10^8\ \mathrm{s} \approx 3.2$ years; we solve on $\hat{t} \in [0, 0.3]$, roughly one year. Temperature stays in °C. In these units the PDE becomes $\partial T/\partial \hat{t} = \partial^2 T/\partial \hat{x}^2$ (diffusivity 1).

Analytic reference. A single sine mode diffuses without changing shape:

with $T_0 = 5$ °C. It satisfies the PDE and the boundary conditions $T=0$ at $\hat{x}=0$ and $\hat{x}=1$, so we can measure the PINN’s error exactly.

Network. $T_\theta(\hat{x}, \hat{t})$ is a small MLP: 2 inputs, three hidden layers of 64 units, 1 output. We use tanh activations rather than ReLU because the PDE residual needs second derivatives, and the second derivative of ReLU is zero almost everywhere.

4.1 Training points¶

Three sets of points, one per loss term:

• Data points: 40 noisy temperature samples at random $(\hat{x}, \hat{t})$ locations, like sparse sensor readings, with 0.2 °C Gaussian noise.
• Collocation points: 1000 random interior points where we enforce the PDE residual. No temperature values are attached to them.
• Initial/boundary points: 200 points on the $\hat{t}=0$ line (initial profile) and 200 on each boundary $\hat{x}=0$ and $\hat{x}=1$ (temperature held at 0 °C).

4.2 Training loop¶

Each step computes three loss terms and sums them with equal weights:

• Data loss: MSE between $T_\theta$ and the noisy samples.
• PDE loss: at the collocation points, grad gives $\partial T_\theta / \partial \hat{x}$ and $\partial T_\theta / \partial \hat{t}$ in one call (columns of the gradient); a second grad call on the first column gives $\partial^2 T_\theta / \partial \hat{x}^2$. The residual is $\partial T_\theta/\partial \hat{t} - \partial^2 T_\theta/\partial \hat{x}^2$.
• IC/BC loss: MSE against the initial sine profile plus the squared temperature at both boundaries.

A few thousand Adam steps are enough for this smooth problem.

4.3 Results¶

First, predicted vs. analytic temperature profiles at four times. The depth axis is back in meters ($x = L\hat{x}$).

The PINN matches the analytic profiles to a few hundredths of a degree from 40 noisy point measurements, because the PDE and the boundary conditions carry most of the information. The data loss plateaus near the noise floor (the network should not fit the 0.2 °C noise), while the PDE and IC/BC losses drop by three to four orders of magnitude. The periodic spikes in those curves are Adam briefly overshooting and recovering, a common sight in PINN training.

Training is kept short on purpose; on your own machine, raise n_steps and n_col for tighter residuals.

Exercise. Set the PDE loss weight to zero (replace loss_pde with 0 * loss_pde in the sum) and retrain. Compare the profiles at $\hat{t} = 0.3$, where there are few data points and the anomaly is small.

4.4 The baseline: 15 lines of finite differences¶

This book’s rule is baselines before models, and the physics chapter is not exempt. The forward problem we just solved, with known diffusivity, known initial and boundary conditions, and a smooth solution, is textbook territory for classical numerics, so the PINN has to face the classical method before we praise it. The simplest scheme is FTCS (forward-time, centered-space): put $\hat{x}$ on a grid, replace $\partial^2 T/\partial \hat{x}^2$ with the centered second difference, and step forward in time with explicit Euler:

The scheme is only stable for $r \le 1/2$; we use $r = 0.4$. That constraint is the price of an explicit method (Crank–Nicolson removes it at the cost of a tridiagonal solve), and on this problem it costs almost nothing. The whole solver, timed:

The verdict is not close. The finite-difference solver finishes in about 20 ms and matches the analytic solution to about $2 \times 10^{-4}$ °C; the PINN needed 14 s of training to get within 0.03 °C. On this forward problem the classical solver wins by a factor of roughly 750 in speed and two orders of magnitude in accuracy, with no learning rate, no architecture, and no seed. That is the honest reading of Section 4.3, and it generalizes: when the coefficients, initial condition, and boundary conditions are known and a standard discretization exists, use the standard discretization. The PINN’s case has to be made elsewhere.

Two things the FD solver did not do, though. It never touched the 40 noisy interior measurements, because a forward solver has no slot for scattered observations. And it required $k$ as an input. Both gaps point to the same place, and that is where we go next.

4.5 The inverse problem: recovering the diffusivity¶

Section 5 will argue that inverse problems are the first regime where PINNs remain the right tool. Here is the demonstration, on the data we already have. Pretend the diffusivity is unknown. In scaled units the generator’s value is exactly $\hat{k} = 1$, because the nondimensionalization used the true $k = 10^{-6}\ \mathrm{m^2/s}$; the model is not told that. We initialize a trainable $\hat{k}$ at 0.2, five times too low, and let the optimizer recover it from the same 40 noisy samples.

The change to the training loop of Section 4.2 is five lines, marked (1)–(4) below: $\hat{k}$ becomes an nn.Parameter (stored as $\log \hat{k}$ so it stays positive), it joins the optimizer’s parameter list, and the residual becomes $\partial T_\theta/\partial \hat{t} - \hat{k}\, \partial^2 T_\theta/\partial \hat{x}^2$. Everything else is untouched. The information flows like this: the initial condition pins the shape of the anomaly, the 40 samples pin how fast it decays, and the PDE ties that decay rate to $\hat{k}$.

The recovered diffusivity lands within 2% of the generator truth, from 40 noisy point measurements and nothing gridded. Be clear about what this run is: a calibration, also called history matching. The finite-difference solver of Section 4.4 cannot do it alone, because it needs $k$ as an input. The classical route wraps the solver in an outer optimization loop that reruns it for every candidate $k$; in groundwater practice that is what PEST does around a MODFLOW model when calibrating hydraulic conductivity against sparse well records. The PINN merges the two loops: one training run fits the temperature field and inverts for the parameter at the same time, and the same trick extends to many unknowns (a spatially variable $k(\hat{x})$ can be a second small network). Sparse, noisy point data plus a trusted equation plus unknown coefficients: this is the regime where the extra machinery pays for itself.

4.6 Breaking the PINN: loss-weight imbalance¶

So far the composite loss has behaved because its three terms happened to sit at comparable scales; equal weights just worked. That is luck, not a law, and Section 5’s warning about loss balancing deserves the same treatment the broken training runs got in notebook 4.5: show the failure, read the symptoms, name the fix.

The exercise in Section 4.3 removed the physics term and watched the model decay into a noisy regression. Here we break the balance in the opposite direction: the physics term swamps the data and initial condition. The run below repeats Section 4.2 with a single change, w_pde = 1e4 on the PDE residual. Watch the printed total: it keeps falling, as if training were going well.

The total loss falls and the answer is garbage. That combination is the signature of loss imbalance, and it is invisible unless you log the components separately. Read them, not the total: after a few hundred steps the PDE residual is pinned below 10^-5, while the data loss sits near 1 (far above the 0.04 noise floor of Section 4.2) and the IC/BC loss stalls near 8, the same order as the 12.5 a network outputting zero everywhere would score (the mean of $(5\sin\pi\hat{x})^2$). The network has found a near-trivial solution: $T \approx 0$ satisfies the heat equation and the boundary conditions exactly, so the weighted objective rewards staying close to it, and any move toward the true initial profile is punished through the dominant term before it can pay off. The profile at $\hat{t}=0$ misses the 5 °C anomaly by about 4 °C.

The fix is the balance we already had: with equal weights, the same architecture and the same 2000 steps reached 0.03 °C in Section 4.2. When no fixed weighting works, because the terms have genuinely different scales, the standard moves are to normalize each term by its initial value, tune the weights against a held-out validation metric, or use an adaptive weighting scheme (Karniadakis et al., 2021); longer training and wider layers only help when the imbalance is mild. The habit to take away: always plot the loss components separately. A single total-loss curve hides exactly this failure.

Exercise (spectral bias). The other failure mode Section 5 names is spectral bias. Add a higher-frequency mode to the initial condition:

whose analytic solution adds $2\sin(7\pi\hat{x})\,e^{-49\pi^2 \hat{t}}$ to the single-mode reference. Regenerate y_ic and the 40 data samples from the two-mode solution, retrain the Section 4.2 PINN (equal weights), and rerun the FTCS solver of Section 4.4 with the new initial profile. Compare both to the analytic solution at $\hat{t} = 0$.

5. Where PINNs stand in 2026¶

PINNs are no longer a novelty, and their failure modes are well documented; two of them are now sitting in this notebook. Loss balancing is a persistent tuning problem: a bad balance makes training converge to a solution that satisfies one term and ignores the others, and Section 4.6 showed it on our own problem, where a factor of 10⁴ on the physics term produced a network stuck near the trivial zero solution, missing the 5 °C anomaly by about 4 °C while its total loss kept falling. Standard MLPs also carry a spectral bias toward smooth, low-frequency functions: they learn smooth solutions quickly and oscillatory or fine-scale structure slowly, if at all (Karniadakis et al., 2021); the exercise in 4.6 makes a tanh MLP flatten a seventh-mode ripple that the finite-difference solver resolves without any change. Stiff PDEs, sharp fronts, and widely separated time scales compound both problems, because the residual then varies over orders of magnitude across the domain and no fixed weighting is right everywhere.

The first rule of use is negative: do not solve a clean forward problem with a PINN. Section 4.4 put numbers on this. Fifteen lines of FTCS beat the trained PINN by a factor of roughly 750 in wall-clock time and two orders of magnitude in accuracy, with no hyperparameters to tune. When the coefficients and boundary conditions are known and a standard discretization exists, the classical solver wins. And for the adjacent problem of building fast surrogates evaluated many times, neural operators have largely taken over: the Fourier Neural Operator (Li et al., 2021) and DeepONet (Lu et al., 2021) learn the solution operator, the map from an initial condition, boundary condition, or coefficient field to the solution, so that after training on many simulation pairs a new case costs milliseconds, where a PINN would retrain from scratch. For parametric studies, uncertainty quantification, and operational forecasting, that is the tool that gets used.

PINNs still earn their keep in two regimes, and both are now demonstrated rather than claimed. The first is inverse problems: Section 4.5 made the diffusivity a trainable parameter and recovered it to within 2% from the 40 noisy samples, the PDE acting as the forward model inside a regression (Raissi et al., 2019). This is joint inversion, the PEST-around-MODFLOW pattern of groundwater calibration collapsed into a single training loop. The second is the sparse-data regime those same 40 points represent: scattered, noisy observations plus a trusted governing equation and no simulation library to train an operator on. A forward solver has no slot for such data; the PINN treats them as one more loss term. Both situations are common in geoscience, which is why the method remains part of the toolkit (Karniadakis et al., 2021).

References¶

• Raissi, M., Perdikaris, P., and Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707.
• Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., and Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3, 422-440.
• Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2021). Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations (ICLR).
• Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. (2021). Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3, 218-229.

Summary¶

• A PINN adds the residual of a governing equation, evaluated by automatic differentiation at collocation points, to the training loss.
• On the cooling-law ablation, only the physics term produced correct extrapolation beyond the data; L2 regularization did not.
• On the 1-D heat equation, a 3-layer tanh MLP recovered the analytic diffusing mode from 40 noisy samples plus the PDE and boundary conditions.
• The classical baseline won the forward problem outright: 15 lines of FTCS were a factor of ~750 faster and two orders of magnitude more accurate than the PINN (Section 4.4).
• Making the diffusivity a trainable parameter turned the same loop into a joint inversion that recovered $\hat{k}$ to within 2% from the 40 noisy samples, a calibration the forward solver cannot do alone (Section 4.5).
• A 10⁴ weight on the physics loss gave a falling total loss and a wrong answer; the failure is only visible in the per-term loss curves, so always plot the components (Section 4.6).
• In 2026: classical solvers for clean forward problems, neural operators (FNO, DeepONet) for parametric surrogates, PINNs for inverse and sparse-data problems.