Ill-Posed Problems

The Central Question

Suppose a forward model maps an unknown quantity

to data

The inverse problem asks us to recover

from

A well-posed inverse problem should have a solution that exists, is unique, and changes continuously when the data change.

Hadamard's Three Conditions

A classical problem is considered well posed when it satisfies three conditions:

1. A solution exists.
2. The solution is unique.
3. The solution depends continuously on the data.

If one or more of these conditions fail, the problem is called ill posed.

In compact form,

Failure of Existence

A solution may fail to exist if the observed data do not lie in the range of the forward model.

Suppose

defines only a restricted set of possible outputs.

If noisy observations fall outside that set, there may be no exact \(\boldsymbol{\theta}\) satisfying the equation.

A Simple Existence Example

Consider

If \(\theta\in\mathbb{R}\), then the model can produce only

An observation

has no exact real-valued inverse solution.

Measurement noise can therefore create data that are inconsistent with the exact forward model.

Failure of Uniqueness

A problem fails uniqueness if two or more distinct unknowns produce exactly the same data.

That is, if

while

then the inverse map is not unique.

A Simple Nonunique Example

Again consider

For every nonzero \(\theta\),

Therefore, data such as

correspond to both

and

This is a failure of uniqueness.

Failure of Continuous Dependence

A solution may exist and be unique yet still depend unstably on the data.

If tiny changes in observations produce very large changes in the recovered unknown, the inverse problem violates continuous dependence.

This is often the most important form of ill-posedness in numerical inverse problems.

A Scalar Stability Example

Consider

where

Then

A data perturbation

creates parameter error

When \(\varepsilon\) is very small, data noise is strongly amplified.

Ill-Posedness and Conditioning

A problem may be formally well posed but numerically difficult because the inverse map is highly sensitive.

This leads to the related concept of conditioning.

A well-conditioned inverse problem changes moderately when the data change.

An ill-conditioned inverse problem can change dramatically under small perturbations.

Condition Numbers

For a nonsingular linear system

the sensitivity of the solution is often summarized by the condition number

In the Euclidean norm,

A very large condition number indicates strong sensitivity to perturbations.

Singular Values and Inverse Stability

Let

be the singular value decomposition.

If

then the inverse solution contains factors

Small singular values therefore produce large inverse amplification.

Noise in Singular Directions

Suppose the observed data are

Expanding the noise in the left singular-vector basis gives

The inverse maps these components approximately to

Noise aligned with small singular values can therefore dominate the reconstructed solution.

Rank Deficiency

If

for some singular direction, then the matrix is rank deficient.

In that direction, the data contain no information about the corresponding parameter combination.

This produces nonuniqueness.

Near Rank Deficiency

If a singular value is not zero but is extremely small, the problem may still be formally unique.

However, the corresponding parameter direction is only weakly constrained.

The result is practical instability rather than exact nonuniqueness.

Ill-Posedness in Infinite Dimensions

Many inverse problems estimate an entire function, field, or image rather than a finite parameter vector.

In such problems, the forward operator may smooth the unknown quantity.

Reversing that smoothing can require reconstructing fine-scale information that the observations contain only weakly.

Smoothing Forward Operators

Suppose the forward model is an integral operator:

The kernel \(K\) may average or smooth the unknown function \(f\).

High-frequency variations in \(f\) can be strongly attenuated in the observed data.

Why Smoothing Causes Trouble

If the forward process suppresses fine-scale structure, the inverse must amplify those weak components to reconstruct them.

Unfortunately, measurement noise often lives in the same fine-scale directions.

The inverse therefore amplifies both the desired signal and the unwanted noise.

Compact Operators

Many smoothing operators in inverse problems are compact operators.

Informally, a compact operator maps broad sets of inputs into comparatively compressed sets of outputs.

Its singular values often tend toward zero:

This creates increasingly weak observable directions.

Decay of Singular Values

The rate at which singular values decay provides information about inverse difficulty.

Slow decay suggests that moderately fine-scale information remains recoverable.

Rapid decay means that many solution components are strongly suppressed in the data and are therefore difficult to reconstruct stably.

Mildly and Severely Ill-Posed Problems

Problems are sometimes classified according to singular-value decay.

Algebraic decay such as

is often associated with mild ill-posedness.

Exponential-type decay such as

is associated with more severe instability.

Inverse Problems Can Lose Resolution

When weak singular directions must be suppressed for stability, the reconstruction cannot recover all fine-scale details.

This creates a practical tradeoff:

The more aggressively noise is suppressed, the more fine structure may be lost.

Example: Numerical Differentiation

Differentiation is a classic example of an unstable inverse operation.

Suppose measurements of a function are noisy:

A finite-difference derivative is

Noise differences are divided by \(h\), so small \(h\) can strongly amplify measurement error.

Differentiation Versus Integration

Integration smooths high-frequency fluctuations.

Differentiation amplifies high-frequency fluctuations.

Therefore, recovering a derivative from noisy integrated or smoothed data can be ill posed even though the forward integration operation is stable.

Example: Deconvolution

Suppose a measured signal is blurred:

where \(k*f\) denotes convolution.

In the frequency domain,

Formally,

Why Deconvolution Can Be Unstable

If

becomes very small at high frequencies, division by

greatly amplifies high-frequency noise.

This is the spectral form of ill-posedness.

Example: Tomographic Reconstruction

In tomography, measurements contain integrated information along paths through an object.

The inverse problem reconstructs an internal image or field from those indirect measurements.

Limited viewing angles, sparse measurements, and noise can make some image features weakly constrained or unstable.

Example: Parameter Compensation

Ill-posedness also appears in lower-dimensional parameter estimation.

Suppose

The data identify only the sum

Individual values of \(a\) and \(b\) cannot be recovered uniquely.

This is a finite-dimensional nonuniqueness problem.

Ill-Posedness and Identifiability

Identifiability and ill-posedness are closely related but not identical.

Failure of uniqueness is directly an identifiability failure.

Poor continuous dependence can occur even when a unique parameter vector exists.

Thus a parameter can be structurally identifiable yet practically unstable.

Ill-Posedness and Measurement Noise

Noise does not create ill-posedness, but it exposes it.

If the inverse map strongly amplifies perturbations, even tiny measurement errors can dominate the recovered solution.

The practical quality of the estimate then depends strongly on the noise level.

Relative Error Amplification

For a linear system, perturbation theory gives the approximate relationship

under suitable conditions.

A large condition number therefore permits large solution error even when the relative data error is small.

Model Error Also Matters

Ill-posed inversion can amplify not only measurement noise but also forward model error.

If the true system is

but estimation uses

a small discrepancy between the two forward models can produce a large parameter bias when the inverse problem is unstable.

Discretization Can Reveal Ill-Posedness

A continuous inverse problem is often converted into a finite-dimensional system

As the discretization becomes finer, new small singular values may appear.

The discrete condition number can therefore grow as resolution increases.

More Resolution Can Mean More Instability

A finer discretization introduces more unknown degrees of freedom.

If the data do not contain corresponding new information, those additional directions may be weakly constrained.

Thus increasing numerical resolution can worsen an inverse problem rather than improve it.

Picard-Type Behavior

In singular-value coordinates, stable inversion requires the data coefficients to decay sufficiently rapidly relative to the singular values.

If noise prevents this decay, terms such as

can grow instead of shrink.

This behavior is a classic signal that direct inversion will be unstable.

Exact Data Versus Noisy Data

An inverse formula may work perfectly for ideal exact data.

But if the same formula becomes useless under arbitrarily small realistic perturbations, then the inverse problem is practically ill posed.

Numerical usefulness depends on stability, not only algebraic invertibility.

Regularization as Controlled Inversion

Regularization modifies the inverse so that weakly informed directions are suppressed.

Instead of applying

to every singular component, regularization replaces this with a bounded filter.

This sacrifices some detail to prevent uncontrolled noise amplification.

Tikhonov Stabilization

In zero-order Tikhonov regularization, the spectral inverse factor becomes

When \(\sigma_j\) is large, this resembles ordinary inversion.

When \(\sigma_j\) is small, the amplification is strongly reduced.

Truncated SVD Stabilization

Truncated SVD retains only singular components above a threshold:

Components below the threshold are discarded.

This avoids reconstructing directions dominated by noise.

Early Stopping

Iterative solvers often recover strong singular directions before weak ones.

Stopping before full convergence can therefore prevent the algorithm from fitting noise-dominated directions.

Early stopping is another form of regularization.

Constraints Can Restore Practical Stability

Physical restrictions can reduce the set of admissible solutions.

Examples include:

• nonnegativity;
• monotonicity;
• boundedness;
• smoothness;
• conservation laws;
• known support.

These constraints may turn an unusably broad inverse solution into a practically meaningful one.

But Constraints Add Information

A constrained solution is determined jointly by the data and the imposed restrictions.

Therefore, one should not claim that the observations alone identified the constrained solution.

Bayesian Stabilization

In Bayesian inversion, prior information can stabilize weak data directions.

The posterior

combines likelihood information with prior structure.

Weakly identified directions may then remain broad or become constrained by the prior.

Ill-Posedness and Uncertainty

A stable point estimate is not enough.

Ill-posed directions should also appear in uncertainty analysis.

Large confidence intervals, broad posterior distributions, or elongated uncertainty regions are often the honest statistical expression of weak inverse information.

Data Cannot Recover Information That Was Never Measured

No inversion algorithm can reconstruct uniquely what the observation system fundamentally fails to encode.

Algorithms can impose plausible structure, but they cannot manufacture independent measurement information.

This is a central principle of inverse modeling.

More Data Can Help

Ill-posedness can sometimes be reduced by changing the experiment.

Useful strategies include:

• adding independent measurement types;
• expanding the observation domain;
• varying experimental inputs;
• improving sensor precision;
• measuring additional state variables;
• using complementary viewing angles or frequencies.

More of the Same Data May Not Help

Repeating a measurement that probes the same weak parameter combination may reduce random noise but still leave the fundamental inverse ambiguity intact.

Experimental diversity can be more valuable than raw sample count.

Experimental Design and Singular Values

In linearized inverse problems, experimental design can aim to increase small singular values or reduce the condition number of the sensitivity matrix.

This makes weak parameter directions more observable.

Null Spaces

If

then adding any multiple of \(\mathbf{v}\) to a parameter vector leaves the predicted data unchanged:

The null space therefore contains exactly unobservable parameter directions.

Near-Null Spaces

A near-null direction satisfies

Large changes in parameters along this direction produce only tiny changes in the observations.

Near-null spaces are a major source of practical ill-posedness.

Resolution Matrices

In a regularized linear inverse problem, the recovered solution may be approximated by

where \(R\) is a resolution matrix.

If

the reconstruction preserves the true parameter components well.

Strong regularization often makes \(R\) smoother and less identity-like.

Stability Is Relative to a Norm

Continuous dependence is measured relative to chosen notions of size.

A mapping can be stable under one norm and unstable under another.

Therefore, inverse-problem stability depends partly on how data errors and solution errors are measured.

Relative Versus Absolute Conditioning

Absolute conditioning measures how

responds to

Relative conditioning compares fractional changes:

Both viewpoints can be useful depending on the application.

Nonlinear Ill-Posedness

For a nonlinear forward map

local instability can be studied through the Jacobian

Small singular values of \(J\) indicate weak local parameter directions.

Local Linearization

Near a parameter value

approximate

The local inverse behavior is therefore governed by the conditioning and rank of \(J\).

Nonlinear Global Nonuniqueness

Local Jacobian rank does not guarantee global uniqueness.

A nonlinear model can be locally invertible near one solution while still possessing another distant parameter vector with the same predicted data.

Both local and global structure should be considered.

Ill-Posedness and Objective Landscapes

In parameter fitting, ill-posedness often appears as:

• flat objective valleys;
• multiple minima;
• long parameter ridges;
• extreme parameter correlation;
• boundary-seeking solutions.

These are optimization signatures of weak inverse information.

Profile Diagnostics

A profile objective or profile likelihood can reveal how strongly one parameter is constrained after re-optimizing the others.

Broad, flat profiles are evidence of practical ill-posedness or weak identifiability.

Perturbation Tests

One practical diagnostic is to perturb the data slightly and refit the model.

If tiny perturbations produce large changes in the estimated parameters, the inverse problem is unstable.

Repeating this test across many perturbations provides a direct empirical picture of sensitivity.

Synthetic Noise Experiments

Generate synthetic observations

Refit the inverse problem for many noise realizations.

The spread of recovered estimates measures practical stability.

Conditioning Before and After Scaling

Poor parameter scaling can make a problem appear more ill conditioned than its physical structure warrants.

Before diagnosing instability, scale variables so that numerical magnitude reflects meaningful relative variation rather than arbitrary units.

Ill-Posedness Is Not the Same as a Bad Algorithm

A solver can fail because of poor implementation.

But even a perfect algorithm cannot stably invert data when the mathematical problem itself is ill posed without adding further information or modifying the target.

Ill-Posedness Is Not the Same as Noise

Noise is a property of the observations.

Ill-posedness is a property of the inverse relationship.

Noise becomes especially damaging when the inverse map amplifies it strongly.

Ill-Posedness Is Not the Same as Nonlinearity

A nonlinear inverse problem can be well posed.

A linear inverse problem can be severely ill posed.

Nonlinearity and ill-posedness are distinct concepts, although they can occur together.

Common Error: Assuming an Inverse Formula Guarantees Stability

A matrix or operator can possess a formal inverse while that inverse is so sensitive that it is useless with noisy data.

Algebraic invertibility is not the same as practical stability.

Common Error: Ignoring Small Singular Values

Small singular values identify directions where the inverse strongly amplifies noise.

They should be inspected rather than treated as harmless numerical details.

Common Error: Increasing Resolution Without Increasing Information

Adding more unknown coefficients can create new weakly constrained directions.

Finer discretization is beneficial only when the data contain enough information to support the added resolution.

Common Error: Treating a Regularized Solution as Exact Recovery

Regularization stabilizes the inverse by suppressing or constraining weak directions.

The resulting estimate is therefore a compromise between data fidelity and imposed structure.

Common Error: Reporting Only a Point Estimate

Ill-posed problems can produce large uncertainty even when a solver returns a visually precise answer.

Report sensitivity, uncertainty, and regularization dependence alongside the fitted value.

Common Error: Confusing More Data with More Independent Information

Repeated measurements of the same weak parameter combination may reduce random error while leaving the null or near-null structure unchanged.

New measurement types can be more valuable than more measurements of the same type.

A Reliable Ill-Posedness Diagnostic Workflow

1. Write the forward map explicitly.
2. Check whether an exact solution can exist for the observed data.
3. Check uniqueness and identifiable parameter combinations.
4. For linear problems, inspect rank and singular values.
5. For nonlinear problems, inspect local Jacobian rank and conditioning.
6. Scale parameters and observations appropriately.
7. Perturb the data and measure solution sensitivity.
8. Run synthetic noise-recovery experiments.
9. Inspect objective or likelihood geometry.
10. Quantify uncertainty in weak parameter directions.
11. Introduce regularization or structural constraints when justified.
12. Reassess stability after regularization.
13. Document which conclusions are data-driven and which depend on imposed structure.

Practice 1: Hadamard Conditions

Name the three classical conditions required for a problem to be well posed.

Practice 2: Failure of Existence

For

does the observation

have an exact solution?

Practice 3: Failure of Uniqueness

For

explain why

is nonunique on \(\mathbb{R}\).

Practice 4: Noise Amplification

Suppose

A measurement error of

produces what error in the inferred value of \(\theta\)?

Practice 5: Condition Number

A matrix has largest singular value \(100\) and smallest singular value \(0.01\).

Compute its Euclidean condition number.

Practice 6: Singular Values

Why does a very small singular value make direct inversion unstable?

Practice 7: Null Space

If

for nonzero \(\mathbf{v}\), what does this mean for parameter uniqueness?

Practice 8: Differentiation

Why can numerical differentiation of noisy data be unstable when the step size \(h\) becomes small?

Practice 9: Regularization

What does regularization do to weak singular directions?

Practice 10: More Data

If repeated measurements all depend only on the parameter combination

will collecting thousands of such measurements identify \(a\) and \(b\) separately?

Practice Answers

Practice 1

The three Hadamard conditions are:

1. existence;
2. uniqueness;
3. continuous dependence on the data.

Practice 2

No. For real \(\theta\),

Therefore,

is outside the range of the forward model.

Practice 3

Both

and

satisfy

Therefore, the inverse is not unique on \(\mathbb{R}\).

Practice 4

Since

the propagated error is

Practice 5

This indicates strong sensitivity.

Practice 6

Direct inversion divides by the singular value. If

is very small, the factor

becomes very large and strongly amplifies noise.

Practice 7

The direction \(\mathbf{v}\) is unobservable. If

then

for any scalar \(c\). The parameter solution is therefore nonunique.

Practice 8

Finite differences divide changes in the measured data by \(h\):

Noise differences are also divided by \(h\), so smaller \(h\) can amplify measurement noise.

Practice 9

Regularization suppresses or damps weak singular directions so that they do not amplify noise excessively.

Practice 10

No. Repeated measurements may estimate

very precisely, but they do not supply independent information separating \(a\) from \(b\).

What to Remember

Ill-posed inverse problems fail one or more of the classical requirements of existence, uniqueness, and stable dependence on data.

• Failure of existence means the observed data are incompatible with the exact model.
• Failure of uniqueness means multiple unknowns produce the same observations.
• Failure of continuous dependence means small data perturbations can cause large solution changes.
• Small singular values reveal weak inverse directions.
• Zero singular values produce exact nonuniqueness.
• Near-zero singular values produce practical instability.
• Smoothing forward operators often create severe inverse instability.
• Finer discretization can add weakly informed solution directions.
• Regularization stabilizes inversion by suppressing unsupported directions.
• Constraints and priors can stabilize solutions but add external information.
• Uncertainty should reflect weak inverse information.
• Better experimental design can reduce ill-posedness by adding independent information.

The central principle is:

Where We Go Next

Ill-posedness explains why even mathematically valid inverse procedures can become unstable when observations contain noise.

The next lesson, Noise and Measurement Error, develops the observational side of the inverse problem: sources of error, bias and variance, covariance, heteroscedasticity, correlated noise, error propagation, and how measurement uncertainty should enter estimation.