The Central Idea
Suppose the unknown parameter vector is
\[ \boldsymbol{\theta} \]and the observed data are
\[ \mathbf{y}. \]Bayesian estimation combines three pieces:
- a prior distribution \(p(\boldsymbol{\theta})\);
- a likelihood \(p(\mathbf{y}\mid\boldsymbol{\theta})\);
- a posterior distribution \(p(\boldsymbol{\theta}\mid\mathbf{y})\).
Bayes' rule gives
\[ \boxed{ p(\boldsymbol{\theta}\mid\mathbf{y}) = \frac{ p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}) }{ p(\mathbf{y}) } }. \]Prior, Likelihood, and Posterior
The prior represents uncertainty about the parameters before the current data are incorporated.
The likelihood measures how compatible the observed data are with different parameter values.
The posterior combines both sources of information.
In proportional form,
\[ \boxed{ p(\boldsymbol{\theta}\mid\mathbf{y}) \propto p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}) }. \]The Evidence
The denominator
\[ p(\mathbf{y}) \]is called the evidence or marginal likelihood.
It is obtained by integrating over all admissible parameter values:
\[ \boxed{ p(\mathbf{y}) = \int p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}) \,d\boldsymbol{\theta} }. \]The evidence ensures that the posterior integrates to \(1\).
Why Proportional Form Is Often Enough
When the goal is parameter estimation, the evidence does not depend on the trial value of \(\boldsymbol{\theta}\).
Therefore, many calculations begin with
\[ p(\boldsymbol{\theta}\mid\mathbf{y}) \propto p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}). \]The normalization constant can be computed later or avoided entirely by sampling methods.
A Simple Coin Example
Suppose \(p\) is the unknown probability of heads.
A convenient prior is
\[ p \sim \operatorname{Beta}(\alpha,\beta). \]If we observe \(h\) heads and \(t\) tails, then the Bernoulli likelihood is proportional to
\[ p^h(1-p)^t. \]Multiplying prior and likelihood gives
\[ p^{\alpha+h-1} (1-p)^{\beta+t-1}. \]Beta-Binomial Updating
The posterior is therefore
\[ \boxed{ p\mid\mathbf{y} \sim \operatorname{Beta} ( \alpha+h, \beta+t ) }. \]The prior parameters behave like prior counts that are updated by the new observations.
This is an example of a conjugate Bayesian model.
Conjugate Priors
A prior is conjugate to a likelihood when the posterior belongs to the same distribution family as the prior.
Conjugacy can make Bayesian updating algebraically simple.
Common examples include:
- Beta prior with Bernoulli or binomial likelihood;
- Gamma prior with Poisson likelihood;
- normal prior with normal likelihood under suitable assumptions.
Bayesian Estimation in an Inverse Problem
Suppose a forward model predicts
\[ F(\boldsymbol{\theta}). \]Observations are generated through a measurement model
\[ \mathbf{y} \sim p( \mathbf{y} \mid F(\boldsymbol{\theta}) ). \]Bayesian inference combines this likelihood with prior information:
\[ \boxed{ p(\boldsymbol{\theta}\mid\mathbf{y}) \propto p( \mathbf{y} \mid F(\boldsymbol{\theta}) ) p(\boldsymbol{\theta}) }. \]Gaussian Observation Model
Suppose
\[ \mathbf{y} = F(\boldsymbol{\theta}) + \boldsymbol{\varepsilon}, \]with
\[ \boldsymbol{\varepsilon} \sim N(\mathbf{0},\Sigma). \]Then the likelihood is proportional to
\[ \exp \left[ -\frac{1}{2} \left( \mathbf{y} - F(\boldsymbol{\theta}) \right)^T \Sigma^{-1} \left( \mathbf{y} - F(\boldsymbol{\theta}) \right) \right]. \]Gaussian Prior
Suppose also
\[ \boldsymbol{\theta} \sim N( \boldsymbol{\mu}_0, C_0 ). \]Then the prior is proportional to
\[ \exp \left[ -\frac{1}{2} ( \boldsymbol{\theta}-\boldsymbol{\mu}_0 )^T C_0^{-1} ( \boldsymbol{\theta}-\boldsymbol{\mu}_0 ) \right]. \]Posterior Under Gaussian Assumptions
Multiplying likelihood and prior gives a posterior proportional to
\[ \exp \left[ -\frac{1}{2} \left( \mathbf{y}-F(\boldsymbol{\theta}) \right)^T \Sigma^{-1} \left( \mathbf{y}-F(\boldsymbol{\theta}) \right) \right] \] \[ \times \exp \left[ -\frac{1}{2} ( \boldsymbol{\theta}-\boldsymbol{\mu}_0 )^T C_0^{-1} ( \boldsymbol{\theta}-\boldsymbol{\mu}_0 ) \right]. \]The first factor measures data mismatch. The second penalizes departure from prior expectations.
Maximum A Posteriori Estimation
A common Bayesian point estimate is the maximum a posteriori, or MAP, estimate:
\[ \boxed{ \hat{\boldsymbol{\theta}}_{\mathrm{MAP}} = \operatorname*{arg\,max}_{\boldsymbol{\theta}} p(\boldsymbol{\theta}\mid\mathbf{y}) }. \]Equivalently,
\[ \hat{\boldsymbol{\theta}}_{\mathrm{MAP}} = \operatorname*{arg\,min}_{\boldsymbol{\theta}} \left[ -\log p(\mathbf{y}\mid\boldsymbol{\theta}) - \log p(\boldsymbol{\theta}) \right]. \]MAP and Regularization
Under Gaussian likelihood and Gaussian prior assumptions, MAP estimation produces an objective of the form
\[ \left\| \mathbf{y} - F(\boldsymbol{\theta}) \right\|_{\Sigma^{-1}}^2 + \left\| \boldsymbol{\theta} - \boldsymbol{\mu}_0 \right\|_{C_0^{-1}}^2. \]This is closely related to regularized least squares.
Bayesian estimation therefore provides a probabilistic interpretation of many regularization methods.
MLE Versus MAP
Maximum likelihood uses only the likelihood:
\[ \hat{\boldsymbol{\theta}}_{\mathrm{MLE}} = \operatorname*{arg\,max} p(\mathbf{y}\mid\boldsymbol{\theta}). \]MAP estimation includes the prior:
\[ \hat{\boldsymbol{\theta}}_{\mathrm{MAP}} = \operatorname*{arg\,max} p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}). \]The two estimates coincide when the prior is effectively constant over the relevant parameter region.
A Posterior Is More Than a Point Estimate
The central Bayesian object is not the MAP estimate but the entire posterior distribution.
The posterior can describe:
- parameter uncertainty;
- parameter correlation;
- asymmetry;
- multiple plausible modes;
- tail behavior;
- probabilities of parameter ranges.
Posterior Mean
Another common Bayesian estimate is the posterior mean:
\[ \boxed{ \mathbb{E} [ \boldsymbol{\theta} \mid \mathbf{y} ] }. \]The posterior mean averages parameter values according to their posterior probability.
It does not generally equal the posterior mode.
Posterior Median
For a scalar parameter \(\theta\), a posterior median \(m\) satisfies
\[ P(\theta\le m\mid\mathbf{y}) \ge \frac{1}{2} \]and
\[ P(\theta\ge m\mid\mathbf{y}) \ge \frac{1}{2}. \]Mean, median, and mode can differ substantially when the posterior is skewed.
Credible Intervals
A Bayesian credible interval is a parameter interval containing a chosen amount of posterior probability.
For example, a \(95\%\) credible interval
\[ [a,b] \]may satisfy
\[ \boxed{ P( a\le\theta\le b \mid \mathbf{y} ) = 0.95 }. \]Credible Interval Versus Confidence Interval
A Bayesian credible interval directly refers to posterior probability for the parameter under the specified Bayesian model.
A frequentist confidence interval has a different interpretation based on repeated sampling behavior of the interval-generating procedure.
The two may be numerically similar in some problems but are conceptually distinct.
Equal-Tail Credible Intervals
An equal-tail \(95\%\) credible interval places
\[ 2.5\% \]of posterior probability below the lower endpoint and
\[ 2.5\% \]above the upper endpoint.
This is easy to compute from posterior quantiles.
Highest Posterior Density Regions
Another choice is a highest posterior density region, which contains the most probable parameter values until the desired total posterior probability is reached.
Such regions can be shorter than equal-tail intervals for skewed distributions, though they may be more complicated for multimodal posteriors.
Posterior Covariance
For a parameter vector, posterior covariance is
\[ \operatorname{Cov} ( \boldsymbol{\theta} \mid \mathbf{y} ). \]Diagonal terms describe marginal parameter uncertainty.
Off-diagonal terms describe posterior dependence between parameters.
Posterior Correlation
Strong posterior correlation indicates that parameter values can compensate for one another while remaining compatible with the data.
In two dimensions, this often appears as an elongated posterior cloud.
Such geometry is closely related to practical identifiability.
Prior Information
A prior can encode information from:
- earlier experiments;
- physical constraints;
- engineering specifications;
- expert knowledge;
- historical measurements;
- known parameter scales.
The prior should represent information available before incorporating the current data.
Weakly Informative Priors
A weakly informative prior constrains parameters to plausible scales without forcing them into a narrow range.
Such priors can prevent extreme or physically unreasonable estimates while allowing the likelihood to dominate when the data are informative.
Strong Priors
A concentrated prior can strongly influence the posterior.
This may be appropriate when reliable prior knowledge exists.
But when the data are weak, the posterior may largely reproduce the prior.
Therefore, strong priors should be justified and their influence reported.
Improper Priors
Some Bayesian analyses use functions that do not integrate to \(1\) as formal priors.
Such an improper prior can still yield a valid posterior if the posterior itself is normalizable.
Care is required because an improper posterior does not define a probability distribution.
Prior Predictive Distribution
Before observing data, the prior and forward model imply a distribution over possible observations:
\[ \boxed{ p(\mathbf{y}) = \int p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}) \,d\boldsymbol{\theta} }. \]Simulating from this distribution is a powerful way to check whether the prior generates physically plausible data.
Prior Predictive Checks
A prior predictive check asks:
If parameters were drawn from the prior and data were generated through the model, would the resulting observations look reasonable?
Implausible simulated outputs often reveal poorly scaled or overly broad priors before any parameter fitting is attempted.
Posterior Predictive Distribution
After observing data, uncertainty in the parameters can be propagated into a future observation \(\mathbf{y}_{\text{new}}\):
\[ \boxed{ p( \mathbf{y}_{\text{new}} \mid \mathbf{y} ) = \int p( \mathbf{y}_{\text{new}} \mid \boldsymbol{\theta} ) p( \boldsymbol{\theta} \mid \mathbf{y} ) \,d\boldsymbol{\theta} }. \]Posterior Predictive Uncertainty
Predictive uncertainty can contain several sources:
- parameter uncertainty;
- measurement noise;
- latent-state uncertainty;
- future process variability.
Bayesian prediction naturally averages over parameter uncertainty rather than conditioning on only one fitted value.
Posterior Predictive Checks
Simulate replicated data from the fitted model:
\[ \mathbf{y}^{\text{rep}} \sim p( \mathbf{y}^{\text{rep}} \mid \mathbf{y} ). \]Compare these simulated data with the actual observations.
Systematic discrepancies can reveal model misspecification even when the posterior distribution is numerically well behaved.
Bayesian Updating Is Sequential
Today's posterior can become tomorrow's prior.
If data arrive in stages,
\[ \mathbf{y}_1, \mathbf{y}_2, \ldots, \]then after observing \(\mathbf{y}_1\),
\[ p( \boldsymbol{\theta} \mid \mathbf{y}_1 ) \]becomes the prior for the next update.
Sequential Updating Formula
After observing a second independent data set,
\[ p( \boldsymbol{\theta} \mid \mathbf{y}_1,\mathbf{y}_2 ) \propto p( \mathbf{y}_2 \mid \boldsymbol{\theta} ) p( \boldsymbol{\theta} \mid \mathbf{y}_1 ). \]This makes Bayesian inference especially natural for ongoing data assimilation and monitoring.
Bayesian Linear Inverse Problems
Consider
\[ \mathbf{y} = X\boldsymbol{\theta} + \boldsymbol{\varepsilon}, \]with
\[ \boldsymbol{\varepsilon} \sim N(\mathbf{0},\Sigma) \]and prior
\[ \boldsymbol{\theta} \sim N( \boldsymbol{\mu}_0, C_0 ). \]In this conjugate case, the posterior is also Gaussian.
Posterior Covariance in the Linear Gaussian Case
The posterior covariance is
\[ \boxed{ C_{\text{post}} = \left( X^T\Sigma^{-1}X + C_0^{-1} \right)^{-1} }. \]The term
\[ X^T\Sigma^{-1}X \]represents information from the data, while
\[ C_0^{-1} \]represents information from the prior.
Posterior Mean in the Linear Gaussian Case
The posterior mean is
\[ \boxed{ \boldsymbol{\mu}_{\text{post}} = C_{\text{post}} \left( X^T\Sigma^{-1}\mathbf{y} + C_0^{-1}\boldsymbol{\mu}_0 \right) }. \]The posterior combines data-driven and prior-driven information according to their relative precision.
Precision Matrices
The inverse of a covariance matrix is called a precision matrix.
In the linear Gaussian posterior,
\[ C_{\text{post}}^{-1} = X^T\Sigma^{-1}X + C_0^{-1}. \]Posterior precision is the sum of data precision and prior precision.
This provides a useful information-accounting interpretation of Bayesian updating.
When the Data Are Very Informative
If the likelihood is sharply concentrated compared with the prior, the posterior is dominated by the data.
In this regime, different reasonable priors may produce very similar posteriors.
When the Data Are Weak
If the likelihood is broad or nearly flat, the prior can strongly affect the posterior.
This is not automatically a defect. It accurately reflects the fact that the observations alone do not contain much information about the parameter.
The contribution of prior information should then be made explicit.
Bayesian Identifiability
A prior can turn a mathematically nonunique likelihood into a proper posterior with a unique mode or finite variance.
However, this does not mean the data have suddenly identified the parameter.
The posterior may be concentrated because of prior information.
Distinguish carefully between:
- identification by the data;
- constraint by the prior;
- constraint by hard parameter bounds.
Prior Sensitivity
A useful diagnostic is to repeat the analysis under several plausible priors.
If the posterior conclusions change substantially, then the data are not strong enough to overwhelm prior assumptions.
Prior sensitivity is therefore an important part of transparent Bayesian analysis.
Hierarchical Models
Bayesian models can contain parameters whose distributions depend on higher level parameters.
For example,
\[ \theta_j \sim N(\mu,\tau^2), \]where \(\mu\) and \(\tau\) are themselves unknown.
Such hierarchical models allow information to be shared across groups, experiments, or related systems.
Partial Pooling
In a hierarchical model, individual group parameters are influenced by both their own data and the larger population.
This creates partial pooling:
- more pooling than fitting every group independently;
- less pooling than forcing every group to have the same parameter.
Partial pooling can improve estimation when groups have limited data.
Hyperparameters
Parameters controlling a prior distribution are often called hyperparameters.
For example, in
\[ \theta \sim N(\mu_0,\tau^2), \]the quantities \(\mu_0\) and \(\tau\) are hyperparameters.
They may be fixed from prior knowledge or estimated within a larger hierarchical model.
Marginalization
Bayesian inference often removes nuisance parameters by integration rather than fixing them.
If
\[ p(\theta,\phi\mid\mathbf{y}) \]is a joint posterior, then the marginal posterior of \(\theta\) is
\[ \boxed{ p(\theta\mid\mathbf{y}) = \int p(\theta,\phi\mid\mathbf{y}) \,d\phi }. \]This propagates uncertainty in the nuisance parameter \(\phi\) into the uncertainty of \(\theta\).
Why Exact Integration Is Often Difficult
In simple conjugate models, posterior integrals can be computed analytically.
In nonlinear inverse problems with many parameters, the posterior may be:
- high dimensional;
- strongly correlated;
- non-Gaussian;
- multimodal;
- expensive to evaluate.
Numerical approximation is then required.
Grid-Based Posterior Calculation
For one or two parameters, the posterior can be evaluated directly on a grid.
At each grid point, compute
\[ p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}). \]The resulting values can be normalized numerically.
Grid methods become impractical as dimension increases.
Laplace Approximation
If the posterior is smooth and concentrated near one mode, it can be approximated locally by a Gaussian distribution.
The mode provides the center, while the inverse negative Hessian of the log posterior provides an approximate covariance.
This is called a Laplace approximation.
Monte Carlo Sampling
Another strategy is to represent the posterior using random samples:
\[ \boldsymbol{\theta}^{(1)}, \ldots, \boldsymbol{\theta}^{(M)} \sim p( \boldsymbol{\theta} \mid \mathbf{y} ). \]Posterior means, intervals, probabilities, and predictions can then be approximated from the sample.
Markov Chain Monte Carlo
Markov chain Monte Carlo, or MCMC, constructs a sequence of parameter values whose long-run distribution is the target posterior.
The samples are generally dependent rather than independent.
Common MCMC ideas include:
- Metropolis-type proposals;
- Gibbs sampling;
- Hamiltonian methods;
- adaptive proposal mechanisms.
The Metropolis Idea
Starting from a current parameter value \(\boldsymbol{\theta}\), propose a new value
\[ \boldsymbol{\theta}'. \]Accept or reject the proposal using a rule based on the posterior density ratio.
Repeated application creates a chain that explores the posterior.
MCMC Is Not Just Optimization
An optimizer seeks a high-probability point.
MCMC seeks to explore the entire posterior distribution.
Therefore, a successful MCMC method must move through regions in proportion to their posterior probability rather than simply converge to the mode.
Burn-In and Warmup
Early samples may reflect the arbitrary initial state more strongly than the target posterior.
Many algorithms use an initial warmup period to adapt tuning parameters or move toward the high-probability region before samples are retained for inference.
Autocorrelation
MCMC samples are usually correlated across iterations.
Strong autocorrelation means the chain moves slowly through parameter space.
As a result, \(10{,}000\) MCMC iterations may contain much less information than \(10{,}000\) independent samples.
Effective Sample Size
The effective sample size estimates how many independent samples would contain roughly the same amount of information as the correlated MCMC output.
Larger effective sample size generally means more precise Monte Carlo estimates.
Multiple Chains
Running several chains from dispersed initial values helps diagnose whether the sampler has explored the same posterior region.
If different chains remain in different locations, the posterior may be multimodal or the sampler may not have converged adequately.
Trace Plots
A trace plot shows sampled parameter values against iteration number.
Useful trace plots typically show:
- stable long-run behavior;
- frequent movement through the sampled region;
- similar behavior across chains;
- no persistent trend.
Posterior Multimodality
A posterior can have several distinct modes.
This may arise from:
- symmetries in the forward model;
- multiple plausible explanations of the data;
- weak identifiability;
- mixture structures;
- nonlinear inverse mappings.
Reporting only one posterior mode can hide this structure.
Posterior Geometry
A posterior may contain:
- compact peaks;
- long correlated ridges;
- curved valleys;
- multiple separated modes;
- heavy tails.
Posterior geometry directly affects both scientific interpretation and computational difficulty.
Bayesian Model Comparison
The evidence
\[ p(\mathbf{y}\mid M) \]can be used to compare models \(M\).
For two models,
\[ \frac{ p(\mathbf{y}\mid M_1) }{ p(\mathbf{y}\mid M_2) } \]is a Bayes factor.
Unlike maximum likelihood alone, the evidence averages over parameter uncertainty and naturally reflects model complexity.
Evidence Depends on the Prior
Because
\[ p(\mathbf{y}) = \int p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}) \,d\boldsymbol{\theta}, \]model evidence can be sensitive to prior choice.
This is especially important when priors are very broad or weakly justified.
Decision-Making from the Posterior
Bayesian inference can feed directly into decisions.
If an action \(a\) has loss
\[ L(a,\boldsymbol{\theta}), \]one may choose the action minimizing posterior expected loss:
\[ \boxed{ a^* = \operatorname*{arg\,min}_{a} \mathbb{E} [ L(a,\boldsymbol{\theta}) \mid \mathbf{y} ] }. \]This links parameter uncertainty to rational decision analysis.
Posterior Probability Statements
Bayesian inference can answer questions such as
\[ P(\theta>0\mid\mathbf{y}), \]or
\[ P(\theta_1>\theta_2\mid\mathbf{y}). \]These statements summarize posterior uncertainty directly.
Bayesian Experimental Design
Future experiments can be chosen to maximize expected information gain or reduce expected posterior uncertainty.
Conceptually,
\[ \text{choose experiment} \longrightarrow \text{observe data} \longrightarrow \text{update posterior}. \]Experimental design can therefore be treated as an optimization problem over anticipated Bayesian learning.
Bayesian Data Assimilation
When new observations arrive over time, Bayesian updating becomes a natural framework for state and parameter estimation.
This idea underlies:
- Kalman filtering;
- ensemble filters;
- particle filtering;
- sequential Bayesian estimation.
A later lesson develops data assimilation directly.
Computational Cost in Inverse Problems
Bayesian inverse problems can be expensive because posterior evaluation often requires a forward-model evaluation.
If MCMC requires thousands or millions of posterior evaluations, a costly simulation can dominate the entire computation.
Strategies include:
- surrogate models;
- reduced-order models;
- parallel computation;
- gradient-based samplers;
- multilevel methods;
- careful parameter reduction.
Common Error: Treating the Prior as Optional Decoration
The prior is mathematically part of the Bayesian model.
Changing the prior changes the posterior, especially when the data are weak.
Priors should therefore be justified, documented, and tested for sensitivity.
Common Error: Saying the Posterior Comes Only from the Data
The posterior reflects both likelihood and prior:
\[ p(\boldsymbol{\theta}\mid\mathbf{y}) \propto p(\mathbf{y}\mid\boldsymbol{\theta}) p(\boldsymbol{\theta}). \]When the prior is influential, that influence should be acknowledged rather than attributed to the observations.
Common Error: Confusing MAP with the Posterior
The MAP estimate is only the location of the posterior mode.
It does not describe:
- posterior width;
- asymmetry;
- correlation;
- multiple modes;
- tail probability.
Common Error: Ignoring Prior Sensitivity
A conclusion that depends strongly on a narrowly chosen prior may be fragile.
Repeating the analysis with several scientifically plausible priors helps reveal how much of the conclusion is data-driven.
Common Error: Trusting MCMC Output Without Diagnostics
A sampler can produce thousands of numbers without adequately exploring the posterior.
Always inspect convergence, effective sample size, trace behavior, and consistency across chains.
Common Error: Treating a Narrow Posterior as Proof of Identifiability
A narrow posterior can result from a strong prior even when the likelihood is weak.
To understand identifiability, inspect both the likelihood and the prior contribution.
Common Error: Ignoring Posterior Predictive Checks
A mathematically well-behaved posterior does not guarantee that the model reproduces important features of the observed data.
Posterior predictive checks test whether the fitted model can generate data resembling what was actually observed.
A Reliable Bayesian Estimation Workflow
- Define the forward model and observation model.
- Identify the unknown parameters and latent quantities.
- Specify scientifically justified priors.
- Perform prior predictive checks.
- Write the likelihood.
- Form the posterior up to proportionality.
- Check structural and practical identifiability.
- Choose an appropriate computational method.
- Diagnose optimization or sampling performance.
- Summarize posterior means, medians, modes, and intervals as appropriate.
- Inspect posterior dependence among parameters.
- Perform posterior predictive checks.
- Test sensitivity to prior assumptions.
- Propagate posterior uncertainty into predictions or decisions.
Practice 1: Bayes' Rule
Write Bayes' rule for a parameter \(\theta\) and observed data \(y\).
Practice 2: Prior and Likelihood
In Bayesian estimation, which term represents information available before the current data are observed?
Practice 3: Beta-Binomial Update
Suppose
\[ p \sim \operatorname{Beta}(2,3) \]and the data contain \(8\) heads and \(2\) tails.
What is the posterior distribution?
Practice 4: MAP Versus MLE
What additional term affects the MAP estimate that does not affect the MLE?
Practice 5: Credible Interval
What does a \(95\%\) Bayesian credible interval represent?
Practice 6: Posterior Predictive Distribution
Why does a posterior predictive distribution integrate over the posterior distribution of the parameters?
Practice 7: Prior Sensitivity
Two scientifically plausible priors produce substantially different posteriors from the same data.
What does this suggest?
Practice 8: Linear Gaussian Posterior
In the linear Gaussian model, what happens to posterior precision when data information and prior information are combined?
Practice 9: MCMC
What is the primary goal of MCMC: finding the posterior mode or exploring the posterior distribution?
Practice 10: Narrow Posterior
A parameter has a narrow posterior but a nearly flat likelihood.
What is the most likely source of the posterior concentration?
Practice Answers
Practice 1
\[ \boxed{ p(\theta\mid y) = \frac{ p(y\mid\theta)p(\theta) }{ p(y) } }. \]Practice 2
The prior distribution
\[ p(\theta) \]represents information or uncertainty before the current data are included.
Practice 3
The Beta prior parameters are updated by the observed head and tail counts:
\[ \alpha_{\text{post}} = 2+8 = 10, \] \[ \beta_{\text{post}} = 3+2 = 5. \]Therefore,
\[ \boxed{ p\mid\mathbf{y} \sim \operatorname{Beta}(10,5) }. \]Practice 4
The MAP estimate includes the prior density:
\[ p(\theta). \]MLE uses the likelihood alone.
Practice 5
Under the specified Bayesian model, a \(95\%\) credible interval contains \(95\%\) of the posterior probability for the parameter.
Practice 6
The parameter is not known exactly after fitting. Integrating over the posterior propagates parameter uncertainty into the predictive uncertainty for future observations.
Practice 7
The data are not strong enough to dominate the prior assumptions. The inference is prior-sensitive and should be reported as such.
Practice 8
The precision contributions add:
\[ \boxed{ C_{\text{post}}^{-1} = X^T\Sigma^{-1}X + C_0^{-1} }. \]Practice 9
The primary goal of MCMC is to explore the posterior distribution, not merely find its mode.
Practice 10
A nearly flat likelihood supplies little data information, so the narrow posterior is most likely being produced by a strong or concentrated prior.
What to Remember
Bayesian estimation updates parameter uncertainty by combining prior information with the likelihood of the observed data.
- Bayes' rule gives \(p(\boldsymbol{\theta}\mid\mathbf{y})\propto p(\mathbf{y}\mid\boldsymbol{\theta})p(\boldsymbol{\theta})\).
- The prior represents information before the current observations.
- The likelihood represents information supplied by the data.
- The posterior represents updated uncertainty after observing the data.
- MAP estimation is a point estimate based on the posterior mode.
- Credible intervals summarize posterior probability directly.
- Posterior covariance and correlation reveal parameter dependence.
- Prior predictive checks test whether priors generate plausible observations.
- Posterior predictive checks test whether the fitted model reproduces observed structure.
- Strong priors can stabilize poorly identified inverse problems.
- MCMC and related methods approximate posteriors that cannot be integrated analytically.
- Prior sensitivity and sampling diagnostics are essential for trustworthy inference.
The central relationship is:
\[ \boxed{ \text{posterior information} = \text{data information} + \text{prior information} } \]in the broad conceptual sense that Bayesian updating combines what was known before with what the new observations reveal.
Where We Go Next
Bayesian estimation provides a complete probability distribution over uncertain parameters, but inverse problems can still be unstable when small observational changes produce large parameter changes or when many parameter vectors explain the data almost equally well.
The next lesson, Regularization, develops methods for stabilizing such problems by adding structural preferences, penalties, or prior-like constraints to the estimation process.