The Central Idea
Suppose a finite Markov chain has ordered state space
\[ S=\{1,2,\ldots,m\}. \]If
\[ p_{ij}=P(X_{n+1}=j\mid X_n=i), \]then the transition matrix is
\[ \boxed{ P= \begin{bmatrix} p_{11} & p_{12} & \cdots & p_{1m}\\ p_{21} & p_{22} & \cdots & p_{2m}\\ \vdots & \vdots & \ddots & \vdots\\ p_{m1} & p_{m2} & \cdots & p_{mm} \end{bmatrix} }. \]In this lesson, rows represent current states and columns represent next states.
From Transition Probabilities to a Matrix
Consider a chain with states
\[ S=\{A,B\}. \]Suppose
\[ p_{AA}=0.8,\qquad p_{AB}=0.2, \] \[ p_{BA}=0.3,\qquad p_{BB}=0.7. \]Using the state order \(A,B\), the transition matrix is
\[ \boxed{ P= \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} }. \]Reading an Entry
In the matrix
\[ P= \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix}, \]the entry in row \(A\), column \(B\) is \(0.2\).
Therefore,
\[ P(X_{n+1}=B\mid X_n=A)=0.2. \]The matrix entry follows the same directional convention as \(p_{ij}\): from the row state to the column state.
Why the State Order Matters
A transition matrix is meaningful only when the state order is known.
If the states are ordered as
\[ A,B,C, \]then row \(2\) and column \(3\) refer to the transition
\[ B\to C. \]Reordering the states changes the appearance of the matrix even when the underlying Markov chain is unchanged.
Transition Matrices Are Row-Stochastic
Under the convention used here, each row contains all possible next-state probabilities from one current state.
Therefore,
\[ \boxed{\sum_{j=1}^{m}p_{ij}=1} \]for every row \(i\).
A matrix with nonnegative entries and row sums equal to \(1\) is called row-stochastic.
Conditions for a Valid Transition Matrix
A finite matrix \(P=[p_{ij}]\) is a valid transition matrix under the row-stochastic convention when:
- every entry satisfies \(0\le p_{ij}\le 1\);
- every row sums to \(1\);
- the row and column labels refer to the same ordered state space.
Example: Checking a Matrix
Consider
\[ P= \begin{bmatrix} 0.4 & 0.5 & 0.1\\ 0.2 & 0.6 & 0.2\\ 0.0 & 0.3 & 0.7 \end{bmatrix}. \]The row sums are
\[ 0.4+0.5+0.1=1, \] \[ 0.2+0.6+0.2=1, \] \[ 0.0+0.3+0.7=1. \]All entries are nonnegative, so the matrix is a valid row-stochastic transition matrix.
Example: An Invalid Matrix
Consider
\[ Q= \begin{bmatrix} 0.6 & 0.5\\ 0.2 & 0.8 \end{bmatrix}. \]The first row sums to
\[ 0.6+0.5=1.1. \]Therefore, \(Q\) cannot represent a complete set of one-step transition probabilities under the row-stochastic convention.
Zeros in a Transition Matrix
An entry
\[ p_{ij}=0 \]means that the transition from state \(i\) directly to state \(j\) is impossible in one step.
It does not necessarily mean that \(j\) can never be reached from \(i\). A multi-step path through intermediate states may still exist.
Ones in a Transition Matrix
If
\[ p_{ij}=1, \]then the next state is certainly \(j\) whenever the current state is \(i\).
In particular, if
\[ p_{ii}=1, \]then \(i\) is an absorbing state.
Absorbing States in Matrix Form
Suppose state \(C\) is absorbing in the ordered state space
\[ S=\{A,B,C\}. \]Its row must be
\[ \begin{bmatrix} 0 & 0 & 1 \end{bmatrix}. \]Once the process reaches \(C\), the transition matrix sends it back to \(C\) with probability \(1\).
Building a Matrix from a Transition Table
Suppose a chain has states \(A,B,C\) and the following transition table.
| Current State | Next \(A\) | Next \(B\) | Next \(C\) |
|---|---|---|---|
| \(A\) | \(0.6\) | \(0.3\) | \(0.1\) |
| \(B\) | \(0.2\) | \(0.5\) | \(0.3\) |
| \(C\) | \(0.0\) | \(0.4\) | \(0.6\) |
Using the order \(A,B,C\),
\[ \boxed{ P= \begin{bmatrix} 0.6 & 0.3 & 0.1\\ 0.2 & 0.5 & 0.3\\ 0.0 & 0.4 & 0.6 \end{bmatrix} }. \]Building a Matrix from a Transition Diagram
A transition diagram and a transition matrix contain the same one-step information.
To convert a diagram into a matrix:
- choose and record a state order;
- use one row for each starting state;
- use one column for each destination state;
- place each arrow probability in the corresponding entry;
- enter \(0\) when a direct transition is impossible;
- verify that every row sums to \(1\).
State Distributions
A transition matrix describes how the chain moves. To describe where the chain currently is, we use a probability distribution over the states.
For states
\[ S=\{1,2,\ldots,m\}, \]define the row vector
\[ \boldsymbol{\pi}_n = \begin{bmatrix} P(X_n=1) & P(X_n=2) & \cdots & P(X_n=m) \end{bmatrix}. \]A State Distribution Is Also Stochastic
Because the chain must be in exactly one state at time \(n\),
\[ \pi_{n,j}\ge 0 \]and
\[ \boxed{\sum_{j=1}^{m}\pi_{n,j}=1}. \]Thus, a state-distribution vector is itself a probability distribution.
A Known Initial State
If a two-state chain begins in state \(A\) with certainty, then using state order \(A,B\),
\[ \boldsymbol{\pi}_0 = \begin{bmatrix} 1 & 0 \end{bmatrix}. \]If it begins in \(B\) with certainty, then
\[ \boldsymbol{\pi}_0 = \begin{bmatrix} 0 & 1 \end{bmatrix}. \]An Uncertain Initial State
Suppose
\[ P(X_0=A)=0.25,\qquad P(X_0=B)=0.75. \]Then
\[ \boldsymbol{\pi}_0 = \begin{bmatrix} 0.25 & 0.75 \end{bmatrix}. \]The vector records uncertainty about the starting state.
Updating a State Distribution
Under the row-vector convention used in this lesson, the distribution one step later is
\[ \boxed{\boldsymbol{\pi}_{n+1}=\boldsymbol{\pi}_nP}. \]Matrix multiplication automatically combines the possible current states with their outgoing transition probabilities.
Why the Update Formula Works
The \(j\)-th entry of
\[ \boldsymbol{\pi}_{n+1}=\boldsymbol{\pi}_nP \]is
\[ \pi_{n+1,j} = \sum_{i=1}^{m}\pi_{n,i}p_{ij}. \]This is exactly the law of total probability:
\[ P(X_{n+1}=j) = \sum_i P(X_n=i)P(X_{n+1}=j\mid X_n=i). \]Example: One-Step Update from a Known State
Let
\[ P= \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} \]and suppose the chain begins in \(A\):
\[ \boldsymbol{\pi}_0 = \begin{bmatrix} 1 & 0 \end{bmatrix}. \]Then
\[ \boldsymbol{\pi}_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} = \boxed{ \begin{bmatrix} 0.8 & 0.2 \end{bmatrix} }. \]This simply reproduces the row of transition probabilities leaving \(A\).
Example: One-Step Update from an Uncertain State
Using the same matrix, let
\[ \boldsymbol{\pi}_0 = \begin{bmatrix} 0.25 & 0.75 \end{bmatrix}. \]Then
\[ \boldsymbol{\pi}_1 = \begin{bmatrix} 0.25 & 0.75 \end{bmatrix} \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix}. \]The first component is
\[ (0.25)(0.8)+(0.75)(0.3)=0.425, \]and the second is
\[ (0.25)(0.2)+(0.75)(0.7)=0.575. \]Therefore,
\[ \boxed{ \boldsymbol{\pi}_1 = \begin{bmatrix} 0.425 & 0.575 \end{bmatrix} }. \]Checking an Updated Distribution
After a matrix update, the resulting vector should still be a valid probability distribution.
In the preceding example,
\[ 0.425+0.575=1. \]A sum different from \(1\) signals either a calculation error or an invalid transition matrix.
Two-Step Transitions
Applying the transition rule twice gives
\[ \boldsymbol{\pi}_2 = \boldsymbol{\pi}_1P. \]Since
\[ \boldsymbol{\pi}_1=\boldsymbol{\pi}_0P, \]we obtain
\[ \boldsymbol{\pi}_2 = (\boldsymbol{\pi}_0P)P = \boxed{\boldsymbol{\pi}_0P^2}. \]The Meaning of \(P^2\)
The matrix
\[ P^2=PP \]contains two-step transition probabilities.
Its \((i,j)\)-entry is
\[ \boxed{ p_{ij}^{(2)} = P(X_{n+2}=j\mid X_n=i) }. \]Matrix multiplication automatically sums the probabilities of all possible two-step paths from \(i\) to \(j\).
Why Matrix Multiplication Matches Path Counting
The \((i,j)\)-entry of \(P^2\) is
\[ (P^2)_{ij} = \sum_{k=1}^{m}p_{ik}p_{kj}. \]Each product
\[ p_{ik}p_{kj} \]is the probability of the path
\[ i\to k\to j. \]The sum combines the mutually exclusive choices of intermediate state \(k\).
Example: Squaring a Two-State Matrix
Let
\[ P= \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix}. \]Then
\[ P^2 = \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix}. \]Multiplying gives
\[ \boxed{ P^2= \begin{bmatrix} 0.70 & 0.30\\ 0.45 & 0.55 \end{bmatrix} }. \]Interpreting an Entry of \(P^2\)
In the preceding example,
\[ (P^2)_{AB}=0.30. \]Therefore,
\[ P(X_{n+2}=B\mid X_n=A)=0.30. \]This probability includes both possible two-step routes:
\[ A\to A\to B \]and
\[ A\to B\to B. \]Verifying a Two-Step Entry by Paths
Starting from \(A\), the probability of
\[ A\to A\to B \]is
\[ (0.8)(0.2)=0.16. \]The probability of
\[ A\to B\to B \]is
\[ (0.2)(0.7)=0.14. \]Adding the alternative routes gives
\[ 0.16+0.14=\boxed{0.30}, \]matching the \((A,B)\)-entry of \(P^2\).
Three-State Example
Consider
\[ P= \begin{bmatrix} 0.6 & 0.3 & 0.1\\ 0.2 & 0.5 & 0.3\\ 0.0 & 0.4 & 0.6 \end{bmatrix}. \]Squaring gives
\[ \boxed{ P^2= \begin{bmatrix} 0.42 & 0.37 & 0.21\\ 0.22 & 0.43 & 0.35\\ 0.08 & 0.44 & 0.48 \end{bmatrix} }. \]Thus, starting from \(A\), the probability of being in \(C\) after two steps is \(0.21\).
Checking the Three-State Calculation
The two-step probability from \(A\) to \(C\) can be computed directly:
\[ p_{AA}p_{AC} + p_{AB}p_{BC} + p_{AC}p_{CC}. \]Substituting the values gives
\[ (0.6)(0.1)+(0.3)(0.3)+(0.1)(0.6) = 0.06+0.09+0.06 = \boxed{0.21}. \]\(k\)-Step Transition Matrices
More generally,
\[ \boxed{P^k} \]is the \(k\)-step transition matrix.
Its entries satisfy
\[ \boxed{ (P^k)_{ij} = P(X_{n+k}=j\mid X_n=i) }. \]This interpretation assumes a time-homogeneous Markov chain so that the same transition matrix applies at every step.
Updating Several Steps at Once
If the initial state distribution is
\[ \boldsymbol{\pi}_0, \]then after \(k\) steps,
\[ \boxed{ \boldsymbol{\pi}_k = \boldsymbol{\pi}_0P^k }. \]This formula combines the starting uncertainty and all \(k\)-step transitions.
Example: Two Steps from a Known State
Using
\[ P^2= \begin{bmatrix} 0.70 & 0.30\\ 0.45 & 0.55 \end{bmatrix} \]and
\[ \boldsymbol{\pi}_0= \begin{bmatrix} 1 & 0 \end{bmatrix}, \]we obtain
\[ \boldsymbol{\pi}_2 = \boldsymbol{\pi}_0P^2 = \boxed{ \begin{bmatrix} 0.70 & 0.30 \end{bmatrix} }. \]Starting from \(A\), there is a \(0.70\) probability of being in \(A\) and a \(0.30\) probability of being in \(B\) after two steps.
The Identity Matrix Represents Zero Steps
Before any transition occurs, each starting state is still itself.
Therefore,
\[ P^0=I, \]where \(I\) is the identity matrix.
The entry interpretation remains consistent:
\[ (P^0)_{ij} = \begin{cases} 1,&i=j,\\ 0,&i\ne j. \end{cases} \]Matrix Powers Preserve the Row-Sum Property
If \(P\) is row-stochastic, then
\[ P^2,P^3,\ldots \]are also row-stochastic.
This makes sense probabilistically: each row of \(P^k\) is a complete probability distribution over the state occupied after \(k\) steps, given a particular starting state.
Matrix Multiplication Is Not Entrywise Multiplication
To compute \(P^2\), do not simply square every entry.
In general,
\[ P^2\ne \begin{bmatrix} p_{11}^2 & p_{12}^2\\ p_{21}^2 & p_{22}^2 \end{bmatrix}. \]Standard matrix multiplication is required because multi-step probabilities must account for intermediate states.
Matrix Multiplication Encodes Total Probability
The formula
\[ (P^2)_{ij} = \sum_k p_{ik}p_{kj} \]is not merely an algebraic rule.
It is the law of total probability written in matrix form. Each possible intermediate state contributes one route to the final state.
Transition Matrices and Reachability
Matrix powers provide a convenient test for multi-step accessibility.
If
\[ (P^k)_{ij}>0 \]for some \(k\ge 1\), then state \(j\) is reachable from state \(i\) in \(k\) steps.
Thus, entries of matrix powers connect numerical computation with the structural idea of reachability.
Transition Matrices and Communication
States \(i\) and \(j\) communicate when each can reach the other.
In matrix terms, this means there exist positive integers \(r\) and \(s\) such that
\[ (P^r)_{ij}>0 \]and
\[ (P^s)_{ji}>0. \]The matrix therefore contains both probability values and information about the connectivity of the chain.
Sparse Transition Matrices
Many Markov chains allow only a few transitions from each state.
Their transition matrices contain many zeros and are called sparse.
Random walks, queue-length models, population models, and reliability systems often have sparse transition structures because a state can change only in limited ways during one step.
Example: Nearest-Neighbor Structure
Suppose states are
\[ S=\{0,1,2,3\}, \]and the process can move only one state left, one state right, or remain where it is.
Then entries corresponding to jumps such as
\[ 0\to 3 \]must be zero.
The matrix pattern itself reveals which one-step movements are impossible.
Transition Counts from Data
Observed Markov data can be summarized in a transition-count matrix
\[ N=[N_{ij}], \]where \(N_{ij}\) counts how many observed transitions went from state \(i\) to state \(j\).
A row total
\[ N_i=\sum_jN_{ij} \]counts how many observed transitions began in state \(i\).
Estimating a Transition Matrix
A natural empirical estimate is obtained by dividing each transition count by its row total:
\[ \boxed{ \widehat p_{ij} = \frac{N_{ij}}{N_i} }. \]In matrix language, each row of transition counts is normalized so that it sums to \(1\).
Example: Estimating from Counts
Suppose the observed transition counts are
\[ N= \begin{bmatrix} 80 & 20\\ 30 & 70 \end{bmatrix}. \]Each row total is \(100\), so
\[ \widehat P = \boxed{ \begin{bmatrix} 0.80 & 0.20\\ 0.30 & 0.70 \end{bmatrix} }. \]This matrix describes observed transition frequencies and serves as an estimate of the underlying transition probabilities.
Zero Observed Transitions Require Care
If an observed count \(N_{ij}\) is zero, the empirical estimate
\[ \widehat p_{ij}=0 \]does not necessarily prove that the transition is theoretically impossible.
It may simply mean that the transition did not appear in the available sample. Model structure and sample size must be considered separately from the raw estimate.
Time-Varying Transition Matrices
A time-inhomogeneous Markov chain can use a different transition matrix at each step:
\[ P_0,P_1,P_2,\ldots \]Then
\[ \boldsymbol{\pi}_{n+1} = \boldsymbol{\pi}_nP_n. \]The multi-step update becomes an ordered product rather than a single matrix power.
Why Order Matters for Time-Varying Matrices
Starting from \(\boldsymbol{\pi}_0\),
\[ \boldsymbol{\pi}_1=\boldsymbol{\pi}_0P_0, \]and
\[ \boldsymbol{\pi}_2 = \boldsymbol{\pi}_0P_0P_1. \]In general, matrix multiplication is not commutative:
\[ P_0P_1\ne P_1P_0. \]The chronological order of the transition rules must therefore be preserved.
Row-Vector and Column-Vector Conventions
Different textbooks use different conventions.
This lesson uses row probability vectors and row-stochastic matrices:
\[ \boldsymbol{\pi}_{n+1} = \boldsymbol{\pi}_nP. \]Some sources use column probability vectors instead. Under a common column-vector convention, the matrix is transposed and the update is written
\[ \boldsymbol{\pi}_{n+1} = P^\mathsf{T}\boldsymbol{\pi}_n. \]Both conventions describe the same probabilities when used consistently.
Never Mix Conventions Mid-Calculation
The most important issue is consistency.
If rows represent current states and probability distributions are row vectors, use
\[ \boldsymbol{\pi}_{n+1} = \boldsymbol{\pi}_nP. \]Switching to a column-vector formula without transposing the matrix changes the calculation.
Transition Matrices as Linear Transformations
Matrix multiplication maps one state-distribution vector to another:
\[ \boldsymbol{\pi} \longmapsto \boldsymbol{\pi}P. \]Because this operation is linear, the tools of linear algebra become available for studying Markov chains.
This connection is one reason transition matrices are so powerful.
Probability Is Preserved by the Transformation
If \(\boldsymbol{\pi}\) is a valid probability vector and \(P\) is row-stochastic, then
\[ \boldsymbol{\pi}P \]is also a valid probability vector.
The matrix transformation preserves nonnegativity and total probability.
Repeated Multiplication and Long-Run Questions
Repeated matrix powers
\[ P,P^2,P^3,\ldots \]reveal how transition probabilities change over longer horizons.
In some chains, the rows of \(P^n\) approach a common probability distribution. In others, periodicity, reducibility, or absorbing behavior produces a different long-run pattern.
The next lesson studies the distributions that remain unchanged under the transition rule.
A Fixed Distribution
A probability vector \(\boldsymbol{\pi}\) is unchanged by one transition when
\[ \boldsymbol{\pi}P=\boldsymbol{\pi}. \]Such distributions are called stationary distributions.
This equation is introduced here only to show the natural next question created by transition-matrix iteration. Its interpretation and computation are developed in the next lesson.
Common Error: Summing Columns Instead of Rows
Under the convention used here, the outgoing probabilities from a current state occupy one row.
Therefore, it is the rows that must sum to \(1\).
Column sums need not equal \(1\).
Common Error: Forgetting the State Order
A matrix such as
\[ \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} \]cannot be interpreted correctly unless the ordering of its states is known.
Always record the state order when constructing or communicating a transition matrix.
Common Error: Reversing Row and Column Meaning
In this lesson,
\[ p_{ij} \]appears in row \(i\), column \(j\), and means
\[ i\to j. \]Reading it as \(j\to i\) reverses the transition.
Common Error: Squaring Entries Instead of the Matrix
The matrix power
\[ P^2 \]means
\[ PP, \]using standard matrix multiplication.
It does not mean squaring each probability separately.
Common Error: Using \(P^n\) for a Time-Varying Chain
The formula
\[ \boldsymbol{\pi}_n = \boldsymbol{\pi}_0P^n \]assumes that the same transition matrix \(P\) applies at every step.
If the transition rules change with time, the appropriate ordered product of matrices must be used instead.
Common Error: Treating Matrix Entries as Counts
A transition matrix contains probabilities, not raw observations.
A count matrix such as
\[ \begin{bmatrix} 80 & 20\\ 30 & 70 \end{bmatrix} \]must be normalized by row totals before it becomes an estimated transition matrix.
Common Error: Expecting Every Matrix to Converge
Repeated powers of a transition matrix do not always settle into one simple limiting matrix.
Long-run behavior depends on structural features such as irreducibility, periodicity, closed classes, and absorbing states.
Stationary distributions and limiting distributions are related concepts, but they are not automatically identical.
A Reliable Procedure for Constructing a Transition Matrix
- List the states in a fixed order.
- Use that same order for both rows and columns.
- Interpret each row as a current state.
- Interpret each column as a next state.
- Place \(p_{ij}\) in row \(i\), column \(j\).
- Use \(0\) for impossible one-step transitions.
- Check that every entry lies between \(0\) and \(1\).
- Check that every row sums to \(1\).
- Record the state order beside the matrix.
A Reliable Procedure for Updating a Distribution
- Write the current distribution as a row vector in the chosen state order.
- Check that its entries are nonnegative and sum to \(1\).
- Multiply the row vector by the transition matrix on the right.
- Interpret each resulting entry as the next probability of the corresponding state.
- Check that the resulting entries still sum to \(1\).
- Repeat the process or use a matrix power for additional steps.
A Reliable Procedure for \(k\)-Step Probabilities
- Confirm that the chain is using one time-homogeneous transition matrix \(P\).
- Compute \(P^k\) using matrix multiplication.
- Read \((P^k)_{ij}\) as the probability of moving from \(i\) to \(j\) in exactly \(k\) steps.
- If an initial distribution is given, compute \(\boldsymbol{\pi}_0P^k\).
- Check that every row of \(P^k\) and the resulting probability vector sum to \(1\).
Example: Full Transition-Matrix Workflow
Suppose a customer may be Active or Inactive, with state order
\[ S=\{A,I\}. \]Assume
\[ P= \begin{bmatrix} 0.85 & 0.15\\ 0.25 & 0.75 \end{bmatrix}. \]Each row sums to \(1\), so the matrix is row-stochastic.
If the customer starts Active,
\[ \boldsymbol{\pi}_0= \begin{bmatrix} 1 & 0 \end{bmatrix}. \]After one period,
\[ \boldsymbol{\pi}_1 = \boldsymbol{\pi}_0P = \begin{bmatrix} 0.85 & 0.15 \end{bmatrix}. \]The two-step transition matrix is
\[ P^2 = \begin{bmatrix} 0.76 & 0.24\\ 0.40 & 0.60 \end{bmatrix}. \]Therefore,
\[ \boldsymbol{\pi}_2 = \boldsymbol{\pi}_0P^2 = \boxed{ \begin{bmatrix} 0.76 & 0.24 \end{bmatrix} }. \]Starting Active, the customer has probability \(0.76\) of being Active two periods later.
Practice 1: Construct a Matrix
A two-state chain has states \(A,B\) and transition probabilities
\[ p_{AA}=0.7,\qquad p_{AB}=0.3, \] \[ p_{BA}=0.4,\qquad p_{BB}=0.6. \]Write the transition matrix using state order \(A,B\).
Practice 2: Check Validity
Is
\[ P= \begin{bmatrix} 0.5 & 0.5\\ 0.3 & 0.8 \end{bmatrix} \]a valid row-stochastic transition matrix?
Practice 3: Find a Missing Entry
A row of a three-state transition matrix is
\[ \begin{bmatrix} 0.25 & 0.50 & x \end{bmatrix}. \]Find \(x\).
Practice 4: Interpret an Entry
In a transition matrix whose state order is \(A,B,C\),
\[ p_{23}=0.18. \]Explain what this means.
Practice 5: One-Step Distribution
Let
\[ \boldsymbol{\pi}_0= \begin{bmatrix} 1 & 0 \end{bmatrix} \]and
\[ P= \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix}. \]Find \(\boldsymbol{\pi}_1\).
Practice 6: Uncertain Starting State
Using the same matrix, let
\[ \boldsymbol{\pi}_0= \begin{bmatrix} 0.2 & 0.8 \end{bmatrix}. \]Find \(\boldsymbol{\pi}_1\).
Practice 7: Two-Step Matrix
For
\[ P= \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix}, \]compute \(P^2\).
Practice 8: Read a Two-Step Probability
If
\[ P^2= \begin{bmatrix} 0.61 & 0.39\\ 0.52 & 0.48 \end{bmatrix}, \]what is the probability of moving from \(A\) to \(B\) in exactly two steps?
Practice 9: Estimate from Counts
Observed transition counts are
\[ N= \begin{bmatrix} 45 & 15\\ 20 & 80 \end{bmatrix}. \]Estimate the transition matrix.
Practice 10: Identify the Error
A student wants a two-step transition matrix and writes
\[ P= \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} \]followed by
\[ P^2= \begin{bmatrix} 0.64 & 0.04\\ 0.09 & 0.49 \end{bmatrix}. \]Explain the error.
Practice Answers
Practice 1
Using state order \(A,B\),
\[ \boxed{ P= \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix} }. \]Practice 2
No. The second row sums to
\[ 0.3+0.8=1.1, \]so the matrix is not row-stochastic.
Practice 3
The row must sum to \(1\):
\[ 0.25+0.50+x=1. \]Therefore,
\[ \boxed{x=0.25}. \]Practice 4
With state order \(A,B,C\), row \(2\) is \(B\) and column \(3\) is \(C\). Therefore,
\[ p_{23}=0.18 \]means that the probability of moving from \(B\) to \(C\) in one step is \(0.18\).
Practice 5
\[ \boldsymbol{\pi}_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix} = \boxed{ \begin{bmatrix} 0.7 & 0.3 \end{bmatrix} }. \]Practice 6
\[ \boldsymbol{\pi}_1 = \begin{bmatrix} 0.2 & 0.8 \end{bmatrix} \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix}. \]The first entry is
\[ (0.2)(0.7)+(0.8)(0.4)=0.46, \]and the second is
\[ (0.2)(0.3)+(0.8)(0.6)=0.54. \]Thus,
\[ \boxed{ \boldsymbol{\pi}_1= \begin{bmatrix} 0.46 & 0.54 \end{bmatrix} }. \]Practice 7
\[ P^2 = \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix} \begin{bmatrix} 0.7 & 0.3\\ 0.4 & 0.6 \end{bmatrix} = \boxed{ \begin{bmatrix} 0.61 & 0.39\\ 0.52 & 0.48 \end{bmatrix} }. \]Practice 8
The \((A,B)\)-entry of \(P^2\) is \(0.39\). Therefore,
\[ \boxed{ P(X_{n+2}=B\mid X_n=A)=0.39 }. \]Practice 9
The first row total is
\[ 45+15=60, \]and the second row total is
\[ 20+80=100. \]Normalize each row:
\[ \widehat P = \boxed{ \begin{bmatrix} 0.75 & 0.25\\ 0.20 & 0.80 \end{bmatrix} }. \]Practice 10
The student squared each entry separately. Matrix powers require standard matrix multiplication.
The correct calculation is
\[ P^2 = \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} \begin{bmatrix} 0.8 & 0.2\\ 0.3 & 0.7 \end{bmatrix} = \boxed{ \begin{bmatrix} 0.70 & 0.30\\ 0.45 & 0.55 \end{bmatrix} }. \]What to Remember
A transition matrix organizes one-step probabilities for a finite Markov chain:
\[ \boxed{P=[p_{ij}]}, \qquad p_{ij}=P(X_{n+1}=j\mid X_n=i). \]- Rows represent current states and columns represent next states in the convention used here.
- The state order must remain fixed and explicit.
- Every entry must lie between \(0\) and \(1\).
- Every row of a row-stochastic transition matrix sums to \(1\).
- A zero entry means that a particular one-step transition is impossible.
- An absorbing state has a row with a \(1\) on its own diagonal entry and \(0\) elsewhere.
- A state distribution can be written as a row probability vector \(\boldsymbol{\pi}_n\).
- One-step updating uses \(\boldsymbol{\pi}_{n+1}=\boldsymbol{\pi}_nP\).
- \(P^k\) contains \(k\)-step transition probabilities for a time-homogeneous chain.
- The entry \((P^k)_{ij}\) is the probability of moving from \(i\) to \(j\) in exactly \(k\) steps.
- Matrix multiplication implements the law of total probability over intermediate states.
- Transition matrices can be estimated by normalizing observed transition counts row by row.
- Different row-vector and column-vector conventions are valid, but they must not be mixed.
The central computational relationship is
\[ \boxed{ \boldsymbol{\pi}_k = \boldsymbol{\pi}_0P^k }. \]It translates a stochastic process into a repeatable matrix operation.
Where We Go Next
Transition matrices describe how probability moves among states. Repeated multiplication raises a natural question: can a probability distribution remain unchanged even while individual sample paths continue to move?
The next lesson, Stationary Distributions, studies probability vectors satisfying
\[ \boldsymbol{\pi}P=\boldsymbol{\pi}, \]and explains their role in the long-run structure of Markov chains.