Discrete‐Time Markov Chains

This chapter considers a collection of random variables defined on a probability space, with values in a countable set and satisfying the Markov property, that is the past and the future of random variables are independent when its present state is known. The set is often called the state space. The chapter explains definitions and properties for discrete-time Markov Chains. Irreducible, recurrent Markov chains, aperiodic Markov chains, finite Markov chains and absorbing Markov chains are considered for discussion. The chapter shows that a discrete-time Markov chain is completely determined by its initial distribution and transition probability matrix. The chapter starts by describing strong Markov property, and then moves onto explain recurrent and transient states of a Markov chain and state space decomposition. It also presents a brief discussion on ergodic theorem.