Suite of functions related to discrete-time discrete-state Markov Chains

Description

A series of functions which aid in both simulating and determining the properties of finite, discrete-time, discrete state markov chains. This package may be of use to practioners who need to simulate Markov Chains, but its primary intended audience is students of an introductory stochastic processes studying class properties and long run behavior patterns of Markov Chains. Two functions (DTMC, MultDTMC) produce n iterations of a Markov Chain(s) based on transition probabilities and an initial distribution. The function FPTime determines the first passage time into each state. The function statdistr determines the stationary distribution of a Markov Chain. Updated 4/10/22 to maintain compatibility with R.

Details

Author(s)

Will Nicholson

Maintainer: <wbn8@cornell.edu>

References

Sidney Resnick, "Adventures in Stochastic Processes"

Examples

data(gr)
data(id)
DTMC(gr,id,10,trace=FALSE) 

Simulation of Discrete-Time/State Markov Chain

Description

This function simulates iterations through a discrete time Markov Chain. A Markov Chain is a discrete Markov Process with a state space that usually consists of positive integers. The advantage of a Markov process in a stochastic modeling context is that conditional dependencies over time are manageable because the probabilistic future of the process depends only on the present state, not the past. Therefore, if we specify an initial distribution as well as a transition matrix, we can simulate many periods into the future without any further information. Future transition probabilities can be computed by raising the transition matrix to higher-and higher powers, but this method is not numerically tractable for large matrices. My method uses a uniform random variable to iterate a user-specified number of iterations of a Markov Chain based on the transition probabilities and the initital distribution. A graphical output is also available in the form of a trace plot.

Usage

DTMC(tmat, io, N, trace)

Arguments

Value

Author(s)

Will Nicholson

References

"Adventures in Stochastic Processes" by Sidney Resnick

See Also

MultDTMC

Examples

data(gr)
data(id)
DTMC(gr,id,10,trace=TRUE) 
# 10 iterations through "Gambler's ruin"

First Passage Time

Description

This function uses the companion function multDTMC to simulate several Markov chains to determine the first passage time into each state, i.e. the first time (after the initial iteration) that a specified state is reached in the Markov Process. First Passage Time can be useful for both determining class properties as well as the stationary/invariant distribution for large Markov Chains in which explicit matrix inversion is not computationally tractable.

Usage

FPTime(state, nchains, tmat, io, n)

Arguments

Value

Author(s)

Will Nicholson

See Also

DTMC

Examples

data(gr)
data(id)
FPTime(1,10,gr,id,10) # First passage time into first state on Gambler's ruin

Multiple Discrete time Markov Chains

Description

An extension of the DTMC package which enables multiple cocurrent Markov Chain simulations. At this time, plotting is not enabled.

Usage

MultDTMC(nchains, tmat, io, n)

Arguments

Value

Author(s)

Will Nicholson

See Also

DTMC

Examples

data(gr)
data(id)
MultDTMC(20,gr,id,10) # 20 chains with 10 iterations using the Gambler's ruin example.

Example Data Set: Gambler's ruin on 4 states

Description

Motivating example, random walk with absorbing boundaries on 4 states. Analogous to a gambler at a casino. The 4 states represent a range of wealth. States 1 and 4 are absorbing with state 1="Broke", state 4="wealthy enough to walk away" and the intermediate states 2 and 3 are transitory. It is assumed that he bets of all his winnings in the intermediate states and has equal probability of winning and losing

Examples

data(gr)
data(id)
DTMC(gr,id,10,trace=FALSE)

Harry the SemiPro

Description

Example Markov Chain from page 139 of Resnick. The protagonist, basketball player "Happy Harry's" productivity fluctuates between three states (0-1 points), (2-5 points), (5 or more points) and the transition between states can be modeled using a Markov Chain. Used as a motivating example to calculate the long run proportion of time spent in each state using the statdist function.

Source

Sidney Resnick "Adventures in Stochastic Processes"

Examples

data(hh)
statdistr(hh)

Initial distribution

Description

A starting distribution for the gambler's ruin example, which assigns equal probability of starting in each state.

Examples

data(id)
data(gr)
DTMC(gr,id,10,trace=FALSE)

Computing Stationary Distribution

Description

This function computes the stationary distribution of a markov chain (assuming one exists) using the formula from proposition 2.14.1 of Resnick: pi=(1,...1)(I-P+ONE)^(-1), where I is an mxm identity matrix, P is an mxm transition matrix, and ONE is an mxm matrix whose entries are all 1. This formula works well if the number of states is small, but since it directly computes the inverse of the matrix, it is not tractable for larger matrices. For larger matrices 1/E(FPTime(n)) is a rough approximation for the long run proportion of time spent in a state n.

Usage

statdistr(tmat)

Arguments

Value

Returns a stationary distribution: mxm matrix which represents the long run percentage of time spent in each state.

Author(s)

Will Nicholson

References

Resnick, "Adventures in Stochastic Processes"

Examples

data(hh)
statdistr(hh)