Essentials of Stochastic Processes

Rick Durrett

Version 3.9, May 2021

1 Markov Chains

• 1.1 Definitions and Examples
• 1.2 Multistep Transition Probabilities
• 1.3 Classification of States
• 1.4 Stationary Distributions
• 1.5 Detailed Balance Condition
• 1.6 Limit Behavior
• 1.7 Returns to a fixed state
• 1.8 Proofs of the convergence theorem*
• 1.9 Exit Distributions
• 1.10 Exit Times
• 1.11 Infinite State Spaces*
• 1.12 Chapter Summary
• 1.13 Exercises

2 Poisson Processes

• 2.1 Exponential Distribution
• 2.2 Defining the Poisson Process
• 2.3 Compound Poisson Processes
• 2.4 Transformations
• 2.5 Chapter Summary
• 2.6 Exercises

3 Renewal Processes

• 3.1 Laws of Large Numbers
• 3.2 Applications to Queueing Theory
• 3.3 Age and Residual Life*
• 3.4 Chapter Summary
• 3.5 Exercises

4 Continuous Time Markov Chains

• 4.1 Definitions and Examples
• 4.2 Computing the Transition Probability
• 4.3 Limiting Behavior
• 4.4 Exit Distributions and Exit Times
• 4.5 Markovian Queues
• 4.6 Queueing Networks*
• 4.7 Chapter Summary
• 4.8 Exercises

5 Martingales

• 5.1 Conditional Expectation
• 5.2 Examples
• 5.3 Gambling Strategies, Stopping Times
• 5.4 Applications
• 5.5 Convergence
• 5.6 Exercises

6 Mathematical Finance

• 6.1 Two Simple Examples
• 6.2 Binomial Model
• 6.3 Concrete Examples
• 6.4 American Options
• 6.5 Black-Scholes formula
• 6.6 Calls and Puts
• 6.7 Exercises

A Review of Probability

• A.1 Probabilities, Independence
• A.2 Random Variables, Distributions
• A.3 Expected Value, Moments
• A.4 Integration to the Limit