TwoStateMDP

Coding a Simple Markov Decision Process

This is a quick tutorial on how to implement a two-state Markov Decision Process (MDP). It has only two states: state 1 and state 2. Applications MDPs often appear in machine learning and reinforcement learning.

At any stage, the agent can take any of the two actions. The actions are to stay or switch. The switch will change the current state to the other one and stay won’t make any changes.

At any point in time, we can print out the current state of the system. The state transition diagram can be visualized as below.

TwoStateMDP
Two state MDP

The following code implements the MDP and its fuctionalities.

class TwoStateMDP(object):
	"""docstring for TwoState"""
	def __init__(self):
		self.states = 2
		self.nactions = 2
		self.currentS = 1

	def stay(self):
		pass

	def toggle(self):
		if self.currentS ==1:
			self.currentS=2
		else:
			self.currentS=1

switch = TwoStateMDP()

print(switch.currentS)
switch.toggle()
print(switch.currentS)
switch.toggle()
print(switch.currentS)

As you can see coding it in the form of class can make the implementation easy. We could define the states of MDP, current states, actions possible, etc.

Separate functions have been written to do the possible actions ‘stay’ and ‘toggle’.

While running the code we get the following output.

1
2
1

The command ‘switch = TwoStateMDP()’ generate an object of the two-state MDP type. By default, it will be of state 1. When we call the ‘toggle()’ function, it gets switched to the other state.

Leave a Comment

Your email address will not be published.