Markovian Policy

**Proof:**For every
and
we define
as follows:

We will first show that this definition results in the same distibution over the group of **actions**:

The first equality is derived from the fact that is Markovian.

The second equality is by definition.

It is left to be shown that the distribution over the group of

We will prove this part by an induction on

Recalling that

concludes this proof.