2019-05-02 21:12:57
This blog is copied from: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146
PG) are frequently used algorithms in reinforcement learning (RL). The principle is very simple.
We observe and act.
A human takes actions based on observations. As a quote from Stephen Curry:
You have to rely on the fact that you put the work in to create the muscle memory and then trust that it will kick in. The reason you practice and work on it so much is so that during the game your instincts take over to a point where it feels weird if you don’t do it the right way.
The training in PG makes actions with high rewards more likely, or vice versa.
We keep what is working and throw away what is not.
In policy gradients, Curry is our agent.
- s).
- s.
- He moves and the opponents react. A new state is formed.
- He takes further actions based on the observed state.
- R(τ) received.
Curry visualizes the situation and instantly knows what to do. Years of training perfects the instinct to maximize the rewards. In RL, the instinct may be mathematically described as:
policy in RL. For example, what is the chance of turning or stopping when you see a car in front:
Objective
How can we formulate our objective mathematically? The expected rewards equal the sum of the probability of a trajectory × corresponding rewards:
τ
that maximizes the expected rewards.
Input features & rewards
which outputs the possibility of an action that it may take.
r given at each time step. In a basketball game, all are 0 except the terminate state which equals 0, 1, 2 or 3.
H steps.
Optimization
log(f(x)).
π.
Also, for a continuous space, expectation can be expressed as:
Now, let’s formalize our optimization problem mathematically. We want to model a policy that creates trajectories that maximize the total rewards.
r which may not be differentiable or formalized?
J as:
policy gradient) becomes:
τ).
τ) is defined as:
Take the log:
θ and can be removed.
So the policy gradient
becomes:
θ.
Intuition
How can we make sense of these equations? The underlined term is the maximum log likelihood. In deep learning, it measures the likelihood of the observed data. In our context, it measures how likely the trajectory is under the current policy. By multiplying it with the rewards, we want to increase the likelihood of a policy if the trajectory results in a high positive reward. On the contrary, we want to decrease the likelihood of a policy if it results in a high negative reward. In short, keep what is working and throw out what is not.
If going up the hill below means higher rewards, we will change the model parameters (policy) to increase the likelihood of trajectories that move higher.
There is one thing significant about the policy gradient. The probability of a trajectory is defined as:
States in a trajectory are strongly related. In Deep Learning, a long sequence of multiplication with factors that are strongly correlated can trigger vanishing or exploding gradient easily. However, the policy gradient only sums up the gradient which breaks the curse of multiplying a long sequence of numbers.
The trick
creates a maximum log likelihood and the log breaks the curse of multiplying a long chain of policy.
Policy Gradient with Monte Carlo rollouts
Monte Carlo rollout to compute the rewards. i.e. play out the whole episode to compute the total rewards.
Policy gradient with automatic differentiation
The policy gradient can be computed easily with many Deep Learning software packages. For example, this is the partial code for TensorFlow:
Yes, as often, coding looks simpler than the explanations.
Continuous control with Gaussian policies
How can we model a continuous control?
Let’s assume the values for actions are Gaussian distributed
and the policy is defined using a Gaussian distribution with means computed from a deep network:
With
We can compute the partial derivative of the log π as:
So we can backpropagate
θ. The algorithm will look exactly the same as before. Just start with a slightly different way in calculating the log of the policy.
Policy Gradients improvements
Policy Gradients suffer from high variance and low convergence.
Monte Carlo plays out the whole trajectory and records the exact rewards of a trajectory. However, the stochastic policy may take different actions in different episodes. One small turn can completely alter the result. So Monte Carlo has no bias but high variance. Variance hurts deep learning optimization. The variance provides conflicting descent direction for the model to learn. One sampled rewards may want to increase the log likelihood and another may want to decrease it. This hurts the convergence. To reduce the variance caused by actions, we want to reduce the variance for the sampled rewards.
Increasing the batch size in PG reduces variance.
However, increasing the batch size significantly reduces sample efficiency. So we cannot increase it too far, we need additional mechanisms to reduce the variance.
Baseline
s).
A.
In deep learning, we want input features to be zero-centered. Intuitively, RL is interested in knowing whether an action is performed better than the average. If rewards are always positive (R>0), PG always try to increase a trajectory probability even if it receives much smaller rewards than others. Consider two different situations:
- Situation 1: Trajectory A receives+10 rewards and Trajectory B receives -10 rewards.
- Situation 2: Trajectory A receives +10 rewards and Trajectory B receives +1 rewards.
In the first situation, PG will increase the probability of Trajectory A while decreasing B. In the second situation, it will increase both. As a human, we will likely decrease the likelihood of trajectory B in both situations.
V, we can recalibrate the rewards relative to the average action.
Vanilla Policy Gradient Algorithm
b.
Causality
Future actions should not change past decision. Present actions only impact the future. Therefore, we can change our objective function to reflect this also.
Reward discount
Reward discount reduces variance which reduces the impact of distant actions. Here, a different formula is used to compute the total rewards.
And the corresponding objective function becomes:
Part 2
This ends part 1 of the policy gradient methods. In the second part, we continue on the Temporal Difference, Hyperparameter tuning, and importance sampling. Temporal Difference will further reduce the variance and the importance sampling will lay down the theoretical foundation for more advanced policy gradient methods like TRPO and PPO.
Credit and references
UCL RL course
UC Berkeley RL course
UC Berkeley RL Bootcamp
A3C paper
GAE paper