Policy gradient method explained

that selects actions without consulting a value function. For policy gradient to apply, the policy function is parameterized by a differentiable parameter .

Overview

In policy-based RL, the actor is a parameterized policy function

, where are the parameters of the actor. The actor takes as argument the state of the environment and produces a probability distribution .

If the action space is discrete, then

. If the action space is continuous, then .

The goal of policy optimization is to find some

that maximizes the expected episodic reward :$$J(\backslash theta)\; =\; \backslash mathbb\_\backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash gamma^t\; R\_t\; \backslash Big|\; S\_0\; =\; s\_0\; \backslash right]$$where is the discount factor, is the reward at step , is the starting state, and is the time-horizon (which can be infinite).

The policy gradient is defined as

. Different policy gradient methods stochastically estimate the policy gradient in different ways. The goal of any policy gradient method is to iteratively maximize by gradient ascent. Since the key part of any policy gradient method is the stochastic estimation of the policy gradient, they are also studied under the title of "Monte Carlo gradient estimation".^[1]

REINFORCE

Policy gradient

The REINFORCE algorithm, introduced by Ronald J. Williams in 1992, was the first policy gradient method.^[2] It is based on the identity for the policy gradient$$\backslash nabla\_\backslash theta\; J(\backslash theta)=\; \backslash mathbb\_\backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t\; \backslash mid\; S\_t)\backslash ;\; \backslash sum\_\{t\; \backslash in\; 0:T\}\; (\backslash gamma^t\; R\_t)\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$ which can be improved via the "causality trick"^[3] $$\backslash nabla\_\backslash theta\; J(\backslash theta)=\; \backslash mathbb\_\backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t\backslash mid\; S\_t)\backslash sum\_\{\backslash tau\; \backslash in\; t:T\}\; (\backslash gamma^\backslash tau\; R\_\backslash tau)\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$

Thus, we have an unbiased estimator of the policy gradient:$$\backslash nabla\_\backslash theta\; J(\backslash theta)\; \backslash approx\; \backslash frac\; 1N\; \backslash sum\_^N\; \backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_\{t,n\}\backslash mid\; S\_\{t,n\})\backslash sum\_\{\backslash tau\; \backslash in\; t:T\}\; (\backslash gamma^\{\backslash tau-t\}\; R\_\{\backslash tau,n\})\; \backslash right]$$where the index

ranges over rollout trajectories using the policy . can be interpreted as the direction in the parameter space that increases the probability of taking action in state . The policy gradient, then, is a weighted average of all possible directions to increase the probability of taking any action in any state, but weighted by reward signals, so that if taking a certain action in a certain state is associated with high reward, then that direction would be highly reinforced, and vice versa.

Algorithm

The REINFORCE algorithm is a loop:

1. Rollout

trajectories in the environment, using as the policy function.

1. Compute the policy gradient estimation:

1. Update the policy by gradient ascent:

Here,

is the learning rate at update step .

Variance reduction

REINFORCE is an on-policy algorithm, meaning that the trajectories used for the update must be sampled from the current policy

. This can lead to high variance in the updates, as the returns can vary significantly between trajectories. Many variants of REINFORCE have been introduced, under the title of variance reduction.

REINFORCE with baseline

A common way for reducing variance is the REINFORCE with baseline algorithm, based on the following identity:$$\backslash nabla\_\backslash theta\; J(\backslash theta)=\; \backslash mathbb\_\backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t|\; S\_t)\backslash left(\backslash sum\_\{\backslash tau\; \backslash in\; t:T\}\; (\backslash gamma^\backslash tau\; R\_\backslash tau)\; -\; b(S\_t)\backslash right)\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$for any function

. This can be proven by applying the previous lemma.

The algorithm uses the modified gradient estimator$$g\_i\; \backslash leftarrow\; \backslash frac\; 1N\; \backslash sum\_^N\; \backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash nabla\_\{\backslash theta\_t\}\backslash ln\backslash pi\_\backslash theta(A\_\{t,n\}|\; S\_\{t,n\})\backslash left(\backslash sum\_\{\backslash tau\; \backslash in\; t:T\}\; (\backslash gamma^\backslash tau\; R\_\{\backslash tau,n\})\; -\; b\_i(S\_\{t,n\})\backslash right)\; \backslash right]$$ and the original REINFORCE algorithm is the special case where

.

Actor-critic methods

See main article: Actor-critic algorithm. If $b\_i$ is chosen well, such that $b\_i(S\_t)\; \backslash approx\; \backslash sum\_\; (\backslash gamma^\backslash tau\; R\_\backslash tau)\; =\; \backslash gamma^t\; V^(S\_t)$, this could significantly decrease variance in the gradient estimation. That is, the baseline should be as close to the value function

as possible, approaching the ideal of:$$\backslash nabla\_\backslash theta\; J(\backslash theta)=\; \backslash mathbb\_\backslash left[\backslash sum\_\{t\backslash in\; 0:T\}\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t|\; S\_t)\backslash left(\backslash sum\_\{\backslash tau\; \backslash in\; t:T\}\; (\backslash gamma^\backslash tau\; R\_\backslash tau)\; -\; \backslash gamma^t\; V^\{\backslash pi\_\backslash theta\}(S\_t)\backslash right)\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$Note that, as the policy updates, the value function updates as well, so the baseline should also be updated. One common approach is to train a separate function that estimates the value function, and use that as the baseline. This is one of the actor-critic methods, where the policy function is the actor and the value function is the critic.

The Q-function

can also be used as the critic, since$$\backslash nabla\_\backslash theta\; J(\backslash theta)=\; E\_\backslash left[\backslash sum\_\{0\backslash leq\; t\; \backslash leq\; T\}\; \backslash gamma^t\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t|\; S\_t)\; \backslash cdot\; Q^\{\backslash pi\_\backslash theta\}(S\_t,\; A\_t)\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$ by a similar argument using the tower law.

Subtracting the value function as a baseline, we find that the advantage function

can be used as the critic as well:$$\backslash nabla\_\backslash theta\; J(\backslash theta)=\; E\_\backslash left[\backslash sum\_\{0\backslash leq\; t\; \backslash leq\; T\}\; \backslash gamma^t\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t|\; S\_t)\; \backslash cdot\; A^\{\backslash pi\_\backslash theta\}(S\_t,\; A\_t)\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$In summary, there are many unbiased estimators for $\backslash nabla\_\backslash theta\; J\_\backslash theta$, all in the form of: $$\backslash nabla\_\backslash theta\; J(\backslash theta)\; =\; E\_\backslash left[\backslash sum\_\{0\backslash leq\; t\; \backslash leq\; T\}\; \backslash nabla\_\backslash theta\backslash ln\backslash pi\_\backslash theta(A\_t|\; S\_t)\; \backslash cdot\; \backslash Psi\_t\; \backslash Big|S\_0\; =\; s\_0\; \backslash right]$$ where $\backslash Psi\_t$ is any linear sum of the following terms:

• $\backslash sum\_\; (\backslash gamma^\backslash tau\; R\_\backslash tau)$: never used.
• $\backslash gamma^t\backslash sum\_\; (\backslash gamma^\; R\_\backslash tau)$: used by the REINFORCE algorithm.
• $\backslash gamma^t\; \backslash sum\_\; (\backslash gamma^\; R\_\backslash tau)\; -\; b(S\_t)$: used by the REINFORCE with baseline algorithm.
• $\backslash gamma^t\; \backslash left(R\_t\; +\; \backslash gamma\; V^(S\_)\; -\; V^(S\_)\backslash right)$: 1-step TD learning.
• $\backslash gamma^t\; Q^(S\_t,\; A\_t)$.
• $\backslash gamma^t\; A^(S\_t,\; A\_t)$.

Some more possible $\backslash Psi\_t$ are as follows, with very similar proofs.

• $\backslash gamma^t\; \backslash left(R\_t\; +\; \backslash gamma\; R\_\; +\; \backslash gamma^2\; V^(S\_)\; -\; V^(S\_)\backslash right)$: 2-step TD learning.
• $\backslash gamma^t\; \backslash left(\backslash sum\_^\; \backslash gamma^k\; R\_\; +\; \backslash gamma^n\; V^(S\_)\; -\; V^(S\_)\backslash right)$: n-step TD learning.
• $\backslash gamma^t\; \backslash sum\_^\backslash infty\; \backslash frac\backslash cdot\; \backslash left(\backslash sum\_^\; \backslash gamma^k\; R\_\; +\; \backslash gamma^n\; V^(S\_)\; -\; V^(S\_)\backslash right)$: TD(λ) learning, also known as GAE (generalized advantage estimate).^[4] This is obtained by an exponentially decaying sum of the n-step TD learning ones.

Natural policy gradient

The natural policy gradient method is a variant of the policy gradient method, proposed by Sham Kakade in 2001.^[5] Unlike standard policy gradient methods, which depend on the choice of parameters

(making updates coordinate-dependent), the natural policy gradient aims to provide a coordinate-free update, which is geometrically "natural".

Motivation

Standard policy gradient updates

solve a constrained optimization problem:$$\backslash begin\backslash max\_\; J(\backslash theta\_i)\; +\; (\backslash theta\_\; -\; \backslash theta\_i)^T\; \backslash nabla\_\backslash theta\; J(\backslash theta\_i)\backslash \backslash \backslash |\backslash theta\_\; -\; \backslash theta\_\backslash |\backslash leq\; \backslash alpha\; \backslash cdot\; \backslash |\backslash nabla\_\backslash theta\; J(\backslash theta\_i)\backslash |\backslash end$$While the objective (linearized improvement) is geometrically meaningful, the Euclidean constraint introduces coordinate dependence. To address this, the natural policy gradient replaces the Euclidean constraint with a Kullback–Leibler divergence (KL) constraint:$$\backslash begin\backslash max\_\; J(\backslash theta\_i)\; +\; (\backslash theta\_\; -\; \backslash theta\_i)^T\; \backslash nabla\_\backslash theta\; J(\backslash theta\_i)\backslash \backslash \backslash bar\_(\backslash pi\_\; \backslash |\; \backslash pi\_)\; \backslash leq\; \backslash epsilon\backslash end$$where the KL divergence between two policies is averaged over the state distribution under policy . That is,$$\backslash bar\_(\backslash pi\_\; \backslash |\; \backslash pi\_)\; :=\; \backslash mathbb\; E\_[D\_\{KL\}(\backslash pi\_\{\backslash theta\_\{i+1\}\}(\backslash cdot\; |\; s)\; \backslash |\; \backslash pi\_\{\backslash theta\_\{i\}\}(\backslash cdot\; |\; s))]$$ This ensures updates are invariant to invertible affine parameter transformations.

Fisher information approximation

For small

, the KL divergence is approximated by the Fisher information metric:$$\backslash bar\_(\backslash pi\_\; \backslash |\; \backslash pi\_)\; \backslash approx\; \backslash frac\; (\backslash theta\_\; -\; \backslash theta\_i)^T\; F(\backslash theta\_i)\; (\backslash theta\_\; -\; \backslash theta\_i)$$where is the Fisher information matrix of the policy, defined as:$$F(\backslash theta)\; =\; \backslash mathbb\_\backslash left[\backslash nabla\_\backslash theta\; \backslash ln\; \backslash pi\_\backslash theta(a|s)\; \backslash left(\backslash nabla\_\backslash theta\; \backslash ln\; \backslash pi\_\backslash theta(a|s)\backslash right)^T\; \backslash right]$$ This transforms the problem into a problem in quadratic programming, yielding the natural policy gradient update:$$\backslash theta\_\; =\; \backslash theta\_i\; +\; \backslash alpha\; F(\backslash theta\_i)^\; \backslash nabla\_\backslash theta\; J(\backslash theta\_i)$$The step size is typically adjusted to maintain the KL constraint, with $\backslash alpha\; \backslash approx\; \backslash sqrt$.

Inverting

is computationally intensive, especially for high-dimensional parameters (e.g., neural networks). Practical implementations often use approximations.

Trust Region Policy Optimization (TRPO)

Trust Region Policy Optimization (TRPO) is a policy gradient method that extends the natural policy gradient approach by enforcing a trust region constraint on policy updates.^[6] Developed by Schulman et al. in 2015, TRPO improves upon the natural policy gradient method.

The natural gradient descent is theoretically optimal, if the objective is truly a quadratic function, but this is only an approximation. TRPO's line search and KL constraint attempts to restrict the solution to within a "trust region" in which this approximation does not break down. This makes TRPO more robust in practice.

Formulation

Like natural policy gradient, TRPO iteratively updates the policy parameters

by solving a constrained optimization problem specified coordinate-free:$$\backslash begin\backslash max\_\; L(\backslash theta,\; \backslash theta\_i)\backslash \backslash \backslash bar\_(\backslash pi\_\; \backslash |\; \backslash pi\_)\; \backslash leq\; \backslash epsilon\; \backslash end$$where is the surrogate advantage, measuring the performance of relative to the old policy . is the trust region radius.Note that in general, other surrogate advantages are possible:$$L(\backslash theta,\; \backslash theta\_i)\; =\; \backslash mathbb\_\backslash left[\backslash frac\{\backslash pi\_\backslash theta(a|s)\}\{\backslash pi\_\{\backslash theta\_i\}(a|s)\}\backslash Psi^\{\backslash pi\_\{\backslash theta\_i\}\}(s,\; a)\; \backslash right]$$where is any linear sum of the previously mentioned type. Indeed, OpenAI recommended using the Generalized Advantage Estimate, instead of the plain advantage .

The surrogate advantage

is designed to align with the policy gradient . Specifically, when ,

\nabla_\thetaL(\theta,\theta_t)

equals the policy gradient derived from the advantage function:$$\backslash nabla\_\backslash theta\; J(\backslash theta)\; =\; \backslash mathbb\_\backslash left[\backslash nabla\_\backslash theta\; \backslash ln\; \backslash pi\_\backslash theta(a\; |\; s)\; \backslash cdot\; A^\{\backslash pi\_\backslash theta\}(s,\; a)\; \backslash right]\; =\; \backslash nabla\_\backslash theta\; L(\backslash theta,\; \backslash theta\_t)$$However, when , this is not necessarily true. Thus it is a "surrogate" of the real objective.

As with natural policy gradient, for small policy updates, TRPO approximates the surrogate advantage and KL divergence using Taylor expansions around

: where: is the policy gradient. is the Fisher information matrix.

This reduces the problem to a quadratic optimization, yielding the natural policy gradient update: $$\backslash theta\_\; =\; \backslash theta\_i\; +\; \backslash sqrt\; F^\; g.$$So far, this is essentially the same as natural gradient method. However, TRPO improves upon it by two modifications:

• Use conjugate gradient method to solve for

in iteratively without explicit matrix inversion.

• Use backtracking line search to ensure the trust-region constraint is satisfied. Specifically, it backtracks the step size to ensure the KL constraint and policy improvement. That is, it tests each of the following test-solutions$$$$

\theta_ = \theta_i + \sqrt x, \; \theta_i + \alpha \sqrt x, \; \theta_i + \alpha^2 \sqrt x, \; \dots until it finds one that both satisfies the KL constraint

and results in a higher . Here, is the backtracking coefficient.

Proximal Policy Optimization (PPO)

A further improvement is proximal policy optimization (PPO), which avoids even computing

and via a first-order approximation using clipped probability ratios.^[7]

Specifically, instead of maximizing the surrogate advantage$$\backslash max\_\backslash theta\; L(\backslash theta,\; \backslash theta\_t)\; =\; \backslash mathbb\_\backslash left[\backslash frac\{\backslash pi\_\backslash theta(a|s)\}\{\backslash pi\_\{\backslash theta\_t\}(a|s)\}\; A^\{\backslash pi\_\{\backslash theta\_t\}\}(s,\; a)\; \backslash right]$$ under a KL divergence constraint, it directly inserts the constraint into the surrogate advantage: and PPO maximizes the surrogate advantage by stochastic gradient descent, as usual.

In words, gradient-ascending the new surrogate advantage function means that, at some state

, if the advantage is positive: , then the gradient should direct towards the direction that increases the probability of performing action under the state . However, as soon as has changed so much that , then the gradient should stop pointing it in that direction. And similarly if . Thus, PPO avoids pushing the parameter update too hard, and avoids changing the policy too much.

To be more precise, to update

to requires multiple update steps on the same batch of data. It would initialize , then repeatedly apply gradient descent (such as the Adam optimizer) to update until the surrogate advantage has stabilized. It would then assign to , and do it again.

During this inner-loop, the first update to

would not hit the bounds, but as is updated further and further away from , it eventually starts hitting the bounds. For each such bound hit, the corresponding gradient becomes zero, and thus PPO avoid updating too far away from .

This is important, because the surrogate loss assumes that the state-action pair

is sampled from what the agent would see if the agent runs the policy , but policy gradient should be on-policy. So, as changes, the surrogate loss becomes more and more off-policy. This is why keeping proximal to is necessary.

If there is a reference policy

that the trained policy should not diverge too far from, then additional KL divergence penalty can be added:$$-\backslash beta\; \backslash mathbb\_\backslash left[\backslash log\backslash left(\backslash frac\{\backslash pi\_\{\backslash theta\}(a|s)\}\{\backslash pi\_\{\backslash text\{ref\}\}(a|s)\}\backslash right)\; \backslash right]$$where adjusts the strength of the penalty. This has been used in training reasoning language models with reinforcement learning from human feedback.^[8] The KL divergence penalty term can be estimated with lower variance using the equivalent form (see f-divergence for details):$$-\backslash beta\; \backslash mathbb\_\backslash left[\backslash log\backslash left(\backslash frac\{\backslash pi\_\{\backslash theta\}(a|s)\}\{\backslash pi\_\{\backslash text\{ref\}\}(a|s)\}\backslash right)\; +\; \backslash frac\{\backslash pi\_\{\backslash text\{ref\}\}(a|s)\}\; \{\backslash pi\_\{\backslash theta\}(a|s)\}\; -1\; \backslash right]$$

Group Relative Policy Optimization (GRPO)

The Group Relative Policy Optimization (GRPO) is a minor variant of PPO that omits the value function estimator

. Instead, for each state , it samples multiple actions from the policy , then calculate the group-relative advantage^[9] $$A^(s,\; a\_)\; =\; \backslash frac$$ where are the mean and standard deviation of . That is, it is the standard score of the rewards.

Then, it maximizes the PPO objective, averaged over all actions:Intuitively, each policy update step in GRPO makes the policy more likely to respond to each state with an action that performed relatively better than other actions tried at that state, and less likely to respond with one that performed relatively worse.

As before, the KL penalty term can be applied to encourage the trained policy to stay close to a reference policy. GRPO was first proposed in the context of training reasoning language models by researchers at DeepSeek.

Policy Optimization and the Mirror Descent perspective (MDPO)

Methods like TRPO, PPO and natural policy gradient share a common idea - while the policy should be updated in the direction of the policy gradient, the update should be done in a safe and stable manner, typically measured by some distance with respect to the policy before the update.

A similar notion of update stability is found in proximal convex optimization techniques like Mirror Descent.^[10] There, $\backslash mathbf$, the proposed minimizer of $f$ in some constraint set $\backslash mathcal$, is iteratively updated in the direction of the gradient $\backslash nabla\; f$, with a proximity penalty with respect to the current $\backslash mathbf\_t$ measured by some Bregman divergence $B\_\backslash omega$, which can formalized by the following formula:$$\backslash mathbf\_\backslash in\backslash arg\; \backslash min\_\; \backslash nabla\; f(\backslash mathbf\_t)^T\; (\backslash mathbf\; -\; \backslash mathbf\_t)\; +\; \backslash frac\; B\_\backslash omega(x,x\_t),$$ where $\backslash eta\_t$ controls the proximity between consecutive iterates, similar to the learning rate in gradient descent.

This leads to reconsidering the policy update procedure as an optimization procedure aimed at finding an optimal policy, in the (non-convex) optimization landscape of the underlying Markov decision process (MDP). This optimization viewpoint of using the policy gradient is termed Mirror Descent Policy Optimization (MDPO),^{[11] [12]} leading to the following update when the KL is the chosen Bregman divergence:$$\backslash pi\_\backslash in\; \backslash arg\; \backslash max\_\; \backslash mathbb\_\; \backslash left[A^\{\backslash pi\_\{t\}\}(s,\; a)\; \backslash right]\; +\; \backslash frac\; D\_(\backslash pi\; ||\backslash pi\_t)$$With a parameterized policy $\backslash pi\_\backslash theta$, the MDPO loss becomes:$$\backslash max\_\; L(\backslash theta,\; \backslash theta\_t)\; =\; \backslash mathbb\_\backslash left[\backslash frac\{\backslash pi\_\backslash theta(a|s)\}\{\backslash pi\_\{\backslash theta\_t\}(a|s)\}\; A^\{\backslash pi\_\{\backslash theta\_t\}\}(s,\; a)\; \backslash right]\; +\; \backslash frac\; D\_(\backslash pi\_\backslash theta\; ||\backslash pi\_)$$This objective can be used together with other common techniques like the clipping done in PPO. In fact, the KL divergence penalty also appears in the original PPO paper, suggesting the MDPO perspective as a theoretical unification of the main derivation concepts behind many concurrent policy gradient techniques.

See also

• Reinforcement learning
• Deep reinforcement learning
• Actor-critic method

References

• Book: Sutton . Richard S. . Reinforcement learning: an introduction . Barto . Andrew G. . 2018 . The MIT Press . 978-0-262-03924-6 . 2 . Adaptive computation and machine learning series . Cambridge, Massachusetts.
• Book: Bertsekas, Dimitri P. . Reinforcement learning and optimal control . 2019 . Athena Scientific . 978-1-886529-39-7 . 2 . Belmont, Massachusetts.
• Book: Grossi, Csaba . Algorithms for Reinforcement Learning . 2010 . Springer International Publishing . 978-3-031-00423-0 . 1 . Synthesis Lectures on Artificial Intelligence and Machine Learning . Cham.
• Mohamed . Shakir . Rosca . Mihaela . Figurnov . Michael . Mnih . Andriy . 2020 . Monte Carlo Gradient Estimation in Machine Learning . Journal of Machine Learning Research . 21 . 132 . 1–62 . 1906.10652 . 1533-7928.

External links

• Web site: Weng . Lilian . 2018-04-08 . Policy Gradient Algorithms . 2025-01-25 . lilianweng.github.io . en.
• Web site: Vanilla Policy Gradient — Spinning Up documentation . 2025-01-25 . spinningup.openai.com.

Notes and References

1. Mohamed . Shakir . Rosca . Mihaela . Figurnov . Michael . Mnih . Andriy . 2020 . Monte Carlo Gradient Estimation in Machine Learning . Journal of Machine Learning Research . 21 . 132 . 1–62 . 1906.10652 . 1533-7928.
2. Williams . Ronald J. . May 1992 . Simple statistical gradient-following algorithms for connectionist reinforcement learning . Machine Learning . en . 8 . 3–4 . 229–256 . 10.1007/BF00992696 . 0885-6125.
3. Sutton . Richard S . McAllester . David . Singh . Satinder . Mansour . Yishay . 1999 . Policy Gradient Methods for Reinforcement Learning with Function Approximation . Advances in Neural Information Processing Systems . MIT Press . 12.
4. Schulman . John . High-Dimensional Continuous Control Using Generalized Advantage Estimation . 2018-10-20 . 1506.02438 . Moritz . Philipp . Sergey Levine . Levine . Sergey . Jordan . Michael . Abbeel . Pieter . cs.LG .
5. Kakade . Sham M . 2001 . A Natural Policy Gradient . Advances in Neural Information Processing Systems . MIT Press . 14.
6. Schulman . John . Levine . Sergey . Moritz . Philipp . Jordan . Michael . Abbeel . Pieter . 2015-07-06 . Trust region policy optimization . Proceedings of the 32nd International Conference on International Conference on Machine Learning . 37 . Lille, France . JMLR.org . 1889–1897.
7. Schulman . John . Proximal Policy Optimization Algorithms . 2017-08-28 . 1707.06347 . Wolski . Filip . Dhariwal . Prafulla . Radford . Alec . Klimov . Oleg . cs.LG .
8. Nisan Stiennon . Long Ouyang . Jeffrey Wu . Daniel Ziegler . Ryan Lowe . Chelsea Voss . Alec Radford . Dario Amodei . Paul F. Christiano . 2020 . Learning to summarize with human feedback . Advances in Neural Information Processing Systems . en . 33.
9. Shao . Zhihong . DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models . 2024-04-27 . 2402.03300 . Wang . Peiyi . Zhu . Qihao . Xu . Runxin . Song . Junxiao . Bi . Xiao . Zhang . Haowei . Zhang . Mingchuan . Li . Y. K. . cs.CL .
10. Arkadi Nemirovsky and David Yudin. Problem Complexity and Method Efficiency in Optimization. John Wiley & Sons, 1983.
11. Shani . Lior . Efroni . Yonathan . Mannor . Shie . 2020-04-03 . Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPS . Proceedings of the AAAI Conference on Artificial Intelligence . 34 . 4 . 5668–5675 . 10.1609/aaai.v34i04.6021 . 2374-3468. 1909.02769 .
12. Tomar . Manan . Shani . Lior . Efroni . Yonathan . Ghavamzadeh . Mohammad . 2020-05-20 . Mirror Descent Policy Optimization . cs.LG . 2005.09814v5 . en.