### Abstract

We consider policy evaluation algorithms within the context of infinite-horizon dynamic programming problems with discounted cost. We focus on discrete-time dynamic systems with a large number of states, and we discuss two methods, which use simulation, temporal differences, and linear cost function approximation. The first method is a new gradient-like algorithm involving least-squares subproblems and a diminishing stepsize, which is based on the λ-policy iteration method of Bertsekas and Ioffe. The second method is the LSTD(λ) algorithm recently proposed by Boyan, which for λ = 0 coincides with the linear least-squares temporal-difference algorithm of Bradtke and Barto. At present, there is only a convergence result by Bradtke and Barto for the LSTD(0) algorithm. Here, we strengthen this result by showing the convergence of LSTD(λ), with probability 1, for every λε[0,1].

Original language | English (US) |
---|---|

Pages (from-to) | 79-110 |

Number of pages | 32 |

Journal | Discrete Event Dynamic Systems: Theory and Applications |

Volume | 13 |

Issue number | 1-2 |

DOIs | |

State | Published - Jan 2003 |

Externally published | Yes |

### Fingerprint

### ASJC Scopus subject areas

- Control and Systems Engineering
- Applied Mathematics
- Management Science and Operations Research
- Safety, Risk, Reliability and Quality

### Cite this

*Discrete Event Dynamic Systems: Theory and Applications*,

*13*(1-2), 79-110. https://doi.org/10.1023/A:1022192903948

**Least squares policy evaluation algorithms with linear function approximation.** / Nedich, Angelia; Bertsekas, D. P.

Research output: Contribution to journal › Article

*Discrete Event Dynamic Systems: Theory and Applications*, vol. 13, no. 1-2, pp. 79-110. https://doi.org/10.1023/A:1022192903948

}

TY - JOUR

T1 - Least squares policy evaluation algorithms with linear function approximation

AU - Nedich, Angelia

AU - Bertsekas, D. P.

PY - 2003/1

Y1 - 2003/1

N2 - We consider policy evaluation algorithms within the context of infinite-horizon dynamic programming problems with discounted cost. We focus on discrete-time dynamic systems with a large number of states, and we discuss two methods, which use simulation, temporal differences, and linear cost function approximation. The first method is a new gradient-like algorithm involving least-squares subproblems and a diminishing stepsize, which is based on the λ-policy iteration method of Bertsekas and Ioffe. The second method is the LSTD(λ) algorithm recently proposed by Boyan, which for λ = 0 coincides with the linear least-squares temporal-difference algorithm of Bradtke and Barto. At present, there is only a convergence result by Bradtke and Barto for the LSTD(0) algorithm. Here, we strengthen this result by showing the convergence of LSTD(λ), with probability 1, for every λε[0,1].

AB - We consider policy evaluation algorithms within the context of infinite-horizon dynamic programming problems with discounted cost. We focus on discrete-time dynamic systems with a large number of states, and we discuss two methods, which use simulation, temporal differences, and linear cost function approximation. The first method is a new gradient-like algorithm involving least-squares subproblems and a diminishing stepsize, which is based on the λ-policy iteration method of Bertsekas and Ioffe. The second method is the LSTD(λ) algorithm recently proposed by Boyan, which for λ = 0 coincides with the linear least-squares temporal-difference algorithm of Bradtke and Barto. At present, there is only a convergence result by Bradtke and Barto for the LSTD(0) algorithm. Here, we strengthen this result by showing the convergence of LSTD(λ), with probability 1, for every λε[0,1].

UR - http://www.scopus.com/inward/record.url?scp=0037288398&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0037288398&partnerID=8YFLogxK

U2 - 10.1023/A:1022192903948

DO - 10.1023/A:1022192903948

M3 - Article

AN - SCOPUS:0037288398

VL - 13

SP - 79

EP - 110

JO - Discrete Event Dynamic Systems: Theory and Applications

JF - Discrete Event Dynamic Systems: Theory and Applications

SN - 0924-6703

IS - 1-2

ER -