The leading and most up-to-date textbook on the far-rangingalgorithmic methododogy of Dynamic Programming, which can be used for optimal control,Markovian decision problems, planning and sequential decision making under uncertainty, anddiscrete/combinatorial optimization. The treatment focuses on basic unifyingthemes, andconceptual foundations. Itillustrates the versatility, power, and generality of the method withmany examples and applicationsfrom engineering, operations research, and other fields. It alsoaddresses extensively the practicalapplication of the methodology, possibly through the use of approximations, andprovides an extensive treatment of the far-reaching methodology ofNeuro-Dynamic Programming/Reinforcement Learning.
The first volume is oriented towards modeling, conceptualization, andfinite-horizon problems, but also includes a substantive introductionto infinite horizon problems that is suitable for classroom use. Thesecond volume is oriented towards mathematical analysis andcomputation, treats infinite horizon problems extensively, and provides an up-to-date account of approximate large-scale dynamic programming and reinforcement learning. Thetext contains many illustrations, worked-out examples, and exercises.
This extensive work, aside from its focus on the mainstream dynamicprogramming and optimal controltopics, relates to our Abstract Dynamic Programming (Athena Scientific, 2013),a synthesis of classical research on the foundations of dynamic programming with modern approximate dynamic programming theory, and the new class of semicontractive models, Stochastic Optimal Control: The Discrete-TimeCase (Athena Scientific, 1996),which deals with the mathematical foundations of the subject, Neuro-Dynamic Programming (Athena Scientific,1996), which develops the fundamental theory for approximation methods in dynamic programming,Reinforcement Learning and Optimal Control (Athena Scientific,2019) and Rollout, Policy Iteration, and Distributed Reinforcement Learning (Athena Scientific,202, which develop approximation methods that are suitable for large-scale problems,and Introduction to Probability (2nd Edition, Athena Scientific,2008), which provides the prerequisite probabilistic background.
New features of the 4th edition of Vol. I (see the Preface fordetails): provides textbook accounts of recent original research onapproximate DP, limited lookahead policies, rollout algorithms, model predictive control, Monte-Carlo tree search and the recent uses of deep neural networks in computer game programs such as Go.
New features of the 4th edition of Vol. II (see the Preface fordetails): Contains a substantial amount of new material, as well as a reorganization of old material. The length has increased by more than 60% from the third edition, andmost of the old material has been restructured and/or revised. Volume II now numbers more than 700 pages and is larger in size than Vol. I. It can arguably be viewed as a new book!
A major expansion of the discussion of approximate DP (neuro-dynamic programming), which allows the practical application of dynamic programming to large and complex problems. Approximate DP has become the central focal point of this volume.
Extensive new material, the outgrowth of research conducted in the six years since the previous edition, has been included. The first account of the emerging methodology of Monte Carlo linear algebra, which extends the approximate DP methodology to broadly applicable problems involving large-scale regression and systems of linear equations.
"Prof. Bertsekas book is an essential contribution that provides practitioners with a 30,000 feet view in Volume I - the second volume takes a closer look at the specific algorithms, strategies and heuristics used - of the vast literature generated by the diverse communities that pursue the advancement of understanding and solving control problems. This is achieved through the presentation of formal models for special