This volume provides the first book entirely devoted to recent developments on the theory and applications of continuous-time Markov decision processes (MDPs). The MDPs presented here include most of the cases that arise in applications.

This book encompasses the study of hybrid switching di usion processes and their applications. The word hybrid" signi es the coexistence of c- tinuous dynamics and discrete events, which is one of the distinct features of the processes under consideration. Much of the book is concerned with the interactions of the continuous dynamics and the discrete events. Our motivations for studying such processes originate from emerging and - isting applications in wireless communications, signal processing, queueing networks, production planning, biological systems, ecosystems, nancial engineering, and...

This book gives a systematic treatment of singularly perturbed systems that naturally arise in control and optimization, queueing networks, manufacturing systems, and financial engineering. It presents results on asymptotic expansions of solutions of Komogorov forward and backward equations, properties of functional occupation measures, exponential upper bounds, and functional limit results for Markov chains with weak and strong interactions. To bridge the gap between theory and applications, a large portion of the book is devoted to applications in controlled dynamic systems, production...

The theory of probability began in the seventeenth century with attempts to calculate the odds of winning in certain games of chance. However, it was not until the middle of the twentieth century that mathematicians de- veloped general techniques for maximizing the chances of beating a casino or winning against an intelligent opponent. These methods of finding op- timal strategies for a player are at the heart of the modern theories of stochastic control and stochastic games. There are numerous applications to engineering and the social sciences, but the liveliest intuition still comes from...

This book has grown out of lectures and courses in calculus of variations and optimization taught for many years at the University of Michigan to graduate students at various stages of their careers, and always to a mixed audience of students in mathematics and engineering. It attempts to present a balanced view of the subject, giving some emphasis to its connections with the classical theory and to a number of those problems of economics and engineering which have motivated so many of the present developments, as well as presenting aspects of the current theory, particularly value theory and...

The stimulus for the present work is the growing need for more accurate numerical methods. The rapid advances in computer technology have not provided the resources for computations which make use of methods with low accuracy. The computational speed of computers is continually increasing, while memory still remains a problem when one handles large arrays. More accurate numerical methods allow us to reduce the overall computation time by of magnitude. several orders The problem of finding the most efficient methods for the numerical solution of equations, under the assumption of fixed array...

Adaptive systems are widely encountered in many applications ranging through adaptive filtering and more generally adaptive signal processing, systems identification and adaptive control, to pattern recognition and machine intelligence: adaptation is now recognised as keystone of "intelligence" within computerised systems. These diverse areas echo the classes of models which conveniently describe each corresponding system. Thus although there can hardly be a "general theory of adaptive systems" encompassing both the modelling task and the design of the adaptation procedure, nevertheless,...

The book is mainly concerned with the mathematical foundations of Bayesian image analysis and its algorithms. This amounts to the study of Markov random fields and dynamic Monte Carlo algorithms like sampling, simulated annealing and stochastic gradient algorithms. The approach is introductory and elementary: given basic concepts from linear algebra and real analysis it is self-contained. No previous knowledge from image analysis is required. Knowledge of elementary probability theory and statistics is certainly beneficial but not absolutely necessary. The necessary background from imaging is...

Historical Comments Two-dimensional random walks in domains with non-smooth boundaries inter- est several groups of the mathematical community. In fact these objects are encountered in pure probabilistic problems, as well as in applications involv- ing queueing theory. This monograph aims at promoting original mathematical methods to determine the invariant measure of such processes. Moreover, as it will emerge later, these methods can also be employed to characterize the transient behavior. It is worth to place our work in its historical context. This book has three sources. l. Boundary...

"This book is concerned with a probabilistic approach for image analysis, mostly from the Bayesian point of view, and the important Markov chain Monte Carlo methods commonly used....This book will be useful, especially to researchers with a strong background in probability and an interest in image analysis. The author has presented the theory with rigor...he doesn't neglect applications, providing numerous examples of applications to illustrate the theory." -- MATHEMATICAL REVIEWS