Algorithms and Programs of Dynamic Mixture Estimation: Unified Approach to Different Types of Components

ISBN-13: 9783319646701 / Angielski / Miękka / 2017 / 113 str.

Ivan Nagy; Evgenia Suzdaleva

Opis

This book provides a general theoretical background for constructing the recursive Bayesian estimation algorithms for mixture models.

Recenzje

"The book presents and discusses dynamic mixture models and their use in estimation and prediction. ... Mixture models have applications in several domains such as industry, engineering, social science, medicine, transportation etc. The book therefore can be of interest to researchers and PhD students in many diverse fields." (Christina Diakaki, zbMATH 1383.62005, 2018)

Spis treści

1 Introduction 7

1.1 On dynamic mixtures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2 General conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Basic Models 13

2.1 Regression model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.1 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.2 Point estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1.3 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2 Categorical model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.1 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.2.2 Point estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.2.3 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3 State-space model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3.1 State estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3 Statistical Analysis of Dynamic Mixtures 21

3.1 Dynamic mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.2 Unified approach to mixture estimation . . . . . . . . . . . . . . . . . . . . . . . 22

3.2.1 The component part . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2.2 The pointer part . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.2.3 Main subtasks of mixture estimation . . . . . . . . . . . . . . . . . . . . . 23

3.2.4 General algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3 Mixture prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3.1 Pointer prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.3.2 Data prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4 Dynamic Mixture Estimation 29

4.1 Normal regression components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1.2 Simple program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.3 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.2 Categorical components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.2.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.2.2 Simple program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.2.3 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3 State-space components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.3.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.3.2 Simple program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.3.3 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5 Program Codes 43

5.1 Main program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.1.1 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.2 Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.2.1 Initialization of estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.2.2 Computation of proximities . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.2.3 Update of component statistics . . . . . . . . . . . . . . . . . . . . . . . . 52

5.3 Collection of programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

6 Experiments 55

6.1 Mixture with regression components . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.1.1 Well separated components . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.1.2 Weak components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

6.1.3 Reduced number of components . . . . . . . . . . . . . . . . . . . . . . . 64

6.1.4 High dimensional output . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

<6.1.5 Big noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6.2 Mixture with categorical components . . . . . . . . . . . . . . . . . . . . . . . . . 69

6.3 Mixture with state-space components . . . . . . . . . . . . . . . . . . . . . . . . . 75

6.4 Case studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

6.4.1 Static normal components . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.4.2 Dynamic normal components . . . . . . . . . . . . . . . . . . . . . . . . . 80

7 Appendix A (supporting notions) 87

7.1 Useful matrix formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

7.2 Matrix trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

7.3 Dirac and Kronecker functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

7.4 Gamma and beta functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

7.5 The Bayes rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

7.6 The chain rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

7.7 The natural conditions of control . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

7.8 Conjugate Dirichlet distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

7.8.1 The normalization constant of Dirichlet distribution . . . . . . . . . . . . 92

7.8.2 Statistics update with the conjugate Dirichlet distribution . . . . . . . . . 92

7.8.3 The parameter point estimate of the categorical model . . . . . . . . . . . 93

7.8.4 Data prediction with Dirichlet distribution . . . . . . . . . . . . . . . . . 94

7.9 Conjugate Gauss-inverse-Wishart distribution . . . . . . . . . . . . . . . . . . . 94

7.9.1 Statistics update for the normal regression model . . . . . . . . . . . . . . 94

7.9.2 The parameter point estimate of the regression model . . . . . . . . . . . 95

7.9.3 The proximity evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

8 Appendix B (supporting programs) 97

8.1 Simulation programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

8.1.1 The simulation of pointer values . . . . . . . . . . . . . . . . . . . . . . . 97

8.1.2 The simulation of mixture with regression components . . . . . . . . . . . 98

8.1.3 The simulation of mixture with discrete components . . . . . . . . . . . . 99

8.1.4 The simulation of mixture with state-space components . . . . . . . . . . 101

8.2 Supporting subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

8.2.1 Scilab start settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

8.2.2 The point estimation of a normal regression model . . . . . . . . . . . . . 103

8.2.3 The value of a normal multivariate distribution . . . . . . . . . . . . . . . 104

8.2.4 Discrete regression vector coding . . . . . . . . . . . . . . . . . . . . . . . 105

8.2.5 Kalman filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

8.2.6 Matrix upper-lower factorization . . . . . . . . . . . . . . . . . . . . . . . 107

8.2.7 Transition table normalization . . . . . . . . . . . . . . . . . . . . . . . . 108

8.2.8 The approximation of normal pdfs by a single pdf . . . . . . . . . . . . . 108

O autorze

Doc. Ing. Ivan Nagy, CSc. (Ph.D.), born 1956 in Prague, Czech Republic, received his CSc. (Ph.D.) in cybernetics from UTIA, Prague in 1983. In 1980, he started working as a researcher at the Institute of Information Theory and Automation of the Czech Academy of Sciences. Since 1998, he has also been a lecturer at the Czech Technical University Faculty of Transportation Sciences in Prague.

Ing. Evgenia Suzdaleva, CSc. (Ph.D.), born 1977 in Krasnoyarsk, Russia, obtained her CSc. (Ph.D.) in 2002 in system analysis at the Siberian State Aerospace University, Krasnoyarsk, Russia. Since 2004, she has been a researcher at the Institute of Information Theory and Automation at the Czech Academy of Sciences. At the same time, she works as a lecturer at the Czech Technical University Faculty of Transportation Sciences in Prague.

Blurb

This book provides a general theoretical background for constructing the recursive Bayesian estimation algorithms for mixture models. It collects the recursive algorithms for estimating dynamic mixtures of various distributions and brings them in the unified form, providing a scheme for constructing the estimation algorithm for a mixture of components modeled by distributions with reproducible statistics. It offers the recursive estimation of dynamic mixtures, which are free of iterative processes and close to analytical solutions as much as possible. In addition, these methods can be used online and simultaneously perform learning, which improves their efficiency during estimation. The book includes detailed program codes for solving the presented theoretical tasks. Codes are implemented in the open source platform for engineering computations. The program codes given serve to illustrate the theory and demonstrate the work of the included algorithms.