Linear Regression

ISBN-13: 9783540401780 / Angielski / Miękka / 2003 / 398 str.

Jurgen Gross; J]rgen Gro_

Opis

In linear regression the ordinary least squares estimator plays a central role and sometimes one may get the impression that it is the only reasonable and applicable estimator available. Nonetheless, there exists a variety of alterna tives, proving useful in specific situations. Purpose and Scope. This book aims at presenting a comprehensive survey of different point estimation methods in linear regression, along with the the oretical background on a advanced courses level. Besides its possible use as a companion for specific courses, it should be helpful for purposes of further reading, giving detailed explanations on many topics in this field. Numerical examples and graphics will aid to deepen the insight into the specifics of the presented methods. For the purpose of self-containment, the basic theory of linear regression models and least squares is presented. The fundamentals of decision theory and matrix algebra are also included. Some prior basic knowledge, however, appears to be necessary for easy reading and understanding."

Spis treści

I Point Estimation and Linear Regression.- Fundamentals.- 1.1 Linear Models.- 1.1.1 Application of Linear Models.- 1.1.2 Types of Linear Models.- 1.1.3 Proceeding with Linear Models.- 1.1.4 A Preliminary Example.- 1.2 Decision Theory and Point Estimation.- 1.2.1 Decision Rule.- 1.2.2 Non-operational Decision Rule.- 1.2.3 Loss and Risk.- 1.2.4 Choosing a Decision Rule.- 1.2.5 Admissibility.- 1.2.6 Squared Error Loss.- 1.2.7 Matrix Valued Squared Error Loss.- 1.2.8 Alternative Loss Functions.- 1.3 Problems.- The Linear Regression Model.- 2.1 Assumptions.- 2.2 Ordinary Least Squares Estimation.- 2.2.1 The Principle of Least Squares.- 2.2.2 Coefficient of Determination R2.- 2.2.3 Predictive Loss.- 2.2.4 Least Squares Variance Estimator.- 2.2.5 Properties of the Ordinary Least Squares Estimator.- 2.2.6 Properties Under Normality.- 2.3 Optimality of Least Squares Estimation.- 2.3.1 Linear Unbiased Estimation.- 2.3.2 Gauss-Markov Theorem.- 2.3.3 Normality Assumption.- 2.3.4 Admissibility.- 2.4 Unreliability of Least Squares Estimation.- 2.4.1 Estimation of the Covariance Matrix.- 2.4.2 Unbiased Versus Biased Estimation.- 2.4.3 Collinearity.- 2.4.4 Consistency.- 2.4.5 Biased Estimation.- 2.5 Inadmissibility of the Ordinary Least Squares Estimator.- 2.5.1 The Reparameterized Regression Model.- 2.5.2 Risk Comparison of Least Squares and Stein Estimator.- 2.5.3 An Example for Stein Estimation.- 2.5.4 Admissibility.- 2.6 Problems.- II Alternatives to Least Squares Estimation.- Alternative Estimators.- 3.1 Restricted Least Squares Estimation.- 3.1.1 The Principle of Restricted Least Squares.- 3.1.2 The Parameter Space.- 3.1.3 Properties of Restricted Least Squares Estimator.- 3.1.4 Risk Comparison of Restricted and Ordinary Least Squares Estimator.- 3.1.5 Pretest Estimation.- 3.2 Other Types of Restriction.- 3.2.1 Stochastic Linear Restrictions.- 3.2.2 Inequality Restrictions.- 3.2.3 Elliptical Restrictions.- 3.3 Principal Components Estimator.- 3.3.1 Preliminary Considerations.- 3.3.2 Properties of the Principal Components Estimator.- 3.3.3 Drawbacks of the Principal Components Estimator.- 3.3.4 The Marquardt Estimator.- 3.4 Ridge Estimator.- 3.4.1 Preliminary Considerations.- 3.4.2 Properties of the Linear Ridge Estimator.- 3.4.3 The Choice of the Ridge Parameter.- 3.4.4 Standardization.- 3.4.5 Ridge and Restricted Least Squares Estimator.- 3.4.6 Ridge and Principal Components Estimator.- 3.4.7 Jackknife Modified Ridge Estimator.- 3.4.8 Iteration Estimator.- 3.4.9 An Example for Ridge Estimation.- 3.5 Shrinkage Estimator.- 3.5.1 Preliminary Considerations.- 3.5.2 Risk Comparison to Ordinary Least Squares.- 3.5.3 The Choice of the Shrinkage Parameter.- 3.5.4 Direction Modified Shrinkage Estimators.- 3.6 General Ridge Estimator.- 3.6.1 A Class of Estimators.- 3.6.2 Risk Comparison of General Ridge and Ordinary Least Squares Estimator.- 3.7 Linear Minimax Estimator.- 3.7.1 Preliminary Considerations.- 3.7.2 Inequality Restrictions.- 3.7.3 Linear Minimax Solutions.- 3.7.4 Alternative Approaches.- 3.7.5 Admissibility.- 3.8 Linear Bayes Estimator.- 3.8.1 Preliminary Considerations.- 3.8.2 Characterization of Linear Bayes Estimators.- 3.8.3 Non-Operational Bayes Solutions.- 3.8.4 A-priori Assumptions.- 3.9 Robust Estimator.- 3.9.1 Preliminary Considerations.- 3.9.2 Weighted Least Squares Estimation.- 3.9.3 The l1 Estimator.- 3.9.4 M Estimator.- 3.9.5 Robust Ridge Estimator.- 3.10 Problems.- Linear Admissibility.- 4.1 Preliminary Considerations.- 4.2 Linear Admissibility in the Non-Restricted Model.- 4.2.1 Linear Admissibility in the Simple Mean Shift Model.- 4.2.2 Characterization of Linearly Admissible Estimators.- 4.2.3 Ordinary Least Squares and Linearly Admissible Estimator.- 4.2.4 Linear Transforms of Ordinary Least Squares Estimator.- 4.2.5 Linear Admissibility of Known Estimators.- 4.2.6 Shrinkage Property and Linear Admissibility.- 4.2.7 Convex Combination of Estimators.- 4.2.8 Linear Bayes Estimator.- 4.3 Linear Admissibility Under Linear Restrictions.- 4.3.1 The Assumption of a Full Rank Restriction Matrix.- 4.3.2 Restricted Estimator.- 4.3.3 Characterization of Linearly Admissible Estimators.- 4.4 Linear Admissibility Under Elliptical Restrictions.- 4.4.1 Characterization of Linearly Admissible Estimators.- 4.4.2 Linear Admissibility of Certain Linear Estimators.- 4.4.3 Admissible Improvements Over Ordinary Least Squares.- 4.5 Problems.- III Miscellaneous Topics.- The Covariance Matrix of the Error Vector.- 5.1 Estimation of the Error Variance.- 5.1.1 The Sample Variance.- 5.1.2 Nonnegative Unbiased Estimation.- 5.1.3 Optimality of the Least Squares Variance Estimator.- 5.1.4 Non-Admissibility of the Least Squares Variance Estimator.- 5.2 Non-Scalar Covariance Matrix.- 5.2.1 Preliminary Considerations.- 5.2.2 The Transformed Model.- 5.2.3 Two-Stage Estimation.- 5.3 Occurrence of Non-Scalar Covariance Matrices.- 5.3.1 Seemingly Unrelated Regression.- 5.3.2 Heteroscedastic Errors.- 5.3.3 Equicorrelated Errors.- 5.3.4 Autocorrelated Errors.- 5.4 Singular Covariance Matrices.- 5.5 Equality of Ordinary and Generalized Least Squares.- 5.6 Problems.- Regression Diagnostics.- 6.1 Selecting Independent Variables.- 6.1.1 Mallows’ Cp.- 6.1.2 Stepwise Regression.- 6.1.3 Alternative Criteria.- 6.2 Assessing Goodness of Fit.- 6.3 Diagnosing Collinearity.- 6.3.1 Variance Inflation Factors.- 6.3.2 Scaled Condition Indexes.- 6.4 Inspecting Residuals.- 6.4.1 Normal Quantile Plot.- 6.4.2 Residuals Versus Fitted Values Plot.- 6.4.3 Further Residual Plots.- 6.5 Finding Influential Observations.- 6.5.1 Leverage.- 6.5.2 Influential Observations.- 6.5.3 Collinearity-Influential Observations.- 6.6 Testing Model Assumptions.- 6.6.1 Preliminary Considerations.- 6.6.2 Testing for Heteroscedasticity.- 6.6.3 Testing for Autocorrelation.- 6.6.4 Testing for Non-Normality.- 6.6.5 Testing for Non-Linearity.- 6.6.6 Testing for Outlier.- 6.7 Problems.- Matrix Algebra.- A.1 Preliminaries.- A.1.1 Matrices and Vectors.- A.1.2 Elementary Operations.- A.1.3 Rank of a Matrix.- A.1.4 Subspaces and Matrices.- A.1.5 Partitioned Matrices.- A.1.6 Kronecker Product.- A.1.7 Moore-Penrose Inverse.- A.2 Common Pitfalls.- A.3 Square Matrices.- A.3.1 Specific Square Matrices.- A.3.2 Trace and Determinant.- A.3.3 Eigenvalue and Eigenvector.- A.3.4 Vector and Matrix Norm.- A.3.5 Definiteness.- A.4 Symmetric Matrix.- A.4.1 Eigenvalues.- A.4.2 Spectral Decomposition.- A.4.3 Rayleigh Ratio.- A.4.4 Definiteness.- A.5 Löwner Partial Ordering.- Stochastic Vectors.- B.1 Expectation and Covariance.- B.2 Multivariate Normal Distribution.- B.3 x2 Distribution.- B.4 F Distribution.- An Example Analysis with R.- C.1 Problem and Goal.- C.2 The Data.- C.3 The Choice of Variables.- C.3.1 The Full Model.- C.3.2 Stepwise Regression.- C.3.3 Collinearity Diagnostics.- C.4 Further Diagnostics.- C.4.1 Residuals.- C.4.2 Influential Observations.- C.5 Prediction.- References.