Multivariate Statistics Classical Foundations and ModernMachine Learning

"This book explores multivariate statistics from both traditional and modern perspectives. The first section covers core topics like multivariate normality, MANOVA, discrimination, PCA, and canonical correlation analysis. The second section includes modern concepts such as gradient boosting, random forests, variable importance, and causal inference. A key theme is leveraging classical multivariate statistics to explain advanced topics and prepare for contemporary methods. For example, linear models provide a foundation for understanding regularization with AIC and BIC, leading to a deeper analysis of regularization through generalization error and the VC theorem. Discriminant analysis introduces the weighted Bayes rule, which leads into modern classification techniques for class-imbalanced machine learning problems. Steepest descent serves as a precursor to matching pursuit and gradient boosting. Axis-aligned trees like CART, a classical tool, set the stage for more recent methods like super greedy trees. Another central theme is the concept of training error. While introductory courses often emphasize that reducing training error can lead to overfitting, training error is also known as empirical risk. In regression, it is the residual sum of squares, and reducing it leads to least squares solutions. But is this always the best approach? Empirical risk is vital in statistical learning theory and is key to determining whether learning is possible. The principle of empirical risk minimization is crucial and shows that reducing training error is not harmful when regularization is applied. This principle is further examined through discussions on penalization, matching pursuit, gradient boosting, and super greedy trees"--