Mixture of Logistic Regression with LASSO Regularization for High-Dimensional Binary Classification with Latent Heterogeneity
DOI:
https://doi.org/10.62933/h3051210Keywords:
Mixture models, LASSO regularisation , high-dimensional classification, EM algorithm, breast cancer diagnosisAbstract
This paper proposes a Mixture of Logistic Regression with LASSO Regularization (MLR-LR) framework for high-dimensional binary classification under latent heterogeneity. The approach integrates finite mixture modelling to capture unobserved subpopulations with ℓ₁-penalizsed logistic regression to induce sparsity, enabling simultaneous subgroup identification and variable selection. Parameter estimation is carried out via a penalized Expectation-Maximization algorithm with component-wise coordinate descent. In contrast to the primarily applied framework of Morvan et al. (2021), we develop a rigorous theoretical foundation for penalized mixture logistic models. Under high-dimensional asymptotic regimes, we establish identifiability, estimation consistency, sparsity recovery, and the oracle property of the proposed estimator. Extensive simulations across six heterogeneity regimes (R1-R6) with 200 replications demonstrate that MLR-LR consistently outperforms homogeneous LASSO when latent structure is present, while remaining competitive in homogeneous settings. In particular, the proposed method achieves systematic improvements in classification performance, with gains in AUC ranging from 0.028 to 0.054 under heterogeneous scenarios. Application to the Wisconsin Diagnostic Breast Cancer dataset further illustrates the practical utility of the method in the presence of multicollinearity, where MLR-LR attains superior predictive performance (AUC=0.9903, F1=0.9550, misclassification=0.0292) relative to LASSO, Random Forest, and XGBoost, while yielding interpretable, component-specific coefficient structures that reveal distinct risk profiles. Overall, the proposed framework provides a unified and theoretically grounded approach for high-dimensional heterogeneous binary classification, bridging the gap between mixture modelling and sparse statistical learning.
References
[1] McLachlan, G., & Peel, D. (2000). Finite mixture models. Wiley.
[2] Titterington, D. M., Smith, A. F. M., & Makov, U. E. (1985). Statistical analysis of finite mixture distributions. John Wiley & Sons.
[3] Wedel, M., & Kamakura, W. A. (2000). Market segmentation: Conceptual and methodological foundations (2nd ed.). Springer.
[4] Bouveyron, C., Celeux, G., Murphy, T. B., & Raftery, A. E. (2019). Model-based clustering and classification for data science. Cambridge University Press.
[5] Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
[6] Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society: Series B, 58(1), 267-288.
[7] Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1-22.
[8] Meier, L., van de Geer, S., & Bühlmann, P. (2008). The group lasso for logistic regression. Journal of the Royal Statistical Society: Series B, 70(1), 53-71.
[9] Khalili, A., & Chen, J. (2007). Variable selection in finite mixture of regression models. Journal of the American Statistical Association, 102(479), 1025-1038.
[10] Städler, N., Bühlmann, P., & van de Geer, S. (2010). ℓ₁-penalization for mixture regression models. TEST, 19(2), 209-256.
[11] Städler, N., Bühlmann, P., & van de Geer, S. (2012). ℓ₁-penalization for mixture regression models. Journal of Statistical Planning and Inference, 142(12), 3192-3205.
[12] Khalili, A. (2011). An overview of the new feature selection methods in finite mixture of regression models. Journal of the Iranian Statistical Society, 10(2), 201-235.
[13] Morvan, M., Bertrand, A., Doutrelon, P., & Bruyère, F. (2021). Penalized mixture of logistic regressions for the prediction of nonalcoholic steatohepatitis. Annals of Applied Statistics, 15(2), 1043-1061.
[14] Sohn, H., Kim, D., & Park, T. (2020). Adaptive penalized mixture of experts models. Journal of Computational and Graphical Statistics, 29(3), 540-551.
[15] Huang, Y., Breheny, P., & Ma, S. (2018). A selective review of group variable selection in high-dimensional models. Statistical Science, 33(4), 552-569.
[16] Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39(1), 1-22.
[17] Teicher, H. (1963). Identifiability of finite mixtures. The Annals of Mathematical Statistics, 34(4), 1265-1269.
[18] Allman, E., Matias, C., & Rhodes, J. (2009). Identifiability of parameters in latent structure models with many observed variables. The Annals of Statistics, 37(6A), 3099-3132.
[19] van der Vaart, A. W. (1998). Asymptotic statistics. Cambridge University Press.
[20] Pan, W., & Shen, X. (2007). Penalized model-based clustering with applications to variable selection. Journal of Machine Learning Research, 8, 1145-1164.
[21] Fraley, C., & Raftery, A. E. (2002). Model-based clustering, discriminant analysis, and density estimation. Journal of the American Statistical Association, 97(458), 611-631.
[22] Hastie, T., Tibshirani, R., & Wainwright, M. (2015). Statistical learning with sparsity: The Lasso and generalizations. Chapman and Hall/CRC.
[23] Keribin, C. (2000). Consistent estimation of the order of mixture models. Sankhyā: The Indian Journal of Statistics, 62(1), 49-66.
[24] Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3(1), 79-87.
[25] Hopkins, B., & Skellam, J. G. (1954). A new method for determining the type of distribution of plant individuals. Annals of Botany, 18(2), 213-227.
[26] Wolberg, W. H. (1995). Breast Cancer Wisconsin (Diagnostic) Data Set. UCI Machine Learning Repository. https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)
Downloads
Published
Issue
Section
License
Copyright (c) 2026 buhari ishaq, Abubakar Usman, Yahaya Zakari, Hillary Ugwu Okwudili (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Licensed under a Creative Commons Attribution 4.0 International License: https://creativecommons.org/licenses/by/4.0/
