Impact Analysis of Stacked Machine Learning Algorithms Based Feature Selections for Deep Learning Algorithm Applied to Regression Analysis

Ensemble learning algorithms have proved to be one of the best machine learning algorithms towards optimal performances in terms of regression and classification tasks for a variety of applications. When applied for small or medium structured datasets, eXtreme Gradient Boosting (XGBoost) has emerged as a popular ensemble learning technique based on its performance merits. In recent years Light Gradient Boosting Machine (LightGBM) has emerged as a promising ensemble strategy that is competing with XGBoost in terms of performance. Also, Lasso Regression has proven capabilities in terms of feature selections and applications for small datasets. This paper illustrates experimentation performed on a diabetes dataset where the authors tested the hypothesis that feature selection has relatively no impact on the performances of Deep Learning Algorithms as they have built-in capabilities in terms of layers to perform feature selection on their own. Therefore, the hypothesis tested stacking using Deep Learning - Multi-Layer Perceptron (DMLP) with optimal algorithms like XGBoost, LightGBM, and Lasso Regression. In the present work, DMLP with all feature variables (DMLP-ALL) outscored DMLP with stacked selected features (DMLP-MS) by 8.78 % in terms of R2. Also, DMLP-ALL outperformed the benchmarked algorithm Automated Machine Learning (AML) by 10.25% in terms of R2. The validation of the proposed stacking models by applying a moderate-sized dataset provides promising results for deep learning models stacked with a powerful Level-0 learner.