AI Model Enhancement through Surrogate Model Selection and Transfer Learning
In recent advancements in machine learning (ML), researchers have been focusing on optimising model performance for diverse applications by incorporating intelligent feature selection and enhancement techniques such as surrogate models and transfer learning. A recent study delved into the detailed process of selecting and refining surrogate models for predictive analytics, showcasing its significance in improving prediction accuracy and efficiency across various unseen configurations.
The study, conducted using MATLAB 2022a on a device equipped with an Intel Core i7-8665U CPU and 16 GB of RAM, focused on evaluating the performance of numerous ML models. The primary objective was to optimally configure these models for surrogate modelling (SM)—a pivotal step towards successfully implementing nested cross-validation (NCV) and updating the SM using Transfer Learning (TL).
Surrogate Model Selection Process
This phase involved training several ML models under two conditions: firstly, utilising only discrete element method (DEM) interaction parameters, and secondly, integrating additional features characterising the initial configurations (ICs) of mixtures in a hopper. Data from 185 samples were employed, featuring 18 distinct factors, to gauge model performance through a 5-fold cross-validation strategy. Results indicated a marked performance enhancement when ICs were included in the model as input features.
By implementing Bayesian optimisation (BO) for hyperparameter tuning across 50 iterations, the models significantly improved, highlighting the necessity of hyperparameter tuning in ML. Noteworthy, among all models evaluated, the Gaussian Process Regression (GPR) and ensemble of trees emerged as the top contenders, with GPR surpassing in all performance metrics, including minimal Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and maximal (R^{2}).
Employing Transfer Learning for Enhanced Model Generalisation
Subsequent to model selection, the incorporation of transfer learning was explored to enhance the model’s ability to adapt to unseen initial configurations. For this, ensemble learning and GPR methods were reassessed, employing a designed cross-validation framework across various ICs established earlier. The objective was to validate model efficacy and update performance on unseen ICs through strategic retraining.
During this phase, models were initially evaluated using all available samples, creating a performance benchmark. In iterative procedures, one IC was systematically excluded, while the model was trained on the remaining ICs and tested on the excluded IC. Notably, the application of transfer learning strategies facilitated substantial improvements even with minimal sample retraining from the unseen IC, showcasing the robustness of these models in handling novel data.
Results and Observations
The study provided detailed insights into the performance of both TL-Ensemble and TL-GPR models. Notably, the TL-GPR model consistently demonstrated superior performance over TL-Ensemble across diverse sample sizes. Specifically, the TL-GPR model displayed significant error reduction, particularly with minimal sample retraining, validating its effectiveness in rapidly adjusting to new ICs and enhancing predictive accuracy.
Performance assessments revealed variations across different ICs. For instance, the ICs with unique material distributions, such as IC3 with fully mixed materials and IC2 with reversed layering, initially posed challenges to the models due to their distinctive data distribution. However, subsequent retraining with even a few data points drastically improved model performance.
Conclusion
The research underscores the importance of strategic feature integration and advanced learning techniques like transfer learning in optimising ML model performance. By effectively employing surrogate model selection and transfer learning, significant improvements in model predictability and adaptability to new configurations were achieved, thus contributing valuable advancements to the field of machine learning and its applications.
Source: Noah Wire Services