Machine Learning Research

| Peer-Reviewed |

Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review

Received: Dec. 20, 2021    Accepted: Jan. 08, 2022    Published: Jan. 14, 2022
Views:       Downloads:

Share

Abstract

Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.

DOI 10.11648/j.mlr.20220701.11
Published in Machine Learning Research ( Volume 7, Issue 1, June 2022 )
Page(s) 1-7
Creative Commons

This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.

Copyright

Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Transfer Learning, Machine Learning, Hyperparameter, Automation, Artificial Intelligence

References
[1] Chauhan, K., Jani, S., Thakkar, D., Dave, R., Bhatia, J., Tanwar, S., & Obaidat, M. S. (2020). Automated Machine Learning: The New Wave of Machine Learning. (Icimia), 205–212.
[2] Tuggener, L., Amirian, M., Rombach, K., Stefan, L., Varlet, A., Westermann, C., … Ag, P. (2019). Automated Machine Learning in Practice: State of the Art and Recent Results. 31–36. https://doi.org/10.1109/SDS.2019.00-11.
[3] Lutkevich, B. (2020). What is automated machine learning (AutoML). Retrieved from https://searchenterpriseai.techtarget.com/definition/automated-machine-learning-AutoML.
[4] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., and Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529 (7587), 484–489. https://doi.org/10.1038/nature16961.
[5] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90.
[6] Yao, Q., Wang, M., Chen, Y., Dai, W., Li, Y., Tu, W., … Yu, Y. (2018). Taking Human out of Learning Applications: A Survey on Automated Machine Learning. 1–20. Retrieved from http://arxiv.org/abs/1810.13306.
[7] Mirtaheri, S. M., & Dinc, M. E. (2014). A Brief History of Web Crawlers. (May).
[8] Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., & Hutter, F. (2019). Auto-sklearn: Efficient and Robust Automated Machine Learning.
[9] Thornton, C., & Leyton-brown, K. (2013). Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. 847–855.
[10] Escalante, H. J. (2020). Automated Machine Learning -- a brief review at the end of the early years. 1–17. Retrieved from http://arxiv.org/abs/2008.08516.
[11] Olson, R. S., Bartley, N., Urbanowicz, R. J., & Moore, J. H. (2016). Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science.
[12] Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural Architecture Search: A Survey. 20, 1–21.
[13] Cai, H., Chen, T., Zhang, W., Yu, Y., & Wang, J. (2017). Efficient Architecture Search by Network Transformation. 2787–2794.
[14] Donges, N. (2021). What Is Transfer Learning A Simple Guide Built In. Retrieved from https://builtin.com/data-science/transfer-learning.
[15] Pan, S. J., & Fellow, Q. Y. (2009). A Survey on Transfer Learning. 1–15.
[16] Kaul, A., Maheshwary, S., & Pudi, V. (2017). AutoLearn - Automated Feature Generation and Selection. https://doi.org/10.1109/ICDM.2017.31.
[17] Krauß, J., Pacheco, B. M., Zang, H. M., Heinrich, R., Design, C., Stief, P., … Siadat, A. (2020). ScienceDirect ScienceDirect methodology to analyze the functional and physical architecture Automated machine learning for predictive quality in production existing products for an assembly oriented product family identification. Procedia CIRP, 93, 443–448. https://doi.org/10.1016/j.procir.2020.04.039.
[18] Yang, C., Fan, J., Wu, Z., & Udell, M. (2020). AutoML Pipeline Selection: Efficiently Navigating the Combinatorial Space.
[19] Swearingen, T., Drevo, W., Cyphers, B., Cuesta-Infante, A., Ross, A., & Veeramachaneni, K. (2017). ATM: A distributed, collaborative, scalable system for automated machine learning. Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017, 2018-January (December), 151–162. https://doi.org/10.1109/BigData.2017.8257923.
[20] Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K., & Miikkulainen, R. (2019). Evolutionary neural automl for deep learning. GECCO 2019 - Proceedings of the 2019 Genetic and Evolutionary Computation Conference, 401–409. https://doi.org/10.1145/3321707.3321721.
[21] Feurer, M., & Hutter, F. (2019). Hyperparameter Optimization. 3–33.
[22] Imbrea, A. (2021). Automated Machine Learning Techniques for Data Streams. Retrieved from http://arxiv.org/abs/2106.07317.
[23] Mohr, F., Wever, M., & Hüllermeier, E. (2018). ML-Plan: Automated machine learning via hierarchical planning. Machine Learning. https://doi.org/10.1007/s10994-018-5735-z.
[24] He, X., Zhao, K., & Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212 (Dl). https://doi.org/10.1016/j.knosys.2020.106622.
[25] Pan, W., Xiang, E. W., Liu, N. N., & Yang, Q. (2010). Transfer Learning in Collaborative Filtering for Sparsity Reduction. 230–235.
[26] Lam, H. T., Thiebaut, J., & Sinn, M. (2016). One button machine for automating feature engineering in relational databases.
[27] Katz, G., & Song, D. (2016). ExploreKit: Automatic Feature Generation and Selection. https://doi.org/10.1109/ICDM.2016.176.
[28] Nargesian, F., Samulowitz, H., Khurana, U., Khalil, E. B., & Turaga, D. (2017). Learning Feature Engineering for Classification. 2529–2535.
[29] Garrido-merch, E. C., Tom, F., Hern, D., & Tom, F. (2016). Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints.
[30] Li, L., Jamieson, K., & Desalvo, G. (2018). Hyperband: A Novel Bandit-Based Approach to. 18, 1–52.
[31] Baker, B., Gupta, O., and Raskar, N. N. (2018). Accelerating neural architecture search using performance prediction. 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings, 1–19.
[32] Mendoza, H. (2016). Towards Automatically-Tuned Neural Networks. 1–8.
[33] Guyon, I., Sun-hosoya, L., Boullé, M., Escalante, H., Liu, Z., Jajetic, D., … Guyon, I. (2018). Analysis of the AutoML Challenge series 2015-2018 To cite this version: HAL Id: hal-01906197 Analysis of the AutoML Challenge series 2015-2018.
[34] Hutter, F., Hoos, H. H., & Leyton-brown, K. (2011). Sequential Model-Based Optimization for General Algorithm Configuration.
[35] Jamieson, K., & Talwalkar, A. (2016). Non-stochastic best arm identification and hyperparameter optimization. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, 240–248.
[36] Maclaurin, D., Duvenaud, D., and R. A. (2015). Hyper-Parameter Initialization for Squared Exponential Kernel-based Gaussian Process Regression Nalika Ulapane - Academia. Retrieved from https://www.academia.edu/45378935/Hyper_Parameter_Initialization_for_Squared_Exponential_Kernel_based_Gaussian_Process_Regression.
Cite This Article
  • APA Style

    Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson. (2022). Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Machine Learning Research, 7(1), 1-7. https://doi.org/10.11648/j.mlr.20220701.11

    Copy | Download

    ACS Style

    Nwokonkwo Obi Chukwuemeka; John-Otumu Adetokunbo MacGregor; Nnadi Leonard Chukwualuka; Ogene Ferguson. Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Mach. Learn. Res. 2022, 7(1), 1-7. doi: 10.11648/j.mlr.20220701.11

    Copy | Download

    AMA Style

    Nwokonkwo Obi Chukwuemeka, John-Otumu Adetokunbo MacGregor, Nnadi Leonard Chukwualuka, Ogene Ferguson. Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review. Mach Learn Res. 2022;7(1):1-7. doi: 10.11648/j.mlr.20220701.11

    Copy | Download

  • @article{10.11648/j.mlr.20220701.11,
      author = {Nwokonkwo Obi Chukwuemeka and John-Otumu Adetokunbo MacGregor and Nnadi Leonard Chukwualuka and Ogene Ferguson},
      title = {Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review},
      journal = {Machine Learning Research},
      volume = {7},
      number = {1},
      pages = {1-7},
      doi = {10.11648/j.mlr.20220701.11},
      url = {https://doi.org/10.11648/j.mlr.20220701.11},
      eprint = {https://download.sciencepg.com/pdf/10.11648.j.mlr.20220701.11},
      abstract = {Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.},
     year = {2022}
    }
    

    Copy | Download

  • TY  - JOUR
    T1  - Automated Machine Learning Models and State-Of-The-Art Effort in Mitigating Combined Algorithm Selection and Hyperparameter Optimization Problems: A Review
    AU  - Nwokonkwo Obi Chukwuemeka
    AU  - John-Otumu Adetokunbo MacGregor
    AU  - Nnadi Leonard Chukwualuka
    AU  - Ogene Ferguson
    Y1  - 2022/01/14
    PY  - 2022
    N1  - https://doi.org/10.11648/j.mlr.20220701.11
    DO  - 10.11648/j.mlr.20220701.11
    T2  - Machine Learning Research
    JF  - Machine Learning Research
    JO  - Machine Learning Research
    SP  - 1
    EP  - 7
    PB  - Science Publishing Group
    SN  - 2637-5680
    UR  - https://doi.org/10.11648/j.mlr.20220701.11
    AB  - Automated machine learning (AutoML) models is one of several machine learning algorithms that can be used to automate the solution of real-world problems. It automates the selection, composition, and parameterization processes of the machine learning models in particular. Machine learning could be more user-friendly when it is automated, and it often produces faster, and more accurate results than hand-coded machine learning methods. For more than ten years, AutoML for supervised learning has been the main focus of research under the discipline of artificial intelligence, and significant progress has been made theeafter; consider the usefulness of AutoML methods in the most popular machine learning toolkits, as well as the AutoML mechanisms in large scale platforms such as Microsoft Azure. This paper provides a methodical analysis of the AutoML workflow as well as the state-of-the-art effort in dealing with the challenges involving Combined Algorithm Selection and Hyperparameter Optimization by gathering information about AutoML from several published articles from different online repositories in order to delve more into the methods used in different domains and the level of accuracy obtained. Findings revealed that the next generation of machine learning and artificial intelligence research is focused on automating the other phases of the whole end-to-end machine learning pipeline, from data comprehension to model deployment. With significantly better deep learning algorithms and big datasets, AutoML is predicted to be able to handle most of the data cleaning process in the future. AutoML will evolve into a highly human-competitive system that will change the way we think about data research.
    VL  - 7
    IS  - 1
    ER  - 

    Copy | Download

Author Information
  • Department of Information Technology, Federal University of Technology, Owerri, Nigeria

  • Department of Computer Science, Morgan State University, Baltimore, USA

  • Department of Information Technology, Federal University of Technology, Owerri, Nigeria

  • Department of Computer Science & Info Tech, Petroleum Training Institute, Effurun, Nigeria

  • Section