A Study on Hyperparameter Tuning of Twin Support Vector Regression Based on Bayesian Optimization
DOI:
https://doi.org/10.54097/8p56d128Keywords:
Twin Support Vector Regression, Bayesian Optimization, Hyperparameter Tuning, Regression.Abstract
This study addresses the challenge of optimizing the predictive performance of Twin Support Vector Regression (TSVR) models on complex datasets by employing Bayesian Optimization (BO) for automatic hyperparameter tuning. As an enhanced variant of Support Vector Regression (SVR), TSVR achieves significantly greater computational efficiency by solving two smaller-scale quadratic programming problems. Within the Bayesian optimization framework, the construction of a posterior probability model enables efficient exploration of the hyperparameter space, thereby overcoming the inefficiency limitations of traditional methods. Experimental validation demonstrates that the BO-optimized TSVR model achieves significant reductions in Mean Squared Error (MSE) across multiple synthetic datasets, while simultaneously exhibiting superior predictive accuracy and generalization capability. The significance of this research lies not only in its improvement of TSVR model performance but also in its provision of a novel approach for hyperparameter optimization in machine learning, possessing substantial theoretical and practical merit.
Downloads
References
[1] Vapnik V N. Statistical learning theory Wiley, 1998.
[2] Deng N Y, Tian Y J, Zhang C H. Support vector machines: theory, algorithms, and extensions CRC Press, 2012. DOI: https://doi.org/10.1201/b14297
[3] Suykens J A K, Lukas L, van Dooren P, De Moor B, Vandewalle J. Least squares support vector machine classifiers: a large-scale algorithm Proceedings of European conference of circuit theory design, 1999: 839–844.
[4] Mangasarian O L, Wild E W. Multisurface proximal support vector classification via generalized eigenvalues IEEE Trans Pattern Anal Mach Intell, 2006, 28(1): 69–74. DOI: https://doi.org/10.1109/TPAMI.2006.17
[5] Peng X. Primal twin support vector regression and its sparse approximation Neurocomputing, 2010, 73(16–18): 2846–2858. DOI: https://doi.org/10.1016/j.neucom.2010.08.013
[6] Shao Y H, Deng N Y. A novel margin-based twin support vector machine with unity norm hyperplanes Neural Comput Appl, 2012. DOI: https://doi.org/10.1007/s00521-012-0894-5
[7] Peng X. TPMSVM: a novel twin parametric-margin support vector machine for pattern recognition Pattern Recognit, 2011, 44(10–11): 2678–2692. DOI: https://doi.org/10.1016/j.patcog.2011.03.031
[8] Chao Yuan, Liming Yang. Capped L2, p-norm metric based robust least squares twin support vector machine for pattern classification. Neural Networks, 2021, 142: 457-478. DOI: https://doi.org/10.1016/j.neunet.2021.06.028
[9] Jun Ma, Liming Yang, Qun Sun. Capped L1-norm distance metric-based fast robust twin bounded support vector machine. Neurocomputing, 2020, 412: 295-311. DOI: https://doi.org/10.1016/j.neucom.2020.06.053
[10] Lan Bai, Xu Chen, Zhen Wang, Yuan-Hai Shao. Safe intuitionistic fuzzy twin support vector machine for semi-supervised learning. Applied Soft Computing, 2022, 123: 108906. DOI: https://doi.org/10.1016/j.asoc.2022.108906
[11] Ran An, Yitian Xu, Xuhua Liu. Multi-task twin bounded support vector machine and its safe screening rule. Applied Soft Computing, 2023, 138: 110188. DOI: https://doi.org/10.1016/j.asoc.2023.110188
[12] Jimenez-C et al. Krein twin support vector machines for imbalanced data classification
[13] Pattern Recognition Letters, 2024, 182: 39-45. DOI: https://doi.org/10.1016/j.patrec.2024.03.017
[14] Si, Q., Yang, Z., Ye, J. Symmetric LINEX loss twin support vector machine for robust classification and its fast iterative algorithm Neural Networks, 2023, 168: 143-160. DOI: https://doi.org/10.1016/j.neunet.2023.08.055
[15] M.A. Ganaie, Vrushank Ahire, Anouck Girard. Granular Ball K-Class Twin Support Vector Classifier. Pattern Recognition, 2025, 166: 111636. DOI: https://doi.org/10.1016/j.patcog.2025.111636
[16] Suykens J A K, Vandewalle J. Least squares support vector machine classifiers Neural Process Lett, 1999, 9(3): 293–300. DOI: https://doi.org/10.1023/A:1018628609742
[17] Peng X. TSVR: an efficient twin support vector machine for regression Neural Netw, 2010, 23(3): 365–372. DOI: https://doi.org/10.1016/j.neunet.2009.07.002
[18] Anuradha Kumari, M. Tanveer. LSTSVR+: Least square twin support vector regression with privileged information. Engineering Applications of Artificial Intelligence, 2024, 136: 108964. DOI: https://doi.org/10.1016/j.engappai.2024.108964
[19] Liu C. Twin proximal support vector regression with heteroscedastic Gaussian noise. Expert Syst Appl, 2024, 250: 123840. DOI: https://doi.org/10.1016/j.eswa.2024.123840
[20] Yafen Ye et al. Twin support vector quantile regression Expert Systems with Applications, 2024.
[21] Salim Rezvani, Xizhao Wang. Intuitionistic fuzzy twin support vector machines for imbalanced data. Neurocomputing, 2022, 507: 16-25. DOI: https://doi.org/10.1016/j.neucom.2022.07.083
[22] Jiamin Xu, Huamin Wang, Libo Zhang, Shiping Wen. Robust twin depth support vector machine based on average depth. Knowledge-Based Systems, 2023, 274: 110627. DOI: https://doi.org/10.1016/j.knosys.2023.110627
[23] Chengjiang Zhou, Hao Li, Jintao Yang, Qihua Yang, Limiao Yang, Shanyou He, Xuyi Yuan. Fuzzy regular least squares twin support vector machine and its application in fault diagnosis. Expert Systems with Applications, 2023, 231: 120804. DOI: https://doi.org/10.1016/j.eswa.2023.120804
[24] Qianfei Liu et al. Multi-view structural twin support vector machine with the consensus and complementarity principles and its safe screening rules Expert Systems with Applications, 2025.
[25] Zhiyong Che et al. Twin support vector machines with privileged information Information Sciences, 2021.
[26] Ye Y. L1-norm twin support vector quantile regression. Appl Soft Comput, 2025, 169: 112580. DOI: https://doi.org/10.1016/j.asoc.2024.112580
[27] Richhariya B. A reduced universum twin support vector machine for class imbalance learning. Pattern Recognit, 2020,102: 107150. DOI: https://doi.org/10.1016/j.patcog.2019.107150
[28] Liu, B., Huang, R., Xiao, Y., Liu, J., Wang, K., Li, L., Chen, Q. Adaptive robust Adaboost-based twin support vector machine with universum data Information Sciences, 2022, 609: 1334-1352. DOI: https://doi.org/10.1016/j.ins.2022.07.155
[29] Xie, X., Li, Y., Sun, S. Deep multi-view multiclass twin support vector machines Information Fusion, 2023, 91: 80-92. DOI: https://doi.org/10.1016/j.inffus.2022.10.005
[30] Lu, J., Xie, X., Xiong, Y. Multi-view hypergraph regularized Lp norm least squares twin support vector machines for semi-supervised learning. Pattern Recognition, 2024, 156: 110753. DOI: https://doi.org/10.1016/j.patcog.2024.110753
[31] Huang, H., Wei, X., Zhou, Y. An overview on twin support vector regression Neurocomputing, 2022, 490: 80-92. DOI: https://doi.org/10.1016/j.neucom.2021.10.125
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Highlights in Science, Engineering and Technology
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
