| Peer-Reviewed

Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention

Received: 22 March 2023     Accepted: 18 April 2023     Published: 24 April 2023
Views:       Downloads:
Abstract

To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.

Published in American Journal of Computer Science and Technology (Volume 6, Issue 2)
DOI 10.11648/j.ajcst.20230602.11
Page(s) 50-56
Creative Commons

This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.

Copyright

Copyright © The Author(s), 2023. Published by Science Publishing Group

Keywords

Sentiment Analysis, BERT Pre-training Model, BiGRU, Gated Attention

References
[1] Zhong Jiawa, Liu Wei, Wang Sili, Yang Heng. A review of text sentiment analysis methods and applications [J]. Data Analysis and Knowledge Discovery, 2021, 5 (06): 1-13.
[2] Hong Wei, Li Min. A review of research on text sentiment analysis methods [J]. Computer Engineering and Science, 2019, 41 (04): 750-757.
[3] Sabour S, Frosst N, Hinton G E. Dynamic routing between capsules [J]. Advances in neural information processing systems, 2017, 30.
[4] Liu, L. W., Yu, S.. Recurrent neural network (RNN) and application research [J]. Science and Technology Perspectives, 2019 (32): 54-55. DOI: 10.19694/j.cnki.issn2095-2457.2019.32.022.2.
[5] S. Xiong, H. Lv, W. Zhao, and D. Ji, "Towards Twitter sentiment classification by multi-level sentiment-enriched word embeddings," Neurocomputing, vol. 275, pp. 2459-2466, 2018, doi: 10.1016/j.neucom.2017.11.023.
[6] Liu H L, He Y F. Seasonal attention in LSTM and its application to text sentiment classification [J/OL]. Systems Science and Mathematics: 1-19 [2023-03-14].
[7] Wang W, Sun YX, Qi QJ, Meng XF. A text sentiment classification model based on BiGRU-attention neural network [J]. Computer Application Research, 2019, 36 (12): 3558-3564. DOI: 10.19734/j.issn.1001-3695.2018.07.0413.
[8] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [J]. arXiv preprint arXiv: 1810.04805, 2018.
[9] Minqing Hu, Bing Liu. Mining and summarizing customer reviews [P]. Knowledge discovery and data mining, 2004.
[10] Tomas Mikolov, Ilya Sutskever, Kai Chen 0010, Greg Corrado, Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. [J]. CoRR, 2013, abs/1310.4546.
[11] Xu Yizhou, Lin Xiao, Lu Li-Ming. A long text sentiment classification model based on hierarchical CNN [J]. Computer Engineering and Design, 2022, 43 (04): 1121-1126. DOI: 10.16208/j.issn1000-7024.2022.04.030.
[12] Lifu Wang, Bo Shen, Bo Hu, Xing Cao. Can Gradient Descent Provably Learn Linear Dynamic Systems? [J]. arXiv: 2211. 10582, 2022.
[13] Cheng Y, Sun H, Chen H, et al. Sentiment analysis using multi-head attention capsules with multi-channel CNN and bidirectional GRU [J]. IEEE Access, 2021, 9: 60383-60395.
[14] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. [J]. CoRR, 2015, abs/1502. 03044.
[15] Lima Luiz Renato, Godeiro Lucas Lúcio. Equity-premium prediction: Attention is all you need [J]. Journal of Applied Econometrics, 2022, 38 (1).
[16] X. Sun and C. Zhang, "Detecting anomalous emotion through big data from social networks based on a deep learning method," Multimedia Tools and Applications, vol. 79, no. 13-14, pp. 9687-9687, 2020, doi: 10.1007/s11042-018-5665-6.
[17] Peters M, Neumann M, Iyyer M, et al. Deep Contextualized Word Representations [J]. 2018.
[18] Phan, Huyen Trang, Ngoc Thanh Nguyen, et al. AspectLevel Sentiment Analysis Using CNN Over BERTGCN [J]. IEEE Access, 2022, 10: 110402-110409.
[19] Shao Nan. Research on discrete recommendation algorithm based on Gumbel-Softmax distribution [D]. University of Electronic Science and Technology, 2020. DOI: 10.27005/d.cnki.gdzku.2020.004485.
[20] Yang Xiuzhang, Wu Shuai, Ren Tianshu, Liu Jianyi, Song Jiwen, Liao Wenjing. Research on sentiment analysis of e-commerce reviews by integrating multi-headed attention mechanism and BiLSTM [J]. Information Technology and Informatization, 2022 (10): 5-9.
[21] Cui Jia-Bin. Research on text sentiment analysis based on BERT-BiGRU model [D]. Shanxi University, 2021. DOI: 10.27284/d.cnki.gsxiu.2021.000092.
Cite This Article
  • APA Style

    Huang Shufen, Liu Changhui, Zhang Yinglin. (2023). Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. American Journal of Computer Science and Technology, 6(2), 50-56. https://doi.org/10.11648/j.ajcst.20230602.11

    Copy | Download

    ACS Style

    Huang Shufen; Liu Changhui; Zhang Yinglin. Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. Am. J. Comput. Sci. Technol. 2023, 6(2), 50-56. doi: 10.11648/j.ajcst.20230602.11

    Copy | Download

    AMA Style

    Huang Shufen, Liu Changhui, Zhang Yinglin. Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention. Am J Comput Sci Technol. 2023;6(2):50-56. doi: 10.11648/j.ajcst.20230602.11

    Copy | Download

  • @article{10.11648/j.ajcst.20230602.11,
      author = {Huang Shufen and Liu Changhui and Zhang Yinglin},
      title = {Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention},
      journal = {American Journal of Computer Science and Technology},
      volume = {6},
      number = {2},
      pages = {50-56},
      doi = {10.11648/j.ajcst.20230602.11},
      url = {https://doi.org/10.11648/j.ajcst.20230602.11},
      eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajcst.20230602.11},
      abstract = {To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.},
     year = {2023}
    }
    

    Copy | Download

  • TY  - JOUR
    T1  - Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention
    AU  - Huang Shufen
    AU  - Liu Changhui
    AU  - Zhang Yinglin
    Y1  - 2023/04/24
    PY  - 2023
    N1  - https://doi.org/10.11648/j.ajcst.20230602.11
    DO  - 10.11648/j.ajcst.20230602.11
    T2  - American Journal of Computer Science and Technology
    JF  - American Journal of Computer Science and Technology
    JO  - American Journal of Computer Science and Technology
    SP  - 50
    EP  - 56
    PB  - Science Publishing Group
    SN  - 2640-012X
    UR  - https://doi.org/10.11648/j.ajcst.20230602.11
    AB  - To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.
    VL  - 6
    IS  - 2
    ER  - 

    Copy | Download

Author Information
  • College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China

  • College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China

  • College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China

  • Sections