Evolutionary Configuration Selection of Deep Neural Networks Accounting Text Structures in a Task of Author’s Gender Profiling
https://doi.org/10.1134/S2304487X20050132
Abstract
The influence of structural features of the text on the author gender classification has been studied. Considered structural features include a linear structure, a syntactic structure, and a structure calculated in the hidden layers of the language model. Existing syntax-accounting methods that have high computational complexity and working time have been develop by analyzing syntactic paths for each word of each sentence, or sequential analysis of sentences structures. The proposed development is based on the use of attention graph layers (GAT) within the neural network architecture, whose input is the matrix of syntax connectivity of all words of the text. An artificially created vector is added to the input feature matrix of each text, which accumulates the activities of all words in the text and is used to characterize the text and classify it. For the proposed network architecture, the method is implemented for evolutionary selection of hyperparameters based on the tree parzen estimator. The results obtained show that the syntax structure of the text for the considered task of author’s gender identification on the open corpora RusPersonality and Gender Imitation Crowdsource “a” increases the accuracy by 2 and 5 %, respectively, according to the f1-score metric with weighted averaging over classes.
About the Authors
A. G. Sboev
National Research Center Kurchatov Institute; National Research Nuclear University MEPhI (Moscow Engineering Physics Institute)
Russian Federation
Russian Federation
123182
115409
Moscow
A. A. Selivanov
National Research Center Kurchatov Institute
Russian Federation
Russian Federation
123182
Moscow
I. A. Moloshnikov
National Research Center Kurchatov Institute
Russian Federation
Russian Federation
123182
Moscow
R. B. Rybka
National Research Center Kurchatov Institute
Russian Federation
Russian Federation
123182
Moscow
References
1. Sboev A., Moloshnikov I., Gudovskikh D., Selivanov A., Rybka R., Litvinova T. Deep learning neural nets versus traditional machine learning in gender identification of authors of RusProfiling texts // Procedia Comput. Sci. 2018, V. 123. P. 424–431.
2. Sboev A., Moloshnikov I., Gudovskikh D., Selivanov A., Rybka R., Litvinova T. Automatic gender identification of author of Russian text by machine learning and neural net algorithms in case of gender deception // Procedia Computer Science. 2018. V. 123. P. 417–423, ISSN 1877-0509. doi: 10.1016/j.procs.2018.01.064
3. Sboev A., Moloshnikov I., Gudovskikh D., Rybka R. A comparison of Data Driven models of solving the task of gender identification of author in Russian language texts for cases without and with the gender deception // IOP Publishing, Journal of Physics: Conf. Ser. 2017. V. 937. P. 1742–6588.
4. Bergstra J. S., Yamins D., Cox D. Algorithms for hyperparameter optimization // Advances in Neural Information Processing Systems. 2011.
5. Bergstra J. S., Yamins D., Cox D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures / Proceedings of the 30th International Conference on Machine Learning. 2013.
6. Tai K. S., Socher R., Manning C. D. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075, 2015.
7. Miyazaki R., Komachi M. Japanese sentiment classification using a tree-structured long short-term memory with attention. arXiv preprint arXiv:1704.00924. 2017.
8. Sboev A., Selivanov A., Rybka R., Moloshnikov I., Bogachev D. A Neural Network Model to Include Textual Dependency Tree Structure in Gender Classification of Russian Text Author. In: Advanced Technologies in Robotics and Intelligent Systems. Mechanisms and Machine Science. Springer, Cham, 2020. V. 80. doi: 10.1007/978-3-030-33491-8_48
9. Sboev A., Bogachev D., Selivanov A., Moloshnikov I., Rybka R. Graph Convolution Network with Attention to Include Syntax Trees into Text Author’s Gender Identification Task (in press).
10. Veličković P., Cucurull G., Casanova A., Romero A., Lio P., Bengio Y. Graph Attention Networks. 2018. Published as a conference paper at ICLR 2018.
11. Haifeng Jin, Qingquan Song, Xia Hu. Auto-keras: An efficient neural architecture search system / Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.
12. Weill C., Gonzalvo J., Kuznetsov V., Yang S., Yak S., Mazzawi H., Hotaj E., Jerfel G., Macko V., Adlam B., Mohri M. AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles. arXiv preprint arXiv:1905.00080. 2019.
13. Miikkulainen R., Liang J., Meyerson E., Rawal A., Fink D., Francon O., Raju B., Shahrzad H., Navruzyan A., Duffy N., Hodjat B. Evolving deep neural networks. In-Artificial Intelligence in the Age of Neural Networks and Brain Computing. Academic Press, 2019. P. 293–312.
14. James B., Yamins D., Cox D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures / Proceedings of the 30th International Conference on Machine Learning. 2013.
15. Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Kaiser Ł., Polosukhin I. Attention is all you need / In: Advances in Neural Information Processing systems. 2017. P. 5998–6008.
16. Devlin J., Chang M. W., Lee K., Toutanova K. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805. 2018.
17. Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting // Journal of Machine Learning Research. 2014. V. 15. № 1. P. 1929–1958.
18. Mosella-Montoro A., Ruiz-Hidalgo J. Residual Attention Graph Convolutional Network for Geometric 3D Scene Classification / In: Proceedings of the IEEE International Conference on Computer Vision Work-shops, 2019.
19. Straka M., Straková J. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe / Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, 2017. P. 88–99.
20. Vig J. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv: 1906.05714. 2019. no. 12.
Review
For citations:
Sboev A.G., Selivanov A.A., Moloshnikov I.A., Rybka R.B. Evolutionary Configuration Selection of Deep Neural Networks Accounting Text Structures in a Task of Author’s Gender Profiling. Vestnik natsional'nogo issledovatel'skogo yadernogo universiteta "MIFI". 2020;9(6):554-560. (In Russ.) https://doi.org/10.1134/S2304487X20050132
Views:
100
Email this article 