Email this article Decomposition of construction method for a language encoder
- Authors: Trofimov I.V.1
-
Affiliations:
- Ailamazyan Program Systems Institute of RAS
- Issue: Vol 14, No 1 (2023)
- Pages: 31-54
- Section: Articles
- URL: https://journals.rcsi.science/2079-3316/article/view/259973
- DOI: https://doi.org/10.25209/2079-3316-2023-14-1-31-54
- ID: 259973
Cite item
Full Text
Abstract
An encoder as part of a language model is a mechanism for converting text information into an effective numerical representation which is suitable for solving a wide range of text processing tasks by means of neural network methods. This paper suggests a way of decomposing of the learning process for a language encoder. The author considers the issues of expediency of such decomposition taking into account reduction of computational costs, quality control at intermediate training stages, provision of the interpretability of the results on each stage. The quality evaluation of the encoder is given.
About the authors
Igor Vladimirovich Trofimov
Ailamazyan Program Systems Institute of RAS
Author for correspondence.
Email: itrofimov@gmail.com
ORCID iD: 0000-0002-6903-4730
References
- Devlin J., Chang M-W., Lee K., Toutanova K.. BERT: Pre-training of Deep Bidirectional Transformers for language understanding, 2018, 16 pp.
- Raffel C., Shazeer N., Roberts A., Lee K., Narang Sh., Matena M., Zhou Ya., Li W., Liu P. J.. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020, 67 pp.
- Brown T. B., Mann B., Ryder N., Subbiah M., Kaplan J., Dhariwal P., Neelakantan A., Shyam P., Sastry G., Askell A., Agarwal S., Herbert-Voss A., Krueger G., Henighan T., Child R., Ramesh A., Ziegler D. M., Wu J., Winter C., Hesse Ch., Chen M., Sigler E., Litwin M., Gray S., Chess B., Clark J., Berner Ch., McCandlish S., Radford A., Sutskever I., Amodei D.. Language models are few-shot learners, 2020, 75 pp.
- Fedus W., Zoph B., Shazeer N.. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity, 2021, 40 pp.
- Kolesnikova A., Kuratov Y., Konovalov V., Burtsev M.. “Knowledge distillation of Russian language models with reduction of vocabulary”, Proceedings of the International Conference «Dialogue 2022» (Moscow, June 15–18, 2022), Computational Linguistics and Intellectual Technologies, vol. 21, 2022, ISBN 978-5-7281-3205-9, pp. 295–310.
- Zafrir O., Boudoukh G., Izsak P., Wasserblat M.. “Q8bert: Quantized 8bit bert”, 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing – NeurIPS Edition (EMC2-NIPS) (13 December 2019, Vancouver, BC, Canada), 2019, pp. 36–39.
- Clark K., Luong M. -T., Le Q. V., Manning Ch. D.. Electra: Pre-training text encoders as discriminators rather than generators, 2020, 18 pp.
- Kudo T., Richardson J.. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, 2018, 6 pp.
- Sennrich R., Haddow B., Birch A.. Neural machine translation of rare words with subword units, 2016, 11 pp.
- Schuster M., Nakajima K.. “Japanese and Korean voice search”, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (25–30 March 2012, Kyoto, Japan), 2012, pp. 5149–5152.
- Mikolov T., Chen K., Corrado G., Dean J.. Efficient estimation of word representations in vector space, 2013, 12 pp.
- Trofimov I. V., Suleymanova E. A.. “A syntax-based distributional model for discriminating between semantic similarity and association”, Proceedings of the International Conference «Dialogue 2017». 1, Computational Linguistics and Intellectual Technologies, vol. 16, 2017, pp. 349–359.
- Трофимов И. В., Сулейманова Е. А.. «Дистрибутивно-семантическая модель для выявления категориального сходства», Программные системы: теория и приложения, 9:4(39) (2018), с. 443–460.
- Власова Н. А., Трофимов И. В., Сердюк Ю. П., Сулейманова Е. А., Воздвиженский И. Н.. «PaRuS — синтаксически аннотированный корпус русского языка», Программные системы: теория и приложения, 10:4(43) (2019), с. 181–199.
- Panchenko A., Lukashevich N. V., Ustalov D., Paperno D., Meyer K. M., Konstantinova N.. “RUSSE: The first workshop on Russian semantic similarity”, Proceedings of the International Conference «Dialogue 2015». 2, Computational Linguistics and Intellectual Technologies, vol. 14, RGGU, 2015, pp. 89–105.
- Трофимов И. В., Сулейманова Е. А., Власова Н. А., Подобряев А. В.. «Разрешение событийно-несобытийной неоднозначности существительных», Программные системы: теория и приложения, 9:4(39) (2018), с. 3–33.
- Трофимов И. В., Сердюк Ю. П., Сулейманова Е. А., Власова Н. А.. «Разрешение событийно-несобытийной неоднозначности существительных: нейросетевой подход», Программные системы: теория и приложения, 11:4(47) (2020), с. 31–53.
- Kuratov Yu., Arkhipov M.. Adaptation of deep bidirectional multilingual transformers for Russian language, 2019, 8 pp.
- Сорокин А. А., Макогонов С. В., Королев С. П.. «Информационная инфраструктура для коллективной работы ученых Дальнего Востока России», Научно-техническая информация. Серия 1: Организация и методика информационной работы, 2017, №12, с. 14—16.
Supplementary files
