AUTOMATIC DETECTION OF SPEECH INTENTIONS USING A LARGE LANGUAGE MODEL
- Authors: Vanin A.V1, Vlasova A.S1,2, Dymova E.N1, Latynov V.V1, Panfilova A.S1, Sereda-Kalinin P.Y1, Tulyankina A.I3
-
Affiliations:
- FSBI "Institute of Psychology of the Russian Academy of Sciences"
- Lomonosov Moscow State University
- FSBI "Institute of Psychology of the Russian Academy of Sciences". Laboratory of Artificial Intelligence Technology in Psychology
- Issue: Vol 46, No 6 (2025)
- Pages: 66–78
- Section: Methodes and procedures
- URL: https://journals.rcsi.science/0205-9592/article/view/361741
- DOI: https://doi.org/10.7868/S0205959225060071
- ID: 361741
Cite item
Open Access
Access granted
Subscription Access
Abstract
The article presents the results of a study of the capabilities of the large language model (GPT-4) to identify the intentional structure of the text. In the course of the research, the following tasks were solved: to develop a methodology for classifying therapist's intentions in psychotherapeutic discourse; to create optimal instructions for working with the model; to analyze the impact of completeness of instructions on the accuracy of identifying intentions by the model; to study the relationship between the accuracy of the classification of intentions by the model with the frequency of occurrence of intentions in the text and the consistency of expert assessments. To analyze the intentional structure of texts, an original Methodology for classifying therapist's intentions in psychotherapeutic discourse has been developed. Using this technique, a team of 3 experts who make decisions about the presence of intent in a remark independently of each other conducted marking up of 14 sessions in Russian (a total of 692 replicas of the therapist). The task of identifying the therapist's intentions was solved by the "GPT-4o-mini" and "GPT-o1" models. The model revealed the intentions of the psychotherapist, realized by him in specific speech utterances (replicas). The conducted research demonstrated the significant capabilities of the large GPT-4 language model in solving the problem of identifying the speaker's speech intentions. The achieved accuracy in classifying the therapist's intentions turned out to be at the level of the best indicators obtained in works on a similar subject. It is shown that improving the instructions significantly increases the quality of the model's operation, and the complexity of the tasks assigned to the model is related to the accuracy of forecasts. Different criteria for the presence of intentions in the therapist's remarks significantly changed the accuracy of the model's predictions.
About the authors
A. V Vanin
FSBI "Institute of Psychology of the Russian Academy of Sciences"
Author for correspondence.
Email: vaninav@ipran.ru
Researcher, Candidate of Psychological Sciences, Master of Information Security Moscow, Russia
A. S Vlasova
FSBI "Institute of Psychology of the Russian Academy of Sciences"; Lomonosov Moscow State University
Email: vlasovaas@ipran.ru
Laboratory assistant. Student Moscow, Russia; Moscow, Russia
E. N Dymova
FSBI "Institute of Psychology of the Russian Academy of Sciences"
Email: dymovaen@ipran.ru
Junior Researcher. Junior Researcher, Laboratory of Developmental Psychology of the subject in normal and post-traumatic conditions Moscow, Russia
V. V Latynov
FSBI "Institute of Psychology of the Russian Academy of Sciences"
Email: latynovv@ipran.ru
Leading researcher. Senior Researcher, Laboratory of Speech Psychology and Psycholinguistics, Candidate of Psychological Sciences Moscow, Russia
A. S Panfilova
FSBI "Institute of Psychology of the Russian Academy of Sciences"
Email: panfilova87@gmail.com
Head of the Laboratory of Artificial Intelligence Technologies in Psychology, Candidate of Technical Sciences Moscow, Russia
P. Y Sereda-Kalinin
FSBI "Institute of Psychology of the Russian Academy of Sciences"
Email: seredapj@ipran.ru
Junior research assistant Moscow, Russia
A. I Tulyankina
FSBI "Institute of Psychology of the Russian Academy of Sciences". Laboratory of Artificial Intelligence Technology in Psychology
Email: anna.tulyankina23@gmail.com
Junior research assistant Moscow, Russia
References
- Pavlova N.D., Afinogenova V.A., Grebenshhikova T.A. Diskurs social'nyh media: intencional'nyj podhod. Psikhologicheskii zhurnal. 2023. T. 44. №. 1. S. 81–90.
- Pavlova N.D., Grebenshhikova T.A. Postsobytijnyj diskurs v internet-soobshhestvah: intencional'naja struktura i priemy vozdejstvija. Jeksperimental'naja psihologija. 2020. T. 13. № 1. S. 138–148.
- Ushakova T.N., Pavlova N.D., Latynov V.V., Cepcov V.A., Alekseev K.I. Slovo v dejstvii: intent-analiz politicheskogo diskursa. Izd-vo Aletejia, 2000.
- Breum S.M., Egdal D.V., Mortensen V.G., Moller A.G., Aiello L.M. The persuasive power of large language models. Proceedings of the International AAAI Conference on Web and Social Media. 2024. V. 18. P. 152–163.
- Chiu Y.Y., Sharma A., Lin I.W., Althoff T. A computational framework for behavioral assessment of LLM therapists. arXiv preprint arXiv:2401.00820. URL: https://arxiv.org/abs/2401.00820 (date of request: 15.02.2025).
- Elyoseph Z., Hadar-Showal D., Asraf K., Lvosky M. ChatGPT outperforms humans in emotional awareness evaluations. Frontiers in Psychology. 2023. V. 14. P. 1199058. DOI: https://doi.org/10.3389/fpsyg.2023.1199058.
- Goldstein J.A., Chao J., Grossman S., Stamos A., Tomz M. How persuasive is AI-generated propaganda? PNAS Nexus. 2024. V. 3. № 2. P. pgae034.
- Han G., Liu W., Huang X., Borsari B. Chain-of-interaction: Enhancing large language models for psychiatric behavior understanding by dyadic contexts. 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI). IEEE, 2024. P. 392–401.
- Hurst A. et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. URL: https://arxiv.org/abs/2410.21276 (date of request: 20.02.2025).
- Karinshak E., Liu S.X., Park J.S., Hancock J.T. Working with AI to persuade: Examining a large language model's ability to generate pro-vaccination messages. Proceedings of the ACM on Human-Computer Interaction. 2023. V. 7. № CSCWI. P. 1–29.
- Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 1977. P. 159–174.
- Lee Y., Goldwasser D., Reese L.S. Towards understanding counseling conversations: Domain knowledge and large language models. arXiv preprint arXiv:2402.14200. URL: https://arxiv.org/abs/2402.14200 (date of request: 12.01.2025).
- Li A., Lu Y., Song N., Zhang S., Ma L., Lan Z. Understanding the therapeutic relationship between counselors and clients in online text-based counseling using LLMs. arXiv preprint arXiv:2402.11958. URL: https://arxiv.org/abs/2402.11958 (date of request: 05.02.2025).
- Matz S.C., Teeny J.D., Vaid S.S., Peters H., Harari G.M., Cerf M. The potential of generative AI for personalized persuasion at scale. Scientific Reports. 2024. V. 14. № 1. P. 4692.
- Shin M., Kim J. Large language models can enhance persuasion through linguistic feature alignment. arXiv preprint arXiv:2311.16466. URL: https://arxiv.org/abs/2311.16466 (date of request: 25.12.2024).
- Wang T., Zhou N., Chen Z. Enhancing computer programming education with LLMs: A study on effective prompt engineering for Python code generation. arXiv preprint arXiv:2407.05437. URL: https://arxiv.org/abs/2407.05437 (date of request: 10.03.2025).
- Wu Y., Hu G. Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings. Proceedings of the Eighth Conference on Machine Translation. 2023. P. 166–169.
- Zhong T. et al. Evaluation of OpenAI O1: Opportunities and challenges of AGI. arXiv preprint arXiv:2409.18486. URL: https://arxiv.org/abs/2409.18486 (date of request: 15.03.2025).
Supplementary files
