The TF-IDF measure and analysis of links between words within N-grams in the formation of knowledge units for open tests

Аннотация

A method is proposed for searching in a text corpus for phrases that are the most similar to an original one in a described knowledge fragment (including linguistic forms of expression) based on a numerical evaluation of the coupling strength between the related words from the original phrase that occur in them. In this regard, the links themselves expand from traditional bigrams to three or more elements and are distinguished according to the results of dividing the words of the original phrase into classes according to the value of the TF-IDF measure as an alternative to syntactic dependences.