Extraction of Data from Mass Media Web Sites
- Authors: Varlamov M.I.1, Turdakov D.Y.1,2,3, Yatskov A.K.1,2
-
Affiliations:
- Ivannikov Institute for System Programming, Russian Academy of Sciences
- Moscow State University
- National Research University—Higher School of Economics
- Issue: Vol 44, No 5 (2018)
- Pages: 344-352
- Section: Article
- URL: https://journals.rcsi.science/0361-7688/article/view/176663
- DOI: https://doi.org/10.1134/S0361768818050092
- ID: 176663
Cite item
Abstract
To understand the current state and dynamics of the development of the Internet information space, fast tools for extracting data for mass media sites that have a large degree of coverage are needed. However, by no means all sites provide data syndication in the RSS format, and the development of specialized tools for extracting data from each Web site is a costly procedure. In this paper, methods for automatic extraction of news texts from arbitrary mass media sites are proposed. Due to classification of Web page types and the subsequent grouping of their URLs, the quality of extracting news texts is improved. A strategy for traversing a site and detecting the pages containing hyperlinks to news pages is proposed. This strategy decreases the number of requests and reduces the site load.
About the authors
M. I. Varlamov
Ivannikov Institute for System Programming, Russian Academy of Sciences
Author for correspondence.
Email: varlamov@ispras.ru
Russian Federation, Moscow, 109004
D. Yu. Turdakov
Ivannikov Institute for System Programming, Russian Academy of Sciences; Moscow State University; National Research University—Higher School of Economics
Author for correspondence.
Email: turdakov@ispras.ru
Russian Federation, Moscow, 109004; Moscow, 119991; Moscow, 109028
A. K. Yatskov
Ivannikov Institute for System Programming, Russian Academy of Sciences; Moscow State University
Author for correspondence.
Email: yatskov@ispras.ru
Russian Federation, Moscow, 109004; Moscow, 119991