Generation of a Social Network Graph by Using Apache Spark

Аннотация

It is planned to create a method of clustering a social network graph. To test the method, it is necessary to generate a graph similar in structure to existing social networks. The article presents an algorithm for the graph-distributed generation. We take into account basic properties such as the power-law distribution of the number of user communities, the dense intersections of social networks, and others. This algorithm also considers the problems that are present in similar works of other authors, for example, the multiple edges problem in the generation process. A special feature of the created algorithm is the implementation depending on the number of communities, rather than on the number of connected users, as is done in other works. This is connected with a peculiarity of the development of the existing social network structure. The properties of its graph are described in the paper. We describe a Table 1 containing the variables needed for the algorithm. A step-by-step generation algorithm is compiled. Appropriate mathematical parameters are calculated for it. The generation is performed in a distributed way by the Apache Spark framework. It is described in detail how the division of tasks with the help of this framework operates. The Erdos–Renyi model for random graphs is used in the algorithm. It is the most suitable and easiest one to implement. The main advantages of the created method are the small amount of resources and faster execution speed in comparison with other similar generators. Speed is achieved through distributed work and the fact that at any time, the network users have their own unique numbers and are ordered by these numbers so that there is no need to sort them out. The designed algorithm will not only promote the creation of an efficient clustering method, but can also be useful in other development areas connected, for example, with social network search engines.