Avaliação da blocagem de grandes bases de dados na plataforma de tempo real Apache Storm

Abstract

Real-time data processing allows large amounts and volumes of data to be processed continuously. In this large volume of data, there may be data representing the same information (data duplication). This problem is an anomaly that wastes resources on processing time, disk space and auxiliary memory.To treat this anomaly, the data deduplication technique is used. Duplicate data are identified and deleted in this technique. To be applied in large databases, during the process of identifying duplicates, the technique of data grouping (or blocking of data) is used. In this technique, the data that have some similarity are grouped according to a defined characteristic. Thus, when grouping the data, in the process of identifying duplicates, only comparisons will be made between the elements that make up each block. In this context, the RIJOIN tool [7] was developed. This tool, developed on the Apache Storm platform, combines data deduplication with data collation, inverted index, filters and real-time data processing. Thus, in this work, improvements will be proposed in this tool, these being mainly directed to the process of creation and use of inverted index, in the stage of creation of records blocks. Through the tests,it was observed that the improvements proposed in this work achieved promising results, among them, the reduction in the number of records reordering.