Towards a Robust and Scalable Information Retrieval Framework in Big Data Context

Abstract

The proliferation of information in cyberspace is increasing exponentially, leading to challenges for information retrieval systems to satisfy demands for performance and accuracy. How-ever, most existing works concentrate more on designing natural language processing (NLP) models than building such systems, which require massive efforts. In this study, we propose a modular framework for an information retrieval system consisting of several large-scale components capable of processing massive data. In addition, the proposed framework provides a high level of customization by assisting end-users in quickly replacing the NLP models to suit different contexts. This shortens the deployment from research to production of novel NLP models. The evaluation results of our prototype integrated with Vietnamese retrieval models show that the proposed framework is highly robust and scalable in big data contexts.