Conceptual Persian Text Summarizer: A New Model in Continuous Vector Space

Mohammad Ebrahim Khademi, Mohammad Fakhredanesh, and Seyed Mojtaba Hoseini

Faculty of Electrical and Computer Engineering, Malek Ashtar University of Technology, Iran

Abstract: Traditional methods of summarization are not cost-effective and possible today. Extractive summarization is a process that helps to extract the most important sentences from a text automatically, and generates a short informative summary. In this work, we propose a novel unsupervised method to summarize Persian texts. The proposed method adopt a hybrid approach that clusters the concepts of the text using deep learning and traditional statistical methods. First we produce a word embedding based on Hamshahri2 corpus and a dictionary of word frequencies. Then the proposed algorithm extracts the keywords of the document, clusters its concepts, and finally ranks the sentences to produce the summary. We evaluated the proposed method on Pasokh single-document corpus using the ROUGE evaluation measure. Without using any hand-crafted features, our proposed method achieves better results than the state-of-the-art related work results. We compared our unsupervised method with the best supervised Persian methods and we achieved an overall improvement of ROUGE-2 recall score of 7.5%.

Keywords: Extractive Text Summarization, Unsupervised Learning, Language Independent Summarization, Continuous Vector Space, Word Embedding, Natural Language Processing.

Received September 17, 2018; accepted Febreuary 5 , 2020