A Novel Mobile Crawler System Based
on Filtering off Non-Modified Pages for
Reducing Load on the Network
Rajender Nath1 and Satinder Bal2
1Department of Computer Science and Applications, Kurukshetra University, India
2Department of Computer Science and Applications, Vaish College of Engineering, India
1Department of Computer Science and Applications, Kurukshetra University, India
2Department of Computer Science and Applications, Vaish College of Engineering, India
Abstract: The studies in the literature show that about 40% of the current Internet traffic and bandwidth consumption is due to web crawlers that retrieve pages for indexing by the different search engines. This traffic and bandwidth consumption will increase in future due to the exponential growth of the web. This paper addresses the problem of bandwidth consumption by introducing an efficient indexing system based on mobile crawlers. The proposed system employs mobile agents to crawl the pages. These mobile agent based crawlers retrieve the pages, process them, compare their data to filter out pages that are not modified after last crawl, and then compress them before sending them to the search engine for indexing. The experimental results of the proposed system are very encouraging.
Keywords: Search engine, mobile crawler, network resources, internet traffic.
Received March 17, 2009; accepted August 3, 2009