A Novel Mobile Crawler System Based on Filtering off Non-Modified Pages for Reducing Load on the

A Novel Mobile Crawler System Based ‎
on Filtering off Non-Modified Pages for ‎
Reducing Load on the Network


Rajender Nath1 and Satinder Bal2‎
‎1Department of Computer Science and Applications, Kurukshetra University, India ‎
‎ 2Department of Computer Science and Applications, Vaish College of Engineering, India ‎



Abstract: The studies in the literature show that about 40% of the current Internet traffic and bandwidth consumption is due to web ‎crawlers that retrieve pages for indexing by the different search engines. This traffic and bandwidth consumption will increase ‎in future due to the exponential growth of the web. This paper addresses the problem of bandwidth consumption by introducing ‎an efficient indexing system based on mobile crawlers. The proposed system employs mobile agents to crawl the pages. These ‎mobile agent based crawlers retrieve the pages, process them, compare their data to filter out pages that are not modified after ‎last crawl, and then compress them before sending them to the search engine for indexing. The experimental results of the ‎proposed system are very encouraging.     ‎

Keywords: Search engine, mobile crawler, network resources, internet traffic.‎

Received March 17, 2009; accepted August 3, 2009‎

  

Full Text

Read 3436 times Last modified on Thursday, 23 June 2011 04:53
Share
Top
We use cookies to improve our website. By continuing to use this website, you are giving consent to cookies being used. More details…