The Evolution of the Web and Implications for an Incremental Crawler
Summary: Incremental, selective updating of crawls to keep index and local collections fresh, not batch refresh. Empirical study over 0.5M pages in 4 months characterizes page evolution, compares crawl strategies, and proposes a hybrid architecture combining best choices for timelier updates. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
Incoming Citations (Sorted by Pagerank)
Showing 4 of 4 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 233 | Crawling the Hidden Web | 2001 | VLDB | 0.00031996412 |
| 8,314 | Effective Change Detection Using Sampling | 2002 | VLDB | 4.5392079e-05 |
| 12,341 | NEAR-Miner: Mining Evolution Associations of Web Site Directories for Efficient Maintenance of Web Archives | 2009 | VLDB | 4.1905499e-05 |
| 13,540 | Dealing with Web Data: History and Look ahead | 2010 | VLDB | - |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 1 of 1 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 1,304 | Synchronizing a database to Improve Freshness | 2000 | SIGMOD | 0.00012684735 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| Overall Rank | Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 5,666 | Personalized PageRank on Evolving Graphs with an Incremental Index-Update Scheme | 2023 | SIGMOD | 5.3824583e-05 |
| 233 | Crawling the Hidden Web | 2001 | VLDB | 0.00031996412 |
| 12,239 | Optimizing Content Freshness of Relations Extracted From the Web Using Keyword Search | 2010 | SIGMOD | 4.1905499e-05 |
| 9,549 | Optimal Algorithms for Crawling a Hidden Database in the Web | 2012 | VLDB | 4.3216687e-05 |
| 8,674 | Progressive Deep Web Crawling Through Keyword Queries For Data Enrichment | 2019 | SIGMOD | 4.4659264e-05 |
| 409 | Focused Crawling Using Context Graphs | 2000 | VLDB | 0.00023928618 |
| 3,685 | Finding replicated web collections | 2000 | SIGMOD | 6.8392492e-05 |
| 5,436 | RankMass Crawler: A Crawler with High Personalized PageRank Coverage Guarantee | 2007 | VLDB | 5.5048419e-05 |
| 7,767 | Accurate and Efficient Crawling for Relevant Websites | 2004 | VLDB | 4.6525495e-05 |
| 13,540 | Dealing with Web Data: History and Look ahead | 2010 | VLDB | - |