Finding replicated web collections

Summary: Identify replicated documents and hyperlinked collections to improve crawlers, archivers, and ranking. Scalable detection over tens of millions of pages and hundreds of GBs; two real-life case studies show gains for a crawler and a search engine on a 25M-page dataset (~150 GB). (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 3260
Venue: SIGMOD
Year: 2000
Pagerank: 7.171112e-05
Overall Rank: 3,725 | 74.45%
DOI: 10.1145/342009.335429

Incoming Non-self Citations Over Time

Authors

1. Junghoo Cho (Stanford University)
2. Narayanan Shivakumar (Stanford University)
3. Hector Garcia-Molina (Stanford University)

BibTeX Citation

@inproceedings{cho_sigmod00,
        title = {{Finding replicated web collections}},
        author = {Cho, Junghoo and Shivakumar, Narayanan and Garcia-Molina, Hector},
        series = {{SIGMOD} '00},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/342009.335429},
        url = {https://dl.acm.org/doi/10.1145/342009.335429},
        year = {2000}
}

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,128	Graph Pattern Matching: From Intractable to Polynomial Time	2010	VLDB	0.0001206219
2,718	Graph Homomorphism Revisited for Graph Matching	2010	VLDB	8.209788e-05
8,549	Resource-Adaptive Real-Time New Event Detection	2007	SIGMOD	5.4119882e-05
9,638	Streaming Similarity Self-Join	2016	VLDB	5.2434488e-05

Outgoing Citations (Sorted by Pagerank)

Showing 2 of 2 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
577	Computing Iceberg Queries Efficiently	1998	VLDB	0.00016235949
1,284	Copy Detection Mechanisms for Digital Documents	1995	SIGMOD	0.00011332822

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,177	Computing PageRank in a Distributed Internet Search System	2004	VLDB
2	9,683	Optimal Algorithms for Crawling a Hidden Database in the Web	2012	VLDB
3	14,009	A Method of Re-ranking Web Search Results Using their Hidden Hyperlink Structure	2002	VLDB
4	12,373	Large-Scale Copy Detection	2011	SIGMOD
5	3,383	Distributed Hypertext Resource Discovery Through Examples	1999	VLDB
6	4,524	Probe, Count, and Classify: Categorizing Hidden-Web Databases	2001	SIGMOD
7	1,483	Distributed Search over the Hidden Web: Hierarchical Database Sampling and Selection	2002	VLDB
8	8,166	Accurate and Efficient Crawling for Relevant Websites	2004	VLDB
9	7,532	The Evolution of the Web and Implications for an Incremental Crawler	2000	VLDB
10	12,862	Self-similarity in the web	2001	VLDB