To Search or to Crawl? Towards a Query Optimizer for Text-Centric Tasks

Summary: Cost-based optimizer for text-centric tasks chooses between crawl and query-based plans using a formal model of time and recall. Uses random-graph theory and statistics to estimate task-specific parameters; validated with large-scale experiments on three tasks and multiple real-life data sets. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: heb2b0042c8ee3732
Venue: SIGMOD
Year: 2006
Pagerank: 0.00012633809
Overall Rank: 997 | 93.30%
DOI: 10.1145/1142473.1142504

Incoming Non-self Citations Over Time

Authors

1. Panagiotis G. Ipeirotis (New York University)
2. Eugene Agichtein (Microsoft)
3. Pranay Jain (Columbia University)
4. Luis Gravano (Columbia University)

BibTeX Citation

@inproceedings{ipeirotis_sigmod06,
        title = {{To Search or to Crawl? Towards a Query Optimizer for Text-Centric Tasks}},
        author = {Ipeirotis, Panagiotis G. and Agichtein, Eugene and Jain, Pranay and Gravano, Luis},
        series = {{SIGMOD} '06},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/1142473.1142504},
        url = {https://dl.acm.org/doi/10.1145/1142473.1142504},
        year = {2006}
}

Incoming Citations (Sorted by Pagerank)

Showing 16 of 16 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
328	Declarative Information Extraction Using Datalog with Embedded Extraction Predicates	2007	VLDB	0.00020928928
622	On the Provenance of Non-Answers to Queries over Extracted Data	2008	VLDB	0.00015484312
1,794	Building Structured Web Community Portals: A Top-Down, Compositional, and Incremental Approach	2007	VLDB	9.6220467e-05
2,880	Expressive and Flexible Access to Web-Extracted Data: A Keyword-based Structured Query Language	2010	SIGMOD	7.9144548e-05
2,888	DB&IR: Both Sides Now (Extended Abstract)	2007	SIGMOD	7.9072366e-05
3,915	Efficiently Incorporating User Feedback into Information Extraction and Integration Programs	2009	SIGMOD	6.9290474e-05
4,536	Power-Law Based Estimation of Set Similarity Join Size	2009	VLDB	6.5557137e-05
4,736	Scalable Ad-hoc Entity Extraction from Text Collections	2008	VLDB	6.4451961e-05
5,845	Effective Keyword-based Selection of Relational Databases	2007	SIGMOD	5.9729417e-05
6,104	From Information to Knowledge: Harvesting Entities and Relationships from Web Sources	2010	PODS	5.886265e-05
7,365	I4E: Interactive Investigation of Iterative Information Extraction	2010	SIGMOD	5.5415736e-05
8,244	When Speed Has a Price: Fast Information Extraction Using Approximate Algorithms	2013	VLDB	5.3698153e-05
10,014	Optimizing Complex Extraction Programs over Evolving Text Data	2009	SIGMOD	5.0970738e-05
11,753	Autonomously Computable Information Extraction	2023	VLDB	4.9793485e-05
12,381	Synthesizing Data Programs	2015	CIDR	4.9793485e-05
12,516	D-Hive: Data Bees Pollinating RDF, Text, and Time	2013	CIDR	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 5 of 5 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
161	Robust and Efficient Fuzzy Match for Online Data Cleaning	2003	SIGMOD	0.00027718195
519	Random Sampling for Histogram Construction: How much is enough?	1998	SIGMOD	0.00016942879
1,265	Focused Crawling Using Context Graphs	2000	VLDB	0.00011272973
1,517	Distributed Search over the Hidden Web: Hierarchical Database Sampling and Selection	2002	VLDB	0.00010400233
1,631	Automatic Discovery of Language Models for Text Databases	1999	SIGMOD	0.00010022634

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	217	Efficient IR-Style Keyword Search over Relational Databases	2003	VLDB
2	3,004	Optimized Query Execution in Large Search Engines with Global Page Ordering	2003	VLDB
3	1,517	Distributed Search over the Hidden Web: Hierarchical Database Sampling and Selection	2002	VLDB
4	8,244	When Speed Has a Price: Fast Information Extraction Using Approximate Algorithms	2013	VLDB
5	1,265	Focused Crawling Using Context Graphs	2000	VLDB
6	12,578	Probabilistic Query Rewriting for Efficient and Effective Keyword Search on Graph Data	2013	VLDB
7	9,856	Optimal Algorithms for Crawling a Hidden Database in the Web	2012	VLDB
8	8,284	A Formal Model of Trade-off between Optimization and Execution Costs in Semantic Query Optimization	1988	VLDB
9	4,731	QUEST: Query Optimization in Unstructured Document Analysis	2025	VLDB
10	2,286	Join Queries with External Text Sources: Execution and Optimization Techniques	1995	SIGMOD