To Search or to Crawl? Towards a Query Optimizer for Text-Centric Tasks

Summary: Cost-based optimizer for text-centric tasks chooses between crawl and query-based plans using a formal model of time and recall. Uses random-graph theory and statistics to estimate task-specific parameters; validated with large-scale experiments on three tasks and multiple real-life data sets. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 3819
Venue: SIGMOD
Year: 2006
Pagerank: 0.00012870746
Overall Rank: 974 | 93.32%
DOI: 10.1145/1142473.1142504

Incoming Non-self Citations Over Time

Authors

1. Panagiotis G. Ipeirotis (New York University)
2. Eugene Agichtein (Microsoft)
3. Pranay Jain (Columbia University)
4. Luis Gravano (Columbia University)

BibTeX Citation

@inproceedings{ipeirotis_sigmod06,
        title = {{To Search or to Crawl? Towards a Query Optimizer for Text-Centric Tasks}},
        author = {Ipeirotis, Panagiotis G. and Agichtein, Eugene and Jain, Pranay and Gravano, Luis},
        series = {{SIGMOD} '06},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/1142473.1142504},
        url = {https://dl.acm.org/doi/10.1145/1142473.1142504},
        year = {2006}
}

Incoming Citations (Sorted by Pagerank)

Showing 16 of 16 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
319	Declarative Information Extraction Using Datalog with Embedded Extraction Predicates	2007	VLDB	0.00021377065
628	On the Provenance of Non-Answers to Queries over Extracted Data	2008	VLDB	0.00015630285
1,848	Building Structured Web Community Portals: A Top-Down, Compositional, and Incremental Approach	2007	VLDB	9.6208418e-05
2,832	Expressive and Flexible Access to Web-Extracted Data: A Keyword-based Structured Query Language	2010	SIGMOD	8.0783003e-05
2,877	DB&IR: Both Sides Now (Extended Abstract)	2007	SIGMOD	8.0070039e-05
3,843	Efficiently Incorporating User Feedback into Information Extraction and Integration Programs	2009	SIGMOD	7.0791732e-05
4,450	Power-Law Based Estimation of Set Similarity Join Size	2009	VLDB	6.6972929e-05
4,992	Scalable Ad-hoc Entity Extraction from Text Collections	2008	VLDB	6.4100657e-05
5,732	Effective Keyword-based Selection of Relational Databases	2007	SIGMOD	6.1064568e-05
6,011	From Information to Knowledge: Harvesting Entities and Relationships from Web Sources	2010	PODS	6.0105397e-05
7,239	I4E: Interactive Investigation of Iterative Information Extraction	2010	SIGMOD	5.6645117e-05
8,195	When Speed Has a Price: Fast Information Extraction Using Approximate Algorithms	2013	VLDB	5.4694257e-05
9,828	Optimizing Complex Extraction Programs over Evolving Text Data	2009	SIGMOD	5.2138468e-05
11,440	Autonomously Computable Information Extraction	2023	VLDB	5.093636e-05
12,088	Synthesizing Data Programs	2015	CIDR	5.093636e-05
12,225	D-Hive: Data Bees Pollinating RDF, Text, and Time	2013	CIDR	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 5 of 5 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
158	Robust and Efficient Fuzzy Match for Online Data Cleaning	2003	SIGMOD	0.00028199923
508	Random Sampling for Histogram Construction: How much is enough?	1998	SIGMOD	0.00017275873
1,403	Focused Crawling Using Context Graphs	2000	VLDB	0.000108876
1,483	Distributed Search over the Hidden Web: Hierarchical Database Sampling and Selection	2002	VLDB	0.00010633832
1,601	Automatic Discovery of Language Models for Text Databases	1999	SIGMOD	0.00010239819

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	212	Efficient IR-Style Keyword Search over Relational Databases	2003	VLDB
2	2,948	Optimized Query Execution in Large Search Engines with Global Page Ordering	2003	VLDB
3	1,483	Distributed Search over the Hidden Web: Hierarchical Database Sampling and Selection	2002	VLDB
4	8,195	When Speed Has a Price: Fast Information Extraction Using Approximate Algorithms	2013	VLDB
5	1,403	Focused Crawling Using Context Graphs	2000	VLDB
6	12,287	Probabilistic Query Rewriting for Efficient and Effective Keyword Search on Graph Data	2013	VLDB
7	9,683	Optimal Algorithms for Crawling a Hidden Database in the Web	2012	VLDB
8	8,122	A Formal Model of Trade-off between Optimization and Execution Costs in Semantic Query Optimization	1988	VLDB
9	6,371	QUEST: Query Optimization in Unstructured Document Analysis	2025	VLDB
10	2,244	Join Queries with External Text Sources: Execution and Optimization Techniques	1995	SIGMOD