DBScholar

Back to papers

Web-scale Data Integration: You can only afford to Pay As You Go

Summary: Argues that traditional tightly-coupled integration fails at web scale (Deep Web, Google Base) due to extreme heterogeneity and scale. Proposes PAYGO, a dataspaces-inspired pay-as-you-go architecture that delivers incremental, best-effort, cost-aware integration to maximize utility under limited resources. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID
h22f1f81c337a5188
Venue
CIDR
Year
2007
Pagerank
0.00012641942
Overall Rank
992 | 93.34%
DOI
-
PDF
Download (CC BY 2.5)

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{madhavan_cidr07,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '07},
        title = {{Web-scale Data Integration: You can only afford to Pay As You Go}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Madhavan, Jayant and Jeffery, Shawn R. and Cohen, Shirley and Dong, Xin (Luna) and Ko, David and Yu, Cong and Halevy, Alon},
        year = {2007}
}

Incoming Citations (Sorted by Pagerank)

Showing 28 of 28 citing papers.

Rank Citing Paper Year Venue Pagerank
319 Making Database Systems Usable 2007 SIGMOD 0.00021155013
579 Incremental Knowledge Base Construction Using DeepDive 2015 VLDB 0.00016083582
587 Management of Probabilistic Data: Foundations and Challenges 2007 PODS 0.00015927027
888 Pay-as-you-go User Feedback for Dataspace Systems 2008 SIGMOD 0.00013250185
1,582 Google’s Deep-Web Crawl 2008 VLDB 0.00010168682
2,471 A Relational Approach to Incrementally Extracting and Querying Structure in Unstructured Data 2007 VLDB 8.4103934e-05
2,886 DB&IR: Both Sides Now (Extended Abstract) 2007 SIGMOD 7.9047629e-05
3,236 How Large Language Models Will Disrupt Data Management 2023 VLDB 7.4996147e-05
3,740 BLAST: a Loosely Schema-aware Meta-blocking Approach for Entity Resolution 2016 VLDB 7.0594489e-05
4,519 Arnold: Declarative Crowd-Machine Data Integration 2013 CIDR 6.5602428e-05
4,551 Harnessing the Deep Web: Present and Future 2009 CIDR 6.5380754e-05
5,131 iTrails: Pay-as-you-go Information Integration in Dataspaces 2007 VLDB 6.2578858e-05
5,145 Schema-agnostic vs Schema-based Configurations for Blocking Methods on Homogeneous Data 2016 VLDB 6.2539412e-05
5,502 HAMSTER: Using Search Clicklogs for Schema and Taxonomy Matching 2009 VLDB 6.1007121e-05
5,679 A Hierarchical Approach to Model Web Query Interfaces for Web Source Integration 2009 VLDB 6.0381177e-05
7,263 Entity Resolution On-Demand 2022 VLDB 5.5693872e-05
8,065 A Grammar-based Entity Representation Framework for Data Cleaning 2009 SIGMOD 5.3940212e-05
8,189 SXPath - Extending XPath towards Spatial Querying on Web Documents 2011 VLDB 5.379443e-05
8,928 Search Driven Analysis of Heterogeneous XML Data 2009 CIDR 5.2534908e-05
8,974 Effective Entity Augmentation By Querying External Data Sources 2023 VLDB 5.2457737e-05
9,179 The Role of Schema Matching in Large Enterprises 2009 CIDR 5.2116341e-05
9,208 Learning to Extract Form Labels 2008 VLDB 5.2063337e-05
10,324 Stop Word and Related Problems in Web Interface Integration 2009 VLDB 5.0348633e-05
12,607 Mob Data Sourcing 2012 SIGMOD 4.9769913e-05
12,676 Pay-As-You-Go Mapping Selection in Dataspaces 2011 SIGMOD 4.9769913e-05
12,715 Schema Clustering and Retrieval for Multi-domain Pay-As-You-Go Data Integration Systems 2010 SIGMOD 4.9769913e-05
12,732 Creating and Exploring Web Form Repositories 2010 SIGMOD 4.9769913e-05
12,810 Vispedia: On-demand Data Integration for Interactive Visualization and Exploration 2009 SIGMOD 4.9769913e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
24 Data Integration: A Theoretical Perspective 2002 PODS 0.00055018384
191 Applying Model Management to Classical Meta Data Problems 2003 CIDR 0.00025757196
230 Reference Reconciliation in Complex Information Spaces 2005 SIGMOD 0.00023861741
269 Reconciling Schemas of Disparate Data Sources: A Machine-Learning Approach 2001 SIGMOD 0.00022610366
348 Crawling the Hidden Web 2001 VLDB 0.00020252678
474 Automated Ranking of Database Query Results 2003 CIDR 0.00017665834
815 An Interactive Clustering-based Approach to Integrating Source Query Interfaces on the Deep Web 2004 SIGMOD 0.00013713173
1,114 Principles of Dataspace Systems 2006 PODS 0.00011959287
1,567 Crossing the Structure Chasm 2003 CIDR 0.00010208293
2,276 A Platform for Personal Information Management and Integration 2005 CIDR 8.7080472e-05
2,462 DBLife: A Community Information Management Platform for the Database Research Community 2007 CIDR 8.427832e-05
2,662 WISE-Integrator: An Automatic Integrator of Web Search Interfaces for E-Commerce 2003 VLDB 8.1565356e-05
3,762 Toward Large Scale Integration: Building a MetaQuerier over Databases on the Web 2005 CIDR 7.0398453e-05
6,663 A Dataspace Odyssey: The iMeMex Personal Dataspace Management System 2007 CIDR 5.7152846e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 13,341 Data Integration in the Large: The Challenge of Reuse 1994 VLDB
2 13,100 Architectural Issues and Solutions in the Development of Data-Intensive Web Applications 2003 CIDR
3 2,160 Open Data Integration 2018 VLDB
4 815 An Interactive Clustering-based Approach to Integrating Source Query Interfaces on the Deep Web 2004 SIGMOD
5 6,701 Web Data Management 2011 SIGMOD
6 2,122 An Analysis of Structured Data on the Web 2012 VLDB
7 12,663 Schema-As-You-Go: On Probabilistic Tagging and Querying of Wide Tables 2011 SIGMOD
8 3,762 Toward Large Scale Integration: Building a MetaQuerier over Databases on the Web 2005 CIDR
9 12,715 Schema Clustering and Retrieval for Multi-domain Pay-As-You-Go Data Integration Systems 2010 SIGMOD
10 1,668 Bootstrapping Pay-As-You-Go Data Integration Systems 2008 SIGMOD