Optimization for iterative queries on MapReduce

Summary: OptIQ optimizes iterative MapReduce queries by partitioning into invariant and variant views and materializing the invariant part. It reuses the invariant and incrementally evaluates the variant view, skipping converged tuples for up to 5x speedups on PageRank and k-means. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 10902
Venue: VLDB
Year: 2014
Pagerank: 4.773119e-05
Overall Rank: 7,294 | 49.26%
DOI: -

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,129	Scalable Big Graph Processing in MapReduce	2014	SIGMOD	7.5008242e-05
4,696	Asynchronous and Fault-Tolerant Recursive Datalog Evaluation in Shared-Nothing Engines	2015	VLDB	5.9911301e-05
6,245	Iterative Query Processing based on Unified Optimization Techniques	2019	SIGMOD	5.1391202e-05
10,404	Dynamic Pruning for Recursive Joins	2025	SIGMOD	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 12 of 12 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
3	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.0024183614
4	Pregel: A System for Large-Scale Graph Processing	2010	SIGMOD	0.0019005923
37	Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud	2012	VLDB	0.0007522744
42	A Comparison of Approaches to Large-Scale Data Analysis	2009	SIGMOD	0.00073498298
95	Maintaining Views Incrementally	1993	SIGMOD	0.00050896659
157	HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads	2009	VLDB	0.00040397359
413	HaLoop: Efficient Iterative Data Processing on Large Clusters	2010	VLDB	0.00023904409
780	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB	0.00016775082
1,238	Incremental Query Evaluation in a Ring of Databases	2010	PODS	0.00013114581
2,458	REX: Recursive, Delta-Based Data-Centric Computation	2012	VLDB	8.7683462e-05
2,630	PLANET: Massively Parallel Learning of Tree Ensembles with MapReduce	2009	VLDB	8.4128091e-05
9,115	MapReduce Algorithms for Big Data Analysis	2012	VLDB	4.3932167e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
2,009	IO-Top-k: Index-access Optimized Top-k Query Processing	2006	VLDB	9.7977564e-05
2,476	A Platform for Scalable One-Pass Analytics using MapReduce	2011	SIGMOD	8.6960139e-05
3,062	Efficient Multi-way Theta-Join Processing Using MapReduce	2012	VLDB	7.6343994e-05
15	Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters	2007	SIGMOD	0.0010654262
1,280	Automatic Optimization for MapReduce Programs	2011	VLDB	0.0001285503
3,462	Efficient and Provable Multi-Query Optimization	2017	PODS	7.0703696e-05
9,359	IQ: The Case for Iterative Querying for Knowledge	2011	CIDR	4.3509599e-05
8,059	An Iterative Method for Distributed Database Design	1991	VLDB	4.5944409e-05
3,714	An Efficient Query Indexing Mechanism for Filtering Geo-Textual Data	2013	SIGMOD	6.8223298e-05
3,703	Multi-Query Optimization in MapReduce Framework	2014	VLDB	6.8289978e-05