Optimization for iterative queries on MapReduce

Summary: OptIQ optimizes iterative MapReduce queries by partitioning into invariant and variant views and materializing the invariant part. It reuses the invariant and incrementally evaluates the variant view, skipping converged tuples for up to 5x speedups on PageRank and k-means. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11090
Venue: VLDB
Year: 2014
Pagerank: 5.6841364e-05
Overall Rank: 7,168 | 50.83%
DOI: 10.14778/2732279.2732286

Incoming Non-self Citations Over Time

Authors

1. Makoto Onizuka (Nippon Telegraph and Telephone Software Innovation Center)
2. Hiroyuki Kato (National Institute of Informatics)
3. Soichiro Hidaka (National Institute of Informatics)
4. Keisuke Nakano (University of Electro-Communications)
5. Zhenjiang Hu (National Institute of Informatics)

BibTeX Citation

@article{onizuka_vldb14,
        title = {{Optimization for iterative queries on MapReduce}},
        author = {Onizuka, Makoto and Kato, Hiroyuki and Hidaka, Soichiro and Nakano, Keisuke and Hu, Zhenjiang},
        journal = {PVLDB},
        series = {{VLDB} '14},
        volume = {7},
        number = {4},
        pages = {241--252},
        doi = {10.14778/2732279.2732286},
        url = {https://doi.org/10.14778/2732279.2732286},
        year = {2014}
}

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,008	Scalable Big Graph Processing in MapReduce	2014	SIGMOD	7.8578871e-05
4,078	Asynchronous and Fault-Tolerant Recursive Datalog Evaluation in Shared-Nothing Engines	2015	VLDB	6.9209348e-05
5,991	Iterative Query Processing based on Unified Optimization Techniques	2019	SIGMOD	6.0171458e-05
10,690	Dynamic Pruning for Recursive Joins	2025	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 12 of 12 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
3	Pregel: A System for Large-Scale Graph Processing	2010	SIGMOD	0.0012250108
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.0010686205
20	Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud	2012	VLDB	0.00056944564
44	A Comparison of Approaches to Large-Scale Data Analysis	2009	SIGMOD	0.00046055057
61	Maintaining Views Incrementally	1993	SIGMOD	0.00039026867
120	HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads	2009	VLDB	0.00031680027
372	HaLoop: Efficient Iterative Data Processing on Large Clusters	2010	VLDB	0.0001981521
642	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB	0.00015395331
960	Incremental Query Evaluation in a Ring of Databases	2010	PODS	0.00012945163
2,454	REX: Recursive, Delta-Based Data-Centric Computation	2012	VLDB	8.5560058e-05
2,560	PLANET: Massively Parallel Learning of Tree Ensembles with MapReduce	2009	VLDB	8.4143663e-05
9,263	MapReduce Algorithms for Big Data Analysis	2012	VLDB	5.2966408e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,967	IO-Top-k: Index-access Optimized Top-k Query Processing	2006	VLDB
2	2,265	A Platform for Scalable One-Pass Analytics using MapReduce	2011	SIGMOD
3	2,887	Efficient Multi-way Theta-Join Processing Using MapReduce	2012	VLDB
4	72	Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters	2007	SIGMOD
5	1,257	Automatic Optimization for MapReduce Programs	2011	VLDB
6	3,268	Efficient and Provable Multi-Query Optimization	2017	PODS
7	9,588	IQ: The Case for Iterative Querying for Knowledge	2011	CIDR
8	8,473	An Iterative Method for Distributed Database Design	1991	VLDB
9	3,646	An Efficient Query Indexing Mechanism for Filtering Geo-Textual Data	2013	SIGMOD
10	3,163	Multi-Query Optimization in MapReduce Framework	2014	VLDB