Database Paper Browser

Back to papers

Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience

Summary: Pig provides SQL-like data manipulation on MapReduce by building explicit dataflows interleaved with UDFs, compiled to Hadoop jobs. It discusses challenges and compares Pig's performance to hand-tuned MapReduce, showing productivity gains with modest overhead. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
9834
Venue
VLDB
Year
2009
Pagerank
0.00016765694
Overall Rank
776 | 94.61%
DOI
-

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 33 of 33 citing papers.

Rank Citing Paper Year Venue Pagerank
442 Efficient Parallel Set-Similarity Joins Using MapReduce 2010 SIGMOD 0.00023095823
536 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020651621
789 Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) 2010 VLDB 0.00016602215
947 MRShare: Sharing Across Multiple Queries in MapReduce 2010 VLDB 0.00015112344
981 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB 0.00014866268
1,114 Parallel Evaluation of Conjunctive Queries 2011 PODS 0.00013871948
1,199 Hadoop-GIS: A High Performance Spatial Data Warehousing System over MapReduce 2013 VLDB 0.00013358019
1,339 SkewTune: Mitigating Skew in MapReduce Applications 2012 SIGMOD 0.00012492978
1,945 Cheetah: A High Performance, Custom Data Warehouse on Top of MapReduce 2010 VLDB 0.0001000706
2,211 ReStore: Reusing Results of MapReduce Jobs 2012 VLDB 9.2823037e-05
2,754 Stubby: A Transformation-based Optimizer for MapReduce Workflows 2012 VLDB 8.1720428e-05
3,207 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD 7.3847098e-05
3,510 Integrating Hadoop and Parallel DBMS 2010 SIGMOD 7.0280199e-05
3,606 Large-Scale Machine Learning at Twitter 2012 SIGMOD 6.9246371e-05
3,709 Multi-Query Optimization in MapReduce Framework 2014 VLDB 6.8211506e-05
4,128 Advanced Join Strategies for Large-Scale Distributed Computation 2014 VLDB 6.4214449e-05
4,424 Nova: Continuous Pig/Hadoop Workflows 2011 SIGMOD 6.19243e-05
4,563 The Unified Logging Infrastructure for Data Analytics at Twitter 2012 VLDB 6.0752535e-05
4,604 Inspector Gadget: A Framework for Custom Monitoring and Debugging of Distributed Dataflows 2011 VLDB 6.0497655e-05
5,012 Dynamically Optimizing Queries over Large Scale Data Platforms 2014 SIGMOD 5.7543101e-05
5,348 Inspector Gadget: A Framework for Custom Monitoring and Debugging of Distributed Dataflows 2011 SIGMOD 5.5554072e-05
5,373 Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing 2022 VLDB 5.5410059e-05
5,566 A Hadoop Based Distributed Loading Approach to Parallel Data Warehouses 2011 SIGMOD 5.4291336e-05
5,908 Building Wavelet Histograms on Large Data in MapReduce 2012 VLDB 5.2731311e-05
6,135 Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture 2013 SIGMOD 5.1908958e-05
6,177 Exploiting Soft and Hard Correlations in Big Data Query Optimization 2016 VLDB 5.1646971e-05
7,293 Optimization for iterative queries on MapReduce 2014 VLDB 4.7668182e-05
8,585 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB 4.4856045e-05
8,985 SpongeFiles: Mitigating Data Skew in MapReduce Using Distributed Memory 2014 SIGMOD 4.4129993e-05
9,360 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB 4.3476444e-05
9,505 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 4.3300131e-05
11,984 Anti-Combining for MapReduce 2014 SIGMOD 4.1905499e-05
12,211 Resiliency-Aware Data Management 2011 VLDB 4.1905499e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 5 of 5 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
3 Pig Latin: A Not-So-Foreign Language for Data Processing 2008 SIGMOD 0.0024217964
22 SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets 2008 VLDB 0.00084679526
315 Distributed Query Processing In A Relational Data Base System 1978 SIGMOD 0.00028022592
548 Practical Skew Handling in Parallel Joins 1992 VLDB 0.00020369531
2,035 Generating Example Data for Dataflow Programs 2009 SIGMOD 9.7131385e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
42 A Comparison of Approaches to Large-Scale Data Analysis 2009 SIGMOD 0.00073570328
70 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00059744625
4,859 The "Big Data" Ecosystem at LinkedIn 2013 SIGMOD 5.8680444e-05
158 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB 0.00040401371
2,476 A Platform for Scalable One-Pass Analytics using MapReduce 2011 SIGMOD 8.6907971e-05
3,606 Large-Scale Machine Learning at Twitter 2012 SIGMOD 6.9246371e-05
12,133 ReStore: Reusing Results of MapReduce Jobs in Pig 2012 SIGMOD 4.1905499e-05
4,424 Nova: Continuous Pig/Hadoop Workflows 2011 SIGMOD 6.19243e-05
2,211 ReStore: Reusing Results of MapReduce Jobs 2012 VLDB 9.2823037e-05
3 Pig Latin: A Not-So-Foreign Language for Data Processing 2008 SIGMOD 0.0024217964