Opening the Black Boxes in Data Flow Optimization

Summary: Optimizes data flows with black-box UDFs by extracting a few operator properties to determine safe reordering, via static analysis of UDF code. The optimizer enables selection reordering, bushy join ordering, and limited aggregation push-down for non-relational flows, achieving relational-like rewrite power without relying on operator semantics. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h1efa77d09424640d
Venue: VLDB
Year: 2012
Pagerank: 8.8781177e-05
Overall Rank: 2,195 | 85.25%
DOI: 10.14778/2350229.2350244

Incoming Non-self Citations Over Time

Authors

1. Fabian Hueske (Technical University of Berlin)
2. Mathias Peters (Humboldt University of Berlin)
3. Matthias J. Sax (Humboldt University of Berlin)
4. Astrid Rheinländer (Humboldt University of Berlin)
5. Rico Bergmann (Humboldt University of Berlin)
6. Aljoscha Krettek (Technical University of Berlin)
7. Kostas Tzoumas (Technical University of Berlin)

BibTeX Citation

@article{hueske_vldb12,
        title = {{Opening the Black Boxes in Data Flow Optimization}},
        author = {Hueske, Fabian and Peters, Mathias and Sax, Matthias J. and Rheinländer, Astrid and Bergmann, Rico and Krettek, Aljoscha and Tzoumas, Kostas},
        journal = {PVLDB},
        series = {{VLDB} '12},
        volume = {5},
        number = {11},
        pages = {1256--1267},
        doi = {10.14778/2350229.2350244},
        url = {https://doi.org/10.14778/2350229.2350244},
        year = {2012}
}

Incoming Citations (Sorted by Pagerank)

Showing 21 of 21 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,081	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB	0.00012123917
1,803	Tuplex: Data Science in Python at Native Code Speed	2021	SIGMOD	9.6068397e-05
2,227	Spinning Fast Iterative Data Flows	2012	VLDB	8.8021772e-05
2,264	An Intermediate Representation for Optimizing Machine Learning Pipelines	2019	VLDB	8.7289107e-05
3,402	RHEEM: Enabling Cross-Platform Data Processing - May The Big Data Be With You! -	2018	VLDB	7.3304477e-05
4,069	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	6.821366e-05
4,293	A Method for Optimizing Opaque Filter Queries	2020	SIGMOD	6.6819917e-05
5,343	The NebulaStream Platform: Data and Application Management for the Internet of Things	2020	CIDR	6.1708665e-05
5,834	Opportunistic Physical Design for Big Data Analytics	2014	SIGMOD	5.9759267e-05
6,144	The BUDS Language for Distributed Bayesian Machine Learning	2017	SIGMOD	5.8717563e-05
6,415	Containerized Execution of UDFs: An Experimental Evaluation	2022	VLDB	5.7926473e-05
6,878	DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines	2022	CIDR	5.657878e-05
7,410	POLAR: Adaptive and Non-invasive Join Order Selection via Plans of Least Resistance	2024	VLDB	5.5342768e-05
8,213	Scaling a Declarative Cluster Manager Architecture with Query Optimization Techniques	2023	VLDB	5.3770961e-05
8,593	Not Black-Box Anymore! Enabling Analytics-Aware Optimizations in Teradata Vantage	2021	VLDB	5.3068556e-05
9,458	HyperBlocker: Accelerating Rule-based Blocking in Entity Resolution using GPUs	2025	VLDB	5.173446e-05
9,692	Versatile Optimization of UDF-heavy Data Flows with Sofa	2014	SIGMOD	5.140168e-05
9,834	PAXQuery: Parallel Analytical XML Processing	2015	SIGMOD	5.1239767e-05
10,000	cedar: Optimized and Unified Machine Learning Input Data Pipelines	2025	VLDB	5.0979044e-05
12,518	How Achaeans Would Construct Columns in Troy	2013	CIDR	4.9793485e-05
12,528	Iterative Parallel Data Processing with Stratosphere: An Inside Look	2013	SIGMOD	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 13 of 13 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
1	Access Path Selection in a Relational Database Management System	1979	SIGMOD	0.0023947656
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.001052036
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050495102
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049839909
73	Including Group-By in Query Optimization	1994	VLDB	0.00037522101
753	Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs	2011	VLDB	0.00014237583
876	Tenzing: A SQL Implementation On The MapReduce Framework	2011	VLDB	0.00013309176
884	Dynamic Programming Strikes Back	2008	SIGMOD	0.00013267935
1,037	Jaql: A Scripting Language for Large Scale Semistructured Data Analysis	2011	VLDB	0.00012376819
1,281	Automatic Optimization for MapReduce Programs	2011	VLDB	0.00011213384
1,285	SQL/MapReduce: A practical approach to self-describing, polymorphic, and parallelizable user-defined functions	2009	VLDB	0.00011201377
2,476	One Size Fits All? – Part 2: Benchmarking Results	2007	CIDR	8.409729e-05
3,931	FERRY - Database-Supported Program Execution	2009	SIGMOD	6.9183836e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	7,261	Rule-Based Translation of Relational Queries into Iterative Programs	1986	SIGMOD
2	5,667	Enabling Incremental Query Re-Optimization	2016	SIGMOD
3	1,098	A General Framework for the Optimization of Object-Oriented Queries	1992	SIGMOD
4	143	Optimization of Nonrecursive Queries	1986	VLDB
5	689	Query Optimization in the Presence of Foreign Functions	1993	VLDB
6	4,693	Static Optimization of Conjunctive Queries with Sliding Windows Over Infinite Streams	2004	SIGMOD
7	6,296	Declarative Sub-Operators for Universal Data Processing	2023	VLDB
8	7,344	SQL Query Optimization: Reordering for a General Class of Queries	1996	SIGMOD
9	4,069	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD
10	2,516	Optimization of Queries with User-defined Predicates	1996	VLDB