Predicate Pushdown for Data Science Pipelines

Summary: MagicPush uses a search-verification approach to predicate pushdown in data science pipelines, discovering input-space predicates and proving pushdown preserves outputs, even with non-relational operators and UDFs. Evaluations on TPC-H and 200 real-world GitHub Notebook pipelines show it beats a strong rule-based baseline, discovers new pushdown opportunities, and yields up to 99% running-time reduction in 42 pipelines while matching baseline opportunities elsewhere. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hb4a7fd26bb386237
Venue: SIGMOD
Year: 2023
Pagerank: 5.3688159e-05
Overall Rank: 8,176 | 45.20%
DOI: 10.1145/3589281

Incoming Non-self Citations Over Time

Authors

1. Cong Yan (Microsoft)
2. Yin Lin (University of Michigan)
3. Yeye He (Microsoft)

BibTeX Citation

@inproceedings{yan_sigmod23,
        title = {{Predicate Pushdown for Data Science Pipelines}},
        author = {Yan, Cong and Lin, Yin and He, Yeye},
        series = {{SIGMOD} '23},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3589281},
        url = {https://dl.acm.org/doi/10.1145/3589281},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 5 of 5 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
9,729	The UDFBench Benchmark for General-purpose UDF Queries	2025	VLDB	5.1202393e-05
10,119	LiquidCache: Efficient Pushdown Caching for Cloud-Native Data Analytics	2025	VLDB	5.0638787e-05
10,702	Data-Semantics-Aware Recommendation of Diverse Pivot Tables	2026	SIGMOD	4.9605343e-05
10,914	I-Rex: An Interactive Debugger for SQL	2026	VLDB	4.9605343e-05
11,195	Dynamic Pruning for Recursive Joins	2025	SIGMOD	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 27 of 27 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00055523999
125	Predicate Migration: Optimizing Queries with Expensive Predicates	1993	SIGMOD	0.00030398363
312	Optimization of Real Conjunctive Queries	1993	PODS	0.00021306615
896	Froid: Optimization of Imperative Programs in a Relational Database	2018	VLDB	0.00013164545
1,122	Query Optimization by Predicate Move-Around	1994	VLDB	0.00011915958
1,132	Qd-tree: Learning Data Layouts for Big Data Analytics	2020	SIGMOD	0.00011873466
1,702	PIVOT and UNPIVOT: Optimization and Execution Strategies in an RDBMS	2004	VLDB	9.8175982e-05
1,812	Tuplex: Data Science in Python at Native Code Speed	2021	SIGMOD	9.5747668e-05
1,888	WeTune: Automatic Discovery and Verification of Query Rewrite Rules	2022	SIGMOD	9.4148912e-05
1,889	SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures	2014	VLDB	9.4145058e-05
1,958	Quantifying TPC-H Choke Points and Their Optimizations	2020	VLDB	9.2894133e-05
2,219	Magpie: Python at Speed and Scale using Cloud Backends	2021	CIDR	8.8186113e-05
2,446	Mison: A Fast JSON Parser for Data Analytics	2017	VLDB	8.4392802e-05
2,474	AnalyticDB: Real-time OLAP Database System at Alibaba Cloud	2019	VLDB	8.3955114e-05
2,645	Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks	2020	SIGMOD	8.1641056e-05
3,081	Pushing Data-Induced Predicates Through Joins in Big-Data Clusters	2020	VLDB	7.6515625e-05
3,194	Demonstration of the Cosette Automated SQL Prover	2017	SIGMOD	7.5310063e-05
3,294	Automated Verification of Query Equivalence Using Satisfiability Modulo Theories	2019	VLDB	7.4297513e-05
3,404	FlexPushdownDB: Hybrid Pushdown and Caching in a Cloud DBMS	2021	VLDB	7.3212594e-05
4,078	Aggify: Lifting the Curse of Cursor Loops using Custom Aggregates	2020	SIGMOD	6.8028789e-05
4,670	Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications	2018	SIGMOD	6.4607221e-05
5,210	Crystal: A Unified Cache Storage System for Analytical Databases	2021	VLDB	6.2142008e-05
6,155	Sia: Optimizing Queries using Learned Predicates	2021	SIGMOD	5.8540214e-05
6,218	YeSQL: "You extend SQL" with Rich and Highly Performant User-Defined Functions in Relational Databases	2022	VLDB	5.8292066e-05
6,294	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	5.8043081e-05
7,167	Optimizing Recursive Queries with Program Synthesis	2022	SIGMOD	5.5794956e-05
10,195	Generating Application-Specific Data Layouts for In-memory Databases	2019	VLDB	5.0516135e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	7,338	Interactive Demonstration of Probabilistic Predicates	2018	SIGMOD
2	2,672	End-to-end Optimization of Machine Learning Prediction Queries	2022	SIGMOD
3	6,400	Selection Pushdown in Column Stores using Bit Manipulation Instructions	2023	SIGMOD
4	3,966	Optimizing Machine Learning Inference Queries with Correlative Proxy Models	2022	VLDB
5	4,527	Conjunctive Queries with Comparisons	2022	SIGMOD
6	3,081	Pushing Data-Induced Predicates Through Joins in Big-Data Clusters	2020	VLDB
7	1,122	Query Optimization by Predicate Move-Around	1994	VLDB
8	125	Predicate Migration: Optimizing Queries with Expensive Predicates	1993	SIGMOD
9	11,568	PLAQUE: Automated Predicate Learning at Query Time	2024	SIGMOD
10	278	Accelerating Machine Learning Inference with Probabilistic Predicates	2018	SIGMOD