Magpie: Python at Speed and Scale using Cloud Backends

Summary: Magpie exposes the Pandas API but lazily pushes dataframe work into cloud query engines (SQL DW, Spark, SCOPE) through a common data layer, avoiding cross-engine transfer and leveraging DB-grade features. It auto-selects optimal backends to deliver database-scale performance to Python analytics; production traces show ~25% of internal computations could benefit. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: h5c05dd2750c60cca
Venue: CIDR
Year: 2021
Pagerank: 8.8487039e-05
Overall Rank: 2,207 | 85.17%
DOI: -

Incoming Non-self Citations Over Time

Authors

1. Alekh Jindal (Microsoft)
2. K. Venkatesh Emani (Microsoft)
3. Maureen Daum (University of Washington)
4. Olga Poppe (Microsoft)
5. Brandon Haynes (Microsoft)
6. Anna Pavlenko (Microsoft)
7. Ayushi Gupta (Apple)
8. Karthik Ramachandra (Microsoft)
9. Carlo Curino (Microsoft)
10. Andreas Mueller (Microsoft)
11. Wentao Wu (Microsoft)
12. Hiren Patel (Microsoft)

BibTeX Citation

@inproceedings{jindal_cidr21,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '21},
        title = {{Magpie: Python at Speed and Scale using Cloud Backends}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Jindal, Alekh and Emani, K. Venkatesh and Daum, Maureen and Poppe, Olga and Haynes, Brandon and Pavlenko, Anna and Gupta, Ayushi and Ramachandra, Karthik and Curino, Carlo and Mueller, Andreas and Wu, Wentao and Patel, Hiren},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 18 of 18 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,662	End-to-end Optimization of Machine Learning Prediction Queries	2022	SIGMOD	8.1596229e-05
3,098	Lux: Always-on Visualization Recommendations for Exploratory Dataframe Workflows	2022	VLDB	7.6517623e-05
3,930	Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System	2022	VLDB	6.9185978e-05
4,815	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB	6.3992929e-05
5,230	Babelfish: Efficient Execution of Polyglot Queries	2022	VLDB	6.2189001e-05
6,116	ConnectorX: Accelerating Data Loading From Databases to Dataframes	2022	VLDB	5.8815309e-05
6,212	YeSQL: "You extend SQL" with Rich and Highly Performant User-Defined Functions in Relational Databases	2022	VLDB	5.8480745e-05
6,245	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.837329e-05
7,269	Decentralized Actor Scheduling and Reference-based Storage in Xorbits: a Native Scalable Data Science Engine	2025	VLDB	5.5694935e-05
7,983	Efficient Execution of User-Defined Functions in SQL Queries	2023	VLDB	5.4131297e-05
8,153	Predicate Pushdown for Data Science Pipelines	2023	SIGMOD	5.3891786e-05
9,588	The Key to Effective UDF Optimization: Before Inlining, First Perform Outlining	2025	VLDB	5.156809e-05
9,637	QURE: AI-Assisted and Automatically Verified UDF Inlining	2025	SIGMOD	5.1462239e-05
10,278	Dias: Dynamic Rewriting of Pandas Code	2024	SIGMOD	5.0455234e-05
10,898	stratum: A System Infrastructure for Massive Agent-Centric ML Workloads	2026	VLDB	4.9793485e-05
10,995	Painless and Efficient Scaling of Pandas Programs: Demo	2026	VLDB	4.9793485e-05
11,498	Proactive Resume and Pause of Resources for Microsoft Azure SQL Database Serverless	2024	SIGMOD	4.9793485e-05
11,570	SplitDF: Splitting Dataframes for Memory-Efficient Data Analysis	2024	VLDB	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
896	Froid: Optimization of Imperative Programs in a Relational Database	2018	VLDB	0.00013209291
1,274	Rewriting Procedures for Batched Bindings	2008	VLDB	0.00011247334
1,279	AI Meets AI: Leveraging Query Executions to Improve Index Recommendations	2019	SIGMOD	0.00011226878
1,369	Towards Scalable Dataframe Systems	2020	VLDB	0.00010899832
1,686	Garlic: A New Flavor of Federated Query Processing for DB2	2002	SIGMOD	9.8709217e-05
2,446	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.4547121e-05
2,778	AIDA - Abstraction for Advanced In-Database Analytics	2018	VLDB	8.0299506e-05
2,834	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	7.9560627e-05
2,954	Automatic Partitioning of Database Applications	2012	VLDB	7.8126137e-05
2,993	Extracting Equivalent SQL from Imperative Code in Database Applications	2016	SIGMOD	7.7721951e-05
3,685	The Myria Big Data Management and Analytics System and Cloud Service	2017	CIDR	7.0988339e-05
3,923	Sloth: Being Lazy is a Virtue (When Issuing Database Queries)	2014	SIGMOD	6.9224617e-05
6,901	Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation	2021	VLDB	5.6520335e-05
7,438	DBridge: Translating Imperative Code to SQL	2017	SIGMOD	5.5272264e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	6,561	MotherDuck: DuckDB in the cloud and in the client	2024	CIDR
2	4,815	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB
3	1,803	Tuplex: Data Science in Python at Native Code Speed	2021	SIGMOD
4	3,316	Cloud Analytics Benchmark	2023	VLDB
5	5,719	Accelerating Python UDFs in Vectorized Query Execution	2022	CIDR
6	10,995	Painless and Efficient Scaling of Pandas Programs: Demo	2026	VLDB
7	9,743	When sweet and cute isn't enough anymore: Solving scalability issues in Python Pandas with Grizzly	2020	CIDR
8	3,384	Putting Pandas in a Box	2021	CIDR
9	1,369	Towards Scalable Dataframe Systems	2020	VLDB
10	932	Starling: A Scalable Query Engine on Cloud Functions	2020	SIGMOD