Magpie: Python at Speed and Scale using Cloud Backends

Summary: Magpie exposes the Pandas API but lazily pushes dataframe work into cloud query engines (SQL DW, Spark, SCOPE) through a common data layer, avoiding cross-engine transfer and leveraging DB-grade features. It auto-selects optimal backends to deliver database-scale performance to Python analytics; production traces show ~25% of internal computations could benefit. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: h5c05dd2750c60cca
Venue: CIDR
Year: 2021
Pagerank: 8.8445332e-05
Overall Rank: 2,208 | 85.17%
DOI: -
PDF: Download (CC BY 3.0)

Incoming Non-self Citations Over Time

Authors

1. Alekh Jindal (Microsoft)
2. K. Venkatesh Emani (Microsoft)
3. Maureen Daum (University of Washington)
4. Olga Poppe (Microsoft)
5. Brandon Haynes (Microsoft)
6. Anna Pavlenko (Microsoft)
7. Ayushi Gupta (Apple)
8. Karthik Ramachandra (Microsoft)
9. Carlo Curino (Microsoft)
10. Andreas Mueller (Microsoft)
11. Wentao Wu (Microsoft)
12. Hiren Patel (Microsoft)

BibTeX Citation

@inproceedings{jindal_cidr21,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '21},
        title = {{Magpie: Python at Speed and Scale using Cloud Backends}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Jindal, Alekh and Emani, K. Venkatesh and Daum, Maureen and Poppe, Olga and Haynes, Brandon and Pavlenko, Anna and Gupta, Ayushi and Ramachandra, Karthik and Curino, Carlo and Mueller, Andreas and Wu, Wentao and Patel, Hiren},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 18 of 18 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,661	End-to-end Optimization of Machine Learning Prediction Queries	2022	SIGMOD	8.1568473e-05
3,100	Lux: Always-on Visualization Recommendations for Exploratory Dataframe Workflows	2022	VLDB	7.64814e-05
3,931	Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System	2022	VLDB	6.9153238e-05
4,817	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB	6.3962636e-05
5,234	Babelfish: Efficient Execution of Polyglot Queries	2022	VLDB	6.2159562e-05
6,117	ConnectorX: Accelerating Data Loading From Databases to Dataframes	2022	VLDB	5.8787466e-05
6,215	YeSQL: "You extend SQL" with Rich and Highly Performant User-Defined Functions in Relational Databases	2022	VLDB	5.8453061e-05
6,248	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.8345657e-05
7,272	Decentralized Actor Scheduling and Reference-based Storage in Xorbits: a Native Scalable Data Science Engine	2025	VLDB	5.5668569e-05
7,988	Efficient Execution of User-Defined Functions in SQL Queries	2023	VLDB	5.4105672e-05
8,159	Predicate Pushdown for Data Science Pipelines	2023	SIGMOD	5.3866275e-05
9,596	The Key to Effective UDF Optimization: Before Inlining, First Perform Outlining	2025	VLDB	5.1543679e-05
9,645	QURE: AI-Assisted and Automatically Verified UDF Inlining	2025	SIGMOD	5.1437878e-05
10,284	Dias: Dynamic Rewriting of Pandas Code	2024	SIGMOD	5.0431349e-05
10,907	stratum: A System Infrastructure for Massive Agent-Centric ML Workloads	2026	VLDB	4.9769913e-05
11,004	Painless and Efficient Scaling of Pandas Programs: Demo	2026	VLDB	4.9769913e-05
11,504	Proactive Resume and Pause of Resources for Microsoft Azure SQL Database Serverless	2024	SIGMOD	4.9769913e-05
11,576	SplitDF: Splitting Dataframes for Memory-Efficient Data Analysis	2024	VLDB	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
896	Froid: Optimization of Imperative Programs in a Relational Database	2018	VLDB	0.00013203085
1,275	Rewriting Procedures for Batched Bindings	2008	VLDB	0.00011242055
1,280	AI Meets AI: Leveraging Query Executions to Improve Index Recommendations	2019	SIGMOD	0.00011224914
1,370	Towards Scalable Dataframe Systems	2020	VLDB	0.00010895207
1,686	Garlic: A New Flavor of Federated Query Processing for DB2	2002	SIGMOD	9.8662635e-05
2,448	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.4508839e-05
2,778	AIDA - Abstraction for Advanced In-Database Analytics	2018	VLDB	8.0261532e-05
2,833	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	7.9539771e-05
2,957	Automatic Partitioning of Database Applications	2012	VLDB	7.808922e-05
2,994	Extracting Equivalent SQL from Imperative Code in Database Applications	2016	SIGMOD	7.7685417e-05
3,687	The Myria Big Data Management and Analytics System and Cloud Service	2017	CIDR	7.0954736e-05
3,925	Sloth: Being Lazy is a Virtue (When Issuing Database Queries)	2014	SIGMOD	6.919284e-05
6,903	Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation	2021	VLDB	5.6493582e-05
7,441	DBridge: Translating Imperative Code to SQL	2017	SIGMOD	5.5246099e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	6,563	MotherDuck: DuckDB in the cloud and in the client	2024	CIDR
2	4,817	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB
3	1,803	Tuplex: Data Science in Python at Native Code Speed	2021	SIGMOD
4	3,311	Cloud Analytics Benchmark	2023	VLDB
5	5,720	Accelerating Python UDFs in Vectorized Query Execution	2022	CIDR
6	11,004	Painless and Efficient Scaling of Pandas Programs: Demo	2026	VLDB
7	9,748	When sweet and cute isn't enough anymore: Solving scalability issues in Python Pandas with Grizzly	2020	CIDR
8	3,385	Putting Pandas in a Box	2021	CIDR
9	1,370	Towards Scalable Dataframe Systems	2020	VLDB
10	932	Starling: A Scalable Query Engine on Cloud Functions	2020	SIGMOD