Magpie: Python at Speed and Scale using Cloud Backends

Summary: Magpie exposes the Pandas API but lazily pushes dataframe work into cloud query engines (SQL DW, Spark, SCOPE) through a common data layer, avoiding cross-engine transfer and leveraging DB-grade features. It auto-selects optimal backends to deliver database-scale performance to Python analytics; production traces show ~25% of internal computations could benefit. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 406
Venue: CIDR
Year: 2021
Pagerank: 8.2918086e-05
Overall Rank: 2,651 | 81.82%
DOI: -

Incoming Non-self Citations Over Time

Authors

1. Alekh Jindal (Microsoft)
2. K. Venkatesh Emani (Microsoft)
3. Maureen Daum (University of Washington)
4. Olga Poppe (Microsoft)
5. Brandon Haynes (Microsoft)
6. Anna Pavlenko (Microsoft)
7. Ayushi Gupta (Apple)
8. Karthik Ramachandra (Microsoft)
9. Carlo Curino (Microsoft)
10. Andreas Mueller (Microsoft)
11. Wentao Wu (Microsoft)
12. Hiren Patel (Microsoft)

BibTeX Citation

@inproceedings{jindal_cidr21,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '21},
        title = {{Magpie: Python at Speed and Scale using Cloud Backends}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Jindal, Alekh and Emani, K. Venkatesh and Daum, Maureen and Poppe, Olga and Haynes, Brandon and Pavlenko, Anna and Gupta, Ayushi and Ramachandra, Karthik and Curino, Carlo and Mueller, Andreas and Wu, Wentao and Patel, Hiren},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 16 of 16 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,865	End-to-end Optimization of Machine Learning Prediction Queries	2022	SIGMOD	8.0180243e-05
3,072	Lux: Always-on Visualization Recommendations for Exploratory Dataframe Workflows	2022	VLDB	7.7873864e-05
4,079	Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System	2022	VLDB	6.9188052e-05
4,728	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB	6.5360246e-05
5,301	Babelfish: Efficient Execution of Polyglot Queries	2022	VLDB	6.2750553e-05
6,121	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.9688569e-05
6,314	YeSQL: "You extend SQL" with Rich and Highly Performant User-Defined Functions in Relational Databases	2022	VLDB	5.9157364e-05
6,628	ConnectorX: Accelerating Data Loading From Databases to Dataframes	2022	VLDB	5.8197537e-05
7,502	Decentralized Actor Scheduling and Reference-based Storage in Xorbits: a Native Scalable Data Science Engine	2025	VLDB	5.6029996e-05
8,096	Efficient Execution of User-Defined Functions in SQL Queries	2023	VLDB	5.4875738e-05
8,465	Predicate Pushdown for Data Science Pipelines	2023	SIGMOD	5.4194578e-05
9,415	The Key to Effective UDF Optimization: Before Inlining, First Perform Outlining	2025	VLDB	5.2742285e-05
9,456	QURE: AI-Assisted and Automatically Verified UDF Inlining	2025	SIGMOD	5.2643416e-05
10,074	Dias: Dynamic Rewriting of Pandas Code	2024	SIGMOD	5.1613298e-05
11,150	Proactive Resume and Pause of Resources for Microsoft Azure SQL Database Serverless	2024	SIGMOD	5.093636e-05
11,235	SplitDF: Splitting Dataframes for Memory-Efficient Data Analysis	2024	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
894	Froid: Optimization of Imperative Programs in a Relational Database	2018	VLDB	0.00013367658
1,261	Rewriting Procedures for Batched Bindings	2008	VLDB	0.0001143305
1,279	AI Meets AI: Leveraging Query Executions to Improve Index Recommendations	2019	SIGMOD	0.00011361878
1,431	Towards Scalable Dataframe Systems	2020	VLDB	0.00010807221
1,675	Garlic: A New Flavor of Federated Query Processing for DB2	2002	SIGMOD	0.00010035333
2,477	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.5239378e-05
2,822	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	8.0898536e-05
2,847	AIDA - Abstraction for Advanced In-Database Analytics	2018	VLDB	8.0583142e-05
2,939	Extracting Equivalent SQL from Imperative Code in Database Applications	2016	SIGMOD	7.9395908e-05
2,941	Automatic Partitioning of Database Applications	2012	VLDB	7.937125e-05
3,618	The Myria Big Data Management and Analytics System and Cloud Service	2017	CIDR	7.2523695e-05
3,900	Sloth: Being Lazy is a Virtue (When Issuing Database Queries)	2014	SIGMOD	7.0361352e-05
6,781	Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation	2021	VLDB	5.7764885e-05
7,297	DBridge: Translating Imperative Code to SQL	2017	SIGMOD	5.6525237e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,666	HippogriffDB: Balancing I/O and GPU Bandwidth in Big Data Analytics	2016	VLDB
2	4,728	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB
3	6,481	MotherDuck: DuckDB in the cloud and in the client	2024	CIDR
4	1,768	Tuplex: Data Science in Python at Native Code Speed	2021	SIGMOD
5	3,838	Cloud Analytics Benchmark	2023	VLDB
6	5,791	Accelerating Python UDFs in Vectorized Query Execution	2022	CIDR
7	9,567	When sweet and cute isn't enough anymore: Solving scalability issues in Python Pandas with Grizzly	2020	CIDR
8	4,292	Putting Pandas in a Box	2021	CIDR
9	1,431	Towards Scalable Dataframe Systems	2020	VLDB
10	939	Starling: A Scalable Query Engine on Cloud Functions	2020	SIGMOD