DBScholar

Back to papers

Weld: A Common Runtime for High Performance Data Analytics

Summary: Weld presents a common data-parallel IR and runtime to optimize across disjoint libraries (SQL, ML, graph, array), eliminating expensive cross-library data movement. By operator fusion and whole-workflow code generation, Weld plugs into Spark/TensorFlow/NumPy/Pandas and yields up to 30× end-to-end speedups. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID
290
Venue
CIDR
Year
2017
Pagerank
0.00011578425
Overall Rank
1,229 | 91.57%
DOI
-

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{palkar_cidr17,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '17},
        title = {{Weld: A Common Runtime for High Performance Data Analytics}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Palkar, Shoumik and Thomas, James J. and Shanbhag, Anil and Narayanan, Deepak and Pirk, Holger and Schwarzkopf, Malte and Amarasinghe, Saman and Zaharia, Matei},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 37 of 37 citing papers.

Rank Citing Paper Year Venue Pagerank
649 Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask 2018 VLDB 0.00015320656
1,569 HELIX: Holistic Optimization for Accelerating Iterative Machine Learning 2019 VLDB 0.00010335423
1,756 SystemDS: A Declarative Machine Learning System for the End-to-End Data Science Lifecycle 2020 CIDR 9.8172465e-05
1,824 Photon: A Fast Query Engine for Lakehouse Systems 2022 SIGMOD 9.6734544e-05
2,239 An Intermediate Representation for Optimizing Machine Learning Pipelines 2019 VLDB 8.8875753e-05
2,316 Evaluating End-to-End Optimization for Data Analytics Applications in Weld 2018 VLDB 8.7596739e-05
3,205 On Optimizing Operator Fusion Plans for Large-Scale Machine Learning in SystemML 2018 VLDB 7.6386536e-05
3,253 Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics 2021 VLDB 7.5936939e-05
3,351 RHEEM: Enabling Cross-Platform Data Processing - May The Big Data Be With You! - 2018 VLDB 7.4937347e-05
3,682 In-RDBMS Hardware Acceleration of Advanced Analytics 2018 VLDB 7.2035518e-05
4,215 Designing an Open Framework for Query Optimization and Compilation 2022 VLDB 6.8275676e-05
4,599 Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications 2018 SIGMOD 6.6122875e-05
5,215 The NebulaStream Platform: Data and Application Management for the Internet of Things 2020 CIDR 6.3124972e-05
5,301 Babelfish: Efficient Execution of Polyglot Queries 2022 VLDB 6.2750553e-05
5,765 Charting the Design Space of Query Execution using VOILA 2021 VLDB 6.0953705e-05
5,839 Doing More with Less: Characterizing Dataset Downsampling for AutoML 2021 VLDB 6.0699037e-05
5,991 Iterative Query Processing based on Unified Optimization Techniques 2019 SIGMOD 6.0171458e-05
6,250 Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines 2021 CIDR 5.9418015e-05
6,253 Declarative Sub-Operators for Universal Data Processing 2023 VLDB 5.940599e-05
6,263 Database Technology for the Masses: Sub-Operators as First-Class Entities 2021 VLDB 5.936052e-05
6,314 YeSQL: "You extend SQL" with Rich and Highly Performant User-Defined Functions in Relational Databases 2022 VLDB 5.9157364e-05
6,614 Mitigating the Impedance Mismatch between Prediction Query Execution and Database Engine 2025 SIGMOD 5.8216658e-05
7,311 Lachesis: Automatic Partitioning for UDF-Centric Analytics 2021 VLDB 5.6491618e-05
7,774 Building Advanced SQL Analytics From Low-Level Plan Operators 2021 SIGMOD 5.5468469e-05
8,057 Architecting a Query Compiler for Spatial Workloads 2020 SIGMOD 5.4982831e-05
8,096 Efficient Execution of User-Defined Functions in SQL Queries 2023 VLDB 5.4875738e-05
8,438 Flare & Lantern: Efficiently Swapping Horses Midstream 2019 VLDB 5.4251194e-05
8,638 Towards A Polyglot Framework for Factorized ML 2021 VLDB 5.395289e-05
9,572 PlinyCompute: A Platform for High-Performance, Distributed, Data-Intensive Tool Development 2018 SIGMOD 5.2528121e-05
9,742 Unified Data Analytics: State-of-the-art and Open Problems 2022 VLDB 5.227679e-05
9,986 Thriving in the No Man’s Land between Compilers and Databases 2019 CIDR 5.1832796e-05
10,040 SQL Engines Excel at the Execution of Imperative Programs 2024 VLDB 5.1723558e-05
10,072 Query Compilation Without Regrets 2024 SIGMOD 5.1624689e-05
10,200 Automating Database-Native Function Code Synthesis with LLMs 2026 SIGMOD 5.093636e-05
10,232 EncoderForge: Generating Efficient SQL for Encoders in Machine Learning Inference Pipelines 2026 SIGMOD 5.093636e-05
10,630 Cephalopod - Virtual Data Model Composition through Partial Query Translation 2025 CIDR 5.093636e-05
11,120 Welding Natural Language Queries to Analytics IRs with LLMs 2024 CIDR 5.093636e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 12 of 12 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
20 Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud 2012 VLDB 0.00056944564
23 Efficiently Compiling Efficient Query Plans for Modern Hardware 2011 VLDB 0.00054886415
24 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00054865648
106 The MADlib Analytics Library or MAD Skills, the SQL 2012 VLDB 0.00033539462
534 Building Efficient Query Engines in a High-Level Language 2014 VLDB 0.00017046514
759 The case against specialized graph analytics engines 2015 CIDR 0.00014273591
1,070 NoDB: Efficient Query Execution on Raw Data Files 2012 SIGMOD 0.0001232307
1,376 A Demonstration of SciDB: A Science-Oriented DBMS 2009 VLDB 0.00010973814
1,442 An Architecture for Compiling UDF-centric Workflows 2015 VLDB 0.00010778486
1,469 Voodoo - A Vector Algebra for Portable Database Performance on Modern Hardware 2016 VLDB 0.00010678751
1,966 GraphMat: High performance graph analytics made productive 2015 VLDB 9.3844743e-05
2,433 Vertexica: Your Relational Friend for Graph Analytics! 2014 VLDB 8.5869161e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 12,170 Palette: Enabling Scalable Analytics for Big-Memory, Multicore Machines 2014 SIGMOD
2 9,742 Unified Data Analytics: State-of-the-art and Open Problems 2022 VLDB
3 2,018 tf.data: A Machine Learning Data Processing Framework 2021 VLDB
4 2,717 Implicit Parallelism through Deep Language Embedding 2015 SIGMOD
5 6,850 Bridging the Gap Between HPC and Big Data Frameworks 2017 VLDB
6 8,438 Flare & Lantern: Efficiently Swapping Horses Midstream 2019 VLDB
7 2,196 Spinning Fast Iterative Data Flows 2012 VLDB
8 11,120 Welding Natural Language Queries to Analytics IRs with LLMs 2024 CIDR
9 1,442 An Architecture for Compiling UDF-centric Workflows 2015 VLDB
10 2,316 Evaluating End-to-End Optimization for Data Analytics Applications in Weld 2018 VLDB