Making Data Engineering Declarative

Summary: Introduces Delta Live Tables: declarative pipelines that model ELT as a DAG of incrementally maintained, full‑SQL materialized views (vs CTAS), enabling automated incrementalization and holistic, pipeline‑aware optimization. Adds CI/CD‑aware MV evolution, autoscaling/orchestration to reduce cost/latency, plus declarative correctness monitoring and policies for handling dirty source data. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 502
Venue: CIDR
Year: 2023
Pagerank: 5.3194157e-05
Overall Rank: 9,120 | 37.43%
DOI: 10.1145/1234567890

Incoming Non-self Citations Over Time

Authors

1. Michael Armbrust (Databricks)
2. Bilal Aslam (Databricks)
3. Yingyi Bu (Databricks)
4. Sourav Chatterji (Databricks)
5. Yuhong Chen (Databricks)
6. Yijia Cui (Databricks)
7. Vuk Ercegovac (Databricks)
8. Ali Ghodsi (Databricks)
9. Rahul Govind (Databricks)
10. Aakash Japi (Databricks)
11. Kiavash Kianfar (Databricks)
12. Eun-Gyu Kim (Databricks)
13. Xi Liang (Databricks)
14. Paul Lappas (Databricks)
15. Jon Mio (Databricks)
16. Mukul Murthy (Databricks)
17. Supun Nakandala (Databricks)
18. Andreas Neumann (Databricks)
19. Yannis Papakonstantinou (Databricks)
20. Nitin Sharma (Databricks)
21. Yannis Sismanis (Databricks)
22. Justin Tang (Databricks)
23. Joseph Torres (Databricks)
24. Reynold Xin (Databricks)
25. Min Yang (Databricks)
26. Li Zhang (Databricks)

BibTeX Citation

@inproceedings{armbrust_cidr23,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '23},
        title = {{Making Data Engineering Declarative}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Armbrust, Michael and Aslam, Bilal and Bu, Yingyi and Chatterji, Sourav and Chen, Yuhong and Cui, Yijia and Ercegovac, Vuk and Ghodsi, Ali and Govind, Rahul and Japi, Aakash and Kianfar, Kiavash and Kim, Eun-Gyu and Liang, Xi and Lappas, Paul and Mio, Jon and Murthy, Mukul and Nakandala, Supun and Neumann, Andreas and Papakonstantinou, Yannis and Sharma, Nitin and Sismanis, Yannis and Tang, Justin and Torres, Joseph and Xin, Reynold and Yang, Min and Zhang, Li},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
8,467	LST-Bench: Benchmarking Log-Structured Tables in the Cloud	2024	SIGMOD	5.4187125e-05

Outgoing Citations (Sorted by Pagerank)

Showing 3 of 3 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
24	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00054865648
520	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB	0.00017136828
1,138	Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics	2021	CIDR	0.00012023643

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,350	Automating Large-Scale Data Quality Verification	2018	VLDB
2	9,801	Breaking the Chains: On Declarative Data Analysis and Data Independence in the Big Data Era	2014	VLDB
3	6,004	Data Ingestion for the Connected World	2017	CIDR
4	201	Declarative Data Cleaning: Language, Model, and Algorithms	2001	VLDB
5	11,017	Streaming View: An Efficient Data Processing Engine for Modern Real-time Data Warehouse of Alibaba Cloud	2025	VLDB
6	10,618	ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines	2026	VLDB
7	1,190	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
8	520	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB
9	9,050	Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables	2025	SIGMOD
10	10,121	End-to-End Declarative Data Analytics: Co-designing Engines, Interfaces, and Cloud Infrastructure	2026	CIDR