Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search

Summary: Auto-Pipeline synthesizes multi-step data-cleaning pipelines from input tables and a target schema, rather than examples. It exploits implicit constraints (FDs, keys) to guide reinforcement learning and search, recovering ~70% of real pipelines up to 10 steps. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: 12619
Venue: VLDB
Year: 2021
Pagerank: 6.4689177e-05
Overall Rank: 4,875 | 66.56%
DOI: 10.14778/3476249.3476303

Incoming Non-self Citations Over Time

Authors

1. Junwen Yang (University of Chicago)
2. Yeye He (Microsoft)
3. Surajit Chaudhuri (Microsoft)

BibTeX Citation

@article{yang_vldb21,
        title = {{Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search}},
        author = {Yang, Junwen and He, Yeye and Chaudhuri, Surajit},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {11},
        pages = {2563--2575},
        doi = {10.14778/3476249.3476303},
        url = {https://doi.org/10.14778/3476249.3476303},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,990	Explaining Dataset Changes for Semantic Data Versioning with Explain-Da-V	2023	VLDB	6.4105738e-05
5,403	Auto-Tables: Synthesizing Multi-Step Transformations to Relationalize Tables without Using Examples	2023	VLDB	6.2306083e-05
8,177	HAIPipe: Combining Human-generated and Machine-generated Pipelines for Data Preparation	2023	SIGMOD	5.4730821e-05
8,797	Auto-Formula: Recommend Formulas in Spreadsheets using Contrastive Learning for Table Representations	2024	SIGMOD	5.3702298e-05
9,380	Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence	2025	VLDB	5.2755515e-05
10,202	BAT: Target-Instance-Free Data Preparation Synthesis via LLM-Driven Tree Search	2026	SIGMOD	5.093636e-05
10,457	FlowPilot: A Suggestion System for Designing Scientific Workflows	2026	SIGMOD	5.093636e-05
11,309	LucidScript: Bottom-up Standardization for Data Preparation	2024	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 12 of 12 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
517	Query by Output	2009	SIGMOD	0.00017169735
618	A Hybrid Approach to Functional Dependency Discovery	2016	SIGMOD	0.00015711835
729	Functional Dependency Discovery: An Experimental Evaluation of Seven Algorithms	2015	VLDB	0.00014539421
963	The Data Civilizer System	2017	CIDR	0.00012935145
1,160	BlinkFill: Semi-supervised Programming By Example for Syntactic String Transformations	2016	VLDB	0.00011883163
1,165	Foofah: Transforming Data By Example	2017	SIGMOD	0.00011860616
1,866	Predictive Interaction for Data Transformation	2015	CIDR	9.5914583e-05
2,744	Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks	2020	SIGMOD	8.1781662e-05
2,978	Transform-Data-by-Example (TDE): An Extensible Search Engine for Data Transformations	2018	VLDB	7.9030989e-05
2,980	Auto-Join: Joining Tables by Leveraging Transformations	2017	VLDB	7.8975073e-05
4,412	Auto-Transform: Learning-to-Transform by Patterns	2020	VLDB	6.7168614e-05
7,078	Transform-Data-by-Example (TDE): Extensible Data Transformation in Excel	2018	SIGMOD	5.7095504e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	2,980	Auto-Join: Joining Tables by Leveraging Transformations	2017	VLDB
2	5,901	A Scalable AutoML Approach Based on Graph Neural Networks	2022	VLDB
3	4,412	Auto-Transform: Learning-to-Transform by Patterns	2020	VLDB
4	2,744	Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks	2020	SIGMOD
5	9,629	Auto-BI: Automatically Build BI-Models Leveraging Local Join Prediction and Global Schema Graph	2023	VLDB
6	8,914	CtxPipe: Context-aware Data Preparation Pipeline Construction for Machine Learning	2024	SIGMOD
7	6,074	Automatic Data Acquisition for Deep Learning	2021	VLDB
8	8,177	HAIPipe: Combining Human-generated and Machine-generated Pipelines for Data Preparation	2023	SIGMOD
9	9,380	Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence	2025	VLDB
10	5,403	Auto-Tables: Synthesizing Multi-Step Transformations to Relationalize Tables without Using Examples	2023	VLDB