Data Lake Management: Challenges and Opportunities

Summary: Data lakes create new management problems—dataset discovery and evolving needs for extraction, cleaning, integration, versioning, and metadata. This tutorial surveys state-of-the-art approaches, challenges, and opportunities unique to data-lake ecosystems for guiding future research. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11919
Venue: VLDB
Year: 2019
Pagerank: 0.00015197838
Overall Rank: 936 | 93.50%
DOI: 10.14778/3352063.3352116

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 40 of 40 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,088	Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes	2024	VLDB	0.00014158762
1,643	Finding Related Tables in Data Lakes for Interactive Data Science	2020	SIGMOD	0.00011031534
2,842	Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning	2023	VLDB	8.0366354e-05
3,001	SANTOS: Relationship-based Semantic Table Union Search	2023	SIGMOD	7.739698e-05
3,003	Chorus: Foundation Models for Unified Data Discovery and Exploration	2024	VLDB	7.7358219e-05
3,105	GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization	2024	VLDB	7.5567226e-05
4,746	Slice Tuner: A Selective Data Acquisition Framework for Accurate and Fair Machine Learning Models	2021	SIGMOD	5.9446518e-05
4,860	Integrating Data Lake Tables	2023	VLDB	5.867964e-05
4,864	Data-Sharing Markets: Model, Protocol, and Algorithms to Incentivize the Formation of Data-Sharing Consortia	2023	SIGMOD	5.8641082e-05
4,962	Doing More with Less: Characterizing Dataset Downsampling for AutoML	2021	VLDB	5.7979872e-05
5,283	Explaining Dataset Changes for Semantic Data Versioning with Explain-Da-V	2023	VLDB	5.5843052e-05
5,538	Data-Driven Domain Discovery for Structured Datasets	2020	VLDB	5.4520759e-05
5,805	Discovering Related Data At Scale	2021	VLDB	5.3197639e-05
5,929	SchemaPile: A Large Collection of Relational Database Schemas	2024	SIGMOD	5.2635297e-05
6,086	Subgraph Matching over Graph Federation	2022	VLDB	5.2157921e-05
6,432	RONIN: Data Lake Exploration	2021	VLDB	5.0571585e-05
6,526	Data Collection and Quality Challenges for Deep Learning	2020	VLDB	5.0219175e-05
7,642	Cross Modal Data Discovery over Structured and Unstructured Data Lakes	2023	VLDB	4.6856127e-05
7,930	Entity Resolution On-Demand	2022	VLDB	4.6089604e-05
8,121	LakeBench: A Benchmark for Discovering Joinable and Unionable Tables in Data Lakes	2024	VLDB	4.5771143e-05
8,607	Unity Catalog: Open and Universal Governance for the Lakehouse and Beyond	2025	SIGMOD	4.4810982e-05
8,725	OneProvenance: Efficient Extraction of Dynamic Coarse-Grained Provenance From Database Query Event Logs	2023	VLDB	4.453957e-05
8,970	DataLoom: Simplifying Data Loading with LLMs	2024	VLDB	4.4154448e-05
9,700	Towards Functional Decomposition of Storage Formats	2025	CIDR	4.2967256e-05
9,774	EquiTensors: Learning Fair Integrations of Heterogeneous Urban Data	2021	SIGMOD	4.2815042e-05
9,960	QueryArtisan: Generating Data Manipulation Codes for Ad-hoc Analysis in Data Lakes	2025	VLDB	4.2254157e-05
10,142	AutoDDG: Automated Dataset Description Generation using Large Language Models	2026	SIGMOD	4.1905499e-05
10,197	Qualitative Join Discovery in Data Lakes using Examples	2026	SIGMOD	4.1905499e-05
10,519	Table Overlap Estimation through Graph Embeddings	2025	SIGMOD	4.1905499e-05
10,653	OpenForge: Probabilistic Metadata Integration	2025	VLDB	4.1905499e-05
10,803	A Demonstration of QueryArtisan: Real-Time Data Lake Analysis via Dynamically Generated Data Manipulation Code	2025	VLDB	4.1905499e-05
10,808	GraphAr: An Efficient Storage Scheme for Graph Data in Data Lakes	2025	VLDB	4.1905499e-05
10,833	Sort it Like You Mean It: Discovering Semantically Interesting Attribute Augmentations to Sort Tables	2025	VLDB	4.1905499e-05
10,858	LiquidCache: Efficient Pushdown Caching for Cloud-Native Data Analytics	2025	VLDB	4.1905499e-05
10,899	Towards an Objective Metric for Data Value Through Relevance	2024	CIDR	4.1905499e-05
10,954	Determining the Largest Overlap between Tables	2024	SIGMOD	4.1905499e-05
11,009	FusionQuery: On-demand Fusion Queries over Multi-source Heterogeneous Data	2024	VLDB	4.1905499e-05
11,066	Searching Data Lakes for Nested and Joined Data	2024	VLDB	4.1905499e-05
11,079	KGFabric: A Scalable Knowledge Graph Warehouse for Enterprise Data Interconnection	2024	VLDB	4.1905499e-05
11,423	Detecting Layout Templates in Complex Multiregion Files	2022	VLDB	4.1905499e-05

Outgoing Citations (Sorted by Pagerank)

Showing 26 of 26 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
108	WebTables: Exploring the Power of Tables on the Web	2008	VLDB	0.00048345996
394	Big Data Integration	2013	VLDB	0.0002447017
420	InfoGather: Entity Augmentation and Attribute Discovery By Holistic Matching with Web Tables	2012	SIGMOD	0.00023700634
519	Data Integration for the Relational Web	2009	VLDB	0.00021148006
609	Goods: Organizing Google's Datasets	2016	SIGMOD	0.00019223217
814	Finding Related Tables	2012	SIGMOD	0.00016298739
830	Guided Data Repair	2011	VLDB	0.00016125759
1,179	Table Union Search on Open Data	2018	VLDB	0.00013458551
1,185	JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes	2019	SIGMOD	0.00013432692
1,274	The Data Civilizer System	2017	CIDR	0.00012869297
1,278	DataHub: Collaborative Data Science & Dataset Version Management at Scale	2015	CIDR	0.00012851949
1,370	Answering Table Queries on the Web using Column Keywords	2012	VLDB	0.00012339543
1,502	Discovering Queries based on Example Tuples	2014	SIGMOD	0.00011614522
1,565	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00011336478
1,862	Bootstrapping Pay-As-You-Go Data Integration Systems	2008	SIGMOD	0.00010292134
2,081	Sample-Driven Schema Mapping	2012	SIGMOD	9.58883e-05
2,142	LSH Ensemble: Internet-Scale Domain Search	2016	VLDB	9.4461701e-05
2,267	Ground: A Data Context Service	2017	CIDR	9.1554363e-05
2,638	Schema Extraction for Tabular Data on the Web	2013	VLDB	8.3995765e-05
2,737	Open Data Integration	2018	VLDB	8.2053894e-05
3,283	Constance: An Intelligent Data Lake System	2016	SIGMOD	7.27606e-05
3,693	Navigating the Data Lake with DATAMARAN: Automatically Extracting Structure from Log Datasets	2018	SIGMOD	6.8326441e-05
4,105	Extracting Databases from Dark Data with DeepDive	2016	SIGMOD	6.4409563e-05
4,802	CLAMS: Bringing Quality to Data Lakes	2016	SIGMOD	5.9064959e-05
5,793	Interactive Navigation of Open Data Linkages	2017	VLDB	5.3240605e-05
6,814	Error Diagnosis and Data Profiling with Data X-Ray	2015	VLDB	4.9130627e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
7,016	LLM for Data Management	2024	VLDB	4.8561622e-05
7,116	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	4.8219732e-05
9,607	Polyglot Data Management: State of the Art & Open Challenges	2022	VLDB	4.3136057e-05
1,643	Finding Related Tables in Data Lakes for Interactive Data Science	2020	SIGMOD	0.00011031534
1,356	Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics	2021	CIDR	0.00012409986
1,422	Data Management Challenges in Production Machine Learning	2017	SIGMOD	0.00012050431
1,534	Data Management in Machine Learning: Challenges, Techniques, and Systems	2017	SIGMOD	0.00011462072
1,831	Data Wrangling: The Challenging Journey from the Wild to the Lake	2015	CIDR	0.0001037061
10,840	Data Discovery in Data Lakes: Operations, Indexes, Systems	2025	VLDB	4.1905499e-05
13,290	The Challenge of Building Effective Data Lakes	2020	SIGMOD	-