SplitDF: Splitting Dataframes for Memory-Efficient Data Analysis

Summary: Introduce “splitting”: lossless join decomposition adding join keys to reduce dataframe redundancy while preserving a unified tabular view, no FD discovery required. SplitDF (Ibis/DuckDB) with automated SplitGen (Velox) achieves 19–61% memory savings with minimal API change. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13637
Venue: VLDB
Year: 2024
Pagerank: 5.093636e-05
Overall Rank: 11,235 | 22.92%
DOI: 10.14778/3665844.3665849

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

1. Aarati Kakaraparthy (University of Wisconsin)
2. Jignesh M. Patel (Carnegie Mellon University)

BibTeX Citation

@article{kakaraparthy_vldb24,
        title = {{SplitDF: Splitting Dataframes for Memory-Efficient Data Analysis}},
        author = {Kakaraparthy, Aarati and Patel, Jignesh M.},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {9},
        pages = {2175--2184},
        doi = {10.14778/3665844.3665849},
        url = {https://doi.org/10.14778/3665844.3665849},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 0 of 0 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
66	The Snowflake Elastic Data Warehouse	2016	SIGMOD	0.00038561587
103	DuckDB: an Embeddable Analytical Database	2019	SIGMOD	0.00034161428
159	CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies	2004	SIGMOD	0.00028129426
618	A Hybrid Approach to Functional Dependency Discovery	2016	SIGMOD	0.00015711835
729	Functional Dependency Discovery: An Experimental Evaluation of Seven Algorithms	2015	VLDB	0.00014539421
1,374	Data Profiling with Metanome	2015	VLDB	0.00010986078
1,431	Towards Scalable Dataframe Systems	2020	VLDB	0.00010807221
1,500	Efficient Discovery of Approximate Dependencies	2018	VLDB	0.00010561098
1,757	Velox: Meta's Unified Execution Engine	2022	VLDB	9.8166984e-05
2,269	GORDIAN: Efficient and Scalable Discovery of Composite Keys	2006	VLDB	8.8328424e-05
2,651	Magpie: Python at Speed and Scale using Cloud Backends	2021	CIDR	8.2918086e-05
3,441	A Statistical Perspective on Discovering Functional Dependencies in Noisy Data	2020	SIGMOD	7.4138323e-05
4,728	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB	6.5360246e-05
9,567	When sweet and cute isn't enough anymore: Solving scalability issues in Python Pandas with Grizzly	2020	CIDR	5.2528121e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	4,456	AdaptDB: Adaptive Partitioning for Distributed Joins	2017	VLDB
2	8,232	Adaptive Factorization Using Linear-Chained Hash Tables	2025	CIDR
3	2,000	Data Management for Data Science: Towards Embedded Analytics	2020	CIDR
4	4,728	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB
5	10,662	Data Chunk Compaction in Vectorized Execution	2025	SIGMOD
6	4,079	Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System	2022	VLDB
7	1,431	Towards Scalable Dataframe Systems	2020	VLDB
8	10,888	Saving Private Hash Join	2025	VLDB
9	6,628	ConnectorX: Accelerating Data Loading From Databases to Dataframes	2022	VLDB
10	9,069	DQDF: Data-Quality-Aware Dataframes	2022	VLDB