TPCx-AI under the Microscope: A Benchmarking Debt Analysis
Summary: Dissects TPCx-AI for "benchmarking debt": kit/spec divergences, data errors, weak metrics, and workload artifacts that distort what is actually being measured. Shows these issues can skew training/serving by 350x/800x and that fixing them yields up to 3.8x higher end-to-end throughput. (summarized by gpt-5.4-mini on Apr 12 2026)
Incoming Non-self Citations Over Time
No non-self incoming citations found for this paper in this database.
Authors
- 1. Ilin Tolovski (Hasso Plattner Institute; University of Potsdam)
- 2. Philipp Hildebrandt (Hasso Plattner Institute; University of Potsdam)
- 3. Khuzaima Daudjee (University of Waterloo)
- 4. Tilmann Rabl (Hasso Plattner Institute; University of Potsdam)
BibTeX Citation
@article{tolovski_vldb26,
title = {{TPCx-AI under the Microscope: A Benchmarking Debt Analysis}},
author = {Tolovski, Ilin and Hildebrandt, Philipp and Daudjee, Khuzaima and Rabl, Tilmann},
journal = {PVLDB},
series = {{VLDB} '26},
volume = {19},
number = {6},
pages = {1305--1318},
doi = {10.14778/3797919.3797936},
url = {https://doi.org/10.14778/3797919.3797936},
year = {2026}
}Incoming Citations (Sorted by Pagerank)
Showing 0 of 0 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 5 of 5 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 2,202 | Quantifying TPC-H Choke Points and Their Optimizations | 2020 | VLDB | 8.9639459e-05 |
| 5,040 | TPCx-AI - An Industry Standard Benchmark for Artificial Intelligence and Machine Learning Systems | 2023 | VLDB | 6.390395e-05 |
| 5,704 | Optimizing Data Pipelines for Machine Learning in Feature Stores | 2023 | VLDB | 6.1146371e-05 |
| 6,614 | Mitigating the Impedance Mismatch between Prediction Query Execution and Database Engine | 2025 | SIGMOD | 5.8216658e-05 |
| 7,502 | Decentralized Actor Scheduling and Reference-based Storage in Xorbits: a Native Scalable Data Science Engine | 2025 | VLDB | 5.6029996e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 9,777 | PBench: Workload Synthesizer with Real Statistics for Cloud Analytics Benchmarking | 2025 | VLDB |
| 2 | 226 | OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases | 2014 | VLDB |
| 3 | 14,454 | Database System Performance Measurement | 1986 | SIGMOD |
| 4 | 3,994 | Generating Databases for Query Workloads | 2010 | VLDB |
| 5 | 3,173 | Why You Should Run TPC-DS:A Workload Analysis | 2007 | VLDB |
| 6 | 9,433 | FEBench: A Benchmark for Real-Time Relational Data Feature Extraction | 2023 | VLDB |
| 7 | 2,823 | Query Processing on Tensor Computation Runtimes | 2022 | VLDB |
| 8 | 2,408 | Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet | 2024 | VLDB |
| 9 | 8,576 | A Study of Database Performance Sensitivity to Experiment Settings | 2022 | VLDB |
| 10 | 5,040 | TPCx-AI - An Industry Standard Benchmark for Artificial Intelligence and Machine Learning Systems | 2023 | VLDB |