Maestro: Automatic Generation of Comprehensive Benchmarks for Question Answering Over Knowledge Graphs

Summary: Maestro automates generation of QA benchmarks for any knowledge graph (with optional text corpus). It guarantees coverage of NL question and query properties documented in the literature and emits high-quality natural-language questions, enabling reusable, evolving benchmarks across KGs. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6742
Venue: SIGMOD
Year: 2023
Pagerank: 6.0247307e-05
Overall Rank: 5,969 | 59.05%
DOI: 10.1145/3589322

Incoming Non-self Citations Over Time

Authors

1. Abdelghny Orogat (Carleton University)
2. Ahmed El-Roby (Carleton University)

BibTeX Citation

@inproceedings{orogat_sigmod23,
        title = {{Maestro: Automatic Generation of Comprehensive Benchmarks for Question Answering Over Knowledge Graphs}},
        author = {Orogat, Abdelghny and El-Roby, Ahmed},
        series = {{SIGMOD} '23},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3589322},
        url = {https://dl.acm.org/doi/10.1145/3589322},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
10,663	Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs	2025	SIGMOD	5.093636e-05
10,909	LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation	2025	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 7 of 7 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
65	Freebase: A Collaboratively Created Graph Database For Structuring Human Knowledge	2008	SIGMOD	0.00038697603
530	An Analytical Study of Large SPARQL Query Logs	2018	VLDB	0.0001709169
3,699	KBQA: Learning Question Answering over QA Corpora and Knowledge Bases	2017	VLDB	7.1882315e-05
5,671	Question Answering Over Knowledge Graphs: Question Understanding Via Template Decomposition	2018	VLDB	6.1262839e-05
9,286	SmartBench: Demonstrating Automatic Generation of Comprehensive Benchmarks for Question Answering Over Knowledge Graphs	2022	VLDB	5.2917932e-05
11,694	CBench: Towards Better Evaluation of Question Answering Over Knowledge Graphs	2021	VLDB	5.093636e-05
13,457	CBench: Demonstrating Comprehensive Evaluation of Question Answering Systems over Knowledge Graphs Through Deep Analysis of Benchmarks	2021	VLDB	-

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	4,372	Schemaless and Structureless Graph Querying	2014	VLDB
2	10,437	Chatty-KG: A Multi-Agent AI System for On-Demand Conversational Question Answering over Knowledge Graphs	2026	SIGMOD
3	3,981	Natural Language Question Answering over RDF — A Graph Data Driven Approach	2014	SIGMOD
4	10,663	Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs	2025	SIGMOD
5	3,699	KBQA: Learning Question Answering over QA Corpora and Knowledge Bases	2017	VLDB
6	5,671	Question Answering Over Knowledge Graphs: Question Understanding Via Template Decomposition	2018	VLDB
7	10,023	A Universal Question-Answering Platform for Knowledge Graphs	2023	SIGMOD
8	13,457	CBench: Demonstrating Comprehensive Evaluation of Question Answering Systems over Knowledge Graphs Through Deep Analysis of Benchmarks	2021	VLDB
9	11,694	CBench: Towards Better Evaluation of Question Answering Over Knowledge Graphs	2021	VLDB
10	9,286	SmartBench: Demonstrating Automatic Generation of Comprehensive Benchmarks for Question Answering Over Knowledge Graphs	2022	VLDB