Compiled and vectorized evaluation in columnar engines

Thu, 18 Jun 2026 00:00:00 +0000

How do modern columnar database management systems process a query like this?

SELECT SUM(t.c) FROM t WHERE t.a > 5 AND t.b < 7;

Columnar systems today are generally either use compiling query engines or vectorized query engines.

Compiling query engines generate code that executes the query, allowing tools like LLVM to optimize queries “end to end.”
Vectorized query engines use hand-optimized subroutines that process large chunks of single columns at a time, then combine the intermediate results together to form the final output.

First, we’ll look at the strategies used by both engine types, and measure their performance over random data (both a > 5 and b < 7 will have 50% selectivity – more on that later). Then, we’ll analyze some of the surprises we find along the way. This blog post is a microcosm of the excellent 2018 VLDB paper, “Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask.” by Kersten et al.

Compiling engine strategy

Most compiling engines, like Amazon Redshift or Tableau Hyper, follow the code generation technique described in Thomas Neumann’s 2011 paper, “Efficiently compiling efficient query plans for modern hardware.” The code generated for a query like ours would be pretty close to what you might write by hand. Ignoring all the scaffolding required to read data, handle multiple chunks, etc., a compiling execution engine might generate code like this:

pub fn compiled_engine(a: &[i32], b: &[i32], c: &[f32]) -> f32 {
    a.iter()
        .zip(b)
        .zip(c)
        .filter_map(|((a_el, b_el), c_el)| {
            (*a_el > 5 && *b_el < 7).then_some(c_el)
        }).sum()
}

If you check the generated assembly (make sure to use optimizations and set the target CPU to something modern!), you’ll see that Rust produces a compact, short-circuiting loop with SIMD instructions for the summation. At first glance, it also seems like this is just about the optimal code we could write to answer this query (aside from system-level changes like parallelism or index structures).

Doesn't Rust do bounds checking?

You might wonder if Rust’s iterators or bounds checking make this code slower than the “C style” version using a for-loop. In this case, the compiler is smart enough to optimize away the bounds check. You can check yourself if you’d like, using this unsafe code:

pub fn compiled_engine(a: &[i32], b: &[i32], c: &[f32]) -> f32 {
    unsafe {
        let mut accum = 0.0;
        for i in 0..a.len() {
            if *a.get_unchecked(i) > 5 && *b.get_unchecked(i) < 7 {
                accum += *c.get_unchecked(i);
            }
        }
        accum
    }
}

Vectorized engine strategy

Vectorized engines, like Actian Vector, MonetDB, and DuckDB, don’t generate or compile code, but instead use hand-optimized “kernel functions” that operate on large chunks of data at once (primarily following Peter Boncz et al.’s X100 paper). There are two ways a vectorized engine might handle our query: full evaluation and partial evaluation.

Full evaluation is the most straightforward, and may even seem naive at first. At a high level, the idea is to build one bitmap for a > 5, another bitmap for b < 7, then bitwise-AND the two bitmaps together. Finally, we can iterate over the set bits in the resulting bitmap and sum up the corresponding entries of c.

pub fn vec_engine_full(a: &[i32], b: &[i32], c: &[f32]) -> f32 {
    // kernel 1: compare column with constant, store result in bitmap
    let a_bm = BooleanBuffer::collect_bool(a.len(), |idx| a[idx] > 5);

    // kernel 2: compare column with constant, store result in bitmap
    let b_bm = BooleanBuffer::collect_bool(b.len(), |idx| b[idx] < 7);

    // kernel 3: bitwise AND of two bitmaps
    let res = &a_bm & &b_bm;

    // kernel 4: sum of a vector at positions given by bitmap
    BitIndexIterator::new(res.values(), res.offset(), res.len())
        .map(|idx| c[idx])
        .sum()
}

Here, we use Apache Arrow’s BooleanBuffer to construct and iterate over the bitmaps. At first glance, one might think this code is a significant regression from the compiled code. Not only are we doing “extra work” by testing entries of b that the compiled version gets to skip via short-circuiting, but we’re also allocating two ~125KB data structures. Well, vectorized database engine designers had the same thought, so they also built kernels for partial evaluation.

Partial evaluation avoids the wasted work by building vectors (sometimes called selections or selection vectors) of indexes that track which positions are not filtered out.¹ Once the vector of indexes that pass the a > 5 filter is computed, we can check just those indexes for b < 7, and then compute the sum over c.

pub fn vec_engine_partial(a: &[i32], b: &[i32], c: &[f32]) -> f32 {
    // kernel 1: find indexes of a where the predicate is true
    let sel: Vec<usize> = a
        .iter()
        .enumerate()
        .filter_map(|(idx, a_el)| (*a_el > 5).then_some(idx))
        .collect();

    // kernel 2: narrow input indexes by predicate
    let sel: Vec<usize> = sel
        .into_iter()
        .filter_map(|idx| (b[idx] < 7).then_some(idx))
        .collect();

    // kernel 3: sum using indexes
    sel.into_iter().map(|idx| c[idx]).sum()
}

Unfortunately, I could not for the life of me get the compiler to properly vectorize this code², so I had to write the SIMD manually. This is totally fine for a vectorized database, since every kernel is expected to be hand-tuned. You can find the SIMD implementation here. But the SIMD version simply does what the above code does, just more efficiently.

Bakeoff #1: 50% Selectivity

OK, how do our times compare? Let’s evaluate over a million randomly generated rows, generating the data so that a > 5 and b < 7 have 50% selectivity. This benchmark was ran on my laptop, equipped with a i7-1370P.

predicates/compiled
    time:   [4.2265 ms 4.2431 ms 4.2578 ms]
    thrpt:  [234.86 Melem/s 235.68 Melem/s 236.60 Melem/s]

predicates/vec_full
    time:   [852.99 µs 858.36 µs 863.24 µs]
    thrpt:  [1.1584 Gelem/s 1.1650 Gelem/s 1.1723 Gelem/s]

predicates/vec_partial
    time:   [1.4747 ms 1.4793 ms 1.4830 ms]
    thrpt:  [674.32 Melem/s 675.98 Melem/s 678.09 Melem/s]

Wow! It might be quite surprising that the vectorized “full” strategy is not just the fastest, but is the fastest but a wide margin. What’s going on here? Since some of the fastest databases in the world use a compiled engine, we should be quite surprised by this seemingly-poor showing.

Let’s try and figure out why our compiled engine is doing so poorly compared with the vectorized strategies. After all, the compiled code doesn’t create any extra data structures, and the compiled code avoids all wasted work via short-circuiting. We can use perf and top-down analysis to figure out what is going on.

Performance counter stats for 'cargo bench compiled':
 12K  page-faults:u           #  830.127 /sec
 74G  cycles/u                #    5.111 GHz
 15G  instructions/u          #    0.46  insn per cycle ❶
  6G  branches/u              #  462.162 M/sec
  2G  branch-misses/u         #   26.75% of all branches ❹
  TopdownL1 (cpu_core)        #    9.9 %  tma_backend_bound
                              #   37.3 %  tma_bad_speculation ❷
                              #   47.7 %  tma_frontend_bound ❸
                              #    5.0 %  tma_retiring

The first thing we notice is that ❶ instructions per cycle is quite low. This means that our CPU is not doing useful work most of the time; instead, the CPU might be recovering from branch mispredictions, or might be stuck waiting for memory accesses (either data or instruction). Second, when looking at the topdown analysis, we see very high ❸ frontend_bound and ❷ bad_speculation. Since frontend_bound includes instruction misses from branch resteers, and our bad_speculation is also pretty high, we can conclude that the primary bottleneck here is likely branch misprediction – ❹ about 26% of branches are missing mispredicted. Instruction count could also be a problem, but the presence of so much bad speculation is a “red flag” that should catch our eye first.

Why is high bad speculation a red flag?

When bad speculation occurs, it could be counted as either a branch resteer (loading a different set of instructions), or a branch misprediction (undoing the effects of an incorrect speculation). Thus, when bad speculation is high, we don’t really know if the true problem is in the front end or the back end. As Ahmad Yasin puts it in “A Top-Down Method for Performance Analysis and Counters Architecture”:

Having Bad Speculation category at the Top-Level is a key principle in our Top-Down Analysis. It determines the fraction of the workload under analysis that is affected by incorrect execution paths, which in turn dictates the accuracy of observations listed in other categories. Furthermore, this permits nodes at lower levels to make use of some of the many traditional counters, given that most counters in out-of-order CPUs count speculatively. Hence, a high value in Bad Speculation would be interpreted by the user as a “red flag” that need to be investigated first, before looking at other categories. In other words, assuring Bad Speculation is minor not only improves utilization of the available resources, but also increases confidence in metrics reported throughout the hierarchy.

Let’s compare this with a top-down analysis of the vectorized full code (using bitmaps):

Performance counter stats for 'cargo bench bitmap':
 12,260  page-faults:u   #  542.632 /sec
 101G    cycles/u        #    4.485 GHz
 381G    instructions/u  #    8.35  insn per cycle ❶
 63G     branches/u      #    2.829 G/sec
 443M    branch-misses/u #    2.41% of all branches
 TopdownL1 (cpu_core)    #   22.3 %  tma_backend_bound ❸
                         #    6.0 %  tma_bad_speculation
                         #   12.4 %  tma_frontend_bound
                         #   59.2 %  tma_retiring ❷

We can immediately see that our bitmap code is making better use of the CPU. First, instead of executing ½ an instruction per cycle, ❶ we’re now executing over 8 instructions per cycle! Our topdown analysis shows that most of our time is ❷ going to “retiring” instructions, meaning that the instructions are leaving the CPU pipeline normally! Our next biggest bottleneck is ❸ “backend,” means that instructions are waiting on resources (like memory reads) in order to proceed.

It’s important to note that higher instructions per cycle (IPC) is not always better. Higher IPC more instructions are making it through the CPU, but two different techniques may require a different number of instructions! In our case, the bitmap approach requires some extra “work” to be done (bitmaps must be allocated, constructed, compared, and destroyed), whereas the compiled approach does very little extra work. The reason that the bitmap approach performs better is not because it has an higher IPC, but because it does work in a way that is better aligned with what the CPU is expecting.

Bakeoff #2: 0.04% Selectivity

If we want to make the compiled version of the code look as good as possible, we note two things:

The compiled version only gets to avoid work if the a > 5 predicate is false – that is, if the conditional short-circuits.
The compiled version had tons of branch misprediction, so making the branch behavior more consistent should improve performance.

We can maximize the number of short-circuits and make the branch behavior more consistent by lowering the selectivity of a > 5. Keeping everything else the same, we’ll lower the selectivity of a > 5 to 0.04:

predicates/compiled
    time:   [152.77 µs 153.70 µs 154.50 µs]
    thrpt:  [6.4723 Gelem/s 6.5063 Gelem/s 6.5456 Gelem/s]

predicates/vec_full
    time:   [401.64 µs 403.45 µs 405.38 µs]
    thrpt:  [2.4668 Gelem/s 2.4786 Gelem/s 2.4898 Gelem/s]

predicates/vec_partial
    time:   [195.99 µs 197.01 µs 197.92 µs]
    thrpt:  [5.0524 Gelem/s 5.0759 Gelem/s 5.1023 Gelem/s]

And, under some circumstances, this code is indeed essentially optimal!

To understand why this code might sometimes perform well or poorly, it is important to make a few observations about this query and the compiled_engine approach:

Memory bound. We’re doing very little with each value that we read from memory. Essentially, we are performing two inequalities and one addition (both cheap operations) per three memory reads. We can thus hypothesize that our code will be bound by the speed of memory, either by the “front end” (loading new instructions) or the “back end” (reading data).
Branching. The short-circuiting evaluation of *a_el > 5 && *b_el < 7 means that we may have a branch. I say “may” because the optimizer could choose to rewrite this code in a branchless fashion, but branching is something we should think about. This is also our first hint that the performance of this code might have something to do with how often a > 5 is true or false.
Inconsistent memory access. We always read the entries of a in order, but we might access b sparsely (since we only access b if a > 5), and c even more sparsely (since we only access c is both conditions are true).

We can confirm or deny our hypotheses using perf stat. If we run with a highly selective a > 5 (that is, most rows are filtered out):

predicates/compiled time:   [144.20 µs 144.76 µs 145.35 µs]
Performance counter stats for 'cargo bench compiled':
 19,254.19 msec task-clock:u       #  1.306 CPUs utilized
      81K          page-faults:u   #  4.239 K/sec
      80G          cycles/u        #  4.158 GHz              (93.14%)
      292G         instructions/u  #  7.34  insn per cycle   (93.14%)
      126G         branches/u      #  6.591 G/sec            (93.14%)
      204,605,913  branch-misses/u #  2.13% of all branches  (93.14%)
      TopdownL1                    #    3.9 %  tma_backend_bound
                                   #    5.5 %  tma_bad_speculation
                                   #   14.6 %  tma_frontend_bound
                                   #   76.0 %  tma_retiring

Compared to if we use a non-selective a > 5 (zero rows are filtered):

predicates/compiled time:   [3.6357 ms 3.6439 ms 3.6532 ms]
Performance counter stats for 'cargo bench compiled':
 16,432.68 msec task-clock:u       #   0.999 CPUs utilized
  12,161      page-faults:u        # 740.050 /sec
     79G      cycles/u             #   4.851 GHz              (99.85%)
     30G      instructions/u       #   0.89  insn per cycle   (99.85%)
     13G      branches/u           # 815.276 M/sec            (99.85%)
      2G      branch-misses/u      #  24.00% of all branches  (99.85%)
         TopdownL1 (cpu_core)      #     9.6 %  tma_backend_bound
                                   #    49.9 %  tma_bad_speculation
                                   #    31.2 %  tma_frontend_bound
                                   #     9.3 %  tma_retiring

When the predicate over a is selective, we get 7.34 instructions per cycle, but when the predicate is not selective, we drop to 0.89 instructions per clock cycle! This reflects in the total time taken to process 1M rows going from 144 microseconds to 3.6 milliseconds! Luckily, perf also points us to the culprit: the topdown analysis shows that, in the slow case, we’re spending 50% of our time recovering from bad speculations (i.e., branch mispredictions). In the fast case, only 5% of total time is spent on mispredictions.

Bakeoff #3: End-to-end optimizations

Compiling execution engines really shine when the compiler (e.g., LLVM) can do something smarter than just lowering our code to machine instructions. In the above examples, LLVM was able to use SIMD instructions for some of our code, unroll loops, etc., but it was never able to fundamentally change the shape of the compilation for this particular query. But what if our query was instead:

SELECT COUNT(*) FROM t WHERE t.a > 5 AND t.b < 7;

… now we only need to count how often both predicates are true. Our vectorized code might look like this:

pub fn vec_count(a: &[i32], b: &[i32], _c: &[f32]) -> usize {
    let a_bm = BooleanBuffer::collect_bool(a.len(), |idx| a[idx] > 5);
    let b_bm = BooleanBuffer::collect_bool(b.len(), |idx| b[idx] < 7);
    let res = &a_bm & &b_bm;
    res.count_set_bits()
}

… and the compiler-generated code might look like this:

pub fn compiled_count(a: &[i32], b: &[i32], _c: &[f32]) -> usize {
    a.iter()
        .zip(b)
        .filter(|(a_el, b_el)| **a_el > 5 && **b_el < 7)
        .count()
}

In this case, LLVM can lower this code into beautiful SIMD code that loads 16 elements from a, 16 elements from b, compares them, and then counts the number of set bits in the resulting mask and adds that count to an accumulator.

This pays off – at any selectivity, the compiled code operates at around rows/sec, compared to the vectorized code which operates at around rows/sec. But such optimizations are not always possible.

The best of both worlds?

Can we get the best of both vectorized and compiled execution engines? Surely, a compiling engine could generate the same code that a vectorized engine uses, so why not? A paper by Menon et al. called “Relaxed Operator Fusion” explores this idea. The short answer is that you can, but it’s not as simple as either strategy alone.

Database, and especially their execution engines, are filled with surprising performance “gotchas” and highly-optimized tricks that have been passed down from system to system. If you’d like, you can read some of my work on engineering a hash table for GROUP BY aggregation, or check out the excellent everything you’ve always wanted to know about vectorized and compiled execution engines, but were afraid to ask.

Notes

You can technically use a partial strategy with bitmaps instead of selection vectors, and you can use selection vectors instead of bitmaps in a full strategy. But intersecting bitmaps is generally cheaper than intersecting vectors, and iterating over (sparse) vectors is generally cheaper than iterating over bitmaps. Of course, there are exceptions. ↩
Vectorizing the selection vector code gets significantly easier on AVX-512 hardware due to the “compress store” primitives, but unfortunately, I’m trapped in AVX2 land. Vectorizing this code with AVX2 requires precomputing a permutation lookup table, which is probably why the compiler doesn’t do it automatically. ↩

Neo, 6 years and 600 citations later

Mon, 15 Dec 2025 00:00:00 +0000

I was excited to see that Neo, the neural query optimizer, got its 600th citation! It’s nice to have a quantification, but what I’m most proud of is the impact Neo had on the research community.

Neo was simultaneously the last paper of my PhD at Brandeis and the first paper of my postdoc at MIT. Working with my advisor Olga and then-1st-year-PhD student Chi at Brandeis, along with Mohammad, Hongzi, Pari, Nesime, and Tim (my future postdoc supervisor) at MIT, was a great experience. I couldn’t think of a better way to end a PhD, or to start a postdoc!

What is Neo?

Learning a query optimizer with deep reinforcement learning is interesting because making a query optimizer is a long and complex task. Modern query optimizers, like the one found in SQL Server, represent decades of engineering effort. If one builds a new type of system, are they doomed to spend decades of engineering effort building a query optimizer as well? Or is there another way?

Neo showed that deep reinforcement learning, with a little bit of demonstration from a simple query optimizer, could eventually learn policies that matched the performance of state-of-the-art (in 2019) query optimizers. Neo thus represented a hope that query optimizers for future data systems could be “learned” instead of manually built.

How’d we get there?

Neo was the culmination of two prior works from my PhD investigating how deep reinforcement learning could be applied to query optimization. Initially, in ReJOIN, we showed that an off-the-shelf PPO algorithm could find plans with low costs according to the query optimizer’s cost model. Concurrently, a team at UW (Ortiz et al.) discovered that the state learned by a deep RL agent contained rich information about the query’s structure. A little later that year, a team from Cal (Krishnan et al.) showed how Q-learning could be used to optimize queries against a cost model.

In order to get from here to Neo, we needed to solve two key problems.

Key Problem 1: Sample Inefficiency

Deep reinforcement learning algorithms are notoriously sample inefficient. While RL models could learn to solve complex problems, they might need millions (or even trillions) of “episodes” in order to get there. This sample inefficiency is easy to get around in traditional RL areas like video games. If you are bad at Pacman, you’ll get eaten by a ghost quickly. This means you can get a lot of episodes per wall-clock hour.¹ This is not the case in query optimization: if you pick a really bad query plan, it could take days to execute instead of seconds. In other words, in query optimization, doing worse takes longer.

Because of this, throwing an off-the-shelf RL algorithm at query optimization is not really viable. In our vision paper early the next year, we laid out three different potential ways to solve the problem. One of those ways – learning from expert demonstration – became Neo. Later, another idea, bootstrapping from a cost model, was explored by Berkeley in Balsa.

Key Problem 2: Inductive Bias

Half the battle of getting deep learning to work well is getting the right inductive bias. Throwing a fully connected network at a problem, while theoretically sufficient, is rarely the right choice. Think about the areas where deep learning has had the most success: in computer vision, convolutional neural networks (CNNs) excel at modeling low-level vision primitives like oriented edge filters, and thus tend to perform better than fully connected networks (although today you’ll likely see a convolutional head on top of a transformer). In natural language processing, transformer models excel at modeling long-range dependencies, and thus tend to perform better than fully connected networks.

So how can we get a good inductive bias for query optimization? Fortunately, researchers in 2016 at Peking University proposed a technique called tree convolution. Originally designed for program language processing, tree convolution is also a great fit for modeling the performance of query plans. Section 4.1 of the Neo paper provides an intuitive overview of tree convolution and how it can be applied to query optimization.

What happened next?

By all measures of an academic project, Neo was a success. I graduated and got a postdoc. The paper has been cited many times, and research on replacing a query optimizer with a deep reinforcement learning agent continues to thrive to this day (for example, check out Roq from IBM). While Neo demonstrated excellent performance, it also revealed (at least to me) two new important facts about query optimization:

The tail matters more than the mean. Neo’s RL algorithm seeks to minimize the mean latency of a query plan, and it does so admirably, often matching or even exceeding the performance of expert systems. However, in real systems, tail performance (e.g., the 99th percentile) tends to matter more than the mean latency. Neo would happily increase P99 latency by a few minutes to cut mean latency by a few seconds – a trade that looks great on paper,² but is often unacceptable in practice.
Regressions are killer. Users don’t want their query that was fast today to potentially be slow tomorrow. Reinforcement learning requires exploration to try new ideas, and Neo does exploration the same way that most RL algorithms do: by considering, for each query, a balance between exploiting existing knowledge to try to get the best plan possible, and exploring new ideas to try to get better plans in the future. This is a non-starter for a lot of database engineers who would get paged at 3am when the system decided to explore.

We addressed both these concerns in our follow-up work, Bao, which sat on top of an existing query optimizer instead of wholly replacing one. Bao would later lead to a limited deployment at Microsoft and Meta. In Bao’s industrial adoptions, offline execution was used to ensure that a new candidate plan was evaluated before being deployed, mitigating the risk of regressions. My lab at UPenn has started to explore how to best allocate this offline execution budget, at both the query level and the workload level.

Overall, the future of learned query optimization looks bright. The biggest recent change is, of course, the emergence of LLMs, which have already been integrated into query rewriting systems, steering systems, and RAG-based query optimizers that learn from their mistakes.

Footnotes

Video games, and most traditional RL environments, can also be simulated concurrently, potentially running many simulations in parallel on the same machine. In query optimization, the machine itself is part of the “state” – if you run 8 queries in parallel on the same machine, they will interfere with each other, messing up the reward signal. ↩
These improvements look good on paper in the sense that plots of the total workload time improve. Obviously, we’ve wisened up since then, and now regularly report tail latency results. ↩

2024's hottest topics in databases (a bibliometric approach)

Fri, 28 Mar 2025 00:00:00 +0000

From database companies to renowned professor of databaseology Andy Pavlo, everyone seems to be writing their “year in review” of the database world. While I could never match Andy’s wit nor his no-doubt earnest adoration for Larry Ellison, I thought I might be able to provide some additional insights into the hottest topics in database research from a bibliometric perspective.

For the past few years, I’ve maintained a “ranking” of database researchers and papers using the citation graph and PageRank. We can use the same data to identify papers with the fastest growing citation counts, which is arguably a reasonable proxy for “hotness.” Note that these papers are the papers being cited a lot, so they might represent the baseline for a particular area, or they might be the first paper in a line of work, and might not represent the actual “hotness” themselves.

Below, I’ve organized the 20 papers with the fastest growing non-self-citations.

Learned indexes

Rank	Title	Year	Non-self Citations	Per Year
1	The Case for Learned Index Structures.	2018	142	20
12	ALEX: An Updatable Adaptive Learned Index.	2020	59	11

Since Tim Kraska’s paper on learned indexes (written during a sabbatical at Google, and preceding his subsequent move from Brown to MIT), the database community has been hard at work developing new learned indexes, benchmarking them, and implementing them into real systems. A recent survey paper from Purdue shows just how explosive the subject has become:

The ALEX paper was (to the best of my knowledge) the first paper to make learned indexes updatable, leveraging an adaptive tree structure. The first author, Jialin Ding, was a PhD student in Kraska’s lab, and is now faculty at Princeton. Jialin is also the led author on a paper describing how multi-dimensional learned indexes were integrated into AWS Redshift.

It’s been a while since Kraska’s paper made its initial splash, but (for a SIGMOD paper) it was surprisingly controversial. Blog posts about if we should throw out our algorithm textbooks or simply use B-trees were everywhere. In my opinion, part of this confusion was because the original SIGMOD paper did not come with any code. And, if you don’t believe in an idea, it’s very easy to whip up a crappy implementation. Fortunately, since 2018, we now have a much better idea why learned indexes work.

(Learned) query optimization

Rank	Title	Year	Non-self Citations	Per Year
2	Learned Cardinalities: Estimating Correlated Joins with Deep Learning.	2019	124	20
3	Neo: A Learned Query Optimizer.¹	2019	116	19
4	How Good Are Query Optimizers, Really?	2015	188	18
7	Bao: Making Learned Query Optimization Practical.¹	2021	57	14
19	An End-to-End Learning-based Cost Estimator.	2019	61	10
20	Are We Ready For Learned Cardinality Estimation?	2021	42	10

Arguably since the 2001 IBM Leo paper, we’ve been trying to figure out how to build query optimizers that learn from their mistakes. After all, if the optimizer chooses a bad query plan, why shouldn’t it pick a better one next time?

The database community has been attacking this problem from (at least) three different directions, each of which is represented in the “hottest” papers:

Using machine learning methods, like deep learning, to replace cardinality estimates,
using semi-supervised or guided learning techniques to replace cost models,
and using reinforcement learning to either replace or steer the query optimization process.

While academics continue to wax poetic about the merits of each approach, practitioners in industry also made some inroads with learned query optimization, including a deployment of “steered” optimizers at Microsoft and Meta¹, as well as integrations of learned performance predictors into AWS Redshift’s intelligent scaling and workload manager¹.

The classic “How Good Are Query Optimizers, Really?” paper also makes an appearance, as the use of the join order benchmark (JOB) has become ubiquitous in query optimization research. Of course, this is also a seminal paper in query optimization!

DBMS engines

Rank	Title	Year	Non-self Citations	Per Year
5	Spark SQL: Relational Data Processing in Spark.	2015	172	17
6	Pregel: a system for large-scale graph processing.	2010	214	14
10	The Snowflake Elastic Data Warehouse.	2016	102	11
11	Hekaton: SQL server's memory-optimized OLTP engine.	2013	138	11
13	Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases.	2017	94	11
16	DuckDB: an Embeddable Analytical Database.	2019	65	10
18	Efficiently Compiling Efficient Query Plans for Modern Hardware.	2011	151	10

The execution model and engines of DBMSes will always be a core part of the database community’s interests. Recently, researchers have started to build on “new” foundations!

Papers building on Spark SQL continue to emphasize ease-of-use and compatibility with distributed and open source (mostly Apache) runtimes.
Papers building on Pregel continue to tackle graph database problems, which is far from solved – graph databases are interesting because they seem to warrant both a new query language and a new execution model.
Papers building on Hekaton continue to push the boundaries of in-memory data processing, often integrating PIM elements or pushing the performance envelope.
The database community’s obsession with clouds continues to show in works building off of Snowflake and Aurora.

Perhaps most notably, at least from my perspective, is the most-recent addition, DuckDB. For a long time, vectorized out-of-core² analytics databases were only available as proprietary products (like Vertica) or locked away in TU Munich’s code repositories (like Hyper, which was eventually sold to Tableau, then Salesforce). DuckDB not only provides a state-of-the-art implementation of a vectorized database, but also packages it up as an easy-to-use, embedded tool. As a result, DuckDB has become the de facto testbed for implementing new OLAP techniques, like new research on join ordering or confidential computing.

ML-powered system tuning

Rank	Title	Year	Non-self Citations	Per Year
8	Automatic Database Management System Tuning Through Large-scale Machine Learning.	2017	109	13
17	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning.	2019	60	10

Databases have knobs. Lots and lots of knobs. Whether it’s the classic PostgreSQL shared_buffers or something more obscure like Vertica’s resource pools, getting the knobs set to the “right” values for your workload has always been an essential performance tuning step.

The two hottest papers in this category both represent machine learning powered techniques, either using reinforcement learning or Bayesian optimization. The approaches building on these two works have made a lot of progress in both cloud and on-premises environments. Of particular note is the DB-BERT line of work from Immanuel Trummer’s lab at Cornell, which integrates LLMs into the already-ML heavy area, resulting in a database tuner that can, almost literally, “read the manual.”

Data cleaning and matching

Rank	Title	Year	Non-self Citations	Per Year
9	Deep Entity Matching with Pre-Trained Language Models.	2020	66	13
14	HoloClean: Holistic Data Repairs with Probabilistic Inference.	2017	90	11
15	Deep Learning for Entity Matching: A Design Space Exploration.	2018	71	10

Database systems are only as useful as the data they contain! Garbage in, garbage out. Tools for data cleaning and entity matching are critical for bringing structure to chaos, enabling analysis and hopefully dredging up some of that good good Actionable Business Insight we love to talk about.

The prevailing trend, as you might imagine, is integrating LLMs and other recent ML inventions into traditional data cleaning and matching tools. While the latency of these systems still leaves something to be desired, the accuracy provided by these new LLM-enabled tools is sometimes orders of magnitude ahead of what we could do before.

Some good examples of this trend include Avanika Narayan’s work in Chris Re’s lab, which show that LLMs (“foundation models”) are quite effective at a number of data wrangling tasks. A slightly more involved approach is Nan Tang’s recent paper on relational pre-trained transformer (which is apparently almost all you need), which is directly trained for data cleaning tasks and can even operate on complex, non-1NF types like JSON (but we always put our databases into 3NF, right?).

What’s next?

Another year goes by and databases get a little faster, a little cleaner, a little easier to use, and a little smarter. Maybe next year the DataFusion team or the DuckDB team will make them a little cheaper! But nobody tell the VCs.

Perhaps you want to hear the no-doubt enlightened opinion of me, your humble author? Well, I think smarter, faster, cheaper, and more accessible databases are great, but how come nobody is looking into making databases more polite? Look what happens if I misspell WHERE:

imdb=# select count(*) from title were production_year=2022;
ERROR:  syntax error at or near "production_year"
LINE 1: select count(*) from title were production_year=2022;

Frankly rude – no respect from PostgreSQL here. I guess if Mike Stonebraker raised me in a basement at Berkeley in the 80s, I might also be missing a few “pleases” and “thank yous” around the edges. Or maybe the middle is already hollow and the papers are all diarrhea, what do I know?

(If you actually want to know what I’ve been up to, check out our latest work on offline query optimization – SIGMOD papers coming soon!)

I am an author of this paper. ↩ ↩² ↩³ ↩⁴
MonetDB is arguably an exception to this claim, since MonetDB could mmap larger-than-memory chunks, but I think it is reasonable to categorize MonetDB as a mostly in-memory analytics database. ↩

Related work search for database papers

Sun, 02 Feb 2025 00:00:00 +0000

This tool takes your paper’s title and abstract, and searches past VLDB, PODS, SIGMOD, and CIDR papers for potential related work. Obviously, this tool does not replace a thorough literature review, but it can help you find papers you might have missed.

Names that are highlighted are on the VLDB or SIGMOD program committee.

Title:

Abstract:

How it works

This tool embeds prior database papers and stores the embeddings in a search index. When you enter your own title and abstract, the tool computes the embeddings for your paper and searches the index for similar papers. Relevant results are then explained and categorized by an OpenAI model (currently gpt-4o-mini).

Ten years of improvements in PostgreSQL's optimizer

Fri, 12 Apr 2024 00:00:00 +0000

As a query optimization researcher, I’ve spent the last 10 years of my life playing with, learning from, and building on top of the most sophisticated open source query optimizer out there, PostgreSQL. I recently wondered how much PostgreSQL had improved over the decade since I started working on databases. While changelogs and opinion pieces were plentiful, I couldn’t find any strong empirical comparisons, so I decided to run the join order benchmark (JOB)¹ on PostgreSQL 8 through 16. I recorded the 90th percentile query latency for each database version.

I built each version² of PostgreSQL using GCC 13.2 inside a Docker container with Arch Linux. Since I wanted to measure the quality of the query optimizer, and not index/IO performance, I set shared_buffers to 8GB (large enough to hold the entire database). I also set work_mem to 8MB for all versions. Each query is executed once to warm the cache, then the median latency of 5 additional runs in recorded.

Overall, PostgreSQL’s tail performance has improved drastically, although versions 13 through 16 have been mostly stable. Comparing version 8 to version 16, PostgreSQL has dropped tail latency by nearly half in the last 10 years!

We can also investigate the entire query distribution (note the log scale):

We can use regression analysis to (1) confirm that the downward slope in latency is significant, and (2) quantify how much improvement is brought by each version of PostgreSQL. If we regress the PostgreSQL major version number against query latency, we see that each new major version of PostgreSQL brings, on average, a 15% performance improvement on the Join Order Benchmark (). However, a linear model is arguably a poor measure of the change ().

Of course, not all of these improvements are attributable to the query optimizer. Improvements to the execution engine – from parallel workers to just-in-time (JIT) compilation – also play a role. It would be interesting to investigate how each query plan in JOB has changed over the year… maybe next time!

Quantifying the improvement aside:

Upgrade your database! Going from PostgreSQL 8 to 16 has the potential to massively improve your workload’s tail latency.
Researchers should note that PostgreSQL is a bit of a moving target. Learned query optimization research has compared with different versions of PostgreSQL over time (e.g., Neo and Bao compare with version 11, whereas newer work compare with version 14, 15, or 16.) So just because an older technique improves on PostgreSQL by 30%, and a newer technique only improves on PostgreSQL by 25%, the newer technique may be comparing against a stronger PostgreSQL.

You can also check out the raw data for yourself.

Notes

The join order benchmark is a set of complex queries with many joins. The benchmark was introduced in the seminal paper “How Good are Query Optimizers, Really?”, in which the authors showcase the difficult nature of the join order benchmark. ↩
I used the most recent minor version for each major version of PostgreSQL. For example, for PostgreSQL 8, I used version 8.4.22. These minor versions are often released after new major versions are available, but generally only contain bug fixes (not new features or performance improvements). ↩

Most influential database papers

Tue, 25 Jul 2023 00:00:00 +0000

Ever wondered which database systems papers have been the most influential? This page explores the PageRank of a paper in the citation graph, one possible measure of influence.

This blog post has been replaced by DBScholar, which uses its own metadata extraction pipeline instead of the (now defunct) APIs used in this post.

All rankings, including this one, codify an ideology and thus won’t match up with everyone’s understanding of “influence.” Most notably, this ranking has nothing to do with industrial adoption, actual usage, or quality of an idea, other than by correlation with citation. Note that the citation graph grows every year, so papers published more recently will, on average, have a lower score.

PageRank is computed with the Python package NetworkX on top of the citation graph of all SIGMOD, VLDB, CIDR, and PODS papers. Self-citations are excluded. Absolute and percentile ranks are then computed for each paper using the PageRank score. Authors are ranked according to the sum of their paper’s PageRank score, where each paper’s score is divided by the number of authors, similar to CSRankings. Click one of the links below to explore the ranking!

Most influential papers of all time
Most influential papers by year
Most influential researchers

Latest update: Mar 4th, 2025.

Most influential papers of all time

This is a list of all papers, sorted by their PageRank score. Search papers by title using the text box below.

Most influential papers, by year

This is a list of papers published in a particular year, sorted by their PageRank score. You can select a different year from the dropdown. Papers published in recent years have fewer citations, and thus more noise in their score.

Most influential people

This table ranks all authors of a paper that has been cited at least once (by another paper). An author’s score is the sum of their paper’s normalized PageRank score. The normalized PageRank score of a paper is the PageRank score of the paper divided by the number of authors on the paper.

Due to the exponential distribution of PageRank scores, the most influential papers have significantly higher scores than the median paper. As a result, authors of the most influential papers will rank highly here, even if they have few papers overall.

Notes

PageRank is computed with and with the initial weight for each paper set to . These hyperparameters come from:

Walker, Dylan, Huafeng Xie, Koon-Kiu Yan, and Sergei Maslov. “Ranking scientific publications using a model of network traffic.” Journal of Statistical Mechanics: Theory and Experiment 2007, no. 06 (2007): P06010.

Generating bios with large language models

Tue, 20 Dec 2022 00:00:00 +0000

Update March 19th, 2024: this page now uses the gpt-3.5-turbo model.

It’s the holiday season of 2022, and everyone is talking about large language models! OpenAI’s GPT-3 and others are generating quite a bit of interest in the CS research community. Larger datasets and models are significantly improving results on many NLP tasks. Even in databases (my field), large language models are being used to tune databases and translate natural language to SQL.

Large language models are so good, some folks are tricking themselves into thinking that large language models are sentient. It is important to remember that there is no there there – GPT-3 and the like only model the distribution of the next word given the previous word.¹ As a result, large language models can at times easily produce misinformation, state biased viewpoints, or conjure total nonsense!

Let’s try it out. Enter the name of a computer scientist below to search for their DBLP entry. Click their name to ask GPT-3 to generate a bio for them using a list of their citations as an input.

A standalone version of this bio generator is available at https://bio.rmarcus.info.

In order to generate a bio, we fetch the 15 most recent publications for the author from DBLP, and then ask GPT-3 to complete the following text (“prompt”):

Given the below list of publications, produce a short summary of the research of {name}. Use succinct language when possible. Focus on common themes across all of their works. Do not enumerate specific topics or papers. Don't use phrases like "such as" or "including." Do not include specific dates, conferences, venues, or journals. Use gender-neutral language.

{publication list}

Bio: 

So GPT-3 starts generating text after Bio:, which normally results in some plausible descriptions of researchers. However, the model can certainly make mistakes. For example, despite the instruction to use gender-neutral language, the language model still outputs the pronoun “he” for Nesime Tatbul:

Nesime Tatbul is a computer scientist and researcher. His work focuses on machine programming, query optimization, explainable anomaly detection, cloud observability, and other topics related to data management. He has been involved in multiple projects, from developing a benchmarking platform for explainable anomaly detection to creating a pluggable metrics storage engine for the age of observability.

Additionally, despite being instructed not to list specific projects or papers, the model frequently does exactly that. This is somewhat understandable, since it might be impossible to determine an author’s general subfield from only their 15 more recent publications.

Because large language models are prone to produce bias, capable of producing nonsense, and are generally unpredictable, you probably shouldn’t use them in any “real” application yet. But that doesn’t mean you can’t play around with them and have fun! If you haven’t tried out ChatGPT, give it a shot here: https://chat.openai.com

Footnotes

A portion of the machine learning community subscribes to the belief that human beings are similar to such models (“there is no ghost in the machine”). We simply map inputs to outputs, and our consciousness is a manifested illusion. Once we multiply sufficiently large matrices together, AGI is achieved! I’m not an expert here, so I’ve gone with my intuition, which is that we are at least not entirely stochastic machines. ↩

Applying Bao to distributed systems

Thu, 17 Jun 2021 00:00:00 +0000

This post was also published on the MIT DSAIL blog.

A good portion of my postdoc at MIT has been spent developing Bao, a system for learned query optimization with an eye towards practicality. Bao was recently published at SIGMOD 2021, where we were thrilled to receive a best paper award.

In our paper, we show how Bao can be applied to the open-source PostgreSQL DBMS, as well as an unnamed commercial system. Both DBMSes ran in a traditional, single-node environment. Here, we’ll give a brief overview of the Bao system and then walk through our early attempts at applying Bao to commercial, cloud-based, distributed database management systems.

For more information on Bao in the traditional, single-node context, check out:

Our research paper, published in SIGMOD 2021.
For a video overview, the recording of Ryan’s SIGMOD talk (20 minute version or 3 minute version).
For an overview of tree convolution as applied to query plans, Ryan’s AIDB 2019 talk.

However, since we wrote the paper almost a year ago, much has happened. First, We worked together with Microsoft to explore how Bao can help with Big Data workloads. This work will also be presented at SIGMOD as part of the industry session, and received a honorable mention for the industry best paper award.

Second, after the discussion we had with Mike Stonebraker around the potential impact Bao could have on distributed database warehouse systems (obviously, he was very skeptical), we ran a whole range of additional experiments on Vertica, Redshift, and Azure Synapse using a real-world dataset we received from an anonymous corporation.

How Bao works

Previous approaches¹ to learned query optimization attempted to replace large parts of traditional query optimizers. In contrast, Bao sits on top of a traditional query optimizer (called the underlying optimizer) and learns to steer the underlying optimizer in the right direction.

The image above illustrates the process Bao uses to steer the underlying query optimizer.

When a query arrives, the underlying optimizer is used to generate a number of plan variants for the query. For example, we might have the optimizer generate a plan using index nested-loop joins, another plan using sort-merge joins, and a third plan using hash joins.²
Once these plan variants are constructed, a predictive model (a deep neural network) predicts the latency of each plan.
The plan with the best predicted latency is selected and executed. The result is sent to the user, and the actual latency is recorded.
The actual latency of the recorded query is added to Bao’s experience set, which is used to further refine the predictive model using Thompson sampling.³

Over time, Bao’s predictive model learns from its mistakes, hopefully making increasingly accurate predictions. Experimentally, we’ve shown that Bao can learn in the presence of dynamic workloads, shifting data distributions, and schema modifications.

From single-node to distributed

In the original Bao paper, we evaluated Bao on a single-node database system (e.g., Oracle and PostgreSQL). However, many data warehouse databases are, in fact, distributed. Luckily, Bao is largely agnostic to the underlying execution engine or storage layout: as long as you have a set of hints, Bao can pick and choose from them and learn from its mistakes. In the paper, we discuss what good hints for a single-node DBMS look like: for example, forcing particular join types, or forcing particular access paths (i.e., index vs. table scan). These choices can impact the performance of the plan on their own, but can also impact the join order selected by the underlying query optimizer, potentially resulting in drastically different run times.

Adapting Bao to a distributed DBMS is simply a matter of finding the right hints. While many aspects of the single-node case apply to the distributed case as well (operator choice matters, join order matters), distributed DBMSes bring about other important performance considerations:

Suppose a fact table is distributed across multiple nodes based on a key column. Should a join of that fact table with a dimension table be done via sending the dimension table to all nodes and using a hash algorithm? By partitioning the dimension table on the foreign key and using a merge join and union? By collecting the matching rows of the fact table on a single node, then performing a non-distributed merge?

Depending on network costs, query selectivity, materialization strategy, what data is already present at each node, and a wide range of other factors, any of these strategies might be applicable. Different DBMSes will lean towards different options depending on which code paths have been optimized. Most distributed DBMSes choose between these different strategies using the same tools that non-distributed DBMSes use: heuristics, cost models, and cardinality estimation. These tools are already highly error-prone (and require significant tuning) in non-distributed settings, so you can imagine how tricky things get when entire clusters are involved!

Next, we’ll walk through how Bao can be applied to three different state-of-the-art commercial cloud distributed DBMSes: Vertica, Amazon RedShift, and Azure Synapse (an analytics-focused offering of SQL Server). After discussing each system, we’ll show a small experiment highlighting potential gains (or lack thereof) from applying Bao. We’ll be running each DBMS on different hardware, so please do not attempt to draw comparisons between these DBMSes.

In each test, we’ll be executing an analytic dashboarding workload called Corp. The workload was donated to our research team by a large corporation under the condition of anonymity. The workload contains about 1 TB of data and 2000 unique queries which change over time. A large schema change (normalizing a fact table) happens in the middle of the workload – we do not count the time required to perform this modification. The workload makes use of analytic functions, (materialized) views, and other advanced SQL features.

Vertica

Vertica is a distributed columnar DBMS, and is the commercial adaptation of the C-Store paper. Vertica’s optimizer is reasonably transparent, and the documented set of hints gives us a lot of control over what the optimizer considers.

For Vertica, we select query hints to:

Force a particular join operator (JTYPE hint),
force a particular group by operator (GBYTYPE hint),
force a particular distributed join algorithm (DISTRIB hint, with options L, R, B, or A, representing a local join, a resegment join, a broadcast join, or letting the optimizer pick, respectively).

We started up two identical 3-node clusters on AWS using the official Vertica image (which, at time of writing, uses r4.4xlarge nodes by default). We loaded the data into both clusters, and used the Vertica DBD tool to create a good layout, which we manually verified through testing. One cluster ran Bao on top of Vertica (Bao), while the other cluster did not run Bao (Vertica). The resulting time and costs are plotted below:

Bao was able to reduce the end-to-end processing time of this workload by over three hours, while reducing cost by over 25% (about $25, in our case). The time savings with Bao are more significant than the cost savings because Bao must periodically retrain its predictive model, which (temporarily) requires a GPU.

Around 45% of Bao’s gains (the plurality) come from forcing the Vertica optimizer to use a broadcast join instead of a resegment join when the Vertica optimizer wrongly over-estimates the cardinality of a particular subplan. The small subplan result can be easily materialized and sent to all nodes, allowing for a faster computation than shuffling around parts of the subplan evenly to each node in the cluster (a resegment).

In the future, we intend to experiment with specific hints to include or exclude projections (PROJS and SKIP_PROJS) to allow Bao to further custom-tailor its strategy to the user’s data.

Azure Synapse (SQL Server)

Azure Synapse is a analytics offering from Microsoft, based on SQL Server. SQL Server is one of the most widespread commercial database systems. Unlike Vertica, SQL Server can store data in either a row-based or a column-based format. Here, we’ll focus on the column store format provided by the Azure Synapse Analytics cloud database. Unfortunately, the types of hints available for Synapse Analytics are quite limited – so we had to get a bit creative.

For Synapse, we select query hints to:

Disable or enable a particular type of join operator (hash, merge, and/or loop),
use a replicated or partitioned versions of all dimension tables,
use an indexed or non-indexed version of the fact table

In order to implement the last two hints, we cheat a little bit: we created two versions of each dimension table, one replicated and one partitioned, along with two versions of the fact table (one indexed and one non-indexed). Based on the hint Bao selects, we rewrite the query to use the correct versions of each table.

We spun up two Azure Synapse Analytic instances, each with a dedicated 1000 DWUs (“data warehouse units,” a measure of query processing resources available to queries). We ran one instance with Bao and the other instance without Bao. After loading the data, we added the specialized tables described above to the instance using Bao. The other instance, which would use the stock optimizer, had all dimension tables replicated and an indexed version of the fact table. The resulting time and costs are plotted below:

Bao was able to reduce the end-to-end runtime of this workload by a little over two hours, while reducing the cost by around 10% (a little under $35). Again, the cost reduction is smaller than the latency reduction because of the cost of training the Bao model.

A plurality of Bao’s gains (40%) came from avoiding the index on the fact table (sometimes by removing loop join as an option, sometimes by rewriting the query to use the non-indexed table). Removing the index from the instance running with the stock optimizer led to a decrease in performance, meaning that while the index normally helped, it hurt some queries, and Bao was able to identify those queries automatically.

In the future, Bao could be applied to Azure Synapse in serverless mode, but this mode currently does not support query hints.

RedShift

RedShift is Amazon’s data analytics offering, and is often cited as one of the first “cloud native” analytics databases. As far as we can tell, RedShift only offers two query optimization hints, which we’ll use alongside the same trick we used for Synapse.⁴

For RedShift, we select query hints to:

Disable or enable AQUA, the RedShift query accelerator (activate_aqua),
disable or enable using (up-to-date) materialized views in query processing (mv_enable_aqmv_for_session),
use a replicated or partitioned version of all dimension tables (as with Synapse).

Unfortunately, RedShift offers by-far the least visibility and control into its query optimizer, leaving our Bao implementation (seemingly) limited. We started two 3-node clusters (ra3.4xlarge nodes), one running Bao and one running the stock optimizer. The results are plotted below:

Bao speeds up total workload latency by over an hour, and reduces costs by around 10% (in this case, about $10). Most of the gains (65%) came from disabling the usage of materialized views within subtree expressions, which the RedShift optimizer seems to do a little too aggressively (e.g., scanning the entire materialized view is slower than applying the predicates to the relations involved and performing the join). Disabling the use of materialized views in query processing globally hurts query performance overall, again showing that Bao is able to learn when the feature helps and when it hurts.

RedShift seems like a really cool system, but unfortunately it does not provide the visibility or control required by researchers for deep study in query optimization. In the future, it would be awesome to see Amazon build a few more windows and knobs (with sane defaults!) into the optimizer for scientists to use.

Discussion

Looking over the results, we find ourself returning to the same set of takeaways:

Don’t spend too much time comparing results between systems. The experiments are ran on different clouds, using different hardware, at different times of day. While we followed best practice guidelines to tune the systems, an expert might still do better, in particular for Redshift and Azure Synapse, as we have less experience with them.
Just because Bao produces large gains on system X but smaller gains on system Y doesn’t mean the default optimizer of system X is better than the default optimizer of system Y. In our opinion, it is more likely that system X provides more visibility into the optimizer, whereas system Y keeps things pretty opaque, limiting Bao’s possible gains.
Everyone wants a “knob-free” query optimizer, but nobody tells you that the cost of such is decreased query performance. Without incorporating feedback from query execution in some way, heuristic optimizers are doomed to have edge cases that make them fall flat. “Knob-free” sometimes just means “not sophisticated enough to give you the tools you need to fix the mistakes.” A real “knob-free” query optimizer should take query latency feedback into account.
The idea behind Bao – use a deep neural network as a predictive model to choose between a number of different variants, then retrain that model progressively using Thompson sampling – seems widely applicable. We focused on database systems, but maybe there are other applications as well?

Of course, Bao does come with some drawbacks. Bao causes query optimization to take a little bit more time (~300ms), requiring quite a bit more computation. We studied this overhead in our SIGMOD paper. For data warehouse workloads, which largely consists of long-running, resource intensive queries, Bao’s increased overhead is hardly noticeable. However, for workloads with a lot of short running queries, like OLTP workloads, this might not be the case. We are currently working on new approaches to mitigate that problem – so stay tuned!

If you feel so inclined, you can read the Bao paper, check out our open source prototype for PostgreSQL, or take a look at the other publications from our group.

Notes

Approaches such as ReJOIN, DQ, and Neo (in chronological order) fall into this category. ↩
The choice of which plan variants to generate is important. Generally, one variant is always just what the underlying query optimizer would have chosen with no intervention, and the other variants are generated by forcing the optimizer to ignore particular optimization paths. See the paper for more details. ↩
Note that training the predictive model using standard supervised learning techniques will not balance the exploration of new policies (i.e., trying something new to see how well it works) against the exploitation of existing knowledge (i.e., doing what we know works well). Thus, reinforcement learning techniques are preferred. It is easy to think that, to avoid regression, you would want to maximize exploitation. But this can get you trapped in a local minima: if you mis-predict the value of the optimal plan, you will never select it. In other words, without exploration, you might initially have fewer regressions, but you’ll never recover from them! ↩
Redshift, like Vertica, does not support traditional indexes on tables like Synapse does, so we do not add those indexed variants here. ↩

Machine learning for systems

Thu, 06 Jun 2019 00:00:00 +0000

Imagine you are writing a system to repeatedly sort large datasets. You could just blindly choose a search algorithm, like Quicksort, and apply it every time. However, if the data is nearly sorted already, something like bubble sort or insertion sort might be a better idea. But how do we know when this is the case? One approach would be to sample the data and compute some statistics over it, like the expected size of a run¹ and the average pairwise difference. Then, you could run a lot of experiments and try to come up with a hand-tuned heuristic to pick a sorting algorithm based on the sampled information.

But how well will the heuristic you designed work when the data distribution changes? Or when your system is moved to new hardware (possibly with larger caches, etc.)? Or you want to add some other sorting algorithm? Some of the thresholds you picked might need to be tuned… every single time.

So why don’t we just learn the function that maps our sampled statistics to the optimal sorting algorithm? And better yet, let’s do it in an online fashion, using a corrective feedback loop where we learn from our mistakes.

State
Quicksort
Insertion
Bubble
ExperienceThe total number of sorts the agent has performed, and thus the amount of training data acquired, so far.
Net regretThe time difference between the action taken by the policy (the agent or Quicksort, QS) and the optimal action over every episode so far.	Agent: / QS:
Net savingsThe amount of time saved (if positive) or lost (if negative) by using the learning agent instead of picking Quicksort every time.

Right now, your computer is busily training and evaluating a deep reinforcement learning model² which, when fed the sampled statistics as input, will predict how long each of three sorting algorithms (Quicksort, bubble sort, and insertion sort) will take to sort the data. The graph shows the regret, the difference between the decision made and the optimal decision, for two policies: the learned model, and a policy that uses Quicksort every time. The “net savings” measures how much time has been saved (or lost) by using the learned policy compared to Quicksort. Over time, this number should increase, indicating that the agent is learning.

What’s going on here?

Every time a new set arrives for sorting (an episode), we feed the sampled statistics (the context) to the neural network, and we use the network’s prediction to select a sorting algorithm (an action). After we select an algorithm, we sort the set and see how good our decision was by measuring the latency (the reward). This problem setup, a restriction of general reinforcement learning, is often referred to as the contextual multi-armed bandit problem³, in which:

At the start of each episode, an agent receives a context (sampled statistics), containing information relevant to the current episode.
The agent selects an action (a sorting algorithm) from a finite set.
The agent receives some reward (the time it takes to sort).

The agent’s goal is to maximize the reward it earns over time (i.e., minimize the latency of sorts over time). In order to do so, the agent must explore the relationship between contexts, actions, and rewards, while simultaneously exploiting the knowledge it has gathered. Our solution to this problem is Thompson sampling, a straight-forward technique that keeps track of (context, action, reward) triples and samples a model to predict the reward of each action based on the context.

We re-train the model every 40 episodes. As the model gains more and more experience (observed triples), the model’s prediction gets better and better, and thus the resulting policy improves as well.

If you watch the graph for long enough (normally around 250 episodes), the regret of the learned policy should go to zero, meaning that the optimal policy has been learned. Since we use Thompson sampling to build a new model every 40 episodes, there are occassionally spikes, or variance, in the learned model’s policy. However, the net savings of the learned model tends to remain significant.

In a sense, this means our learned model is trading worst-case performance for average-case performance: most of the time, the learned model will choose a good algorithm, but every once in a while, Thompson sampling will produce a catestrophic policy,⁴ and performance will degrade significantly for a short period of time. But, on average, the learned agent will perform better than the static policy.

The Big Picture

Machine learning for systems – that is, applying machine learning techniques to problems within computer systems – is coming, and it’s coming fast. Whether its index structures, scheduling, or even database query optimization (including cardinality estimation), machine learning is creeping its way into the domain of systems. It has been called Software-Defined Software, Software 2.0, learned systems components, or self-assembling systems. But regardless of the name, “machine learning for systems” brings about both exciting possibilities and significant new challenges.

The advantages of machine learning powered systems could be huge: for database systems, query optimizers contain a huge number of heuristics that must be painstakingly maintained by hand, and often even require application-specific tuning by a DBA. Some of our recent research shows how deep reinforcement learning can produce query optimizers that automatically tune themselves to specific applications, matching and sometimes exceeding the performance of complex commercial optimizers. In their recent paper, Kraska et al. argue that similar applications of machine learning could improve almost every aspect of a database, including data access, query optimization, query execution, and advanced analytics.

However, there are numerous challenges to overcome.

Like in the example shown here, learned systems can perform better on average, but might have significantly higher variance, especially in early episodes. Figuring out how to reduce this variation will be critical for systems applications that depend on low tail latencies. Several approaches, like learning from demonstration or bootstrapping a model from a cost function, have recently been investigated.
Understanding when models will – and won’t – work is still a major open question. While we know that neural networks can approximate any continuous function, and that algorithms like table-based Q learning converge to optimal policies for any MDP, we are still a long way from having theoretically-justified bounds on practical learning systems. It is possible that such bounds may never be known, and that it is easier to adjust the existing learning systems to have better behavior at the margins. Such techniques could be a simple as using different models for outliers, or may be as complex as modifying database internals to work adaptively.
Doing worse takes longer than doing well. If you are playing a video game and not doing very well, the score printout on the screen will tell you just how bad you doing. If you are doing well, the score printout will convey this information as well, and equally quickly. For systems problems, however, if your initial policy is poor, evaluating it may take a long time. For example, for join order selection in a relational database, a bad join ordering may take orders of magnitude longer to complete than a good one. Thus, how long it takes to “read the score” is a function of the score itself. This means that the initial episodes of learning (in which many RL algorithms start with a random policy) could take a prohibitively long amount of time.

These challenges, and many others, are discussed in our recent workshop paper at RL4RealLife (ICML ‘19). We’ve also released an open source platform called Park to enable researchers to test reinforcement learning algorithms on systems problems, including network congestion control, Spark scheduling, load balancing, and a dozen more. The future of systems is learned!

FAQs

This seems really dumb. Why don’t you just… This is not supposed to be a practical system. It’s supposed to be a demonstration of a learned policy being useful in a systems context. Clearly, a real world system would need significantly more testing, and would likely use a much simpler model. Please see the many linked works in the first paragraph of “The Big Picture” section for more practical examples.

Are you including the featurization and inference time in your measurement? Inference time is not included, but featurization time is included: the always-quicksort policy benefits from not having to compute features, and the learned policy is punished. Note that the zero-regret line on the plot assumes the featurization times – in other words, an oracle policy that didn’t even require the features as inputs would have a (slightly negative) regret.

Shouldn’t you just be using Timsort? Maybe. Depends on the hardware, data distribution… probably a large number of other factors. This system is flexible – you could add in Timsort, and, if a particular piece of hardware and data distribution caused Timsort to always be optimal, Timsort would always be selected. In this case, you would be wasting time computing features. It might be interesting to have the policy select which features it wanted to gather for itself, so that the network could learn which when/which features were useful.

A run is a contiguous sorted subsequence, i.e., a cluster of data that is already sorted within the larger dataset. ↩
With just two inputs and one output, we could’ve easily (and, if this were a real system, should’ve!) used something simpler, like a decision tree. But what fun would that be? ↩
There are many overviews available. For one, see: Chapelle and Li. “An Empirical Evaluation of Thompson Sampling.” In Advances in Neural Information Processing Systems. NIPS’11. ↩
Testing these catastrophic policies is a necessary part of most reinforcement learning – after all, if we do not test the policy, how will we ever learn it is catastrophic? We don’t want the performance of real world systems to suffer while our models explores no-good possibilities. Figuring out how to avoid such catastrophic detours is an active area of research. ↩

Good comment, bad comment

Mon, 05 Nov 2018 00:00:00 +0000

Having spent five years heavily involved in teaching the 2nd semester programming course at Brandeis, I’ve found myself repeating the same maxim over and over again:

Code is an essay written for two audiences: the computer, and your fellow programmers.

Thinking about the dual-audience nature of code immediately highlights several important aspects of “good code”:

When “read” by the computer, good code leads to a correct, effective, and efficient behavior. The desired outcome is achieved (correct) in a reasonable amount of time (effective) without consuming an unreasonable amount of resources (efficient).
When read by your fellow programmers, good code reads like an essay that intuitively, succinctly, and precisely describes the semantics of the problem and how it is solved. In other words, the code is not overly long and verbose (succinct), the code conveys the exact details of what is happening to the reader (precise), and the code does so in a way that makes sense to the reader quickly (intuitive).

In my experience reading thousands (really!) of different solutions to the same problem, I find that the best code follows all six of these principles. In this post, I want to go through each principle, showing both good and bad examples. Each example will be a “real” piece of code written by a student (anonymized, of course), including myself (I have to get the bad examples from somewhere).

For the computer
For your fellow programmers
Conclusions

For the computer

All in all, contemporary computer science departments do a pretty good job of teaching students to write correct, effective, and efficient code. In fact, new programmers tend to optimize for their robotic audiences over their fellow humans – “if it works, who cares?” If a computer wasn’t part of our audience, we might not choose to write code at all.

However, new programmers are often very frustrated by a computer “not doing what I meant.” Understanding how to translate our thoughts into code is a fundamental skill that looks a whole lot like the writing process. The next three sections explain what I believe to be the three properties of good code from the point of view of the computer audience.

Correct

When thinking about the computer as the audience for our code, we want our code to cause the computer to compute the correct result. Correctness may seem like the most obvious principle we would want good code to embody – but correctness is often subtle, underspecified, or misunderstood. For example, consider the following prompt and solution:

Write a method randItem(int[] arr) that returns an item selected uniformally at random from the passed array.

public static int randItem(int[] arr) {
    return arr[(int) (Math.random() * arr.length)];
}

At first glance, you might think this code is just fine – Math.random() returns a uniform value from 0 to 1, multiplying that by the array length and truncating to an int should give a uniformally distributed array index. You might raise a complaint that the programming didn’t use Java’s Random class, but that’s not a correctness issue, right?

It turns out the above code is very subtly incorrect. To understand why, let’s simplify the problem a little bit. Imagine that our input array arr is of size 5, and that Math.random() returned a 4-bit random number between 0 and 1. Let’s look at the possible values of our array index:

Binary	Decimal	Array Index
0000	0.0000	0
0001	0.0625	0
0010	0.1250	0
0011	0.1875	0
0100	0.2500	1
0101	0.3125	1
0110	0.3750	1
0111	0.4375	2
1000	0.5000	2
1001	0.5625	2
1010	0.6250	3
1011	0.6875	3
1100	0.7500	3
1101	0.8125	4
1110	0.8750	4
1111	0.9375	4

Spot the problem? In this case, four values of Math.random() map to index 0, but only three values map to all the other elements! That means this method will return the first element of the array with probabilty , while the other elements will each be returned with a probability of ! That’s not uniform at all.

Now, Math.random() returns a double, which has a lot more than 4 bits, and only a subset of all possible double values are within the range (0, 1). But, saying there are exactly possible return values for Math.random() and elements in our array arr, this method only potentially works if divides , i.e. n % k == 0.

A more correct way to write this code would be:

private static final Random r = new Random();
public static int randItem(int[] arr) {
    return arr[r.nextInt(arr.length)];
}

Since the nextInt method ensures that we are drawing from a uniform distribution,¹ this code produces a result more in-line with the prompt. The point here is that correctness can be subtle, and that a computer will happily do whatever you tell it – and sometimes the wrong thing can look a whole lot like the right thing at first glance.

Effective

In addition to being correct, we also want our code to cause our computer audiences to compute results effectively. We all know we want code that runs fast, and we learn tools like big-O analysis to help us make good decisions along the way. But sometimes, something seemingly-mundane, or that you think should be fast, can end up taking a very long time. There’s even a Tumblr about this phenomenon in very real code. Here’s an example of this in code written by myself a long time ago, and by numerous other students ever since:

Write a method combine(String[] args) that returns a String containing each element of args connected with spaces, in order. If args is empty, return the empty string.

public static String combine(String[] args) {
    if (args.length == 0) return "";
    
    String collector = "";
    for (String el : args) {
        collector += el + " ";
    }
    
    // Cut off the last space.
    return collector.substring(0, collector.length() - 1);
}

Seems straight-forward enough, right? Cutting off that last space was a little tricky. New developers might not even think to analyze the runtime of this method, or if they did, they might declare it to be linear in the size of args. More seasoned (read: cynical) developers might begin to smell trouble, though.

The way Java represents strings is different from many other languages. In Java, strings are immutable. Once they are created, they can’t be modified. So if they can’t be modified, what happens when the program executes collector += el + " "? Well, unfortunately, Java will create a brand new string, copying the contents of collector, el, and " " into it. Java will perform this create and copy operation at every loop iteration, which means that our runtime is actually quadratic in the size of args!

Since the good folks working on Java know that developers sometimes need to concatenate strings, Java has a handy StringBuilder class for this exact situation!

public static String combine(String[] args) {
    if (args.length == 0) return "";
    
    StringBuilder collector = new StringBuilder();
    for (String el : args) {
        collector.append(el);
        collector.append(" ");
    }
    
    // Cut off the last space.
    return collector.substring(0, collector.length() - 1);
}

Now our code is much more effective.² The point here is to show that performance bugs can often be hard to notice, and can creep into code in subtle ways that might elude someone who isn’t looking closely for them.

Efficient

When I was a new programmer, I didn’t really think about the resource usage of my application. It either ran, or it didn’t. Most of the programs I was writing were “short-lived”: I started them and waited for them to finish, normally a few moments later. Even if one of my programs used a ton of memory or temporary disk space, I wouldn’t notice, because my operating system would liberate all these resources from my greedy program after my program terminated.

However, as I gained more experience and started writing longer-living programs, like servers (or “services”, as the kids call ‘em nowadays), I started to feel the pain of all my inefficient code. This pain came mostly in the form of memory leaks: bugs in my programs that caused more and more memory to be used up until there was none left.

Many programmers who swear by languages like Java or Python believe that their language’s garbage collector – the part of the language responsible for reclaiming objects we aren’t using anymore – makes them immune to memory leaks. While it is true that these garbage collectors prevent the majority of memory leaks, I was a poor enough Java programmer to create several! Here’s an example of one written for a class (not by me, but making the same mistake I did):

Write a class BoundedStack<T> with methods push and pop that behaves like a stack with a bound on the number of items it can contain. Take the bound as a parameter in the constructor.

public class BoundedStack<T> {
    private Object[] storage;
    private int topIdx = 0;
    
    public BoundedStack(int bound) {
        storage = new Object[bound];
    }
    
    public void push(T arg) {
        if (topIdx >= storage.length) 
            throw new IllegalStateException();
        storage[topIdx++] = arg;
    }

    public T pop() {
        if (topIdx <= 0) 
            throw new IllegalStateException();
        return (T) storage[--topIdx];
    }
}

Overall, looks fine, right? But this implementation of a bounded stack actually has a memory leak. Consider the following client code:

public static void main(String[] args) {
    BoundedStack<ReallyBigType> s = new BoundedStack<>(5);
    
    s.push(new ReallyBigType(1));
    s.push(new ReallyBigType(2));
    s.push(new ReallyBigType(3));
    s.pop(); s.pop(); s.pop();
    
    performTask();
}

Imagine that ReallyBigType is a class that uses a whole bunch of memory. When performTask is called, the programmer might be expecting that all the ReallyBigType instances created beforehand will be freed up, so that performTask can use as much memory as it needs. However, this isn’t really the case – all three instances of ReallyBigType are guaranteed to still be around when performTask is called!

The reason all three instances of ReallyBigType are still around is that our BoundedStack, s, is still holding a reference to them. Even though we decremented the topIdx variable whenever pop() was called, the values in storage (in positions 0, 1, and 2) still point to our instances of ReallyBigType! That’s a memory leak. If this were a large, long-running application, small errors like that could build up, resulting in a system that needs to be frequently rebooted.

Luckily, this memory leak is easy enough to fix:

public class BoundedStack<T> {
    private Object[] storage;
    private int topIdx = 0;
    
    public BoundedStack(int bound) {
        storage = new Object[bound];
    }
    
    public void push(T arg) {
        if (topIdx >= storage.length) 
            throw new IllegalStateException();
        storage[topIdx++] = arg;
    }

    public T pop() {
        if (topIdx <= 0) 
            throw new IllegalStateException();
            
        T tmp = (T) storage[topIdx];
        storage[topIdx] = null;
        topIdx--;
        
        return tmp;
    }
}

By setting the now-unused cells of the storage array to null, we avoid holding a reference, allowing the Java garbage collector to do its work. The point here is that just writing fast, correct code isn’t enough: good code behaves well even in long running, resource-scarce environments.

For your fellow programmers

Despite all the difficulties of writing to your computer audience, writing to your human audience is often much harder. After all, writing and composition³ have entire classes of their own! Good code shares a lot of properties with good writing. Specifically, I’ve found that most good (technical) writing and good code I’ve read follows three principles. First, I’ll give an overview of each of these principles, and the proceeding subsections will expand on them.

Intuitive: It is well-structured and easy to understand. For human readers, packages, classes, methods, and code chunks serve a similar purpose to chapters, sections, subsections, and paragraphs. Breaking down complex ideas into small, composable pieces is a fundamental skill in both programming and writing. This is how we turn complex ideas into intuitive concepts.
Succinct: It is just long enough. The phrase “if I had more time, I would write a shorter letter” comes to mind here. In high school, you may have learned to “pad out” your essays with unnecessary verbiage. Unfortunately, you’ll need to unlearn that, both for writing code and papers. Nobody wants to spend three hours reading your multi-page comment. The goal should be to get the idea across in the fewest words possible (so perhaps this blog post isn’t the best example).
Precise: It isn’t too “hand-wavy.” The code should communicate to the human reader exactly what is going on, and not attempt to “gloss over” or cover up the details.⁴

Now, luckily for us, we have a bit of a “cheat code” when communicating with human audiences through our code. Specifically, we have comments, which are invisible to our robot audiences, and variable names, which are meaningless to our robot audiences. We can take code the computer is already happy with (correct, effective, efficient), and we can use these two tools to annotate that code to make our human audience happy as well.

As a warning, this part is going to be a lot more subjective than the previous part. I’ll do my best to give examples that I think most of the professional software engineers I’ve worked with would agree on. It’s important to realize that a lot of what makes code readable (and what makes writing good or bad) depends on your audience. So if your audience has certain norms – like indention style, variable naming style, etc. – following them is in your best interest. Otherwise, you’ll be written off from the get-go.

Take a quick look at the two papers below.

You don’t have to be able to see the words to know which paper “looks” better. We have established conventions for writing, and when they aren’t followed, it is immediately jarring. And while eschewing these structures can sometimes be considered “art,” you generally want to match the expectations of your audience. Software engineers are no different – if your code doesn’t “look” right (perhaps it isn’t indented properly, or your variable names are unconventional), the actual code you wrote probably won’t matter… the engineer will just dismiss it out of hand.

Part of becoming a good programmer is learning and practicing those norms. You can take advantage of them to convey information quickly and succinctly, or you can fight an uphill battle against them. Unless you’ve got a really good reason, don’t try to make your readers learn a new convention and your complex idea at the same time! Instead, try to use the conventions as rails for your code.

These next sections will continue with a mix of Java and Python, but I’ll do my best to include only the norms that are common among most languages. Any specific language you use likely has its own norms (for example, JavaDoc), and you should learn them – but we’ll keep it general here.

Intuitive

Most (scientific / technical) writing has the goal of conveying intuition. The writer has an idea in their head that they’ve thought about deeply. They’ve analyzed it, broken it down, tried a million different things, and, at long last, they have a really good understanding of it. The writer’s goal is to create a paper that, when read by other scientists, “fast-forwards” them through that potentially multi-year process, transferring the intuition in their head to paper. In this way, reading a paper can be a lot like time travel: it lets you learn what took years to discover in just a few short hours. This is what Newton meant when he said “if I have seen further it is by standing on the shoulders of Giants.” Good code works the same way.

This is also an extremely difficult task, and the people who are the best at it often become professors at research institutions. It’s a high bar for writing, and it’s an even higher bar for code, but the same idea applies. Developers who write code that can convey intuition to other developers drastically boost the productivity of their teams, and ensure the longevity of their projects. The people who are best at it become the director of AI for Google or senior technical fellows at Microsoft.

Consider the following prompt and solution:

Write a method isUnique which, when given a Map<String, String>, returns true if no two keys map to the same value, and false otherwise. Return true if the map is empty.

public static boolean isUnique(Map<String, String> arg) {
    return (new HashSet<>(arg.values())).size() == arg.size();
}

Now, this code is correct, and is reasonably efficient and effective. It is, however, still very bad code. The reason why it is bad code is that the code isn’t intuitive – there’s no connection from the line as written to the prompt. Here’s a way to improve it:

public static boolean isUnique(Map<String, String> arg) {
    // By placing all the map's values into a set, we 
    // collapse any duplicates. Then, if the number
    // of uniqueValues is equal to the total number 
    // of values, all values must be unique.
    Set<String> uniqueValues = new HashSet<>(args.values());
    return uniqueValues.size() == arg.size();
}

In my opinion, this comment and the new, named intermediate variable greatly clarify the code. Of course, it still needs a “top level” comment above the method signature to describe what it does, but that will look a lot like the prompt. Here’s another way to improve the original code:

public static boolean isUnique(Map<String, String> arg) {
    // Test if bijection using incl/excl.
    Set<String> AuB = new HashSet<>(args.values());
    return AuB.size() == arg.size();
}

Now, is this “good code”? Maybe if you were writing a mathematics library and all of your potential readers were extremely well-acquainted with the inclusion-exclusion principle, but in most situations this might be even worse than where we started. The majority of software engineers would scoff at this, likely assuming it was pretentious.

Another way to make code more intuitive is to connect each variable, method, etc. to a semantic concept in the real world. Remember, most code is trying to model or represent some real-world thing or idea. For example, consider the following code:

def compute_point(a, b, c):
  a %= 12
  a /= 12
  a += (b / 60) + (c / 60*60)
  a *= math.pi * 2
  return (math.cos(a), math.sin(a))

Now, what does this code do? Well, I know. Because I wrote it. Or I should say, I know today, and maybe next week, but I might not know in a year. However, by just changing the variable names, we can convey a whole lot more intuition:

def compute_hour_hand_position(hrs, mins, secs):
  theta = hrs % 12
  theta /= 12
  theta += (mins / 60) + (secs / 60*60)
  theta *= math.pi * 2
  return (math.cos(theta), math.sin(theta))

Now, the resulting code isn’t perfect, but it’s way better. I can now definitely tell, for example, that this has something to do with time. It probably has something to do with an analog clock. If I have some more context from the file, I might know immediately that this code is computing the (x, y) coordinates for an hour hand on clock. With just a few comments, we can make this obvious:

def compute_hour_hand_position(hrs, mins, secs):
  theta = hrs % 12
  theta /= 12
  theta += (mins / 60) + (secs / 60*60)
  theta *= math.pi * 2 # convert to radians
  
  # Compute x and y coordinates from the angle.
  return (math.cos(theta), math.sin(theta))

By connecting the code to the real world, I now understand why the first line includes a hrs % 12 (the clock wraps around), and I understand why mins is divided by 60, and why secs is divided by 60*60. Vastly improved over the initial version.

While none of the examples shown here are large or overly-complex, conveying the intuition behind your code only becomes more and more important as you “scale up” (i.e., get bigger and more complex). Practicing these techniques on small projects or assignments is critical to slowly developing the skills you need to write more complex code in an intuitive way.

Making code intuitive is easily the hardest task facing any software engineer. It’s so extremely difficult that some developers⁵ have disavowed it entirely, claiming that their code “just makes sense to smarter people.” There’s a ton of this fake-flex, over-confident compensation in computer science. Try to assure yourself that they’re deluding themselves, and then try to avoid them as best you can.

Succinct

One of the biggest mistakes I see new programmers make is being overly verbose with comments. Here’s a real(!) example of a simple program that prompts the user for their name and then greets them:

// the main entry method for the program, which
// the java command line program will execute.
public static void main(String[] args) {
    // prompt the user for their name using System.out, which 
    // is a PrintStream class. The PrintStream class has a
    // method called println, which will output the text
    // passed to the console (so that the user can see it) 
    // and then print a newline.
    System.out.println("Welcome to my program! What is your name? ");
    
    // this makes a new scanner, which can read from 
    // STDIN, located at System.in. The scanner lets us look
    // for tokens, aka stuff the user has entered.
    Scanner sc = new Scanner(System.in);
    
    // this will create a new string variable called "name" 
    // and set the value to whatever the user types next
    String name = sc.nextLine();
    
    // finally we use println again, and string 
    // concatenation, with the + operator, to 
    // output or (sic) message!!
    System.out.println("Hello " + name + " ! ");
}

Now, clearly this was written by a new programmer who has paid attention in class, and was told to explain things in comments. To a new developer, all these language features (like the main method) are new, and seem worth… well, commenting on. No points against the new programmer. Part of the learning process.

However, this is a great example of over-commenting. Any software engineer reading this will be quite annoyed wading through the long, verbose comments that are explaining Java language features. As a general rule, you should never use comments to describe a language feature: just assume your reader knows the language as well (perhaps an exception could be made if you use a really weird feature or something). Try to use comments to talk about the why, rather than the what. For example, we could rewrite this program like so:

public static void main(String[] args) {
    // Prompt the user for their name and greet them.
    System.out.println("Welcome to my program! What is your name? ");

    Scanner sc = new Scanner(System.in);
    String name = sc.nextLine();

    System.out.println("Hello " + name + " ! ");
}

We use a single comment to say “the why” of what we are doing. That is, we are describing the semantics of the code as opposed to the syntax of the code. Then we use some line breaks (“paragraphing”) to separate out the input and output sections. Much easier to read.

Verbosity can come from more than just comments. Often, the code itself can be repetitive or overly verbose. Consider the following program:

Write a method checkNames that takes in a List<String> of names and returns the valid names. A name is valid if it begins with a alphabetical character and has a length of at least 3.

public static List<String> checkNames(List<String> names) {
  // creating a new arraylist
  List<String> goodNames = new ArrayList<String>();
  //looping over the input
  for (int i = 0; i < names.length; i++) {
    String currentName = names.get(0);
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          if (currentName.length() > 2) {
            goodNames.add(currentName);
          }
        } else {
          continue;
        }
      } else {
        continue;
      }
    } else {
      continue;
    }
  }
  return goodNames;
}

There’s a lot going on here, so we’ll try to break it down. The first thing almost any programmer will notice is that the code is very “crowded” – there’s no “paragraphing” or blank lines to indicate the logical chunks of the code. The second .length() > 2 check is also completely redundant. The only two comments are also entirely useless. So let’s fix those things first:

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (int i = 0; i < names.length; i++) {
    String currentName = names.get(0);
      
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          goodNames.add(currentName);
        } else {
          continue;
        }
      } else {
        continue;
      }
    } else {
      continue;
    }
  }
  
  return goodNames;
}

Already, the code is starting to look a little bit more like code. The next thing we notice is that all of those else blocks are useless – the loop continues without the explicit continue statement anyway. So we can remove them.

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (int i = 0; i < names.length; i++) {
    String currentName = names.get(0);
    
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          goodNames.add(currentName);
        }
      }
    }
  }
  
  return goodNames;
}

Much shorter, but we can go a bit further. The index variable of the for loop, and the currentName variable, can be replaced with the “for each” pattern.

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (String currentName : names) {
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          goodNames.add(currentName);
        }
      }
    }
  }
    
  return goodNames;
}

Now we’re getting somewhere. The next refactor is a bit of a matter of opinion, with older software developers generally siding “against” and younger software developers generally siding “for.” One heuristic you can use to try to tell if your code is too verbose is to count the number of levels of indentation. If you’re over three, something has probably gone wrong. We can fix it like this:

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (String currentName : names) {
    if (currentName.length() <= 2) continue;
    if (currentName.charAt(0) < 'a') continue;
    if (currentName.charAt(0) > 'Z') continue;

    goodNames.add(currentName);
  }
  
  return goodNames;
}

Every level of indentation adds another level of “context” that the developer has to remember, e.g. “ok, I’m inside this for loop, and this if statement, and this if statement…” By inverting each condition, each if statement now reads more like “ignore names that fail this test.” Putting three of them in a row using the exact same formatting takes advantage of parallel structure, just like in writing. By the time you’ve read through the three if statements, its clear you’re left with a name that meets the conditions.

Finally, we can use a little Java API knowledge to make it slightly better, and add some semantic comments, to finish it off:

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();
    
  // Find the names that: (1) have three or more characters
  // and (2) start with a letter.
  for (String currentName : names) {
    if (currentName.length() <= 2) continue;
    if (!Character.isLetter(currentName.charAt(0))) continue;

    goodNames.add(currentName);
  }
    
  return goodNames;
}

If we want to get a little advanced, we could’ve written the whole thing in an even more succinct way using Java streams:

public static List<String> checkNames(List<String> names) {
  // Find the names that: (1) have three or more characters
  // and (2) start with a letter.
  return names.stream()
          .filter(name -> name.length() > 2)
          .filter(name -> Character.isLetter(name.charAt(0)))
          .collect(Collectors.toList());
}

The point of writing succinct code isn’t to show off. In fact, sometimes shorter code can be much more complex and harder to read. For example, consider swapping two variables in Java:

int a = 5;
int b = 9;

int tmp = a;
a = b;
b = tmp;

Every developer will recognize this as “swap code.” It’s a common pattern we see everywhere. However, an overly-clever developer might write:

int a = 5;
int b = 9;

a = a ^ b;
b = b ^ a;
a = a ^ b;

“It’s shorter by a whole four characters!”, proudly declares a young developer who is about to get a talking-to. This is known as the XOR swap algorithm, and it is a neat trick, but its very opaque. Nobody knows what it is doing the first time they see it, and even after they know the trick, they might not be sure everything is in exactly the right order.⁶ This makes the code very unintuitive. Making code less succinct doesn’t conflict with making code intuitive as often as you’d think. Normally, making things shorter, while staying inside the conventions and norms of your language and community, makes things better. Normally. Experience will teach you when something falls into the “XOR swap” category or not.

Overall, making things short and succinct makes your code more readable. Other developer’s eyes won’t instantly glaze over as soon as they see your code. Since anything but a superhuman-level software developer can only keep so many lines of code in their head a time, making code shorter can often decrease the cognitive load required to read and understand your code. It’s tempting to measure the progress of a project by the number of lines of code you’ve written, but I promise, senior devs will respect a “small codebase that can do a lot” much more than a large codebase that only does a little.

Precise

The last principle for our human audience is precision. Writing comments that tell the user vaguely what is happening is relatively easy – but writing a comment the succinctly conveys exactly what a program is doing is much, much harder. Luckily, code has to be precise, so at a minimum, an unlucky developer can simply wade through every line to try and figure out what is going on. But we should have loftier goals.

Consider the following prompt and solution:

Write a method that, given a height from which a ball was dropped in meters, computes the velocity of the ball when it hits the ground.

public static double velocity(double height) {
    // compute the v with the energy formula
    return Math.sqrt(2.0 * 9.8 * height)
}

That’s short and sweet, and it might even be good enough if your audience has a physics background. But for us non-physics folks… a little more explanation might be nice. Note that the comment “compute the v with the energy formula” is entirely correct. It describes what the code does, semantically, but it is a little hand-wavy. What energy formula? There’s a few. What’s 9.8? How exactly did you reach this solution?

A better version might be:

private static final double GRAVITY = 9.8;
public static double velocity(double height) {
    // Compute v using the conservation of energy formula:
    //     mass * gravity * height = (1/2) mass * velocity^2
    // Mass cancels. Solve for velocity:
    //    sqrt(2 * gravity * height) = velocity
    
    return Math.sqrt(2.0 * GRAVITY * height)
}

Now the code is a bit longer, but it is also a lot more precise – a reader without a ton of physics background can at least step through the algebra to convince themselves that the code is correct. Extracting 9.8 to a variable also makes the code more precisely match the formula. Again, if your target audience is a bunch of physicists, the commentary is probably not required. Context is king.

There’s another, very easy way to add precision to code’s explanation, although it is only applicable in a few cases. Check out this chunk of code:

double[][] E = new double[k][n];
int[][] J = new int[k][n];

for (int j = 0; j < n; j++) {
	E[0][j] = err(0, j);
	J[0][j] = 0;
}

for (int i = 1; i < k; i++) {
	E[i][i] = 0; 
	J[i][i] = i - 1; 
	for (int j = i + 1; j < n; j++) {
		E[i][j] = Double.POSITIVE_INFINITY;

		for (int x = j - 1; x > i-1; x--) {
			double e = err(x, j);
			double se = E[i-1][x] + e;
			if (se < E[i][j]) {
				E[i][j] = se;
				J[i][j] = x;
			}
			
			if (E[i][j] < e) break;
		}
	}		
}

Understand what’s going? If so, I’m extremely impressed (and perhaps you need to see the last paragraph of the “Intuitive” section). To me, this is just about as bad as it gets. Short, one letter variable names. Absolutely no commenting. Random, complex manipulation of 2D arrays.

… but what if I told you that this one weird comment would transform this code from terrible to completely acceptable? Ready? Here it is…

// Algorithm for optimal piecewise approximation. 
// Transcribed exactly from:
// Mahlknecht, Dignös, and Gamper,  “A Scalable Dynamic 
// Programming Scheme for the Computation of Optimal 
// k-Segments for Ordered Data.”
// https://doi.org/10.1016/j.is.2016.08.002
double[][] E = new double[k][n];
int[][] J = new int[k][n];

for (int j = 0; j < n; j++) {
	E[0][j] = err(0, j);
	J[0][j] = 0;
}

for (int i = 1; i < k; i++) {
	E[i][i] = 0; 
	J[i][i] = i - 1; 
	for (int j = i + 1; j < n; j++) {
		E[i][j] = Double.POSITIVE_INFINITY;

		for (int x = j - 1; x > i-1; x--) {
			double e = err(x, j);
			double se = E[i-1][x] + e;
			if (se < E[i][j]) {
				E[i][j] = se;
				J[i][j] = x;
			}
			
			if (E[i][j] < e) break;
		}
	}		
}

This is a (in my opinion) complex dynamic programming algorithm for approximating a function of one variable with k line segments. There’s absolutely no reason to re-explain an entire scientific paper in the comments of your code, unless you feel like you can do it way better than the original authors. In terms of describing an algorithm, citing the paper (or textbook, with page number) from which it was transcribed is just about as precise (and succinct!) as you can get. Of course, this situation doesn’t come up too often, but when it does, it will be one of the most useful-per-character comments you’ll ever write.

Conclusions

My experience is limited, and I’m far from the best software developer in the world, but I have worked at quite a few places, programmed for a reasonably long time, and read a lot of programming assignments! So perhaps you don’t want to treat these as steadfast rules, but I do think they are good general principles. Here’s the cheat-sheet:

Code is an essay written for two audiences: the computer, and your fellow programmers. It should strive to obey these principles.

Correct: the code should produce the correct output.
Effective: the code should run quickly.
Efficient: the code should minimize resource usage.
Intuitive: the code should convey the intuition behind it.
Succinct: the code should be short.
Precise: the code should allow humans to understand exactly what it is doing.

Of course, it is worth noting that you don’t always have to write good code. In fact, most of the time, you won’t. If you go over to my GitHub, you’ll see most of the code I’ve written is… bad. Real bad. Like, super bad. These are things to strive for, and things to practice. You won’t always be able to write code that is especially pristine. But hopefully, bit by bit, year by year, we can get better and better.

Footnotes

You might be wondering how nextInt manages to draw from a uniform distribution – one approach is to use rejection sampling. First, pick the largest number m less than Integer.MAX_VALUE that the size of your array, n, still divides. Then, sample a random integer from the range (0, Integer.MAX_VALUE). If the sampled integer is less than m, return m % n. Otherwise, try again. ↩
We could also use Java’s streams, and the joining collector, to make the code much shorter and more obvious to a human, but we’re only discussing effectiveness here. ↩
There’s a reason it’s called “composition.” Writing code and writing papers both require careful compositions of abstractions. In computer science, these abstractions are linked lists, stacks, queues, etc. In “traditional” writing, these are paragraphs, citations, ideas, etc. Computer scientists who dismiss literature and other humanities will find themselves at a huge disadvantage. ↩
This does not mean you cannot create your own abstractions, or use others. It also does not mean you have to explain how an ArrayList works every time you use one. But someone looking at your code should be able to find those details somewhere (perhaps in your code, perhaps in the Java documentation), if they look for them. ↩
I’ve been guilty of this in the past, and I hold no grudge against those who avoid me to this day because of it. ↩
It turns out the XOR swap is also a tad slower than the “normal” swapping code – mostly due to compiler optimizations. ↩

Compiled and vectorized evaluation in columnar engines

Compiling engine strategy

Vectorized engine strategy

Bakeoff #1: 50% Selectivity

Bakeoff #2: 0.04% Selectivity

Bakeoff #3: End-to-end optimizations

The best of both worlds?

Notes

Neo, 6 years and 600 citations later

What is Neo?

How’d we get there?

Key Problem 1: Sample Inefficiency

Key Problem 2: Inductive Bias

What happened next?

Footnotes

2024's hottest topics in databases (a bibliometric approach)

Learned indexes

(Learned) query optimization

DBMS engines

ML-powered system tuning

Data cleaning and matching

What’s next?

Related work search for database papers

Search related work

Highly relevant

Somewhat relevant

Tangentially relevant

How it works

Ten years of improvements in PostgreSQL's optimizer

Notes

Most influential database papers

Most influential papers of all time

Most influential papers, by year

Most influential people

Incoming citations ()

Outgoing citations ()

Similar papers

Notes

Generating bios with large language models

Footnotes

Applying Bao to distributed systems

How Bao works

From single-node to distributed

Vertica

Azure Synapse (SQL Server)

RedShift

Discussion

Notes

Machine learning for systems

What’s going on here?

The Big Picture

FAQs

Good comment, bad comment

For the computer

Correct

Effective

Efficient

For your fellow programmers

Intuitive

Succinct

Precise

Conclusions

Footnotes