Ryan Marcus

Most influential database papers

2023-07-25T00:00:00+00:00

Ever wondered which database systems papers have been the most influential? This page explores the PageRank of a paper in the citation graph, one possible measure of influence.

All rankings, including this one, codify an ideology and thus won’t match up with everyone’s understanding of “influence.” Most notably, this ranking has nothing to do with industrial adoption, actual usage, or quality of an idea, other than by correlation with citation. Note that the citation graph grows every year, so papers published more recently will, on average, have a lower score.

PageRank is computed with the Python package NetworkX on top of the citation graph of all SIGMOD, VLDB, CIDR, and PODS papers. Self-citations are excluded. Absolute and percentile ranks are then computed for each paper using the PageRank score. Authors are ranked according to the sum of their paper’s PageRank score, where each paper’s score is divided by the number of authors, similar to CSRankings. Click one of the links below to explore the ranking!

Most influential papers of all time
Most influential papers by year
Most influential researchers

Most influential papers of all time

This is a list of all papers, sorted by their PageRank score. Search papers by title using the text box below.

Most influential papers, by year

This is a list of papers published in a particular year, sorted by their PageRank score. You can select a different year from the dropdown. Papers published in recent years have fewer citations, and thus more noise in their score.

Most influential people

This table ranks all authors of a paper that has been cited at least once (by another paper). An author’s score is the sum of their paper’s normalized PageRank score. The normalized PageRank score of a paper is the PageRank score of the paper divided by the number of authors on the paper.

Due to the exponential distribution of PageRank scores, the most influential papers have significantly higher scores than the median paper. As a result, authors of the most influential papers will rank highly here, even if they have few papers overall.

Generating bios with large language models

2022-12-20T00:00:00+00:00

It’s the holiday season of 2022, and everyone is talking about large language models! OpenAI’s GPT-3 and others are generating quite a bit of interest in the CS research community. Larger datasets and models are significantly improving results on many NLP tasks. Even in databases (my field), large language models are being used to tune databases and translate natural language to SQL.

Large language models are so good, some folks are tricking themselves into thinking that large language models are sentient. It is important to remember that there is no there there – GPT-3 and the like only model the distribution of the next word given the previous word.¹ As a result, large language models can at times easily produce misinformation, state biased viewpoints, or conjure total nonsense!

Let’s try it out. Enter the name of a computer scientist below to search for their DBLP entry. Click their name to ask GPT-3 to generate a bio for them using a list of their citations as an input.

A standalone version of this bio generator is available at https://bio.rmarcus.info.

In order to generate a bio, we fetch the 15 most recent publications for the author from DBLP, and then ask GPT-3 to complete the following text (“prompt”):

Given the below list of publications, produce a short summary of the research of {name}. Use succinct language when possible. Focus on common themes across all of their works. Do not enumerate specific topics or papers. Don't use phrases like "such as" or "including." Do not include specific dates, conferences, venues, or journals. Use gender-neutral language.

{publication list}

Bio: 

So GPT-3 starts generating text after Bio:, which normally results in some plausible descriptions of researchers. However, the model can certainly make mistakes. For example, despite the instruction to use gender-neutral language, the language model still outputs the pronoun “he” for Nesime Tatbul:

Nesime Tatbul is a computer scientist and researcher. His work focuses on machine programming, query optimization, explainable anomaly detection, cloud observability, and other topics related to data management. He has been involved in multiple projects, from developing a benchmarking platform for explainable anomaly detection to creating a pluggable metrics storage engine for the age of observability.

Additionally, despite being instructed not to list specific projects or papers, the model frequently does exactly that. This is somewhat understandable, since it might be impossible to determine an author’s general subfield from only their 15 more recent publications.

Because large language models are prone to produce bias, capable of producing nonsense, and are generally unpredictable, you probably shouldn’t use them in any “real” application yet. But that doesn’t mean you can’t play around with them and have fun! If you haven’t tried out ChatGPT, give it a shot here: https://chat.openai.com

Footnotes

A portion of the machine learning community subscribes to the belief that human beings are similar to such models (“there is no ghost in the machine”). We simply map inputs to outputs, and our consciousness is a manifested illusion. Once we multiply sufficiently large matrices together, AGI is achieved! I’m not an expert here, so I’ve gone with my intuition, which is that we are at least not entirely stochastic machines. ↩

Applying Bao to distributed systems

2021-06-17T00:00:00+00:00

This post was also published on the MIT DSAIL blog.

A good portion of my postdoc at MIT has been spent developing Bao, a system for learned query optimization with an eye towards practicality. Bao was recently published at SIGMOD 2021, where we were thrilled to receive a best paper award.

In our paper, we show how Bao can be applied to the open-source PostgreSQL DBMS, as well as an unnamed commercial system. Both DBMSes ran in a traditional, single-node environment. Here, we’ll give a brief overview of the Bao system and then walk through our early attempts at applying Bao to commercial, cloud-based, distributed database management systems.

For more information on Bao in the traditional, single-node context, check out:

Our research paper, published in SIGMOD 2021.
For a video overview, the recording of Ryan’s SIGMOD talk (20 minute version or 3 minute version).
For an overview of tree convolution as applied to query plans, Ryan’s AIDB 2019 talk.

However, since we wrote the paper almost a year ago, much has happened. First, We worked together with Microsoft to explore how Bao can help with Big Data workloads. This work will also be presented at SIGMOD as part of the industry session, and received a honorable mention for the industry best paper award.

Second, after the discussion we had with Mike Stonebraker around the potential impact Bao could have on distributed database warehouse systems (obviously, he was very skeptical), we ran a whole range of additional experiments on Vertica, Redshift, and Azure Synapse using a real-world dataset we received from an anonymous corporation.

How Bao works

Previous approaches¹ to learned query optimization attempted to replace large parts of traditional query optimizers. In contrast, Bao sits on top of a traditional query optimizer (called the underlying optimizer) and learns to steer the underlying optimizer in the right direction.

The image above illustrates the process Bao uses to steer the underlying query optimizer.

When a query arrives, the underlying optimizer is used to generate a number of plan variants for the query. For example, we might have the optimizer generate a plan using index nested-loop joins, another plan using sort-merge joins, and a third plan using hash joins.²
Once these plan variants are constructed, a predictive model (a deep neural network) predicts the latency of each plan.
The plan with the best predicted latency is selected and executed. The result is sent to the user, and the actual latency is recorded.
The actual latency of the recorded query is added to Bao’s experience set, which is used to further refine the predictive model using Thompson sampling.³

Over time, Bao’s predictive model learns from its mistakes, hopefully making increasingly accurate predictions. Experimentally, we’ve shown that Bao can learn in the presence of dynamic workloads, shifting data distributions, and schema modifications.

From single-node to distributed

In the original Bao paper, we evaluated Bao on a single-node database system (e.g., Oracle and PostgreSQL). However, many data warehouse databases are, in fact, distributed. Luckily, Bao is largely agnostic to the underlying execution engine or storage layout: as long as you have a set of hints, Bao can pick and choose from them and learn from its mistakes. In the paper, we discuss what good hints for a single-node DBMS look like: for example, forcing particular join types, or forcing particular access paths (i.e., index vs. table scan). These choices can impact the performance of the plan on their own, but can also impact the join order selected by the underlying query optimizer, potentially resulting in drastically different run times.

Adapting Bao to a distributed DBMS is simply a matter of finding the right hints. While many aspects of the single-node case apply to the distributed case as well (operator choice matters, join order matters), distributed DBMSes bring about other important performance considerations:

Suppose a fact table is distributed across multiple nodes based on a key column. Should a join of that fact table with a dimension table be done via sending the dimension table to all nodes and using a hash algorithm? By partitioning the dimension table on the foreign key and using a merge join and union? By collecting the matching rows of the fact table on a single node, then performing a non-distributed merge?

Depending on network costs, query selectivity, materialization strategy, what data is already present at each node, and a wide range of other factors, any of these strategies might be applicable. Different DBMSes will lean towards different options depending on which code paths have been optimized. Most distributed DBMSes choose between these different strategies using the same tools that non-distributed DBMSes use: heuristics, cost models, and cardinality estimation. These tools are already highly error-prone (and require significant tuning) in non-distributed settings, so you can imagine how tricky things get when entire clusters are involved!

Next, we’ll walk through how Bao can be applied to three different state-of-the-art commercial cloud distributed DBMSes: Vertica, Amazon RedShift, and Azure Synapse (an analytics-focused offering of SQL Server). After discussing each system, we’ll show a small experiment highlighting potential gains (or lack thereof) from applying Bao. We’ll be running each DBMS on different hardware, so please do not attempt to draw comparisons between these DBMSes.

In each test, we’ll be executing an analytic dashboarding workload called Corp. The workload was donated to our research team by a large corporation under the condition of anonymity. The workload contains about 1 TB of data and 2000 unique queries which change over time. A large schema change (normalizing a fact table) happens in the middle of the workload – we do not count the time required to perform this modification. The workload makes use of analytic functions, (materialized) views, and other advanced SQL features.

Vertica

Vertica is a distributed columnar DBMS, and is the commercial adaptation of the C-Store paper. Vertica’s optimizer is reasonably transparent, and the documented set of hints gives us a lot of control over what the optimizer considers.

For Vertica, we select query hints to:

Force a particular join operator (JTYPE hint),
force a particular group by operator (GBYTYPE hint),
force a particular distributed join algorithm (DISTRIB hint, with options L, R, B, or A, representing a local join, a resegment join, a broadcast join, or letting the optimizer pick, respectively).

We started up two identical 3-node clusters on AWS using the official Vertica image (which, at time of writing, uses r4.4xlarge nodes by default). We loaded the data into both clusters, and used the Vertica DBD tool to create a good layout, which we manually verified through testing. One cluster ran Bao on top of Vertica (Bao), while the other cluster did not run Bao (Vertica). The resulting time and costs are plotted below:

Bao was able to reduce the end-to-end processing time of this workload by over three hours, while reducing cost by over 25% (about $25, in our case). The time savings with Bao are more significant than the cost savings because Bao must periodically retrain its predictive model, which (temporarily) requires a GPU.

Around 45% of Bao’s gains (the plurality) come from forcing the Vertica optimizer to use a broadcast join instead of a resegment join when the Vertica optimizer wrongly over-estimates the cardinality of a particular subplan. The small subplan result can be easily materialized and sent to all nodes, allowing for a faster computation than shuffling around parts of the subplan evenly to each node in the cluster (a resegment).

In the future, we intend to experiment with specific hints to include or exclude projections (PROJS and SKIP_PROJS) to allow Bao to further custom-tailor its strategy to the user’s data.

Azure Synapse (SQL Server)

Azure Synapse is a analytics offering from Microsoft, based on SQL Server. SQL Server is one of the most widespread commercial database systems. Unlike Vertica, SQL Server can store data in either a row-based or a column-based format. Here, we’ll focus on the column store format provided by the Azure Synapse Analytics cloud database. Unfortunately, the types of hints available for Synapse Analytics are quite limited – so we had to get a bit creative.

For Synapse, we select query hints to:

Disable or enable a particular type of join operator (hash, merge, and/or loop),
use a replicated or partitioned versions of all dimension tables,
use an indexed or non-indexed version of the fact table

In order to implement the last two hints, we cheat a little bit: we created two versions of each dimension table, one replicated and one partitioned, along with two versions of the fact table (one indexed and one non-indexed). Based on the hint Bao selects, we rewrite the query to use the correct versions of each table.

We spun up two Azure Synapse Analytic instances, each with a dedicated 1000 DWUs (“data warehouse units,” a measure of query processing resources available to queries). We ran one instance with Bao and the other instance without Bao. After loading the data, we added the specialized tables described above to the instance using Bao. The other instance, which would use the stock optimizer, had all dimension tables replicated and an indexed version of the fact table. The resulting time and costs are plotted below:

Bao was able to reduce the end-to-end runtime of this workload by a little over two hours, while reducing the cost by around 10% (a little under $35). Again, the cost reduction is smaller than the latency reduction because of the cost of training the Bao model.

A plurality of Bao’s gains (40%) came from avoiding the index on the fact table (sometimes by removing loop join as an option, sometimes by rewriting the query to use the non-indexed table). Removing the index from the instance running with the stock optimizer led to a decrease in performance, meaning that while the index normally helped, it hurt some queries, and Bao was able to identify those queries automatically.

In the future, Bao could be applied to Azure Synapse in serverless mode, but this mode currently does not support query hints.

RedShift

RedShift is Amazon’s data analytics offering, and is often cited as one of the first “cloud native” analytics databases. As far as we can tell, RedShift only offers two query optimization hints, which we’ll use alongside the same trick we used for Synapse.⁴

For RedShift, we select query hints to:

Disable or enable AQUA, the RedShift query accelerator (activate_aqua),
disable or enable using (up-to-date) materialized views in query processing (mv_enable_aqmv_for_session),
use a replicated or partitioned version of all dimension tables (as with Synapse).

Unfortunately, RedShift offers by-far the least visibility and control into its query optimizer, leaving our Bao implementation (seemingly) limited. We started two 3-node clusters (ra3.4xlarge nodes), one running Bao and one running the stock optimizer. The results are plotted below:

Bao speeds up total workload latency by over an hour, and reduces costs by around 10% (in this case, about $10). Most of the gains (65%) came from disabling the usage of materialized views within subtree expressions, which the RedShift optimizer seems to do a little too aggressively (e.g., scanning the entire materialized view is slower than applying the predicates to the relations involved and performing the join). Disabling the use of materialized views in query processing globally hurts query performance overall, again showing that Bao is able to learn when the feature helps and when it hurts.

RedShift seems like a really cool system, but unfortunately it does not provide the visibility or control required by researchers for deep study in query optimization. In the future, it would be awesome to see Amazon build a few more windows and knobs (with sane defaults!) into the optimizer for scientists to use.

Discussion

Looking over the results, we find ourself returning to the same set of takeaways:

Don’t spend too much time comparing results between systems. The experiments are ran on different clouds, using different hardware, at different times of day. While we followed best practice guidelines to tune the systems, an expert might still do better, in particular for Redshift and Azure Synapse, as we have less experience with them.
Just because Bao produces large gains on system X but smaller gains on system Y doesn’t mean the default optimizer of system X is better than the default optimizer of system Y. In our opinion, it is more likely that system X provides more visibility into the optimizer, whereas system Y keeps things pretty opaque, limiting Bao’s possible gains.
Everyone wants a “knob-free” query optimizer, but nobody tells you that the cost of such is decreased query performance. Without incorporating feedback from query execution in some way, heuristic optimizers are doomed to have edge cases that make them fall flat. “Knob-free” sometimes just means “not sophisticated enough to give you the tools you need to fix the mistakes.” A real “knob-free” query optimizer should take query latency feedback into account.
The idea behind Bao – use a deep neural network as a predictive model to choose between a number of different variants, then retrain that model progressively using Thompson sampling – seems widely applicable. We focused on database systems, but maybe there are other applications as well?

Of course, Bao does come with some drawbacks. Bao causes query optimization to take a little bit more time (~300ms), requiring quite a bit more computation. We studied this overhead in our SIGMOD paper. For data warehouse workloads, which largely consists of long-running, resource intensive queries, Bao’s increased overhead is hardly noticeable. However, for workloads with a lot of short running queries, like OLTP workloads, this might not be the case. We are currently working on new approaches to mitigate that problem – so stay tuned!

If you feel so inclined, you can read the Bao paper, check out our open source prototype for PostgreSQL, or take a look at the other publications from our group.

Notes

Approaches such as ReJOIN, DQ, and Neo (in chronological order) fall into this category. ↩
The choice of which plan variants to generate is important. Generally, one variant is always just what the underlying query optimizer would have chosen with no intervention, and the other variants are generated by forcing the optimizer to ignore particular optimization paths. See the paper for more details. ↩
Note that training the predictive model using standard supervised learning techniques will not balance the exploration of new policies (i.e., trying something new to see how well it works) against the exploitation of existing knowledge (i.e., doing what we know works well). Thus, reinforcement learning techniques are preferred. It is easy to think that, to avoid regression, you would want to maximize exploitation. But this can get you trapped in a local minima: if you mis-predict the value of the optimal plan, you will never select it. In other words, without exploration, you might initially have fewer regressions, but you’ll never recover from them! ↩
Redshift, like Vertica, does not support traditional indexes on tables like Synapse does, so we do not add those indexed variants here. ↩

Machine learning for systems

2019-06-06T00:00:00+00:00

Imagine you are writing a system to repeatedly sort large datasets. You could just blindly choose a search algorithm, like Quicksort, and apply it every time. However, if the data is nearly sorted already, something like bubble sort or insertion sort might be a better idea. But how do we know when this is the case? One approach would be to sample the data and compute some statistics over it, like the expected size of a run¹ and the average pairwise difference. Then, you could run a lot of experiments and try to come up with a hand-tuned heuristic to pick a sorting algorithm based on the sampled information.

But how well will the heuristic you designed work when the data distribution changes? Or when your system is moved to new hardware (possibly with larger caches, etc.)? Or you want to add some other sorting algorithm? Some of the thresholds you picked might need to be tuned… every single time.

So why don’t we just learn the function that maps our sampled statistics to the optimal sorting algorithm? And better yet, let’s do it in an online fashion, using a corrective feedback loop where we learn from our mistakes.

State
Quicksort
Insertion
Bubble
ExperienceThe total number of sorts the agent has performed, and thus the amount of training data acquired, so far.
Net regretThe time difference between the action taken by the policy (the agent or Quicksort, QS) and the optimal action over every episode so far.	Agent: / QS:
Net savingsThe amount of time saved (if positive) or lost (if negative) by using the learning agent instead of picking Quicksort every time.

Right now, your computer is busily training and evaluating a deep reinforcement learning model² which, when fed the sampled statistics as input, will predict how long each of three sorting algorithms (Quicksort, bubble sort, and insertion sort) will take to sort the data. The graph shows the regret, the difference between the decision made and the optimal decision, for two policies: the learned model, and a policy that uses Quicksort every time. The “net savings” measures how much time has been saved (or lost) by using the learned policy compared to Quicksort. Over time, this number should increase, indicating that the agent is learning.

What’s going on here?

Every time a new set arrives for sorting (an episode), we feed the sampled statistics (the context) to the neural network, and we use the network’s prediction to select a sorting algorithm (an action). After we select an algorithm, we sort the set and see how good our decision was by measuring the latency (the reward). This problem setup, a restriction of general reinforcement learning, is often referred to as the contextual multi-armed bandit problem³, in which:

At the start of each episode, an agent receives a context (sampled statistics), containing information relevant to the current episode.
The agent selects an action (a sorting algorithm) from a finite set.
The agent receives some reward (the time it takes to sort).

The agent’s goal is to maximize the reward it earns over time (i.e., minimize the latency of sorts over time). In order to do so, the agent must explore the relationship between contexts, actions, and rewards, while simultaneously exploiting the knowledge it has gathered. Our solution to this problem is Thompson sampling, a straight-forward technique that keeps track of (context, action, reward) triples and samples a model to predict the reward of each action based on the context.

We re-train the model every 40 episodes. As the model gains more and more experience (observed triples), the model’s prediction gets better and better, and thus the resulting policy improves as well.

If you watch the graph for long enough (normally around 250 episodes), the regret of the learned policy should go to zero, meaning that the optimal policy has been learned. Since we use Thompson sampling to build a new model every 40 episodes, there are occassionally spikes, or variance, in the learned model’s policy. However, the net savings of the learned model tends to remain significant.

In a sense, this means our learned model is trading worst-case performance for average-case performance: most of the time, the learned model will choose a good algorithm, but every once in a while, Thompson sampling will produce a catestrophic policy,⁴ and performance will degrade significantly for a short period of time. But, on average, the learned agent will perform better than the static policy.

The Big Picture

Machine learning for systems – that is, applying machine learning techniques to problems within computer systems – is coming, and it’s coming fast. Whether its index structures, scheduling, or even database query optimization (including cardinality estimation), machine learning is creeping its way into the domain of systems. It has been called Software-Defined Software, Software 2.0, learned systems components, or self-assembling systems. But regardless of the name, “machine learning for systems” brings about both exciting possibilities and significant new challenges.

The advantages of machine learning powered systems could be huge: for database systems, query optimizers contain a huge number of heuristics that must be painstakingly maintained by hand, and often even require application-specific tuning by a DBA. Some of our recent research shows how deep reinforcement learning can produce query optimizers that automatically tune themselves to specific applications, matching and sometimes exceeding the performance of complex commercial optimizers. In their recent paper, Kraska et al. argue that similar applications of machine learning could improve almost every aspect of a database, including data access, query optimization, query execution, and advanced analytics.

However, there are numerous challenges to overcome.

Like in the example shown here, learned systems can perform better on average, but might have significantly higher variance, especially in early episodes. Figuring out how to reduce this variation will be critical for systems applications that depend on low tail latencies. Several approaches, like learning from demonstration or bootstrapping a model from a cost function, have recently been investigated.
Understanding when models will – and won’t – work is still a major open question. While we know that neural networks can approximate any continuous function, and that algorithms like table-based Q learning converge to optimal policies for any MDP, we are still a long way from having theoretically-justified bounds on practical learning systems. It is possible that such bounds may never be known, and that it is easier to adjust the existing learning systems to have better behavior at the margins. Such techniques could be a simple as using different models for outliers, or may be as complex as modifying database internals to work adaptively.
Doing worse takes longer than doing well. If you are playing a video game and not doing very well, the score printout on the screen will tell you just how bad you doing. If you are doing well, the score printout will convey this information as well, and equally quickly. For systems problems, however, if your initial policy is poor, evaluating it may take a long time. For example, for join order selection in a relational database, a bad join ordering may take orders of magnitude longer to complete than a good one. Thus, how long it takes to “read the score” is a function of the score itself. This means that the initial episodes of learning (in which many RL algorithms start with a random policy) could take a prohibitively long amount of time.

These challenges, and many others, are discussed in our recent workshop paper at RL4RealLife (ICML ‘19). We’ve also released an open source platform called Park to enable researchers to test reinforcement learning algorithms on systems problems, including network congestion control, Spark scheduling, load balancing, and a dozen more. The future of systems is learned!

FAQs

This seems really dumb. Why don’t you just… This is not supposed to be a practical system. It’s supposed to be a demonstration of a learned policy being useful in a systems context. Clearly, a real world system would need significantly more testing, and would likely use a much simpler model. Please see the many linked works in the first paragraph of “The Big Picture” section for more practical examples.

Are you including the featurization and inference time in your measurement? Inference time is not included, but featurization time is included: the always-quicksort policy benefits from not having to compute features, and the learned policy is punished. Note that the zero-regret line on the plot assumes the featurization times – in other words, an oracle policy that didn’t even require the features as inputs would have a (slightly negative) regret.

Shouldn’t you just be using Timsort? Maybe. Depends on the hardware, data distribution… probably a large number of other factors. This system is flexible – you could add in Timsort, and, if a particular piece of hardware and data distribution caused Timsort to always be optimal, Timsort would always be selected. In this case, you would be wasting time computing features. It might be interesting to have the policy select which features it wanted to gather for itself, so that the network could learn which when/which features were useful.

A run is a contiguous sorted subsequence, i.e., a cluster of data that is already sorted within the larger dataset. ↩
With just two inputs and one output, we could’ve easily (and, if this were a real system, should’ve!) used something simpler, like a decision tree. But what fun would that be? ↩
There are many overviews available. For one, see: Chapelle and Li. “An Empirical Evaluation of Thompson Sampling.” In Advances in Neural Information Processing Systems. NIPS’11. ↩
Testing these catastrophic policies is a necessary part of most reinforcement learning – after all, if we do not test the policy, how will we ever learn it is catastrophic? We don’t want the performance of real world systems to suffer while our models explores no-good possibilities. Figuring out how to avoid such catastrophic detours is an active area of research. ↩

Good comment, bad comment

2018-11-05T00:00:00+00:00

Having spent five years heavily involved in teaching the 2nd semester programming course at Brandeis, I’ve found myself repeating the same maxim over and over again:

Code is an essay written for two audiences: the computer, and your fellow programmers.

Thinking about the dual-audience nature of code immediately highlights several important aspects of “good code”:

When “read” by the computer, good code leads to a correct, effective, and efficient behavior. The desired outcome is achieved (correct) in a reasonable amount of time (effective) without consuming an unreasonable amount of resources (efficient).
When read by your fellow programmers, good code reads like an essay that intuitively, succinctly, and precisely describes the semantics of the problem and how it is solved. In other words, the code is not overly long and verbose (succinct), the code conveys the exact details of what is happening to the reader (precise), and the code does so in a way that makes sense to the reader quickly (intuitive).

In my experience reading thousands (really!) of different solutions to the same problem, I find that the best code follows all six of these principles. In this post, I want to go through each principle, showing both good and bad examples. Each example will be a “real” piece of code written by a student (anonymized, of course), including myself (I have to get the bad examples from somewhere).

For the computer
For your fellow programmers
Conclusions

For the computer

All in all, contemporary computer science departments do a pretty good job of teaching students to write correct, effective, and efficient code. In fact, new programmers tend to optimize for their robotic audiences over their fellow humans – “if it works, who cares?” If a computer wasn’t part of our audience, we might not choose to write code at all.

However, new programmers are often very frustrated by a computer “not doing what I meant.” Understanding how to translate our thoughts into code is a fundamental skill that looks a whole lot like the writing process. The next three sections explain what I believe to be the three properties of good code from the point of view of the computer audience.

Correct

When thinking about the computer as the audience for our code, we want our code to cause the computer to compute the correct result. Correctness may seem like the most obvious principle we would want good code to embody – but correctness is often subtle, underspecified, or misunderstood. For example, consider the following prompt and solution:

Write a method randItem(int[] arr) that returns an item selected uniformally at random from the passed array.

public static int randItem(int[] arr) {
    return arr[(int) (Math.random() * arr.length)];
}

At first glance, you might think this code is just fine – Math.random() returns a uniform value from 0 to 1, multiplying that by the array length and truncating to an int should give a uniformally distributed array index. You might raise a complaint that the programming didn’t use Java’s Random class, but that’s not a correctness issue, right?

It turns out the above code is very subtly incorrect. To understand why, let’s simplify the problem a little bit. Imagine that our input array arr is of size 5, and that Math.random() returned a 4-bit random number between 0 and 1. Let’s look at the possible values of our array index:

Binary	Decimal	Array Index
0000	0.0000	0
0001	0.0625	0
0010	0.1250	0
0011	0.1875	0
0100	0.2500	1
0101	0.3125	1
0110	0.3750	1
0111	0.4375	2
1000	0.5000	2
1001	0.5625	2
1010	0.6250	3
1011	0.6875	3
1100	0.7500	3
1101	0.8125	4
1110	0.8750	4
1111	0.9375	4

Spot the problem? In this case, four values of Math.random() map to index 0, but only three values map to all the other elements! That means this method will return the first element of the array with probabilty , while the other elements will each be returned with a probability of ! That’s not uniform at all.

Now, Math.random() returns a double, which has a lot more than 4 bits, and only a subset of all possible double values are within the range (0, 1). But, saying there are exactly possible return values for Math.random() and elements in our array arr, this method only potentially works if divides , i.e. n % k == 0.

A more correct way to write this code would be:

private static final Random r = new Random();
public static int randItem(int[] arr) {
    return arr[r.nextInt(arr.length)];
}

Since the nextInt method ensures that we are drawing from a uniform distribution,¹ this code produces a result more in-line with the prompt. The point here is that correctness can be subtle, and that a computer will happily do whatever you tell it – and sometimes the wrong thing can look a whole lot like the right thing at first glance.

Effective

In addition to being correct, we also want our code to cause our computer audiences to compute results effectively. We all know we want code that runs fast, and we learn tools like big-O analysis to help us make good decisions along the way. But sometimes, something seemingly-mundane, or that you think should be fast, can end up taking a very long time. There’s even a Tumblr about this phenomenon in very real code. Here’s an example of this in code written by myself a long time ago, and by numerous other students ever since:

Write a method combine(String[] args) that returns a String containing each element of args connected with spaces, in order. If args is empty, return the empty string.

public static String combine(String[] args) {
    if (args.length == 0) return "";
    
    String collector = "";
    for (String el : args) {
        collector += el + " ";
    }
    
    // Cut off the last space.
    return collector.substring(0, collector.length() - 1);
}

Seems straight-forward enough, right? Cutting off that last space was a little tricky. New developers might not even think to analyze the runtime of this method, or if they did, they might declare it to be linear in the size of args. More seasoned (read: cynical) developers might begin to smell trouble, though.

The way Java represents strings is different from many other languages. In Java, strings are immutable. Once they are created, they can’t be modified. So if they can’t be modified, what happens when the program executes collector += el + " "? Well, unfortunately, Java will create a brand new string, copying the contents of collector, el, and " " into it. Java will perform this create and copy operation at every loop iteration, which means that our runtime is actually quadratic in the size of args!

Since the good folks working on Java know that developers sometimes need to concatenate strings, Java has a handy StringBuilder class for this exact situation!

public static String combine(String[] args) {
    if (args.length == 0) return "";
    
    StringBuilder collector = new StringBuilder();
    for (String el : args) {
        collector.append(el);
        collector.append(" ");
    }
    
    // Cut off the last space.
    return collector.substring(0, collector.length() - 1);
}

Now our code is much more effective.² The point here is to show that performance bugs can often be hard to notice, and can creep into code in subtle ways that might elude someone who isn’t looking closely for them.

Efficient

When I was a new programmer, I didn’t really think about the resource usage of my application. It either ran, or it didn’t. Most of the programs I was writing were “short-lived”: I started them and waited for them to finish, normally a few moments later. Even if one of my programs used a ton of memory or temporary disk space, I wouldn’t notice, because my operating system would liberate all these resources from my greedy program after my program terminated.

However, as I gained more experience and started writing longer-living programs, like servers (or “services”, as the kids call ‘em nowadays), I started to feel the pain of all my inefficient code. This pain came mostly in the form of memory leaks: bugs in my programs that caused more and more memory to be used up until there was none left.

Many programmers who swear by languages like Java or Python believe that their language’s garbage collector – the part of the language responsible for reclaiming objects we aren’t using anymore – makes them immune to memory leaks. While it is true that these garbage collectors prevent the majority of memory leaks, I was a poor enough Java programmer to create several! Here’s an example of one written for a class (not by me, but making the same mistake I did):

Write a class BoundedStack<T> with methods push and pop that behaves like a stack with a bound on the number of items it can contain. Take the bound as a parameter in the constructor.

public class BoundedStack<T> {
    private Object[] storage;
    private int topIdx = 0;
    
    public BoundedStack(int bound) {
        storage = new Object[bound];
    }
    
    public void push(T arg) {
        if (topIdx >= storage.length) 
            throw new IllegalStateException();
        storage[topIdx++] = arg;
    }

    public T pop() {
        if (topIdx <= 0) 
            throw new IllegalStateException();
        return (T) storage[--topIdx];
    }
}

Overall, looks fine, right? But this implementation of a bounded stack actually has a memory leak. Consider the following client code:

public static void main(String[] args) {
    BoundedStack<ReallyBigType> s = new BoundedStack<>(5);
    
    s.push(new ReallyBigType(1));
    s.push(new ReallyBigType(2));
    s.push(new ReallyBigType(3));
    s.pop(); s.pop(); s.pop();
    
    performTask();
}

Imagine that ReallyBigType is a class that uses a whole bunch of memory. When performTask is called, the programmer might be expecting that all the ReallyBigType instances created beforehand will be freed up, so that performTask can use as much memory as it needs. However, this isn’t really the case – all three instances of ReallyBigType are guaranteed to still be around when performTask is called!

The reason all three instances of ReallyBigType are still around is that our BoundedStack, s, is still holding a reference to them. Even though we decremented the topIdx variable whenever pop() was called, the values in storage (in positions 0, 1, and 2) still point to our instances of ReallyBigType! That’s a memory leak. If this were a large, long-running application, small errors like that could build up, resulting in a system that needs to be frequently rebooted.

Luckily, this memory leak is easy enough to fix:

public class BoundedStack<T> {
    private Object[] storage;
    private int topIdx = 0;
    
    public BoundedStack(int bound) {
        storage = new Object[bound];
    }
    
    public void push(T arg) {
        if (topIdx >= storage.length) 
            throw new IllegalStateException();
        storage[topIdx++] = arg;
    }

    public T pop() {
        if (topIdx <= 0) 
            throw new IllegalStateException();
            
        T tmp = (T) storage[topIdx];
        storage[topIdx] = null;
        topIdx--;
        
        return tmp;
    }
}

By setting the now-unused cells of the storage array to null, we avoid holding a reference, allowing the Java garbage collector to do its work. The point here is that just writing fast, correct code isn’t enough: good code behaves well even in long running, resource-scarce environments.

For your fellow programmers

Despite all the difficulties of writing to your computer audience, writing to your human audience is often much harder. After all, writing and composition³ have entire classes of their own! Good code shares a lot of properties with good writing. Specifically, I’ve found that most good (technical) writing and good code I’ve read follows three principles. First, I’ll give an overview of each of these principles, and the proceeding subsections will expand on them.

Intuitive: It is well-structured and easy to understand. For human readers, packages, classes, methods, and code chunks serve a similar purpose to chapters, sections, subsections, and paragraphs. Breaking down complex ideas into small, composable pieces is a fundamental skill in both programming and writing. This is how we turn complex ideas into intuitive concepts.
Succinct: It is just long enough. The phrase “if I had more time, I would write a shorter letter” comes to mind here. In high school, you may have learned to “pad out” your essays with unnecessary verbiage. Unfortunately, you’ll need to unlearn that, both for writing code and papers. Nobody wants to spend three hours reading your multi-page comment. The goal should be to get the idea across in the fewest words possible (so perhaps this blog post isn’t the best example).
Precise: It isn’t too “hand-wavy.” The code should communicate to the human reader exactly what is going on, and not attempt to “gloss over” or cover up the details.⁴

Now, luckily for us, we have a bit of a “cheat code” when communicating with human audiences through our code. Specifically, we have comments, which are invisible to our robot audiences, and variable names, which are meaningless to our robot audiences. We can take code the computer is already happy with (correct, effective, efficient), and we can use these two tools to annotate that code to make our human audience happy as well.

As a warning, this part is going to be a lot more subjective than the previous part. I’ll do my best to give examples that I think most of the professional software engineers I’ve worked with would agree on. It’s important to realize that a lot of what makes code readable (and what makes writing good or bad) depends on your audience. So if your audience has certain norms – like indention style, variable naming style, etc. – following them is in your best interest. Otherwise, you’ll be written off from the get-go.

Take a quick look at the two papers below.

You don’t have to be able to see the words to know which paper “looks” better. We have established conventions for writing, and when they aren’t followed, it is immediately jarring. And while eschewing these structures can sometimes be considered “art,” you generally want to match the expectations of your audience. Software engineers are no different – if your code doesn’t “look” right (perhaps it isn’t indented properly, or your variable names are unconventional), the actual code you wrote probably won’t matter… the engineer will just dismiss it out of hand.

Part of becoming a good programmer is learning and practicing those norms. You can take advantage of them to convey information quickly and succinctly, or you can fight an uphill battle against them. Unless you’ve got a really good reason, don’t try to make your readers learn a new convention and your complex idea at the same time! Instead, try to use the conventions as rails for your code.

These next sections will continue with a mix of Java and Python, but I’ll do my best to include only the norms that are common among most languages. Any specific language you use likely has its own norms (for example, JavaDoc), and you should learn them – but we’ll keep it general here.

Intuitive

Most (scientific / technical) writing has the goal of conveying intuition. The writer has an idea in their head that they’ve thought about deeply. They’ve analyzed it, broken it down, tried a million different things, and, at long last, they have a really good understanding of it. The writer’s goal is to create a paper that, when read by other scientists, “fast-forwards” them through that potentially multi-year process, transferring the intuition in their head to paper. In this way, reading a paper can be a lot like time travel: it lets you learn what took years to discover in just a few short hours. This is what Newton meant when he said “if I have seen further it is by standing on the shoulders of Giants.” Good code works the same way.

This is also an extremely difficult task, and the people who are the best at it often become professors at research institutions. It’s a high bar for writing, and it’s an even higher bar for code, but the same idea applies. Developers who write code that can convey intuition to other developers drastically boost the productivity of their teams, and ensure the longevity of their projects. The people who are best at it become the director of AI for Google or senior technical fellows at Microsoft.

Consider the following prompt and solution:

Write a method isUnique which, when given a Map<String, String>, returns true if no two keys map to the same value, and false otherwise. Return true if the map is empty.

public static boolean isUnique(Map<String, String> arg) {
    return (new HashSet<>(arg.values())).size() == arg.size();
}

Now, this code is correct, and is reasonably efficient and effective. It is, however, still very bad code. The reason why it is bad code is that the code isn’t intuitive – there’s no connection from the line as written to the prompt. Here’s a way to improve it:

public static boolean isUnique(Map<String, String> arg) {
    // By placing all the map's values into a set, we 
    // collapse any duplicates. Then, if the number
    // of uniqueValues is equal to the total number 
    // of values, all values must be unique.
    Set<String> uniqueValues = new HashSet<>(args.values());
    return uniqueValues.size() == arg.size();
}

In my opinion, this comment and the new, named intermediate variable greatly clarify the code. Of course, it still needs a “top level” comment above the method signature to describe what it does, but that will look a lot like the prompt. Here’s another way to improve the original code:

public static boolean isUnique(Map<String, String> arg) {
    // Test if bijection using incl/excl.
    Set<String> AuB = new HashSet<>(args.values());
    return AuB.size() == arg.size();
}

Now, is this “good code”? Maybe if you were writing a mathematics library and all of your potential readers were extremely well-acquainted with the inclusion-exclusion principle, but in most situations this might be even worse than where we started. The majority of software engineers would scoff at this, likely assuming it was pretentious.

Another way to make code more intuitive is to connect each variable, method, etc. to a semantic concept in the real world. Remember, most code is trying to model or represent some real-world thing or idea. For example, consider the following code:

def compute_point(a, b, c):
  a %= 12
  a /= 12
  a += (b / 60) + (c / 60*60)
  a *= math.pi * 2
  return (math.cos(a), math.sin(a))

Now, what does this code do? Well, I know. Because I wrote it. Or I should say, I know today, and maybe next week, but I might not know in a year. However, by just changing the variable names, we can convey a whole lot more intuition:

def compute_hour_hand_position(hrs, mins, secs):
  theta = hrs % 12
  theta /= 12
  theta += (mins / 60) + (secs / 60*60)
  theta *= math.pi * 2
  return (math.cos(theta), math.sin(theta))

Now, the resulting code isn’t perfect, but it’s way better. I can now definitely tell, for example, that this has something to do with time. It probably has something to do with an analog clock. If I have some more context from the file, I might know immediately that this code is computing the (x, y) coordinates for an hour hand on clock. With just a few comments, we can make this obvious:

def compute_hour_hand_position(hrs, mins, secs):
  theta = hrs % 12
  theta /= 12
  theta += (mins / 60) + (secs / 60*60)
  theta *= math.pi * 2 # convert to radians
  
  # Compute x and y coordinates from the angle.
  return (math.cos(theta), math.sin(theta))

By connecting the code to the real world, I now understand why the first line includes a hrs % 12 (the clock wraps around), and I understand why mins is divided by 60, and why secs is divided by 60*60. Vastly improved over the initial version.

While none of the examples shown here are large or overly-complex, conveying the intuition behind your code only becomes more and more important as you “scale up” (i.e., get bigger and more complex). Practicing these techniques on small projects or assignments is critical to slowly developing the skills you need to write more complex code in an intuitive way.

Making code intuitive is easily the hardest task facing any software engineer. It’s so extremely difficult that some developers⁵ have disavowed it entirely, claiming that their code “just makes sense to smarter people.” There’s a ton of this fake-flex, over-confident compensation in computer science. Try to assure yourself that they’re deluding themselves, and then try to avoid them as best you can.

Succinct

One of the biggest mistakes I see new programmers make is being overly verbose with comments. Here’s a real(!) example of a simple program that prompts the user for their name and then greets them:

// the main entry method for the program, which
// the java command line program will execute.
public static void main(String[] args) {
    // prompt the user for their name using System.out, which 
    // is a PrintStream class. The PrintStream class has a
    // method called println, which will output the text
    // passed to the console (so that the user can see it) 
    // and then print a newline.
    System.out.println("Welcome to my program! What is your name? ");
    
    // this makes a new scanner, which can read from 
    // STDIN, located at System.in. The scanner lets us look
    // for tokens, aka stuff the user has entered.
    Scanner sc = new Scanner(System.in);
    
    // this will create a new string variable called "name" 
    // and set the value to whatever the user types next
    String name = sc.nextLine();
    
    // finally we use println again, and string 
    // concatenation, with the + operator, to 
    // output or (sic) message!!
    System.out.println("Hello " + name + " ! ");
}

Now, clearly this was written by a new programmer who has paid attention in class, and was told to explain things in comments. To a new developer, all these language features (like the main method) are new, and seem worth… well, commenting on. No points against the new programmer. Part of the learning process.

However, this is a great example of over-commenting. Any software engineer reading this will be quite annoyed wading through the long, verbose comments that are explaining Java language features. As a general rule, you should never use comments to describe a language feature: just assume your reader knows the language as well (perhaps an exception could be made if you use a really weird feature or something). Try to use comments to talk about the why, rather than the what. For example, we could rewrite this program like so:

public static void main(String[] args) {
    // Prompt the user for their name and greet them.
    System.out.println("Welcome to my program! What is your name? ");

    Scanner sc = new Scanner(System.in);
    String name = sc.nextLine();

    System.out.println("Hello " + name + " ! ");
}

We use a single comment to say “the why” of what we are doing. That is, we are describing the semantics of the code as opposed to the syntax of the code. Then we use some line breaks (“paragraphing”) to separate out the input and output sections. Much easier to read.

Verbosity can come from more than just comments. Often, the code itself can be repetitive or overly verbose. Consider the following program:

Write a method checkNames that takes in a List<String> of names and returns the valid names. A name is valid if it begins with a alphabetical character and has a length of at least 3.

public static List<String> checkNames(List<String> names) {
  // creating a new arraylist
  List<String> goodNames = new ArrayList<String>();
  //looping over the input
  for (int i = 0; i < names.length; i++) {
    String currentName = names.get(0);
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          if (currentName.length() > 2) {
            goodNames.add(currentName);
          }
        } else {
          continue;
        }
      } else {
        continue;
      }
    } else {
      continue;
    }
  }
  return goodNames;
}

There’s a lot going on here, so we’ll try to break it down. The first thing almost any programmer will notice is that the code is very “crowded” – there’s no “paragraphing” or blank lines to indicate the logical chunks of the code. The second .length() > 2 check is also completely redundant. The only two comments are also entirely useless. So let’s fix those things first:

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (int i = 0; i < names.length; i++) {
    String currentName = names.get(0);
      
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          goodNames.add(currentName);
        } else {
          continue;
        }
      } else {
        continue;
      }
    } else {
      continue;
    }
  }
  
  return goodNames;
}

Already, the code is starting to look a little bit more like code. The next thing we notice is that all of those else blocks are useless – the loop continues without the explicit continue statement anyway. So we can remove them.

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (int i = 0; i < names.length; i++) {
    String currentName = names.get(0);
    
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          goodNames.add(currentName);
        }
      }
    }
  }
  
  return goodNames;
}

Much shorter, but we can go a bit further. The index variable of the for loop, and the currentName variable, can be replaced with the “for each” pattern.

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (String currentName : names) {
    if (currentName.length() > 2) {
      if (currentName.charAt(0) >= 'a') {
        if (currentName.charAt(0) <= 'Z') {
          goodNames.add(currentName);
        }
      }
    }
  }
    
  return goodNames;
}

Now we’re getting somewhere. The next refactor is a bit of a matter of opinion, with older software developers generally siding “against” and younger software developers generally siding “for.” One heuristic you can use to try to tell if your code is too verbose is to count the number of levels of indentation. If you’re over three, something has probably gone wrong. We can fix it like this:

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();

  for (String currentName : names) {
    if (currentName.length() <= 2) continue;
    if (currentName.charAt(0) < 'a') continue;
    if (currentName.charAt(0) > 'Z') continue;

    goodNames.add(currentName);
  }
  
  return goodNames;
}

Every level of indentation adds another level of “context” that the developer has to remember, e.g. “ok, I’m inside this for loop, and this if statement, and this if statement…” By inverting each condition, each if statement now reads more like “ignore names that fail this test.” Putting three of them in a row using the exact same formatting takes advantage of parallel structure, just like in writing. By the time you’ve read through the three if statements, its clear you’re left with a name that meets the conditions.

Finally, we can use a little Java API knowledge to make it slightly better, and add some semantic comments, to finish it off:

public static List<String> checkNames(List<String> names) {
  List<String> goodNames = new ArrayList<String>();
    
  // Find the names that: (1) have three or more characters
  // and (2) start with a letter.
  for (String currentName : names) {
    if (currentName.length() <= 2) continue;
    if (!Character.isLetter(currentName.charAt(0))) continue;

    goodNames.add(currentName);
  }
    
  return goodNames;
}

If we want to get a little advanced, we could’ve written the whole thing in an even more succinct way using Java streams:

public static List<String> checkNames(List<String> names) {
  // Find the names that: (1) have three or more characters
  // and (2) start with a letter.
  return names.stream()
          .filter(name -> name.length() > 2)
          .filter(name -> Character.isLetter(name.charAt(0)))
          .collect(Collectors.toList());
}

The point of writing succinct code isn’t to show off. In fact, sometimes shorter code can be much more complex and harder to read. For example, consider swapping two variables in Java:

int a = 5;
int b = 9;

int tmp = a;
a = b;
b = tmp;

Every developer will recognize this as “swap code.” It’s a common pattern we see everywhere. However, an overly-clever developer might write:

int a = 5;
int b = 9;

a = a ^ b;
b = b ^ a;
a = a ^ b;

“It’s shorter by a whole four characters!”, proudly declares a young developer who is about to get a talking-to. This is known as the XOR swap algorithm, and it is a neat trick, but its very opaque. Nobody knows what it is doing the first time they see it, and even after they know the trick, they might not be sure everything is in exactly the right order.⁶ This makes the code very unintuitive. Making code less succinct doesn’t conflict with making code intuitive as often as you’d think. Normally, making things shorter, while staying inside the conventions and norms of your language and community, makes things better. Normally. Experience will teach you when something falls into the “XOR swap” category or not.

Overall, making things short and succinct makes your code more readable. Other developer’s eyes won’t instantly glaze over as soon as they see your code. Since anything but a superhuman-level software developer can only keep so many lines of code in their head a time, making code shorter can often decrease the cognitive load required to read and understand your code. It’s tempting to measure the progress of a project by the number of lines of code you’ve written, but I promise, senior devs will respect a “small codebase that can do a lot” much more than a large codebase that only does a little.

Precise

The last principle for our human audience is precision. Writing comments that tell the user vaguely what is happening is relatively easy – but writing a comment the succinctly conveys exactly what a program is doing is much, much harder. Luckily, code has to be precise, so at a minimum, an unlucky developer can simply wade through every line to try and figure out what is going on. But we should have loftier goals.

Consider the following prompt and solution:

Write a method that, given a height from which a ball was dropped in meters, computes the velocity of the ball when it hits the ground.

public static double velocity(double height) {
    // compute the v with the energy formula
    return Math.sqrt(2.0 * 9.8 * height)
}

That’s short and sweet, and it might even be good enough if your audience has a physics background. But for us non-physics folks… a little more explanation might be nice. Note that the comment “compute the v with the energy formula” is entirely correct. It describes what the code does, semantically, but it is a little hand-wavy. What energy formula? There’s a few. What’s 9.8? How exactly did you reach this solution?

A better version might be:

private static final double GRAVITY = 9.8;
public static double velocity(double height) {
    // Compute v using the conservation of energy formula:
    //     mass * gravity * height = (1/2) mass * velocity^2
    // Mass cancels. Solve for velocity:
    //    sqrt(2 * gravity * height) = velocity
    
    return Math.sqrt(2.0 * GRAVITY * height)
}

Now the code is a bit longer, but it is also a lot more precise – a reader without a ton of physics background can at least step through the algebra to convince themselves that the code is correct. Extracting 9.8 to a variable also makes the code more precisely match the formula. Again, if your target audience is a bunch of physicists, the commentary is probably not required. Context is king.

There’s another, very easy way to add precision to code’s explanation, although it is only applicable in a few cases. Check out this chunk of code:

double[][] E = new double[k][n];
int[][] J = new int[k][n];

for (int j = 0; j < n; j++) {
	E[0][j] = err(0, j);
	J[0][j] = 0;
}

for (int i = 1; i < k; i++) {
	E[i][i] = 0; 
	J[i][i] = i - 1; 
	for (int j = i + 1; j < n; j++) {
		E[i][j] = Double.POSITIVE_INFINITY;

		for (int x = j - 1; x > i-1; x--) {
			double e = err(x, j);
			double se = E[i-1][x] + e;
			if (se < E[i][j]) {
				E[i][j] = se;
				J[i][j] = x;
			}
			
			if (E[i][j] < e) break;
		}
	}		
}

Understand what’s going? If so, I’m extremely impressed (and perhaps you need to see the last paragraph of the “Intuitive” section). To me, this is just about as bad as it gets. Short, one letter variable names. Absolutely no commenting. Random, complex manipulation of 2D arrays.

… but what if I told you that this one weird comment would transform this code from terrible to completely acceptable? Ready? Here it is…

// Algorithm for optimal piecewise approximation. 
// Transcribed exactly from:
// Mahlknecht, Dignös, and Gamper,  “A Scalable Dynamic 
// Programming Scheme for the Computation of Optimal 
// k-Segments for Ordered Data.”
// https://doi.org/10.1016/j.is.2016.08.002
double[][] E = new double[k][n];
int[][] J = new int[k][n];

for (int j = 0; j < n; j++) {
	E[0][j] = err(0, j);
	J[0][j] = 0;
}

for (int i = 1; i < k; i++) {
	E[i][i] = 0; 
	J[i][i] = i - 1; 
	for (int j = i + 1; j < n; j++) {
		E[i][j] = Double.POSITIVE_INFINITY;

		for (int x = j - 1; x > i-1; x--) {
			double e = err(x, j);
			double se = E[i-1][x] + e;
			if (se < E[i][j]) {
				E[i][j] = se;
				J[i][j] = x;
			}
			
			if (E[i][j] < e) break;
		}
	}		
}

This is a (in my opinion) complex dynamic programming algorithm for approximating a function of one variable with k line segments. There’s absolutely no reason to re-explain an entire scientific paper in the comments of your code, unless you feel like you can do it way better than the original authors. In terms of describing an algorithm, citing the paper (or textbook, with page number) from which it was transcribed is just about as precise (and succinct!) as you can get. Of course, this situation doesn’t come up too often, but when it does, it will be one of the most useful-per-character comments you’ll ever write.

Conclusions

My experience is limited, and I’m far from the best software developer in the world, but I have worked at quite a few places, programmed for a reasonably long time, and read a lot of programming assignments! So perhaps you don’t want to treat these as steadfast rules, but I do think they are good general principles. Here’s the cheat-sheet:

Code is an essay written for two audiences: the computer, and your fellow programmers. It should strive to obey these principles.

Correct: the code should produce the correct output.
Effective: the code should run quickly.
Efficient: the code should minimize resource usage.
Intuitive: the code should convey the intuition behind it.
Succinct: the code should be short.
Precise: the code should allow humans to understand exactly what it is doing.

Of course, it is worth noting that you don’t always have to write good code. In fact, most of the time, you won’t. If you go over to my GitHub, you’ll see most of the code I’ve written is… bad. Real bad. Like, super bad. These are things to strive for, and things to practice. You won’t always be able to write code that is especially pristine. But hopefully, bit by bit, year by year, we can get better and better.

Footnotes

You might be wondering how nextInt manages to draw from a uniform distribution – one approach is to use rejection sampling. First, pick the largest number m less than Integer.MAX_VALUE that the size of your array, n, still divides. Then, sample a random integer from the range (0, Integer.MAX_VALUE). If the sampled integer is less than m, return m % n. Otherwise, try again. ↩
We could also use Java’s streams, and the joining collector, to make the code much shorter and more obvious to a human, but we’re only discussing effectiveness here. ↩
There’s a reason it’s called “composition.” Writing code and writing papers both require careful compositions of abstractions. In computer science, these abstractions are linked lists, stacks, queues, etc. In “traditional” writing, these are paragraphs, citations, ideas, etc. Computer scientists who dismiss literature and other humanities will find themselves at a huge disadvantage. ↩
This does not mean you cannot create your own abstractions, or use others. It also does not mean you have to explain how an ArrayList works every time you use one. But someone looking at your code should be able to find those details somewhere (perhaps in your code, perhaps in the Java documentation), if they look for them. ↩
I’ve been guilty of this in the past, and I hold no grudge against those who avoid me to this day because of it. ↩
It turns out the XOR swap is also a tad slower than the “normal” swapping code – mostly due to compiler optimizations. ↩

Overflow in consistent hashing

2018-09-14T00:00:00+00:00

Consistent hashing was first proposed in 1997 by David Karger et al., and is used today in many large-scale data management systems, including (for example) Apache Cassandra. Consistent hashing helps reduce the number of items that need to be moved from one machine to another when the number of machines in a cluster changes.

The basic idea is to use two hash functions¹ – one, , which maps each data item to a point on a circle (generally represented as an angle between and ), and another, , which maps each machine to a point on the circle. Each machine is responsible for serving all the data items with hash values that fall between and the machine with the next angle along the circle (clockwise).

Server	Load	Items

In the example above, each server is hashed by to three coordinates on the circle (server 1 is hashed to the east-most point on the circle). Items that fall between server A’s hashed position and the next server (moving clockwise) are assigned to server A. Click and drag the data items to change their assigned segment.²

As shown above, this technique generally produces a uniform distribution of data items onto servers. Of course, bad luck might end up putting far more data items onto one server than another. But that’s pretty unlikely… right?

As far as I can tell, this problem was first given an efficient answer in 1987 by M.V. Ramakrishna. Ramakrishna phrases the problem as an “urn” problem (paraphrased):

There are balls tossed uniformly at random into one of urns, each urn having a capacity of at most balls. What is the probability that no bin overflows?

It turns out this probability is difficult enough to compute that an instance of it appears in a high-numbered (307) Project Euler problem. This blog post won’t go into the details behind efficiently computing the exact probability, but it will:

explore some unintuitive results that arise from these probabilities,
discuss some (very good) approximations that can be computed near-instantly,
provide a handy calculator for computing the probability,
and provide some general pointers for cluster sizing and tuning.

It can’t overflow that often, can it?

Since an example is worth a thousand equations, here’s an animated figure:

Each time a segment turns red, that segment is overflowing. The table below controls the parameters of the experiment:

Parameter	Value
Number of items
Number of bins
Bin capacity
Overflow probability

The probability listed in the table above is computed precisely using Ramakrishna’s technique. With 5 servers, and a server capacity of 4, you have a theoretical maximum capacity of 40 items. However, with just 10 items, there’s a 16.37% chance of one server overflowing!

Perhaps even more counter-intuitively, with 10 bins of size 3 and 15 items, there’s a nearly 50% chance of an overflow! That’s even with the theoretical capacity of the cluster being twice as large as the number of items we’re trying to store (a load factor of 0.5).

When we talk about hash tables, distributed or otherwise, we often talk about the load factor, which is the number of items divided by the total number of spots an item could occupy. In our case, the load factor is . However, playing with extreme cases makes it clear that there’s more to this problem than the load factor. For example, consider two scenarios: 5 bins with a capacity of 2 and 2 bins with a capacity of 5. With 5 items, the overflow probability for the second scenario is zero: no arrangement of items will cause any bin to overflow. The first scenario, however, overflows about 25% of the time.

Next, we’ll look at a way to approximate the probability of a bin overflowing. This process will also teach us quite a bit about what is really going on.

Approximations

Ramakrishna works out a few nice approximate closed-form expression for the overflow probability. To build this approximation, we first note that the probability of an arbitrary but fixed bin overflowing is binomial (e.g., the probability of at least successful trials occurring within total trials with success probability ). Letting be the load factor (the number of items divided by the maximum capacity), we can approximate this probability using a Poisson distribution:

When our bin capacity is large, and our load factor is not too close to one, the tail of the distribution will collapse rapidly, so we can approximate this sum by taking the first few terms (and applying some algebra):

Ramakrishna next makes the assumption that each bin overflows independently of other bins (which is definitely false, but remember, we’re building an approximation), which allows us to write the probability that no bin overflows as:

Ramakrishna shows that this approximation is very good in the case of and , and larger values of allow for lower values of . In addition to being a useful approximation for calculation, we can also differentiate this approximation to gain a better understanding of how the variables work.

Imagine we had a load factor of 0.5 and . This equation tells us that a unit increase in will result in a 20-fold increase to ! So, for example, if we were to increase our load factor from 0.5 to 0.6, we could expect . Counter-intuitively, this means that larger bin sizes are more sensitive to changes in load factor.

Let’s take a closer look at how the load factor, number of items, bin capacity, and number of bins interact. We’ll begin by fixing the capacity of each bin at 20. The sliders below allow you to vary the number of bins represented by each curve.

Fixed bin capacity (20)

Curve	Bins
Red
Green
Blue

The graph can be a little perplexing at first glance. To interpret it, first set the values to 5, 25, and 200. Notice that the red curve, representing 5 bins, at a load factor of 0.8, has an overflow probability of around 50%. This means that in a cluster configuration with 5 nodes, a node capacity of 20, and 80 items, there is (approximately) a 50% chance of a node overflowing. Compare this to the blue curve, representing 200 bins – if I have the same node capacity (20) and the same load factor of 0.8 (implying that I have 3200 items), my overflow probability is practically one. This is very counter-intuitive if you are used to thinking of the load factor as something that should be kept constant as a system grows. However, as we’ve seen here, keeping the load factor constant as you increase the number of nodes and data items will cause your overflow probability to skyrocket.

Imagine you have 5 cups, each currently filled with 8 pieces of candy. You instruct your friend, a computer scientist, to take the remaining 10 pieces of candy and evenly distribute them into each cup, such that each cup will have 10 pieces of candy. The computer scientist, having recently taken a data structures course, begins dropping each remaining piece of candy into a cup at random. Confused and unsatisfied, you are forced to redistribute the candies yourself (the computer scientist’s explanation of an “amortized cup” does not comfort you).

Clearly, the computer scientist’s strategy would’ve worked if your friend had gotten a little lucky. But exactly how lucky? Well, getting it exactly right would’ve been like rolling 10 dice (each representing a piece of candy), each with 5 sides (each representing a cup), and getting exactly two “ones”, two “twos”, two “threes”, two “fours”, and two “fives.” We know that there are possible ways the dice could land, and we know that the total number of ways the dice could land in the desired configuration is:

Taking the quotient, we end up with a success rate of around 1%. Not very good odds.³

Now imagine that instead of 5 cups filled with 8 pieces of candy with 10 pieces of candy leftover, you instead have 50 cups, each with 8 pieces of candy, with 100 pieces of candy still to be distributed. Each cup is still 80% full, but do you think your friend’s odds of success have changed? Now, your friend must roll exactly 20 “ones”, 20 “twos”, 20 “threes”, 20 “fours”, and 20 “fives.” Again, there are different ways for the dice to land. This time, the number of valid configurations is:

Since we’re starting to get some 60+ digit numbers, we’ll let Wolfram Alpha do the math for us. The result? A smashing 0.013% success rate. Just like keeping the cup only 80% full didn’t keep the success probability the same, keeping the load factor the same for consistent hashing schemes of different scales doesn’t give you the same overflow probability.⁴

Next, we will fix the number of bins, and instead alter the bin capacity.

Fixed number of bins (50)

Curve	Bin Capacity
Red
Green
Blue

Interpreting the graph is much the same. First, reset the graph to its default values. Notice that, with a bin capacity of 5 (red line), the overflow probability goes above 10% at about a load factor of 0.25. This corresponds to around 63 items. However, with a bin size of 25 (green line), the overflow probability does not go over 10% until a load factor of 0.55! This corresponds to around 688 items. So, by increasing each bin size by a factor of 5 (from 5 to 25), the total number of items I could store (with a fixed overflow probability) went up by a factor of 10!

We can also learn quite a bit from the slopes of the curves. As we predicted before, based on the derivative of the approximation, the blue curve (representing a bin size of 200) has a much steeper slope than the red curve (representing a bin size of 5). So, on the one hand, increasing bin capacity gives a very nice improvement to the maximum load factor that can be tolerated, but it also makes the system more sensitive to fluctuations in that load factor.

Handy calculator

Parameter	Value
Number of items
Number of bins
Bin capacity
Result type	exact
Overflow probability	0.2095

The calculator will use an exact method when the inputs are small enough, and fall back to using an estimate when the values become too large.

Takeaways & suggestions

When you are using a consistent hashing system, it’s good to keep overflows in mind. Unless any node can hold all your data, there’s a possibility that you’ll have overflow.

Since there’s (almost) always going to be some chance of overflow, you should probably do something about it. Google has put out an interesting paper showing that a simple strategy of “pushing” extra items from one bin to the next doesn’t require too many “pushes.” Vimeo put this idea into practice to great success.

Regardless of how you handle them, overflowing nodes are never a good thing. First, decide what your system needs: tolerance to changes in load factors, or maximal storage with low overflow probability. If you need the former, build many, smaller nodes. If you need the later, build larger nodes, but keep in mind that a small change in load factor may cause your overflow rate to skyrocket!

This problem gets more interesting when you consider what might happen in a heterogeneous cluster… it certainly makes the math a lot more complicated!

I’ll assume that both hash functions are perfectly uniform, which, of course, is never the case in practice. ↩
When you click and drag an item, the figure will neatly re-arrange each segment such that each data item is evenly spaced. In practice, the hash values of each data item is unlikely to spread across the segment so nicely. ↩
If you’d like, you can confirm the calculation using this simulation. ↩
More skeptical readers may claim that the load factor in this example is really 100%, not 80% – the argument holds equally well in both cases. ↩

Pretty pictures with perlin noise fields

2018-03-04T00:00:00+00:00

Perlin noise can be used to create some interesting and fun visual effects, like the image below (click for high resolution):

The above image was created by interpreting each point in 2D Perlin noise as an angle in a vector field, and then tracing particles through the resulting vector field. Each pixel is colored based on its flux. In order to explain how it is built, we’ll go over the basics of Perlin noise, and flow field particle tracing. You can check out the code on GitHub.

Perlin noise

Perlin noise was invented by Ken Perlin in 1983, and it didn’t take long before it became widely used in computer graphics applications.

Part of what makes Perlin noise so appealing is that it is:

random, meaning that it exhibits a lot of diversity,
smooth, meaning that there are no sharp edges or unnatural bumps,
and globally structured, meaning that it appears to have features and properties that span the whole image.

The first step of generating 2D Perlin noise is creating a lattice of grid points, and randomly assigning one of eight possible vectors to each point. These possible vectors correspond to the cardinal and ordinal directions: north, northeast, east, southeast, south, southwest, west, northwest. For example, a 3x3 grid might look like this:

After generating the grid, Perlin noise is generated in a point-wise fashion. For a given point , we draw vectors , , , and from the point to the nearest four grid points (see the demonstration below). We call the random vectors assigned to the nearest four grid points , , , and . We then compute four dot products – , , , and – between the randomly drawn vectors at each grid point and the distance vectors from our point to each grid point. For example, we have: .

Once we’ve computed , , , and , we use linear interpolation, commonly referred to in computer graphics applications as , in order to combine the dot product values. Linear interpolation is defined as:

The values and are interpolated between, and the value is a weight – higher values of bias the interpolation towards , whereas lower values of bias the intrpolation towards . The basic strategy for picking the color of a point was to compute the dot products, and then interpolate between the dot product of the top two sets of vectors, and , weighted based on the horizontal distance of the point to be colored to the grid points. This weighted value is called . The value is computed similarly for the bottom two sets of vectors (i.e., by interpolating between and ). Finally, the value of a pixel, , is computed by interpolating between and , this time weighting the interpolation by the vertical distance from the point to be colored to the grid points.

However, Perlin found this simple form of interpolation didn’t quite work well enough – it produced overly smooth images. As a result, Perlin first “fades” the weight values for each interpolation with the following function:

The demonstration below shows all the quantities described above for the point depicted. You can use your mouse to select a specific point as well.




Color

Since the vectors at each grid point are selected randomly, the dot product values have diverse properties. Notice how the color value only changes a little bit between two neighboring points – this gives Perlin noise a “smooth” look. By changing the size of the grid, one can control the size of features (i.e., global structures) of the noise.

The four images below show Perlin noise with a grid size of (starting from the top left) 3, 4, 5, and 6.

The grid size of Perlin noise is oftern referred to as the frequency of the noise, since higher values correspond to more details and finer-grained edges (and thus the higher frequency components of the image). Multiple frequencies can be combined together into a single image (normally, frequencies that are a power of two, or octaves, are chosen) in order to create multi-level structures:

You can also extend Perlin noise into three dimensions by creating a grid of cubes instead of a grid of squares. You’ll have 6 vectors instead of 4, but the procedure is essentially the same. You could extend Perlin noise to even higher dimensions, although the results might be difficult to interpret.

While Perlin noise values are normally converted directly into pixel values, as we’ve seen so far, we’ll next look at using each Perlin noise value as an angle in a vector field.

Vector fields

A vector field is a simple concept – essentially, a function mapping a point in space to a vector. These vectors can represent a lot of different things, but a very popular use of vector fields is representing velocities. The image below shows a vector field with particles tracing through it.

The plot next to the vector field shows the normalized flux of the vector field – the brightest pixel represents the cell that has seen the largest number of particles travel through it. The plot reset every twenty seconds, but you can also click here to reset it manually.

The implementation is pretty straightforward – an array holds the coordinates of each particle, and at every timestep the position of each particle is incremented by the vector indicated by the vector field. When particles go out of bounds, they are removed.

Turns out, we can interpret Perlin noise itself as a flow field! By considering the value of each point to be an angle instead of a color, converting between the two is trivial. Doing this, plotting the normalized flux, and then choosing a few fun coloring functions can yield some pretty pictures:

We can also make a video of the flux plot as it builds over time, which can look a bit like a fluid permeating through space.

You can see the full quality video here.

Full quality

All of the examples in this blog post can be generated with my Perlin noise Rust implementation, which is open source. Check out the repository for exact details on the algorithm.

Other types of noise

Noise is generally defined as an unknown and random modification to a signal. For example, static in the background of an audio recording or TV “snow” may be considered noise. One of the simplest types of noise, white noise, is just values drawn uniformally (often, from the range ). We can plot white noise as an image:

Here, each channel of each pixel is just a random value. The following Python program will generate it:

import numpy as np
import skimage.io
img = np.random.rand(256, 256, 3)
skimage.io.imsave("noise.png", img)

White noise is simple enough, but it isn’t useful because it doesn’t really look like anything. Researchers in computer graphics have been studying how to procedurally (i.e., with a computer) generate “natural” looking noise. One such attempt is Perlin noise, as we’ve seen, but there’s also Worley noise, which can often look like stone, water, or cells. With a little work, we can make it look like it is 3D:

For more examples of Worley noise, click here.

Notes

The best explaination of how to generate Perlin noise I’ve seen so far comes from Adrian Biagioli, who’s got a ton of cool projects.
You can read the original Perlin noise paper, “An Image Synthesizer”, and the second paper improving on the original algorithm.
I first saw the use of Perlin noise as a vector field on Andy Saia’s blog. You can find him on Twitter.

Computer-generated lines with a human feel

2017-10-23T00:00:00+00:00

Sometimes, the perfect version of something is too perfect. Consider two drawings of a cube shown below.

The left-hand side cube is generated exactly as you’d expect – by drawing a number of straight lines. The cube on the right-hand side, however, is generated using an algorithm from Meraj et al.’s 2008 paper, Mimicking Hand-Drawn Pencil Lines.

This work may seem entirely unmotivated; why, after all, would someone want an imperfect drawing when one could have a precise one? Meraj et al. argue that there are two primary reasons one might wish to be able to generate images that appear to be drawn by a human:

1) communication of uncertainty — precisely rendered computer graphics imply an exactness and perfection that may overstate the fidelity of a simulation or representation; and

2) communication of abstract ideas – simple line drawings such as diagrams used in textbooks can communicate abstract ideas in ways that a photograph cannot.

I would add that a hand-drawn look can also give an image an artistic quality. For example, we can use Meraj et al.’s algorithm to render an animated L-system. The result is something that resembles a hand-drawn flip book animation.

While the paper provides many details, the proposed algorithm is fairly simple. The authors cite prior work that found a parametric polynomial that describes the arc and acceleration of the human arm as it draws a line starting from to :

Here, varies from zero to one. While these polynomials come from a fairly large experimental study, Meraj et al. argue that the variation from a straight line introduced by these polynomials isn’t sufficient to produce a “hand-drawn” look.

So, in addition to this polynomial, Meraj et al. add random deviations at various points along the curve given by the polynomials. To do this, primary control points are placed every steps along the curve. At the midpoint of each line segment that is created, a secondary control point is added, and random noise is added to this third control point. The variable is set based on the total length of the line (longer lines get more control points).

In the example above, the black squares represent the primary control points and the red squares represent the secondary (noise) control points. The final line is drawn by fitting a quadratic spline between each set of three points. Note that the control points are denser around the edges of the line (humans make more mistakes when starting and stopping) than in the middle.

The results can also be used to sketch fractals, like the Sierpinski triangle. You can press the plus and minus button the change the degree of the fractal. The current degree is 1.

At degree five, the deviations from straight lines becomes very noticeable. I find the lack of crisp detail closely matches my attempts at drawing such small triangles. The results are satisfyingly realistic.

Meraj et al. dedicate a significant portion of their paper to mimicking the texture and pattern of pencil markings. They use statistical techniques based on local correlations in scanned images of pencil markings. Unfortunately, the authors did not publish their data, so I am not able to reproduce the results.

Here’s an animated rendering of another L-system that looks like seaweed:

You can use my implementation of Meraj et al.’s algorithm by installing the NPM package.

npm install --save humanlines

You can also check out the code on GitHub. Feel free to report any issues there or to me directly via Twitter.

The often-overlooked random forest kernel

2017-10-04T00:00:00+00:00

Machine learning problems often require a kernel function that measures the similarity between two different input vectors, and . Such kernel functions can enable linear algorithms (like SVM, PCA, and k-means) to find non-linear solutions to classification, regression, or clustering problems. Normally, a kernel function is selected from a standard set (although the most common by far appears to be the radial basis function) and tuned slightly based on the dataset. Here, we’ll discuss a type of kernel called the random forest kernel, which takes advantage of a pre-trained random forest in order to provide a custom-tailored kernel.

The Colonel’s Cocktail Party

Imagine you are a decorated colonel who hosts an annual cocktail party. Your guests mingle and periodically shift between groups of their peers, all the while admiring your tasteful mid-century decoration and savoring delicious hor d’oeuvres.

You, being both a subtle and naturally curious colonel, wish to determine whether or not two of your guests (guest 3 and guest 6) have any “affinity” for each other. However, to avoid detection, you cannot intervene. Observe the cocktail party below.

Portion 3 and 6 are together:	0.00
Portion 1 and 4 are together:	0.00
Average togetherness:	0.00

You keep track of how often guest 3 and guest 6 appear in the same group and compare that against how often two other guests (1 and 4), and the average guest, appear in the same group. As the party progresses, what can you, the colonel, conclude about the affinity guests 3 and 6 have for being in the same group?

We can think of each timestep of the cocktail party as a random partition of your guests. At any particular time , the five sets of party-goers are disjoint (assume invitees move between tables instantly). If there is an affinity for one individual to end up in the same group as another individual , then the percentage of time that we see and together will be higher than the average number of times we see every distinct pair together. If we define:

… then can be used a measure of affinity for and . The higher the value of , the higher the affinity between and . It turns out that can be used as a kernel function: specifically, given a set of partitionings, the function will produce a positive semidefinite Gram matrix. The specific mathematical requirements are interesting, but it suffices to accept that this is a sufficient and necessary condition for to be used as a kernel function.

The question then becomes: how does one create a set of partionings such that elements that are similar (in some context) are frequently grouped together? One can find answers in both Geurts et al. and Davies et al., but the short answer is to use a random forest!

Random Forest Kernel

Random forests are a powerful machine learning ensemble method that combine the results of several decision (or regression) trees into a single model. Consider the example below.

The two-forest random forest shown above has been trained on some dataset, and we can now use it to classify new vectors. Consider the vectors , , , and . You can touch or hover over a vector to see how the forest classifies it.

To use the random forest as a kernel, we view each level of each tree as a partitioning. For example, the first level of the first tree (left hand side) partitions potential vectors into two groups, and . The second level of second tree (right hand side) partitions potential vectors into four groups, , , , and . The random forest kernel, , is the number of times and fall into the same partition divided by the total number of partitionings.

The above tree contains five different partitionings. We can thus evaluate by counting the number of times they share a partition. Since and share all but one partition, . Since and share no partitions in common, the kernel is zero, i.e. .

If two items are in the same partition frequently, they are guaranteed to share some properties (i.e. their values share a set of constraints), but because the random forest is built using semantic information (e.g., the class or label of the data), these groupings of properties will have semantic meaning. Thus, items that share more partitions are more similar, and items that share few partitions are dissimilar.

Examples

It might seem odd to train a random forest model on a dataset and then use that random forset as a kernel, instead of using the random forest directly. However, this can be useful in a number of circumstances.

For example, consider the MNSIT digit recognition dataset. A large random forest can obtain acceptable (greater than 90% accuracy) if it is trained on all ten digit classes. In other words, if the random forest algorithm is allowed to see many examples of each digit class, it can learn to classify each of the ten digits. But what if the random forest algorithm is only allowed to see examples of 7s and 9s? Clearly, such a random forest would not be very useful on a dataset of 3s and 4s. However, using the random forest kernel, we can take advantage of a random forest trained on only 7s and 9s to help us understand the differences between 3s and 4s (this is often called transfer learning).

Consider two different PCA projections of the 3s and 4s from the MNIST dataset:

The left-hand side image shows the results of using Kernel PCA to find the two most significant (potentially non-linear) components of the 3s and 4s data using the random forest kernel. The right-hand side shows the two most significant (linear) components as determined by PCA.

For the most part, the components found using the random forest kernel provide a much better separation of the data. Remember, the random forest here only got to see 7s and 9s – it has never seen a 3 or a 4, yet the partitions learned by the random forest still carry semantic meaning about digits that can be transferred to 3s and 4s.

In fact, if one trains an SVM model to find a linear boundary on the first 5 principal components of the data, the accuracy for the random forest kernel assisted PCA + SVM is 94%, compared to the 87% of the linear PCA + SVM scheme.

The random forest kernel can be a quick and easy way to transfer knowledge from a random forest model to a related domain when techniques like deep belief nets are too expensive or simply overkill. The kernel can be evaluated quickly, and does not require special hardware or a significant amount of memory to compute. It isn’t frequently useful, but it is a nice trick to keep in your back pocket.

Notes

The metaphor of the cocktail party in this context was first introduced by A. Davies and Z. Ghahramani in their paper, “The Random Forest Kernel and other kernels for big data from random partitions” (available on arXiv). They explain:

We consider The Colonel, who is the host of a cocktail party. He needs to determine the strength of the affinity between two of his guests, Alice and Bob. Neither Alice and Bob, nor the other guests, must suspect the Colonel’s intentions, so he is only able to do so through surreptuous (sic) observation.

At the beginning of the evening, his N guests naturally form into different groups to have conversations. As the evening progresses, these groups naturally evolve as people leave and join different conversations. At m opportunities during the course of the evening, our host notes whether Alice and Bob are together in the same conversation.

As the Colonel farewells his guests (sic), he has a very good idea of Alice and Bob’s affinity for one another.

No, really, what's a monad?

2016-12-14T00:00:00+00:00

When seasoned Haskell developers try to explain monads to the rest of us, they’ll often find themselves resorting to ridiculously incomprehensible definitions. James Iry satirized this unique semi-meme as:

A monad is just a monoid in the category of endofunctors, what’s the problem?

The original quote, which reads with an equally satirical tone, comes from the book Categories for the Working Mathematician:

All told, a monad in X is just a monoid in the category of endofunctors of X, with product × replaced by composition of endofunctors and unit set by the identity endofunctor.

There’s also much debate over if monads are or are not burritos. There are several different examples of “monad tutorials” out there as well, although I’ve found they tend to be either overly simplistic (they only give an example of a monad) or mathematically impenetrable.

What follows is yet another attempt at explaining monads. In general, abstract concepts like monads require a bit of struggling to fully comprehend, but I’ve designed this particular guide with a few goals in mind:

First, the code examples can be viewed in either Python, Java, JavaScript or C++. You shouldn’t be struggling to learn a new language and monads at the same time. All the code samples are interactive and runnable thanks to repl.it.
Second, my approach tries to mirror, as closely as possible, how I have taught monads to students in the past.
Third, I will try to avoid abstract theory without practical examples. I’ve found that looking purely at abstract mathematics or code examples isn’t enough.
Finally, I’ll show how monads can be thought of as a design pattern to simplify code.

Before we get started, pick your preferred programming language. Personally, I find the Python code to be the most compact and readable, but you should pick whichever language with which you are most familiar.

Our first monads

First, consider the simple program below.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run():
  x = "friend"
  x = step1(x)
  x = step2(x)
  x = step3(x)
  return x

print(run())

class Main {
  public static void main(String[] args) {
    String x = "friend";
    x = step1(x);
    x = step2(x);
    x = step3(x);
    System.out.println(x);
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function run() {
  x = "friend";
  x = step1(x);
  x = step2(x);
  x = step3(x);
  return x;
}

console.log(run());

#include <iostream>
#include <string>

using namespace std;

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

string run() {
  string x = "friend";
  x = step1(x);
  x = step2(x);
  x = step3(x);
  return x;
}

int main() {
    cout << run() << "\n";
}

There are no tricks here: we just use a few functions static methods functions functions to put some strings together into a message, and then we print out that message. Notice how x is set to an initial value ("friend"), and then we set x to the result of calling a few other functionsmethodsfunctionsfunctions .

Next, let’s take this pattern and codify it a bit. We’ll still set x to some initial value, but then we’ll introduce two new functionsmethodsfunctionsfunctions: wrap and wrap_call wrapCall wrapCall wrapCall .

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"
  
def wrap(x):
    return x
    
def wrap_call(x, func):
    return func(x)
  
def run():
  x = "friend"
  
  # first, wrap up the value of x
  x = wrap(x)
  
  # now use wrap_call to run each step
  x = wrap_call(x, step1)
  x = wrap_call(x, step2)
  x = wrap_call(x, step3)
  
  return x
  
print(run())

import java.util.function.Function;

class Main {
  public static void main(String[] args) {
    String x = "friend";
    
    // first, wrap up the value of x
    x = wrap(x);
    
    x = wrap_call(x, Main::step1);
    x = wrap_call(x, Main::step2);
    x = wrap_call(x, Main::step3);
    System.out.println(x);
  }
  
  public static String wrap(String x) {
    return x;
  }
  
  public static String wrap_call(String s, Function<String, String> func) {
    return func.apply(s);
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function wrap(x) {
  return x;
}

function wrapCall(x, func) {
  return func(x);
}

function run() {
  x = "friend";
  
  // first, wrap up the value of x
  x = wrap(x);
  
  // now use wrapCall to run each step
  x = wrapCall(x, step1);
  x = wrapCall(x, step2);
  x = wrapCall(x, step3);
  return x;
}

console.log(run());

#include <iostream>
#include <string>

using namespace std;

string wrap(string x) {
  return x;
}

string wrapCall(string x, string (*func) (string)) {
  return func(x);
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

string run() {
  string x = "friend";
  
  // first, wrap up the value of x
  x = wrap(x);
  
  // now use wrapCall to run each step
  x = wrapCall(x, step1);
  x = wrapCall(x, step2);
  x = wrapCall(x, step3);
  return x;
}

int main() {
    cout << run() << "\n";
}

These two functions don’t do much: wrap just returns whatever we gave it, and wrap_callwrapCallwrapCallwrapCall just calls func with x as a parameter. You can run the code to verify it produces the same output as before. Take a minute to really convince yourself that the addition of wrap and wrap_callwrapCallwrapCallwrapCall leave the behavior of the program essentially unchanged.

Next, let’s get crazy and make wrap and wrap_callwrapCallwrapCallwrapCall do something a little less trivial.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"
  
def wrap(x):
    return "[" + x + "]"
    
def wrap_call(x, func):
    return "[" + func(x[1:-1]) + "]"
  
def run():
  x = "friend"
  
  # first, wrap up the value of x
  x = wrap(x)
  
  # now use wrap_call to run each step
  x = wrap_call(x, step1)
  x = wrap_call(x, step2)
  x = wrap_call(x, step3)
  
  return x
  
print(run())

import java.util.function.Function;

class Main {
  public static void main(String[] args) {
    String x = "friend";
    
    // first, wrap up the value of x
    x = wrap(x);
    
    x = wrap_call(x, Main::step1);
    x = wrap_call(x, Main::step2);
    x = wrap_call(x, Main::step3);
    System.out.println(x);
  }
  
  public static String wrap(String x) {
    return "[" + x + "]";
  }
  
  public static String wrap_call(String s, Function<String, String> func) {
    return "[" + func.apply(s.substring(1,s.length() - 1)) + "]";
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function wrap(x) {
  return "[" + x + "]";
}

function wrapCall(x, func) {
  return "[" + func(x.substring(1, x.length-1)) + "]";
}

function run() {
  x = "friend";
  
  // first, wrap up the value of x
  x = wrap(x);
  
  // now use wrapCall to run each step
  x = wrapCall(x, step1);
  x = wrapCall(x, step2);
  x = wrapCall(x, step3);
  return x;
}

console.log(run());

#include <iostream>
#include <string>

using namespace std;

string wrap(string x) {
  return "[" + x + "]";
}

string wrapCall(string x, string (*func) (string)) {
  return "[" + func(x.substr(1, x.length() - 2)) + "]";
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

string run() {
  string x = "friend";
  
  // first, wrap up the value of x
  x = wrap(x);
  
  // now use wrapCall to run each step
  x = wrapCall(x, step1);
  x = wrapCall(x, step2);
  x = wrapCall(x, step3);
  return x;
}

int main() {
    cout << run() << "\n";
}

All we’ve done is modify wrap to surround x with square brackets, and we’ve modified wrap_callwrapCallwrapCallwrapCall to remove the square brackets using Python’s string indexingsubstringsubstringsubstr , call func, and add the square brackets back.

The resulting program prints the same string as the previous program, but adds square brackets around it. Without a doubt, this is a very silly change to make to a program, since we could’ve just added the square brackets in step3. Notice, however, that this approach will work no matter how many steps we add to the program.

OK, one more step before we do something that’s actually useful. Let’s rename a few things, and make wrap and wrap_callwrapCallwrapCallwrapCall parameters.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  return "[" + x + "]"
    
def bind(x, func):
  return "[" +func(x[1:-1]) + "]"
  
print(run(ret, bind))

import java.util.function.Function;
import java.util.function.BiFunction;


class Main {
  public static void main(String[] args) {
    System.out.println(run(Main::ret, Main::bind));
  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  public static String ret(String x) {
    return "[" + x + "]";
  }
  
  public static String bind(String s, Function<String, String> func) {
    return "[" + func.apply(s.substring(1,s.length() - 1)) + "]";
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  return "[" + x + "]";
}

function bind(x, func) {
  return "[" + func(x.substring(1, x.length-1)) + "]";
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

console.log(run(ret, bind));

#include <iostream>
#include <string>

using namespace std;

string ret(string x) {
  return "[" + x + "]";
}

string bind(string x, string (*func) (string)) {
  return "[" + func(x.substr(1, x.length() - 2)) + "]";
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  T x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
    cout << run<string>(ret, bind) << "\n";
}

All we did was rename wrap to ret (which is short for return), rename wrap_callwrapCallwrapCallwrapCall to bind, and pass the two functions methods functions functions as parameters. Believe it or not, we’ve just written a monad (specifically, the monad is the pair of bind and ret)!

Symbolically, we can think of ret as a function that takes a value of some type and produces a value of some type . In our example, is “strings” and is “strings surrounded by square brackets.”

We can think of bind as a function that takes:

A value of type
A function that maps items of type to items of type

… and produces an item of type .

Going farther down the rabbit hole, we can say that bind is a mapping from the cross product of:

all the things of type
and all the functions of to

… to things of type .

It might help to “fill in the blanks” and think of ret and bind more concretely. In our example, ret took a string-typed () value and transformed it into a square-bracket-surrounded-string-typed () value. The bind function took a square-bracket-surrounded-string-typed () and a function that mapped strings () to strings (), such as step1, step2, and step3. bind produced a square-bracket-surrounded-string-typed () value. So the mathematics line up with our code.

While the mathematical formulation is specific, it often isn’t particularly clarifying about when a monad can or should be used. To answer these more practical questions (and to build our intuitive understanding of what a monad really is in the first place), let’s look at some less trivial examples.

In the code sample below, we create a monad that prints out debugging information from each step of our string building process.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  print("Initial value:", x)
  return "[" + x + "]"
  
def bind(x, func):
  print("Input to next step is:", x)
  result = func(x[1:-1])
  print("Result is:", result)
  return "[" + result + "]"
  
print(run(ret, bind))

import java.util.function.Function;
import java.util.function.BiFunction;


class Main {
  public static void main(String[] args) {
    System.out.println(run(Main::ret, Main::bind));
  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  public static String ret(String x) {
    System.out.println("Initial value: " + x);
    return "[" + x + "]";
  }
  
  public static String bind(String s, Function<String, String> func) {
    System.out.println("Input to next step is: " + s);
    String result = func.apply(s.substring(1,s.length() - 1));
    System.out.println("Result is: " + result);
    return "[" + result + "]";
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  console.log("Initial value: " + x);
  return "[" + x + "]";
}

function bind(x, func) {
  console.log("Input to next step is: " + x)
  result = "[" + func(x.substring(1, x.length-1)) + "]";
  console.log("Result is: " + result);
  return result;
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

console.log(run(ret, bind));

#include <iostream>
#include <string>

using namespace std;

string ret(string x) {
  cout << "Initial value: " << x << "\n";
  return "[" + x + "]";
}

string bind(string x, string (*func) (string)) {
  cout << "Input to next step is: " << x << "\n";
  string r = "[" + func(x.substr(1, x.length() - 2)) + "]";
  cout << "Result is: " << r << "\n";
  return r;
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  T x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
    cout << run<string>(ret, bind) << "\n";
}

By adding a few simple lines to ret and bind, we can print out the value of x before and after each step. Run the code above to see an example. Notice that without using a monad, we would have to write code for each step. With three steps, this isn’t too bad, but if we had a much larger number of steps, using a monad might’ve saved us significant time! And, since normally we only want to add print statements like this to our code temporarily, monads let us just swap the ret and bind functions around to change between production and debugging.

So far, the “actual” type of the value passed between each function has been a strStringStringstd::string : we invented the idea of the “square-bracket-surrounded-string type”, but the language didn’t know about it. This doesn’t have to be the case. Consider another example of a simple monad, the counting monad, below.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  return {"value": x, "count": 0}
  
def bind(x, func):
  return {"value": func(x["value"]), "count": x["count"] + 1}
  
print(run(ret, bind))

import java.util.function.Function;
import java.util.function.BiFunction;


class Main {
  public static void main(String[] args) {
    System.out.println(run(Main::ret, Main::bind));
  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  static class Record {
    String value;
    int count;
    
    public String toString() {
      return "value: " + value + " count: " + count;
    }
  }
  
  public static Record ret(String x) {
    Record r = new Record();
    r.value = x;
    r.count = 0;
    return r;
  }
  
  public static Record bind(Record input, Function<String, String> func) {
    Record r = new Record();
    r.value = func.apply(input.value);
    r.count = input.count + 1;
    return r;
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  return {value: x, count: 0};
}

function bind(x, func) {
  return {value: func(x.value), count: x.count + 1}
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

console.log(run(ret, bind));

#include <iostream>
#include <string>
#include <memory>

using namespace std;

struct Record {
  string x;
  int count;
};

shared_ptr<Record> ret(string x) {
  shared_ptr<Record> toR(new Record);
  toR->x = x;
  toR->count = 0;
  return toR;
}

shared_ptr<Record> bind(shared_ptr<Record> p, 
                        string (*func) (string)) {
  p->x = func(p->x);
  p->count++;
  return p;
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  // first, wrap up the value of x
  T x = ret("friend");
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
    cout << run<shared_ptr<Record>>(ret, bind)->x << "\n";
}

Above, our ret functionmethodfunctionfunction gives back a structure containing a value and a counter. Our bind updates the value stored inside the structure, and increments the counter. As a result, run now returns this special structure, and not a string.

Now, we get both the return value of the original run functionmethodfunctionfunction as well as the number of steps taken. This might not seem useful at first, but consider that bind now has access to the current “step count” each time it is called! Now we can write this monad:

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  return {"value": x, "count": 0}
  
def bind(x, func):
  with open("f" + str(x["count"]) + ".txt", "w") as f:
    f.write("Initial value: " + str(x["value"]) + "\n")
    result = func(x["value"])
    f.write("Result: " + str(result) + "\n")
    
  return {"value": result, "count": x["count"] + 1}
  
import os
print(run(ret, bind))
print(os.listdir("."))

import java.util.function.Function;
import java.util.function.BiFunction;
import java.io.*;


class Main {
  public static void main(String[] args) {
    System.out.println(run(Main::ret, Main::bind).value);
    File[] list = (new File(".")).listFiles();
    
    for (File f : list) {
      System.out.println(f);
    }
  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  static class Record {
    String value;
    int count;
  }
  
  
  public static Record ret(String x) {
      Record r = new Record();
      r.value = x;
      r.count = 0;
      return r;
  }
  
  public static Record bind(Record s, Function<String, String> func) {
    String result;
    try (Writer writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream("f" + s.count  + ".txt"), "utf-8"))) {
      writer.write("Initial value: " + s.value + "\n");
      result = func.apply(s.value);
      writer.write("Result: " + result + "\n");
    } catch (Exception e) {
      // logging failed, proceed normally
      result = func.apply(s.value);
    }
    
    Record r = new Record();
    r.value = result;
    r.count = s.count + 1;
    return r;
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

const fs = require("fs");

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  return {value: x, count: 0};
}

function bind(x, func) {
  result = func(x.value)
  fs.writeFileSync("f" + x.count + ".txt", "Initial value: " + x.value + "\n Result: " + result + "\n");
  return {value: result, count: x.count + 1};
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

console.log(run(ret, bind));
console.log(fs.readdirSync("."));

#include <iostream>
#include <string>
#include <memory>
#include <fstream>
#include <sstream> 
#include <stdlib.h>

// sadly, we need this because repl.it's GCC is too old
namespace std {
    template<typename T>
    std::string to_string(const T &n) {
        std::ostringstream s;
        s << n;
        return s.str();
    }
}

using namespace std;

struct Record {
  string x;
  int count;
};

shared_ptr<Record> ret(string x) {
  shared_ptr<Record> toR(new Record);
  toR->x = x;
  toR->count = 0;
  return toR;
}

shared_ptr<Record> bind(shared_ptr<Record> p, 
            string (*func) (string)) {
              
  ofstream myfile;
  myfile.open("f" + std::to_string(p->count) + ".txt");
  myfile << "Initial value: " << p->x << "\n";
  p->x = func(p->x);
  p->count++;
  myfile << "Result: " << p->x << "\n";
  return p;
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  // first, wrap up the value of x
  T x = ret("friend");
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
    cout << run<shared_ptr<Record>>(ret, bind)->x << "\n" << flush;
    system("ls");
}

Above, we use the count information from the previous step to name a file (f1.txt for step1, f2.txt for step2, f3.txt for step3) that records debugging information about each step of run.

Finally, we can use the identity monad, as seen below, to run the program normally (with no files written or extra prints):

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  return x
  
def bind(x, func):
  return func(x)
  
print(run(ret, bind))

import java.util.function.Function;
import java.util.function.BiFunction;


class Main {
  public static void main(String[] args) {
    System.out.println(run(Main::ret, Main::bind));
  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  
  public static String ret(String x) {
     return x;
  }
  
  public static String bind(String s, Function<String, String> func) {
    return func.apply(s);
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  return x;
}

function bind(x, func) {
  return func(x)
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

console.log(run(ret, bind));

#include <iostream>
#include <string>
#include <memory>

using namespace std;

string ret(string x) {
  return x;
}

string bind(string p, 
            string (*func) (string)) {
  return func(p);
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  // first, wrap up the value of x
  T x = ret("friend");
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
    cout << run<string>(ret, bind) << "\n";
}

Now we can change the behavior of run to either:

Run normally
Output debugging information to the console
Output debugging information to files

… just by changing which ret and bind we pass to run! Notice that there’s no need to change run at all to change which monad we use. Now, of course, you could’ve used a logging library with a configuration file or a global variable to control where to send debugging information, but I think that monads provide a very clean solution to the problem. Next, we will look at a monad that can’t so easily be replaced with a logging library.

With this basic structure, we could also easily build monads that:

Check some kind of halting condition (like the user pressing “cancel”) between each step
Update a progress indicator after each step

Lazy monads

A common use case for monads is transforming eager programs into lazy programs. An eager program is a program which immediately materializes its result. Most programs written in Python, except for code that uses generators,Java, except for parts of the Streams API,JavaScript that are synchronousC++ are eager. A lazy program is one that doesn’t materialize its result until it absolutely must. Examples include Python generators, Java streams, or JavaScript generators, introduced in ECMAScript 2015.

Imagine that your professor asks you to write the numbers 1 through 100 on placards so they can hold them up in class to discuss each number. You could create all 100 placards ahead of time, but you know that class is only an hour long and your professor is likely to spend at least twenty minutes talking about the delicate intricacies of the number 3. Since writing a number on a placard is a quick process, you decide to just come to class with blank placards, and quickly write a number on one as your professor needs it. Making all 100 placards ahead of time is an eager approach (you materialize each card ahead of time), whereas making each placard as required is a lazy approach.

The two most common examples of such monads are the “thunk” monad and the continuation monad. The “thunk” monad wraps a bunch of functions together to be executed at the same time (think waiting for the professor to ask for the first card, then making all the cards right afterwards). The continuation monad turns a bunch of functions into a single function that can be called multiple times to apply each of the original functions (each call is like making a single card).

The “thunk” monad

Thunks are one of those concepts in computer science from ancient history that has been “reclaimed” by functional programming advocates and SICP acolytes to describe a wide range of concepts, all of which vaguely deal with changing how a function is called (traditionally, thunks have been used to switch between call-by-value and call-by-reference semantics, but this isn’t relevant to us). Generally, when we talk about a “thunk” monad, we are talking about a monad that transforms an eager program into a single lazy value. So instead of evaluating step3(step2(step1("friend"))) all at once, we create a LazyWrapper classLazyWrapper classJavaScript objectRecord struct , which holds a listListarraystd::vector of functions to apply and a get functionmethodfunctionfunction that applies all of these functions to the initial value and returns it.

Check out the code below.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  class LazyWrapper:
    def __init__(self, x):
      self.initial = x
      self.__funcs = []
      
    def get(self):
      x = self.initial
      for f in self.__funcs:
        x = f(x)
      return x
      
    def add_func(self, f):
      self.__funcs.append(f)
      
  return LazyWrapper(x)
  
def bind(x, func):
  x.add_func(func)
  return x
  
lazy = run(ret, bind)
print("none of the steps have executed yet!")
print(lazy.get()) # this will execute them all

import java.util.function.Function;
import java.util.function.BiFunction;
import java.util.*;


class Main {
  public static void main(String[] args) {
    LazyWrapper lw = run(Main::ret, Main::bind);
    System.out.println("Nothing has been computed yet!");
    System.out.println(lw.get());
  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  static class LazyWrapper {
    private String initialValue;
    private List<Function<String, String>> funcs;
    
    public LazyWrapper(String initial) {
      initialValue = initial;
      funcs = new LinkedList<>();
    }
    
    public void addFunc(Function<String, String> f) {
      funcs.add(f);
    }
    
    public String get() {
      for (Function<String, String> f : funcs) {
        initialValue = f.apply(initialValue);
      }
      
      return initialValue;
    }
  }
  
  
  public static LazyWrapper ret(String x) {
      return new LazyWrapper(x);
  }
  
  public static LazyWrapper bind(LazyWrapper s, Function<String, String> func) {
    s.addFunc(func);
    return s;
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  return { initial: x, funcs: [], 
  add: function(x) { this.funcs.push(x); },
  get: function () { 
    let x = this.initial;
    this.funcs.forEach((f) => x = f(x));
    return x;
  }};
}

function bind(x, func) {
  x.add(func);
  return x;
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

const lazy = run(ret, bind);
console.log("nothing has run yet!");
console.log(lazy.get());

#include <iostream>
#include <string>
#include <memory>
#include <fstream>
#include <vector>


using namespace std;

struct Record {
  string initial;
  vector<string (*) (string)> funcs;
  
  string get() {
    for (auto f : funcs) {
      initial = f(initial);
    }
    
    return initial;
  }
};

shared_ptr<Record> ret(string x) {
  shared_ptr<Record> toR(new Record);
  toR->initial;
  toR->funcs.clear();
  return toR;
}

shared_ptr<Record> bind(shared_ptr<Record> p, 
                        string (*func) (string)) {
              
  p->funcs.push_back(func);
  return p;
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  // first, wrap up the value of x
  T x = ret("friend");
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
  shared_ptr<Record> result = run<shared_ptr<Record>>(ret, bind);
  cout << "nothing has been computed yet!" << "\n";
  
  cout << result->get() << "\n";
}

This sort of monad lets you prepare a complex operation (potentially composing together an arbitrary number of functionsmethodsfunctionsfunctions lazily, e.g., without actually executing any of the functionsmethodsfunctionsfunctions . Then, you can execute them all at once with a single call to get.

Why is this useful? Couldn’t you just only execute step3(step2(step1("friend"))) when you actually need to evaluate the result? Certainly – this monad is really just a different way to bundle up (compose) a bunch of functionsmethodsfunctionsfunctions .

The continuation monad

The continuation monad is a lot like the thunk monad we made previously, but instead of executing all the functionsmethodsfunctionsfunctions at once when get is called, it executes each functionmethodfunctionfunction one at a time.

def step1(x):
  return "Hello " + x
  
def step2(x):
  return x + ", monads aren't that complicated."
  
def step3(x):
  return "***" + x + "***"

def run(ret, bind):
  x = "friend"
  
  # first, wrap up the value of x
  x = ret(x)
  
  # next, use bind to call each step
  x = bind(x, step1)
  x = bind(x, step2)
  x = bind(x, step3)

  return x
  
  
def ret(x):
  class ContinuationWrapper:
    def __init__(self, x):
      self.x = x
      self.__funcs = [lambda a: a]
      
    def get(self):
      # WARNING: bug! will keep trying to pop forever!
      self.x = self.__funcs.pop(0)(self.x)
      return self.x
      
    def add_func(self, f):
      self.__funcs.append(f)
      
  return ContinuationWrapper(x)
  
def bind(x, func):
  x.add_func(func)
  return x
  
lazy = run(ret, bind)
print("none of the steps have executed yet!")
print(lazy.get()) # this will execute the first step (the initial value, "friend")
print(lazy.get()) # this will execute the second step
print(lazy.get()) # this will execute the third step
print(lazy.get()) # this will execute the fourth step

import java.util.function.Function;
import java.util.function.BiFunction;
import java.util.*;


class Main {
  public static void main(String[] args) {
    ContinuationWrapper lw = run(Main::ret, Main::bind);
    System.out.println("Nothing has been computed yet!");
    System.out.println(lw.get()); // the initial value
    System.out.println(lw.get()); // step1
    System.out.println(lw.get()); // step2
    System.out.println(lw.get()); // step3

  }
  
  public static <T> T run(Function<String, T> ret, BiFunction<T, Function<String, String>, T> bind) {
    String x = "friend";
    
    // first, wrap up the value of x
    T res = ret.apply(x);
    
    res = bind.apply(res, Main::step1);
    res = bind.apply(res, Main::step2);
    res = bind.apply(res, Main::step3);
    return res;
  }
  
  static class ContinuationWrapper {
    private String x;
    private Deque<Function<String, String>> funcs;
    
    public ContinuationWrapper(String initial) {
      this.x = initial;
      funcs = new LinkedList<>();
      funcs.add((x) -> x);
    }
    
    public void addFunc(Function<String, String> f) {
      funcs.add(f);
    }
    
    public String get() {
      // WARNING: bug! will keep trying to pop forever!
      x = funcs.removeFirst().apply(x);
      return x;
    }
  }
  
  
  public static ContinuationWrapper ret(String x) {
      return new ContinuationWrapper(x);
  }
  
  public static ContinuationWrapper bind(ContinuationWrapper s, Function<String, String> func) {
    s.addFunc(func);
    return s;
  }
  
  public static String step1(String x) {
    return "Hello " + x;
  }
  
  public static String step2(String x) {
    return x + ", monads aren't that complicated.";
  }
  
  public static String step3(String x) {
    return "***" + x + "***";
  }
}

function step1(x) {
  return "Hello " + x;
}

function step2(x) {
  return x + ", monads aren't the complicated.";
}

function step3(x) {
  return "***" + x + "***";
}

function ret(x) {
  return { initial: x, funcs: [(x) => x], 
           add: function(x) { this.funcs.push(x); },
           get: function () { 
            // WARNING: bug! will keep trying to pop forever!
            this.initial = this.funcs.shift()(this.initial);
            return this.initial;
  }};
}

function bind(x, func) {
  x.add(func);
  return x;
}

function run(ret, bind) {
  x = "friend";
  
  // first, wrap up the value of x
  x = ret(x);
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

const lazy = run(ret, bind);
console.log("nothing has run yet!");
console.log(lazy.get()); // initial value
console.log(lazy.get()); // step 1
console.log(lazy.get()); // step 2
console.log(lazy.get()); // step 3

#include <iostream>
#include <string>
#include <memory>
#include <fstream>
#include <vector>


using namespace std;

struct Record {
  string initial;
  vector<string (*) (string)> funcs;
  
  string get() {
    // WARNING: bug! will keep trying to pop forever!
    initial = funcs[0](initial);
    funcs.erase(funcs.begin());
    return initial;
  }
};

string id(string x) { return x; }

shared_ptr<Record> ret(string x) {
  shared_ptr<Record> toR(new Record);
  toR->initial = x;
  toR->funcs.clear();
  toR->funcs.push_back(id);
  return toR;
}

shared_ptr<Record> bind(shared_ptr<Record> p, 
                        string (*func) (string)) {
              
  p->funcs.push_back(func);
  return p;
}

string step1(string x) {
  return "Hello " + x;
}

string step2(string x) {
  return x + ", monads aren't that complicated."; 
}

string step3(string x) {
  return "***" + x + "***";
}

template<typename T> T run(T (*ret) (string), 
                           T (*bind) (T, string (*) (string))) {
  
  // first, wrap up the value of x
  T x = ret("friend");
  
  // now use wrapCall to run each step
  x = bind(x, step1);
  x = bind(x, step2);
  x = bind(x, step3);
  return x;
}

int main() {
  shared_ptr<Record> result = run<shared_ptr<Record>>(ret, bind);
  cout << "nothing has been computed yet!" << "\n";
  
  cout << result->get() << "\n"; // initial value
  cout << result->get() << "\n"; // step1
  cout << result->get() << "\n"; // step2
  cout << result->get() << "\n"; // step3

}

The result is that the first call to get gives you the result from step1, the next call to get gives you the result from step2, and the final call gives you the result of step3. Many of you may recognize this as a coroutine. My implementation is not very safe, and further calls to get will result in an error.

The final behavior of the monad is a lot like a Java iterator, a Python generator, JavaScript generator functions, “Goroutines” with channels in Golang (almost), Boost coroutine for C++, or Ruby Fiber. So the continuation monad doesn’t necessarily give you anything that your language didn’t have beforehand. Again, the point is that every monad we’ve discussed – the debugging monad, the continuation monad, etc. – is entirely generic and be substituted easily.

The monadic design pattern

While many of the more formal monads above can be useful, real-world systems are generally very large, and refactoring them to “support” monads (like the ret and bind functionsmethodsfunctionsfunctions in the examples above) is probably not feasible. But the insight gained from these simple examples can be applied as a “design pattern” to many problems often encountered in real-world code.

For example, consider the code below that accesses the value 6, which is deeply nested inside dictionariesMapsa JSON objectstd::maps . In the real world, the path might look something more like users -> data -> contact -> phone -> mobile, but the idea is the same.

import json
jdata = """{"a": {"b": {"c": {"d": {"e": 6}}}}}"""
jdata = json.loads(jdata)


def extract_value(data):
  return data["a"]["b"]["c"]["d"]["e"]

print(extract_value(jdata))

import java.util.*;

class Main {
  
  // {"a": {"b": {"c": {"d": {"e": 6}}}}}
  static Record root;

  
  public static void main(String[] args) {
    System.out.println(extractValue(root));
  }
  
  public static String extractValue(Record data) {
    return data.get("a").get("b").get("c").get("d").get("e").value;
  }
  
  
  static class Record {
    public Record() {
      children = new HashMap<>();
    }
  
    public Map<String, Record> children;
    public String value;
    
    public Record get(String s) { return children.get(s); }
  }
  
  static {
    Record r1 = new Record();
    r1.value = "6";
    
    Record r2 = new Record();
    r2.children.put("e", r1);
    
    Record r3 = new Record();
    r3.children.put("d", r2);
    
    Record r4 = new Record();
    r4.children.put("c", r3);
    
    Record r5 = new Record();
    r5.children.put("b", r4);
    
    root = new Record();
    root.children.put("a", r5);
  }
}

jdata = {a: {b: {c: {d: {e: 6}}}}};

function extractValue(data) {
  return data.a.b.c.d.e;
}

console.log(extractValue(jdata));

#include <iostream>
#include <string>
#include <map>

using namespace std;

map<string, void*>* get(string key, map<string, void*>* m){
  map<string, void*> x = *m;
  return (map<string, void*>*) x[key];
}


int* extractValue(map<string, void*>* m) {
  map<string, void*>* x1 = get("a", m);
  map<string, void*>* x2 = get("b", x1);
  map<string, void*>* x3 = get("c", x2);
  map<string, void*>* x4 = get("d", x3);
  int* x5 = (int*) get("e", x4);
  
  return x5;
}

int main() {
  int i = 6;
  map<string, void*> m;
  m["e"] = &i;
  
  map<string, void*> m1;
  m1["d"] = &m;
  
  map<string, void*> m2;
  m2["c"] = &m1;
  
  map<string, void*> m3;
  m3["b"] = &m2;
  
  map<string, void*> m4;
  m4["a"] = &m3;
  // m4 is the map: {"a": {"b": {"c": {"d": {"e": 6}}}}}

  int* value = extractValue(&m4);
  
  if (value == NULL) {
    cout << "value was null!" << "\n";
  } else {
    cout << "value: " << *value << "\n";
  }
  
}

Of course, this code isn’t safe, since if the input data doesn’t have exactly the right path, an error will occur. Verify this by changing either the access path or the data itself. So instead, we have to write code that “looks before leaping”:

import json
jdata = """{"a": {"b": {"c": {"d": {"e": 6}}}}}"""
jdata = json.loads(jdata)


def extract_value(data):
  if "a" not in data:
    return None
    
  if "b" not in data["a"]:
    return None
    
  if "c" not in data["a"]["b"]:
    return None
    
  if "d" not in data["a"]["b"]["c"]:
    return None
    
  if "e" not in data["a"]["b"]["c"]["d"]:
    return None
    
  return data["a"]["b"]["c"]["d"]["e"]

print(extract_value(jdata))

import java.util.*;

class Main {
  
  // {"a": {"b": {"c": {"d": {"e": 6}}}}}
  static Record root;

  
  public static void main(String[] args) {
    System.out.println(extractValue(root));
  }
  
  public static String extractValue(Record data) {
    if (!data.has("a"))
      return null;
  
    if (!data.get("a").has("b"))
      return null;
      
    if (!data.get("a").get("b").has("c"))
      return null;
    
    if (!data.get("a").get("b").get("c").has("d"))
      return null;
      
    if (!data.get("a").get("b").get("c").get("d").has("e"))
      return null;
      
    return data.get("a").get("b").get("c").get("d").get("e").value;
  }
  
  
  static class Record {
    public Record() {
      children = new HashMap<>();
    }
  
    public Map<String, Record> children;
    public String value;
    
    public Record get(String s) { return children.get(s); }
    public boolean has(String s) { return children.containsKey(s); }
  }
  
  static {
    Record r1 = new Record();
    r1.value = "6";
    
    Record r2 = new Record();
    r2.children.put("e", r1);
    
    Record r3 = new Record();
    r3.children.put("d", r2);
    
    Record r4 = new Record();
    r4.children.put("c", r3);
    
    Record r5 = new Record();
    r5.children.put("b", r4);
    
    root = new Record();
    root.children.put("a", r5);
  }
}

jdata = {a: {b: {c: {d: {e: 6}}}}};

function extractValue(data) {
  if (! ("a" in data))
    return null;
    
  if (! ("b" in data.a))
    return null;
    
  if (! ("c" in data.a.b))
    return null;
    
    
  if (! ("d" in data.a.b.c))
    return null;
    
  if (! ("e" in data.a.b.c.d))
    return null;
  
  return data.a.b.c.d.e;
}

console.log(extractValue(jdata));

#include <iostream>
#include <string>
#include <map>

using namespace std;

map<string, void*>* get(string key, map<string, void*>* m){
  map<string, void*> x = *m;
  return (map<string, void*>*) x[key];
}


int* extractValue(map<string, void*>* m) {
  
  if (m->find("a") == m->end())
     return NULL;
     
  map<string, void*>* x1 = get("a", m);
  
  if (x1->find("b") == x1->end())
    return NULL;
  
  map<string, void*>* x2 = get("b", x1);
  
  if (x2->find("c") == x2->end())
    return NULL;
  
  map<string, void*>* x3 = get("c", x2);
  
  if (x3->find("d") == x3->end())
    return NULL;
  
  map<string, void*>* x4 = get("d", x3);
  
  if (x4->find("e") == x4->end())
    return NULL;
  
  int* x5 = (int*) get("e", x4);
  return x5;
}

int main() {
  int i = 6;
  map<string, void*> m;
  m["e"] = &i;
  
  map<string, void*> m1;
  m1["d"] = &m;
  
  map<string, void*> m2;
  m2["c"] = &m1;
  
  map<string, void*> m3;
  m3["b"] = &m2;
  
  map<string, void*> m4;
  m4["a"] = &m3;
  // m4 is the map: {"a": {"b": {"c": {"d": {"e": 6}}}}}

  int* value = extractValue(&m4);
  
  if (value == NULL) {
    cout << "value was null!" << "\n";
  } else {
    cout << "value: " << *value << "\n";
  }
  
}

Instead, we could use try/excepttry/catchtry/catchtry/catch to “ask for forgiveness instead of permission”:

import json
jdata = """{"a": {"b": {"c": {"d": {"e": 6}}}}}"""
jdata = json.loads(jdata)


def extract_value(data):
  try:
    return data["a"]["b"]["c"]["d"]["e"]
  except:
    return None
    
print(extract_value(jdata))

import java.util.*;

class Main {
  
  // {"a": {"b": {"c": {"d": {"e": 6}}}}}
  static Record root;

  
  public static void main(String[] args) {
    System.out.println(extractValue(root));
  }
  
  public static String extractValue(Record data) {
    try {
      return data.get("a").get("b").get("c").get("d").get("e").value;
    } catch (Exception e) {
      return null;
    }
  }
  
  
  static class Record {
    public Record() {
      children = new HashMap<>();
    }
  
    public Map<String, Record> children;
    public String value;
    
    public Record get(String s) { return children.get(s); }
    public boolean has(String s) { return children.containsKey(s); }
  }
  
  static {
    Record r1 = new Record();
    r1.value = "6";
    
    Record r2 = new Record();
    r2.children.put("e", r1);
    
    Record r3 = new Record();
    r3.children.put("d", r2);
    
    Record r4 = new Record();
    r4.children.put("c", r3);
    
    Record r5 = new Record();
    r5.children.put("b", r4);
    
    root = new Record();
    root.children.put("a", r5);
  }
}

jdata = {a: {b: {c: {d: {e: 6}}}}};

function extractValue(data) {
  try {
    return data.a.b.c.d.e;
  } catch (e) {
    return null;
  }
}

console.log(extractValue(jdata));

#include <iostream>
#include <string>
#include <map>

using namespace std;

map<string, void*>* get(string key, map<string, void*>* m){
  map<string, void*> x = *m;
  return (map<string, void*>*) x.at(key);
}


int* extractValue(map<string, void*>* m) {
  try {
    map<string, void*>* x1 = get("a", m);
    map<string, void*>* x2 = get("b", x1);
    map<string, void*>* x3 = get("c", x2);
    map<string, void*>* x4 = get("d", x3);
    int* x5 = (int*) get("e", x4);
    return x5;
  } catch (const std::exception& e) {
    return NULL;
  }
}

int main() {
  int i = 6;
  map<string, void*> m;
  m["e"] = &i;
  
  map<string, void*> m1;
  m1["d"] = &m;
  
  map<string, void*> m2;
  m2["c"] = &m1;
  
  map<string, void*> m3;
  m3["b"] = &m2;
  
  map<string, void*> m4;
  m4["a"] = &m3;
  // m4 is the map: {"a": {"b": {"c": {"d": {"e": 6}}}}}

  int* value = extractValue(&m4);
  
  if (value == NULL) {
    cout << "value was null!" << "\n";
  } else {
    cout << "value: " << *value << "\n";
  }
  
}

… but this approach has performance implications in many languages (stack unwinding) and requires us a write a “lot” of boilerplate code. Let’s see if we can come up with something better by thinking with monads! Instead of thinking of bind as a function that takes in a wrapped value and another function, think of bind as a function that takes in a dictMapJSON objectstd::map and a key and produces another dictMapJSON objectstd::map by looking up that key. If the dictMapJSON objectstd::map passed to bind is NonenullfalseNULL , we simply return NonenullnullNULL again.

import json
jdata = """{"a": {"b": {"c": {"d": {"e": 6}}}}}"""
jdata = json.loads(jdata)


def extract_value(data, ret, bind):
  data = ret(data)
  data = bind(data, "a")
  data = bind(data, "b")
  data = bind(data, "c")
  data = bind(data, "d")
  data = bind(data, "e")
  return data
  
def ret(x):
  return x
  
def bind(data, field):
  if data == None or field not in data:
    return None
  return data[field]
  
print(extract_value(jdata, ret, bind))

import java.util.*;
import java.util.function.*;

class Main {
  
  // {"a": {"b": {"c": {"d": {"e": 6}}}}}
  static Record root;

  
  public static void main(String[] args) {
    System.out.println(extractValue(root, Main::ret, Main::bind));
  }
  
  public static String extractValue(Record data, Function<Record, Record> ret, BiFunction<Record, String, Record> bind) {
    
    Record r = ret.apply(data);
    r = bind.apply(r, "a");
    r = bind.apply(r, "b");
    r = bind.apply(r, "c");
    r = bind.apply(r, "d");
    r = bind.apply(r, "e");
    
    return (r == null ? null : r.value);
  }
  
  public static Record ret(Record r) {
    return r;
  }
  
  public static Record bind(Record r, String field) {
    if (r == null || !r.has(field))
      return null;
      
    return r.get(field);
  }
  
  
  static class Record {
    public Record() {
      children = new HashMap<>();
    }
  
    public Map<String, Record> children;
    public String value;
    
    public Record get(String s) { return children.get(s); }
    public boolean has(String s) { return children.containsKey(s); }
  }
  
  static {
    Record r1 = new Record();
    r1.value = "6";
    
    Record r2 = new Record();
    r2.children.put("e", r1);
    
    Record r3 = new Record();
    r3.children.put("d", r2);
    
    Record r4 = new Record();
    r4.children.put("c", r3);
    
    Record r5 = new Record();
    r5.children.put("b", r4);
    
    root = new Record();
    root.children.put("a", r5);
  }
}

jdata = {a: {b: {c: {d: {e: 6}}}}};

function extractValue(data, ret, bind) {
  data = ret(data);
  data = bind(data, "a");
  data = bind(data, "b");
  data = bind(data, "c");
  data = bind(data, "d");
  data = bind(data, "e");
  return data;
}

function ret(x) {
  return x;
}

function bind(data, field) {
  if (data === null || !(field in data))
    return null;
    
  return data[field];
}

console.log(extractValue(jdata, ret, bind));

#include <iostream>
#include <string>
#include <map>

using namespace std;

map<string, void*>* ret(map<string, void*>* m) {
  return m;
}

map<string, void*>* bind(map<string, void*>* m, string key){
  if (m == NULL || m->find(key) == m->end())
    return NULL;
    
  map<string, void*> x = *m;
  return (map<string, void*>*) x[key];
}


int* extractValue(map<string, void*>* m) {
  auto x = ret(m);
  auto x1 = bind(x,  "a");
  auto x2 = bind(x1, "b");
  auto x3 = bind(x2, "c");
  auto x4 = bind(x3, "d");
  auto x5 = bind(x4, "e");
  return (int*)x5;
}

int main() {
  int i = 6;
  map<string, void*> m;
  m["e"] = &i;
  
  map<string, void*> m1;
  m1["d"] = &m;
  
  map<string, void*> m2;
  m2["c"] = &m1;
  
  map<string, void*> m3;
  m3["b"] = &m2;
  
  map<string, void*> m4;
  m4["a"] = &m3;
  // m4 is the map: {"a": {"b": {"c": {"d": {"e": 6}}}}}

  int* value = extractValue(&m4);
  
  if (value == NULL) {
    cout << "value was null!" << "\n";
  } else {
    cout << "value: " << *value << "\n";
  }
  
}

Now we use several calls to bind in order to access each key in the dictMapJSON objectstd::map . If the key doesn’t exist, bind gives back NonenullnullNULL , and all future calls to bind also give back NonenullnullNULL . Now the code is much safer, and changing the data or the access path won’t result in a crash. This type of monad (where the value can suddenly become “bad” and all future bind calls return the same “bad” value) is often called the Maybe monad, and it solves the problem.

… but we certainly didn’t reduce the amount of boilerplate from the “ask for forgiveness” example! In fact, we’ve added a lot more! Next, let’s combine some object-orientedobject-orientedobject-oriented programming know-how to make our monad easier to use.

import json
jdata = """{"a": {"b": {"c": {"d": {"e": 6}}}}}"""
jdata = json.loads(jdata)


class MonadicDict:
    def __init__(self, data):
      self.data = data
    
    def __getitem__(self, field):
      if self.data == None or field not in self.data:
        return MonadicDict(None)
      return MonadicDict(self.data[field])
      
    def get(self):
      return self.data
      

print(MonadicDict(jdata)["a"]["b"]["c"]["d"]["e"].get())

import java.util.*;

class Main {
  
  // {"a": {"b": {"c": {"d": {"e": 6}}}}}
  static Record root;

  
  public static void main(String[] args) {
    System.out.println(extractValue(root));
  }
  
  public static String extractValue(Record data) {
    MonadicRecord mr = new MonadicRecord(data);
    
    return mr.get("a").get("b").get("c").get("d").get("e").getValue();
  }
  
  static class MonadicRecord {
    private Record r;
    public MonadicRecord(Record r) {
      this.r = r;
    }
    
    public MonadicRecord get(String field) {
      if (r == null)
        return new MonadicRecord(null);
        
      return new MonadicRecord(r.get(field));
    }
    
    public String getValue() {
      if (r == null)
        return null;
        
      return r.value;
    }
  }
  
  static class Record {
    public Record() {
      children = new HashMap<>();
    }
  
    public Map<String, Record> children;
    public String value;
    
    public Record get(String s) { return children.get(s); }
    public boolean has(String s) { return children.containsKey(s); }
  }
  
  static {
    Record r1 = new Record();
    r1.value = "6";
    
    Record r2 = new Record();
    r2.children.put("e", r1);
    
    Record r3 = new Record();
    r3.children.put("d", r2);
    
    Record r4 = new Record();
    r4.children.put("c", r3);
    
    Record r5 = new Record();
    r5.children.put("b", r4);
    
    root = new Record();
    root.children.put("a", r5);
  }
}

jdata = {a: {b: {c: {d: {e: 6}}}}};

class MonadicJSON {
  constructor(data) {
    this.data = data;
  }
  
  getField(x) {
    if (this.data === null)
      return this;
    
    if (x in this.data)
      return new MonadicJSON(this.data[x]);
    
    return new MonadicJSON(null);
  }
}

const mdata = new MonadicJSON(jdata);

const result = mdata.getField("a").getField("b").getField("c").getField("d").getField("e");

console.log(result.data);

#include <iostream>
#include <string>
#include <map>
#include <memory>

using namespace std;


class Record {
public:
  Record(bool isMap, void* ptr) : isMap(isMap), ptr(ptr) { }
  bool isMap;
  void* ptr;
};

class MonadicMap {
  void* m;
  bool isValue;
public:
  MonadicMap(void* m) : m(m) { isValue = false; }
  void* asPtr() { return m; }
  
  MonadicMap operator[](const string& s) {
    if (isValue || m == NULL)
      return MonadicMap(NULL);
      
    map<string, Record*> x = *(map<string, Record*>*)m;
    
    if (x.find(s) == x.end())
      return MonadicMap(NULL);
    
    MonadicMap m = MonadicMap(x[s]->ptr);
    m.isValue = ! x[s]->isMap;
    return m;
  }
};


int* extractValue(map<string, Record*>* data) {
  MonadicMap m(data);
  return (int*) (m["a"]["b"]["c"]["d"]["e"].asPtr());
  
}

int main() {
  
  int i = 6;
  map<string, Record*> m;
  map<string, Record*> m1;
  map<string, Record*> m2;
  map<string, Record*> m3;
  map<string, Record*> m4;

  Record r1(true, &m1);
  Record r2(true, &m2);
  Record r3(true, &m3);
  Record r4(true, &m4);
  Record r5(false, &i);

  m["a"] = &r1;
  m1["b"] = &r2;
  m2["c"] = &r3;
  m3["d"] = &r4;
  m4["e"] = &r5;

  
  // m4 is the map: {"a": {"b": {"c": {"d": {"e": 6}}}}}

  int* value = extractValue(&m);
  
  if (value == NULL) {
    cout << "value was null!" << "\n";
  } else {
    cout << "value: " << *value << "\n";
  }
  
}

Here, we’ve created a class with a __getitem__ functionget methodgetField methodoperator[] function that does the same thing as bind did in our previous example. Now we can use the . operatorthis get methodthis getField methodthe overloaded [] operator to safely access arbitrary fields in nested data! No stack unwinding, no boilerplate, totally safe.

Purists will raise their voices in anger and insist that this isn’t really a monad (actually, most of the purists have probably made a few complaints already). Depending on how pedantic you want to be, this class may or may not be considered monadic, but it is a useful construction that came about from monadic thinking.

How I Learned to Stop Worrying and Love the Monad

Hopefully you have now gained at least a little sympathy for the Haskell purist who laments Python’sJava’sJavaScript’sC++’s lack of 1st class support for monads. In languages like Haskell, applying a monad to a function doesn’t require you to refactor that function into a bunch of calls to bind: every function actually already uses the identity/trivial monad, and you can swap out the implementation anytime you want. At the same time, monadic concepts can be used effectively in many other languages, including popular ones like Python, Java, JavaScript, and C++. While your language may lack “proper” support for monads, don’t be afraid to try “monads as a design pattern” when it seems appropriate.

Finally, try not to become part of the problem. Unless your in a context where you know everyone will know what you are talking about, don’t use the word “endofunctor.” When someone asks what Java’s Optional is, don’t just smugly reply “oh, it’s the Maybe monad.” And, please, please, when someone asks “what’s a monad?”, don’t say “a monad is like a burrito.”

Ryan Marcus

Most influential database papers

Most influential papers of all time

Most influential papers, by year

Most influential people

Incoming citations ()

Outgoing citations ()

Generating bios with large language models

Footnotes

Applying Bao to distributed systems

How Bao works

From single-node to distributed

Vertica

Azure Synapse (SQL Server)

RedShift

Discussion

Notes

Machine learning for systems

What’s going on here?

The Big Picture

FAQs

Good comment, bad comment

For the computer

Correct

Effective

Efficient

For your fellow programmers

Intuitive

Succinct

Precise

Conclusions

Footnotes

Overflow in consistent hashing

It can’t overflow that often, can it?

Approximations

Handy calculator

Takeaways & suggestions

Pretty pictures with perlin noise fields

Perlin noise

Vector fields

Other types of noise

Notes

Computer-generated lines with a human feel

The often-overlooked random forest kernel

The Colonel’s Cocktail Party

Random Forest Kernel

Examples

Notes

No, really, what's a monad?

Our first monads

Lazy monads

The “thunk” monad

The continuation monad

The monadic design pattern

How I Learned to Stop Worrying and Love the Monad