Jekyll2023-11-10T15:22:54+00:00https://juancn.github.io/feed.xmlCodng.comTrying to change the world one post at a timeSorting and paging on distributed data2018-03-12T03:00:00+00:002018-03-12T03:00:00+00:00https://juancn.github.io/post/2018/03/12/sorting-and-paging-on-distributed-data<blockquote>
<p><strong>NOTE:</strong> This is a revised version of an article I wrote for <a href="https://engineering.medallia.com/blog/posts/sorting-and-paging-on-distributed-data/">Medallia’s engineering blog</a></p>
</blockquote>
<p>Sorting and paging is a very common problem. The gist of it is that you have a large amount of data that you filter and sort and for rendering purposes, you want a fixed size window into the sorted and filtered data.</p>
<p>There are two ways to model apis to request a page:</p>
<ul>
<li>use a reference element (call it <code class="language-plaintext highlighter-rouge">r</code>) and a <code class="language-plaintext highlighter-rouge">pageSize</code>, so you request a page of size <code class="language-plaintext highlighter-rouge">pageSize</code>, with elements after <code class="language-plaintext highlighter-rouge">r</code></li>
<li>use a <code class="language-plaintext highlighter-rouge">pageStartIdx</code> into the sorted data and a <code class="language-plaintext highlighter-rouge">pageSize</code></li>
</ul>
<p>The first case is relatively easy to distribute, and is the kind of strategy (with variations) that most sane large scale applications use.
This first scenario is usually sharded by having each shard accumulate only values after <code class="language-plaintext highlighter-rouge">r</code> in <code class="language-plaintext highlighter-rouge">pageSize</code> sized priority queues, and then sending each to a coordinator node so they’re merged and any extra data is discarded (there are optimizations where you can build a tree of aggregations to reduce total traffic and distribute merging somewhat if you have many nodes).</p>
<p>The second scenario (using an offset and a page size) is much more interesting in my opinion and there’s little to none literature on how to solve it efficiently. This is the one is the one we will tackle in this article.</p>
<p>If you’re starting from scratch, do yourself a favor and <strong>pick the first strategy</strong>, it will save you countless headaches down the road, even if scale is not an issue yet. If on the other hand, you’re stuck with option two, keep reading.</p>
<h1 id="doing-it-the-hard-way">Doing it the hard way</h1>
<p>In the restricted case of an offset/page sort, all page offsets are page size aligned, that is:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pageStartIdx</span> <span class="o">%</span> <span class="n">pageSize</span> <span class="o">==</span> <span class="mi">0</span>
</code></pre></div></div>
<p>I’ll focus on the general case, where you want a fixed size page starting at an arbitrary offset into sorted data, if we have:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">k</code>: an offset into the sorted data as if it were in one place (i.e. as if it were large array)</li>
<li><code class="language-plaintext highlighter-rouge">pageSize</code>: desired page size</li>
</ul>
<p>Let’s use an example, if the data were in one place (e.g. an array), and you want to go to offset 4 (<code class="language-plaintext highlighter-rouge">k=4</code>), with a page size of 3 (<code class="language-plaintext highlighter-rouge">pageSize=3</code>), it’s enough to just slice the sorted array:</p>
<p><img src="/images/2018-03-12-sorting-and-paging-on-distributed-data/sorted-data.png" alt="Representation of a page on sorted data" /></p>
<p>Or in pseudo-code:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">copyOfRange</span><span class="o">(</span><span class="n">data</span><span class="o">,</span> <span class="n">k</span><span class="o">,</span> <span class="n">min</span><span class="o">(</span><span class="n">k</span> <span class="o">+</span> <span class="n">pageSize</span><span class="o">,</span> <span class="n">length</span><span class="o">(</span><span class="n">data</span><span class="o">)))</span>
</code></pre></div></div>
<p>The function returns a new array and it has the following signature:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">copyOfRange</span><span class="o">(</span><span class="kt">int</span><span class="o">[]</span> <span class="n">array</span><span class="o">,</span> <span class="kt">int</span> <span class="n">fromInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">toExclusive</span><span class="o">)</span>
</code></pre></div></div>
<p>But what happens when your data is distributed?</p>
<p>Picture the situation when your data is spread on a number of shards, these shards are somewhat evenly sized and the data is randomly distributed among them (it’s not strictly required, but it helps and is usually somewhat close to reality).</p>
<p>The simplest approach is to have all shards send all data to a coordinator that merges and discards as necessary. This is terribly wasteful, because the coordinator would end up having to see all the sorted data, or at least the sorted keys of the data, compute the set of records that are actually on the window, and then fetch that.</p>
<p>Another simple approach that may be good enough for some cases is to have each shard generate a locally sorted list of elements from 0 to the maximum element position (<code class="language-plaintext highlighter-rouge">k+pageSize</code>).</p>
<p>For example, if we want the 100th through 102nd elements sorted over the full data (<code class="language-plaintext highlighter-rouge">k=100</code>, <code class="language-plaintext highlighter-rouge">pageSize=2</code>), each shard would build the sorted list of the first 102 elements in the shard.</p>
<p>The lists from each shard can be merged (in sort order) and only the first 102 elements retained.</p>
<p>When all lists from all shards have been merged, the elements in the 100th through 102nd positions represent the desired result.</p>
<p>Note that even if a shard does not have enough elements in its local list, the merged results from all the shards may have enough elements.</p>
<p>The problem with this approach is that while easy to implement, the further down you paginate into it (k becomes bigger), more data is needed to be shipped and merged. If you have a few hundred million records and you try to paginate about the middle of them it’s not pretty,</p>
<p>We can do better.</p>
<h1 id="a-distributed-paginated-sort">A distributed paginated sort</h1>
<p>With more communication between the coordinator and the shards we can greatly reduce the amount of data that needs to be shipped around the network (at the cost of some latency due to additional round trips).</p>
<p>The general intuition for this algorithm is that the leftmost elements sent by each shard in the algorithm I just described (the not so simple one) are mostly redundant, and with some care we can avoid sending them to the coordinator.</p>
<p>This algorithm assumes each shard can hold its locally filtered and sorted set of keys without much trouble.</p>
<p>There are three stages to it, or round trips between the coordinator and the shards, where the coordinator helps the shards figure out which information is redundant.</p>
<p>We also introduce the notion of filtering the data before sorting, since it’s a common operation (think <code class="language-plaintext highlighter-rouge">SELECT * FROM ... WHERE ... ORDER BY ...</code>), applications don’t typically sort entired datasets.</p>
<h1 id="round-trip-1">Round trip 1</h1>
<p>We first filter on each shard (we could also sort, but it’s not strictly necessary at this point) and return the number of elements that match the filter to the coordinator which adds them up for all shards.</p>
<p>This total across shards we will refer to as <code class="language-plaintext highlighter-rouge">totalLen</code>, and is the total number of elements in the filtered and sorted data. We need this to estimate where the window is on the global sorted data (if it existed in one place).</p>
<p>The insight is that <code class="language-plaintext highlighter-rouge">k/totalLen</code> is a number between 0 and 1 that can be used to approximate the beginning of the desired data in each shard if we assume data across shards is randomly distributed.</p>
<p>This step can be skipped if <code class="language-plaintext highlighter-rouge">totalLen</code> can be obtained in some other way.</p>
<h1 id="round-trip-2">Round trip 2</h1>
<p>On each shard, we sort the filtered data and compute a local window. The sort could have happened in the previous round trip or while we returned the local length (so as to increase parallelism).</p>
<p>We’ll refer to the filtered sorted keys in the shard as <code class="language-plaintext highlighter-rouge">data</code>. Assume it’s an array or something that’s randomly accessible by a zero based index.</p>
<p>The local window is defined by two keys, a minimum key and a maximum key. We compute each key’s offset into the sorted data as follows:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">minKeyOff: max(0, (k * length(data)) / totalLen)</code></li>
<li><code class="language-plaintext highlighter-rouge">maxKeyOff: min(minKeyOff + pageSize, length(data) - 1)</code></li>
</ul>
<p>And return <code class="language-plaintext highlighter-rouge">data[minKeyOff]</code> and <code class="language-plaintext highlighter-rouge">data[maxKeyOff]</code></p>
<p>The intuition here is that if we know the total length of the data, we can guess where our local window should be. But this is not enough information to paginate into the data, we need all local windows to be adjacent to each other without holes between them (i.e. missing elements). We’ll see why later.</p>
<p>Once all the local window edges are known, the coordinator merges them by picking the smallest <code class="language-plaintext highlighter-rouge">data[minKeyOff]</code> returned and the largest <code class="language-plaintext highlighter-rouge">data[maxKeyOff]</code>.</p>
<p>We’ll call these <code class="language-plaintext highlighter-rouge">minKey</code> and <code class="language-plaintext highlighter-rouge">maxKey</code>. These two keys define a global window that should contain our page and we’ll use in the third round trip to ensure local windows are contiguous in the globally sorted data.</p>
<h1 id="round-trip-3">Round trip 3</h1>
<p>Once again we go to each shard, but this time we send <code class="language-plaintext highlighter-rouge">minKey</code> and <code class="language-plaintext highlighter-rouge">maxKey</code>. Each shard expands its local window (as computed in round 2) to include all elements between <code class="language-plaintext highlighter-rouge">minKey</code> and <code class="language-plaintext highlighter-rouge">maxKey</code>.</p>
<p>This expansion ensures that between the slices of each shard there are no gaps where we could miss some element that would be present if we could hold all the data in one place.</p>
<p>Each shard then returns:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">nLeft</code>: number of elements left of the corrected local window</li>
<li><code class="language-plaintext highlighter-rouge">windowElements</code>: all elements in the local window.</li>
</ul>
<p>Once we’re on the coordinator, we add up all the <code class="language-plaintext highlighter-rouge">nLeft</code> values for each shard and merge all the <code class="language-plaintext highlighter-rouge">windowElements</code> (<em>merge</em> as merge sort uses the term), we’ll call this <code class="language-plaintext highlighter-rouge">merged</code>.</p>
<p>Then we can carve a page out of <code class="language-plaintext highlighter-rouge">merged</code> as follows:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">copyOfRange</span><span class="o">(</span><span class="n">merged</span><span class="o">,</span>
<span class="n">k</span> <span class="o">-</span> <span class="n">nLeftSum</span><span class="o">,</span>
<span class="n">min</span><span class="o">(</span><span class="n">k</span> <span class="o">-</span> <span class="n">nLeftSum</span> <span class="o">+</span> <span class="n">pageSize</span><span class="o">,</span> <span class="n">length</span><span class="o">(</span><span class="n">merged</span><span class="o">)))</span>
</code></pre></div></div>
<p>If the data is well behaved, the data shipped is roughly proportional to the page size and the number of shards and doesn’t depend on how far we paginate nor on how much data we have (with the exception of a dataset with many duplicate keys).</p>
<h1 id="additional-notes">Additional Notes</h1>
<p>Why does this work?</p>
<p>The trick is in the interaction between the initial local window selection and the subsequent window expansion.</p>
<p>The local window start for each shard is chosen in round two as follows:</p>
\[\lfloor \frac{S_i \, k}{totalLen} \rfloor\]
<p>Where \(S_i\) is the ith shard’s data length (the floor is to account for using integer arithmetic). Note that local windows can sometimes be expanded. For the start value it means this initial value is an upper bound since expansion makes this number smaller.</p>
<p>We know by round one that \(totalLen\) is the sum of all shard’s lengths:</p>
\[totalLen = \sum S_i\]
<p>and \(nLeftSum\) is smaller than or equal to the sum of all window starts (due to the initial choice of window):</p>
\[nLeftSum \leq \sum \lfloor \frac{S_i \, k}{totalLen} \rfloor \leq \sum \frac{S_i \, k}{totalLen}\]
<p>which we can simplify:</p>
\[\sum \frac{S_i \, k}{totalLen} = \frac{k}{totalLen} \sum S_i = k \frac{totalLen}{totalLen} = k\]
<p>And use it as a bound for \(nLeftSum\):</p>
\[nLeftSum \leq k\]
<p>The only case when \(nLeftSum\) is actually smaller than \(k\), is when some of the local windows were expanded to include additional elements, making <code class="language-plaintext highlighter-rouge">merged</code> larger to compensate in the same amount as \(nLeftSum\) is smaller.</p>
<p>Since each shard sends at least \(pageSize\) elements, the desired window is always contained in \(merged\) (with the exception of when \(k\) is close to the end of the sorted data).</p>
<p>This is easy to see with an example. Let’s say we have two similar shards after sorting and filtering, and we want the page with <code class="language-plaintext highlighter-rouge">k=4</code>, <code class="language-plaintext highlighter-rouge">pageSize=3</code>, the initial local window selection yields:</p>
<p><img src="/images/2018-03-12-sorting-and-paging-on-distributed-data/shards1.png" alt="" /></p>
<blockquote>
<p><strong>NOTE:</strong> Observe that in the example the data has 10 elements, but the desired offset <code class="language-plaintext highlighter-rouge">k=4</code>, so each local window starts after the second element. That is because when <code class="language-plaintext highlighter-rouge">k=4</code> is scaled to the size of each shard’s local data, it’s roughly divided in half, since there are two shards</p>
</blockquote>
<p>If these were the definitive windows, it’s <code class="language-plaintext highlighter-rouge">nLeftSum</code> would be 4 which is equal to k and merged would have 6 elements. But since we need to ensure one extra D is included in shard 2, we expand the windows to:</p>
<p><img src="/images/2018-03-12-sorting-and-paging-on-distributed-data/shards2.png" alt="" /></p>
<p>Now, <code class="language-plaintext highlighter-rouge">nLeftSum</code> is equal to 3, one less than <code class="language-plaintext highlighter-rouge">k</code>, but <code class="language-plaintext highlighter-rouge">merged</code> has been expanded by one element; the same amount which <code class="language-plaintext highlighter-rouge">nLeftSum</code> decreased. The merged set is always expanded to the left by <code class="language-plaintext highlighter-rouge">k - nLeftSum</code> and it typically holds at least <code class="language-plaintext highlighter-rouge">nShards*pageSize</code> elements (this doesn’t hold when we’re picking a window at the end of the sorted data).</p>
<p>The worst case for this algorithm in terms of transferred data is when the shards are fragments of a sorted set. Put another way, there’s some concatenation of the shards that yields the globally sorted data. For example, with <code class="language-plaintext highlighter-rouge">k=6</code>, <code class="language-plaintext highlighter-rouge">pageSize=2</code> and three shards:</p>
<p><img src="/images/2018-03-12-sorting-and-paging-on-distributed-data/shards3.png" alt="" /></p>
<p>In this case, window expansion ensures any element larger than C and smaller than N has to be sent to the coordinator:</p>
<p><img src="/images/2018-03-12-sorting-and-paging-on-distributed-data/shards4.png" alt="" /></p>
<p>Essentially transferring most of the data to the coordinator.</p>
<p>On the other hand, if the shards are mostly randomly distributed, the amount of data shipped is on the order of <code class="language-plaintext highlighter-rouge">nShards*pageSize</code>.</p>
<h1 id="implementation">Implementation</h1>
<p>The following is a simple implementation of the basic concepts of the algorithm:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">static</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Math</span><span class="o">.</span><span class="na">max</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">static</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Math</span><span class="o">.</span><span class="na">min</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.ArrayList</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Comparator</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">DistributedSort</span> <span class="o">{</span>
<span class="cm">/**
* Window bounds
*
* @param <K> key type
*/</span>
<span class="kd">static</span> <span class="kd">class</span> <span class="nc">Bounds</span><span class="o"><</span><span class="no">K</span> <span class="kd">extends</span> <span class="nc">Comparable</span><span class="o"><</span><span class="no">K</span><span class="o">>></span> <span class="o">{</span>
<span class="kd">final</span> <span class="no">K</span> <span class="n">low</span><span class="o">;</span>
<span class="kd">final</span> <span class="no">K</span> <span class="n">high</span><span class="o">;</span>
<span class="nc">Bounds</span><span class="o">(</span><span class="no">K</span> <span class="n">low</span><span class="o">,</span> <span class="no">K</span> <span class="n">high</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">low</span> <span class="o">=</span> <span class="n">low</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">high</span> <span class="o">=</span> <span class="n">high</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="cm">/**
* The resulting local window
*
* @param <K> key type
*/</span>
<span class="kd">static</span> <span class="kd">class</span> <span class="nc">LocalWindow</span><span class="o"><</span><span class="no">K</span> <span class="kd">extends</span> <span class="nc">Comparable</span><span class="o"><</span><span class="no">K</span><span class="o">>></span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">nLeft</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">windowElements</span><span class="o">;</span>
<span class="nc">LocalWindow</span><span class="o">(</span><span class="kt">int</span> <span class="n">nLeft</span><span class="o">,</span> <span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">windowElements</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">nLeft</span> <span class="o">=</span> <span class="n">nLeft</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">windowElements</span> <span class="o">=</span> <span class="n">windowElements</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="cm">/**
* State of a shard for a single sort operation
*
* @param <K> key type
*/</span>
<span class="kd">static</span> <span class="kd">class</span> <span class="nc">Shard</span><span class="o"><</span><span class="no">K</span> <span class="kd">extends</span> <span class="nc">Comparable</span><span class="o"><</span><span class="no">K</span><span class="o">>></span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">data</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">minKeyOff</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">maxKeyOff</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="o">;</span>
<span class="nc">Shard</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">data</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">;</span>
<span class="o">}</span>
<span class="kt">int</span> <span class="nf">sortAndFilter</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// We just sort the shard's data</span>
<span class="n">data</span><span class="o">.</span><span class="na">sort</span><span class="o">(</span><span class="nc">Comparator</span><span class="o">.</span><span class="na">naturalOrder</span><span class="o">());</span>
<span class="k">return</span> <span class="n">data</span><span class="o">.</span><span class="na">size</span><span class="o">();</span>
<span class="o">}</span>
<span class="nc">Bounds</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="nf">getLocalWindowBounds</span><span class="o">(</span><span class="kt">int</span> <span class="n">k</span><span class="o">,</span> <span class="kt">int</span> <span class="n">pageSize</span><span class="o">,</span> <span class="kt">int</span> <span class="n">totalLen</span><span class="o">)</span> <span class="o">{</span>
<span class="n">minKeyOff</span> <span class="o">=</span> <span class="n">max</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="o">(</span><span class="n">k</span> <span class="o">*</span> <span class="n">data</span><span class="o">.</span><span class="na">size</span><span class="o">())</span> <span class="o">/</span> <span class="n">totalLen</span><span class="o">);</span>
<span class="n">maxKeyOff</span> <span class="o">=</span> <span class="n">min</span><span class="o">(</span><span class="n">minKeyOff</span> <span class="o">+</span> <span class="n">pageSize</span><span class="o">,</span> <span class="n">data</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">-</span> <span class="mi">1</span><span class="o">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">Bounds</span><span class="o"><>(</span><span class="n">data</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">minKeyOff</span><span class="o">),</span> <span class="n">data</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">maxKeyOff</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">LocalWindow</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="nf">getResult</span><span class="o">(</span><span class="no">K</span> <span class="n">minKey</span><span class="o">,</span> <span class="no">K</span> <span class="n">maxKey</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Expand local window to the left</span>
<span class="k">while</span> <span class="o">(</span><span class="n">minKeyOff</span> <span class="o">></span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">data</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">minKeyOff</span> <span class="o">-</span> <span class="mi">1</span><span class="o">).</span><span class="na">compareTo</span><span class="o">(</span><span class="n">minKey</span><span class="o">)</span> <span class="o">>=</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="o">--</span><span class="n">minKeyOff</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// Expand local window to the right</span>
<span class="k">while</span> <span class="o">(</span><span class="n">maxKeyOff</span> <span class="o">+</span> <span class="mi">1</span> <span class="o"><</span> <span class="n">data</span><span class="o">.</span><span class="na">size</span><span class="o">()</span> <span class="o">&&</span> <span class="n">data</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">maxKeyOff</span> <span class="o">+</span> <span class="mi">1</span><span class="o">).</span><span class="na">compareTo</span><span class="o">(</span><span class="n">maxKey</span><span class="o">)</span> <span class="o"><=</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="o">++</span><span class="n">maxKeyOff</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">LocalWindow</span><span class="o"><>(</span><span class="n">minKeyOff</span><span class="o">,</span> <span class="n">copyOfRange</span><span class="o">(</span><span class="n">data</span><span class="o">,</span> <span class="n">minKeyOff</span><span class="o">,</span> <span class="n">maxKeyOff</span> <span class="o">+</span> <span class="mi">1</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="cm">/**
* The coordinator's state for a single sort operation
*
* @param <K> key type
*/</span>
<span class="kd">static</span> <span class="kd">class</span> <span class="nc">Coordinator</span><span class="o"><</span><span class="no">K</span> <span class="kd">extends</span> <span class="nc">Comparable</span><span class="o"><</span><span class="no">K</span><span class="o">>></span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">List</span><span class="o"><</span><span class="nc">Shard</span><span class="o"><</span><span class="no">K</span><span class="o">>></span> <span class="n">shards</span><span class="o">;</span>
<span class="nc">Coordinator</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="nc">Shard</span><span class="o"><</span><span class="no">K</span><span class="o">>></span> <span class="n">shards</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">shards</span> <span class="o">=</span> <span class="n">shards</span><span class="o">;</span>
<span class="o">}</span>
<span class="cm">/**
* Sort entry point. This method simulates the communication with the shards.
* It does not introduce parallelism for clarity, calls in each round can be issued
* in parallel, only synchronizing when processing results.
*
* @param k start offset
* @param pageSize desired page size
* @return the sorted page
*/</span>
<span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="nf">sortedPage</span><span class="o">(</span><span class="kt">int</span> <span class="n">k</span><span class="o">,</span> <span class="kt">int</span> <span class="n">pageSize</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Round one, how much data is there, trigger the sort</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">totalLen</span> <span class="o">=</span> <span class="n">shards</span><span class="o">.</span><span class="na">stream</span><span class="o">()</span>
<span class="o">.</span><span class="na">mapToInt</span><span class="o">(</span><span class="nl">Shard:</span><span class="o">:</span><span class="n">sortAndFilter</span><span class="o">)</span>
<span class="o">.</span><span class="na">sum</span><span class="o">();</span>
<span class="c1">// Round two, find global window bounds</span>
<span class="no">K</span> <span class="n">minKey</span> <span class="o">=</span> <span class="kc">null</span><span class="o">,</span> <span class="n">maxKey</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
<span class="k">for</span> <span class="o">(</span><span class="nc">Shard</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">shard</span> <span class="o">:</span> <span class="n">shards</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">Bounds</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">localWindow</span> <span class="o">=</span> <span class="n">shard</span><span class="o">.</span><span class="na">getLocalWindowBounds</span><span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">pageSize</span><span class="o">,</span> <span class="n">totalLen</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">minKey</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">localWindow</span><span class="o">.</span><span class="na">low</span><span class="o">.</span><span class="na">compareTo</span><span class="o">(</span><span class="n">minKey</span><span class="o">)</span> <span class="o"><</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">minKey</span> <span class="o">=</span> <span class="n">localWindow</span><span class="o">.</span><span class="na">low</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">maxKey</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">localWindow</span><span class="o">.</span><span class="na">high</span><span class="o">.</span><span class="na">compareTo</span><span class="o">(</span><span class="n">maxKey</span><span class="o">)</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">maxKey</span> <span class="o">=</span> <span class="n">localWindow</span><span class="o">.</span><span class="na">high</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="c1">// Round three, merge results</span>
<span class="kd">final</span> <span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">merged</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>();</span>
<span class="kt">int</span> <span class="n">nLeftSum</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="k">for</span> <span class="o">(</span><span class="nc">Shard</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">shard</span> <span class="o">:</span> <span class="n">shards</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">LocalWindow</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">localWindow</span> <span class="o">=</span> <span class="n">shard</span><span class="o">.</span><span class="na">getResult</span><span class="o">(</span><span class="n">minKey</span><span class="o">,</span> <span class="n">maxKey</span><span class="o">);</span>
<span class="n">nLeftSum</span> <span class="o">+=</span> <span class="n">localWindow</span><span class="o">.</span><span class="na">nLeft</span><span class="o">;</span>
<span class="c1">// We'll do a simple add and sort for simplicity,</span>
<span class="c1">// a mergeSort style merge would be better</span>
<span class="n">merged</span><span class="o">.</span><span class="na">addAll</span><span class="o">(</span><span class="n">localWindow</span><span class="o">.</span><span class="na">windowElements</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">merged</span><span class="o">.</span><span class="na">sort</span><span class="o">(</span><span class="nc">Comparator</span><span class="o">.</span><span class="na">naturalOrder</span><span class="o">());</span>
<span class="k">return</span> <span class="nf">copyOfRange</span><span class="o">(</span><span class="n">merged</span><span class="o">,</span>
<span class="n">k</span> <span class="o">-</span> <span class="n">nLeftSum</span><span class="o">,</span>
<span class="n">min</span><span class="o">(</span><span class="n">k</span> <span class="o">-</span> <span class="n">nLeftSum</span> <span class="o">+</span> <span class="n">pageSize</span><span class="o">,</span> <span class="n">merged</span><span class="o">.</span><span class="na">size</span><span class="o">()));</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="nf">copyOfRange</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="no">K</span><span class="o">></span> <span class="n">list</span><span class="o">,</span> <span class="kt">int</span> <span class="n">fromInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">toExclusive</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>(</span><span class="n">list</span><span class="o">.</span><span class="na">subList</span><span class="o">(</span><span class="n">fromInclusive</span><span class="o">,</span> <span class="n">toExclusive</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Coordinator</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">c</span><span class="o">;</span>
<span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">page</span><span class="o">;</span>
<span class="c1">// Example 1</span>
<span class="n">c</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Coordinator</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
<span class="k">new</span> <span class="nc">Shard</span><span class="o"><>(</span><span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"A"</span><span class="o">,</span> <span class="s">"B"</span><span class="o">,</span> <span class="s">"C"</span><span class="o">,</span> <span class="s">"D"</span><span class="o">,</span> <span class="s">"E"</span><span class="o">))),</span>
<span class="k">new</span> <span class="nc">Shard</span><span class="o"><>(</span><span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"A"</span><span class="o">,</span> <span class="s">"D"</span><span class="o">,</span> <span class="s">"D"</span><span class="o">,</span> <span class="s">"E"</span><span class="o">,</span> <span class="s">"F"</span><span class="o">)))</span>
<span class="o">));</span>
<span class="n">page</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">sortedPage</span><span class="o">(</span><span class="mi">4</span><span class="o">,</span> <span class="mi">3</span><span class="o">);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Example 1: "</span> <span class="o">+</span> <span class="n">page</span><span class="o">);</span>
<span class="c1">// Example 2</span>
<span class="n">c</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Coordinator</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
<span class="k">new</span> <span class="nc">Shard</span><span class="o"><>(</span><span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"A"</span><span class="o">,</span> <span class="s">"B"</span><span class="o">,</span> <span class="s">"C"</span><span class="o">,</span> <span class="s">"D"</span><span class="o">,</span> <span class="s">"E"</span><span class="o">))),</span>
<span class="k">new</span> <span class="nc">Shard</span><span class="o"><>(</span><span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"F"</span><span class="o">,</span> <span class="s">"G"</span><span class="o">,</span> <span class="s">"H"</span><span class="o">,</span> <span class="s">"I"</span><span class="o">,</span> <span class="s">"J"</span><span class="o">))),</span>
<span class="k">new</span> <span class="nc">Shard</span><span class="o"><>(</span><span class="k">new</span> <span class="nc">ArrayList</span><span class="o"><>(</span><span class="nc">List</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"K"</span><span class="o">,</span> <span class="s">"L"</span><span class="o">,</span> <span class="s">"M"</span><span class="o">,</span> <span class="s">"N"</span><span class="o">,</span> <span class="s">"O"</span><span class="o">)))</span>
<span class="o">));</span>
<span class="n">page</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">sortedPage</span><span class="o">(</span><span class="mi">6</span><span class="o">,</span> <span class="mi">2</span><span class="o">);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Example 2: "</span> <span class="o">+</span> <span class="n">page</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>NOTE: This is a revised version of an article I wrote for Medallia’s engineering blogParallelism & Performance: Why O-notation matters2011-09-07T21:28:00+00:002011-09-07T21:28:00+00:00https://juancn.github.io/post/2011/09/07/parallelism-performance-why-o-notation<blockquote>
<p><strong>UPDATE:</strong> There was a bug in the way the suffix map was built in the original post that <a href="http://www.reddit.com/user/lolplz">lolplz</a> found. I updated this post with that bug fixed, however, the content of the post has not changed significantly.</p>
</blockquote>
<p>Aleksandar Prokopec presented a <a href="http://days2011.scala-lang.org/node/138/272">very interesting talk about Scala Parallel Collections</a> in <a href="http://days2011.scala-lang.org/">Scala Days 2011</a>. I saw it today and gave me some food for thought,</p>
<p>He opens his talk with an intriguing code snippet:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="o">{</span>
<span class="n">s</span> <span class="k"><-</span> <span class="n">surnames</span>
<span class="n">n</span> <span class="k"><-</span> <span class="n">names</span>
<span class="k">if</span> <span class="n">s</span> <span class="n">endsWith</span> <span class="n">n</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
</code></pre></div></div>
<p>What this code does, given what presumably are two sequences of strings, it builds all pairs (name, surname) where the surname ends with the name.
The algorithm used is very brute force, it’s order is roughly O(N^2) (it’s basically a <a href="http://en.wikipedia.org/wiki/Cartesian_product">Cartesian Product</a> being filtered). He then goes on to show that by just using parallel collections:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="o">{</span>
<span class="n">s</span> <span class="k"><-</span> <span class="nv">surnames</span><span class="o">.</span><span class="py">par</span>
<span class="n">n</span> <span class="k"><-</span> <span class="nv">names</span><span class="o">.</span><span class="py">par</span>
<span class="k">if</span> <span class="n">s</span> <span class="n">endsWith</span> <span class="n">n</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
</code></pre></div></div>
<p>He can leverage all the cores and reduce the runtime by two or four, depending on the number of cores available in the machine where it’s running. For example, the non-parallel version runs in <strong>1040ms</strong>, the parallel one runs in <strong>575ms</strong> with two cores, and in <strong>305ms</strong> with four. Which is indeed very impressive for such a minor change.</p>
<p>What concerns me is that, no matter how many cores you add, the problem as presented is still O(N^2). It is true that many useful problems can only be speeded up by throwing more hardware at it, but most of the times, using the right data representation can yield even bigger gains.</p>
<p>If we use this problem as an example, we can build a slightly more complex implementation, but that hopefully is a lot faster. The approach I’m taking is that of building a suffix map for surnames. <a href="http://en.wikipedia.org/wiki/Suffix_tree">There are more efficient data structures (memory wise)</a> to do this, but for simplicity I’ll use a Map:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="nv">suffixMap</span> <span class="k">=</span> <span class="nv">collection</span><span class="o">.</span><span class="py">mutable</span><span class="o">.</span><span class="py">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]().</span><span class="py">withDefaultValue</span><span class="o">(</span><span class="nc">Nil</span><span class="o">)</span>
<span class="nf">for</span> <span class="o">(</span><span class="n">s</span> <span class="k"><-</span> <span class="n">surnames</span><span class="o">;</span> <span class="n">i</span> <span class="k"><-</span> <span class="mi">0</span> <span class="n">until</span> <span class="nv">s</span><span class="o">.</span><span class="py">length</span><span class="o">;</span> <span class="n">suffix</span> <span class="k">=</span> <span class="nv">s</span><span class="o">.</span><span class="py">substring</span><span class="o">(</span><span class="n">i</span><span class="o">))</span>
<span class="nf">suffixMap</span><span class="o">(</span><span class="n">suffix</span><span class="o">)</span> <span class="k">=</span> <span class="n">s</span> <span class="o">::</span> <span class="nf">suffixMap</span><span class="o">(</span><span class="n">suffix</span><span class="o">)</span>
</code></pre></div></div>
<p>Having built the prefix map, we can naively rewrite the loop to use it (instead of the Cartesian Product):</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="o">{</span>
<span class="n">n</span> <span class="k"><-</span> <span class="n">names</span>
<span class="n">s</span> <span class="k"><-</span> <span class="nf">suffixMap</span><span class="o">(</span><span class="n">n</span><span class="o">)</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
</code></pre></div></div>
<p>In theory, this loop is roughly O(N) (assuming the map is a HashMap), since it now does a constant number of operations on each name it processes, rather than processing all the names for each surname (I’m ignoring the fact that the map returns lists).</p>
<blockquote>
<p><strong>Note:</strong> The algorithmic order does not change much if we take into account the suffix map construction. Let’s assume that we have S surnames and N names, the order of building the suffix map is O(S) and the order building the pairs is O(N), the total order of the algorithm is O(N+S). If we assume that N is approximately equal to S, then the order is O(N+N) which is the same than O(2N), which can be simplified to O(N).</p>
</blockquote>
<p>So, let’s see if this holds up in real life. For this I wrote the following scala script that runs a few passes of several different implementations:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">#!/</span><span class="n">bin</span><span class="o">/</span><span class="n">sh</span>
<span class="n">exec</span> <span class="n">scala</span> <span class="o">-</span><span class="n">deprecation</span> <span class="o">-</span><span class="n">savecompiled</span> <span class="s">"$0"</span> <span class="s">"$@"</span>
<span class="o">!#</span>
<span class="k">def</span> <span class="nf">benchmark</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)(</span><span class="n">body</span><span class="k">:</span> <span class="o">=></span><span class="n">T</span><span class="o">)</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">start</span> <span class="k">=</span> <span class="nv">System</span><span class="o">.</span><span class="py">currentTimeMillis</span><span class="o">()</span>
<span class="k">val</span> <span class="nv">result</span> <span class="k">=</span> <span class="n">body</span>
<span class="k">val</span> <span class="nv">end</span> <span class="k">=</span> <span class="nv">System</span><span class="o">.</span><span class="py">currentTimeMillis</span><span class="o">()</span>
<span class="nf">println</span><span class="o">(</span><span class="n">name</span> <span class="o">+</span> <span class="s">": "</span> <span class="o">+</span> <span class="o">(</span><span class="n">end</span> <span class="o">-</span> <span class="n">start</span><span class="o">))</span>
<span class="n">result</span>
<span class="o">}</span>
<span class="k">val</span> <span class="nv">surnames</span> <span class="k">=</span> <span class="o">(</span><span class="mi">1</span> <span class="n">to</span> <span class="mi">10000</span><span class="o">).</span><span class="py">map</span><span class="o">(</span><span class="s">"Name"</span> <span class="o">+</span> <span class="k">_</span><span class="o">)</span>
<span class="k">val</span> <span class="nv">names</span> <span class="k">=</span> <span class="o">(</span><span class="mi">1</span> <span class="n">to</span> <span class="mi">10000</span><span class="o">).</span><span class="py">map</span><span class="o">(</span><span class="s">"Name"</span> <span class="o">+</span> <span class="k">_</span><span class="o">)</span>
<span class="k">val</span> <span class="nv">suffixMap</span> <span class="k">=</span> <span class="nv">collection</span><span class="o">.</span><span class="py">mutable</span><span class="o">.</span><span class="py">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]().</span><span class="py">withDefaultValue</span><span class="o">(</span><span class="nc">Nil</span><span class="o">)</span>
<span class="nf">for</span> <span class="o">(</span><span class="n">s</span> <span class="k"><-</span> <span class="n">surnames</span><span class="o">;</span> <span class="n">i</span> <span class="k"><-</span> <span class="mi">0</span> <span class="n">until</span> <span class="nv">s</span><span class="o">.</span><span class="py">length</span><span class="o">;</span> <span class="n">suffix</span> <span class="k">=</span> <span class="nv">s</span><span class="o">.</span><span class="py">substring</span><span class="o">(</span><span class="n">i</span><span class="o">))</span>
<span class="nf">suffixMap</span><span class="o">(</span><span class="n">suffix</span><span class="o">)</span> <span class="k">=</span> <span class="n">s</span> <span class="o">::</span> <span class="nf">suffixMap</span><span class="o">(</span><span class="n">suffix</span><span class="o">)</span>
<span class="nf">for</span><span class="o">(</span> <span class="n">i</span> <span class="k"><-</span> <span class="mi">1</span> <span class="n">to</span> <span class="mi">5</span> <span class="o">)</span> <span class="o">{</span>
<span class="nf">println</span><span class="o">(</span><span class="s">"Run #"</span> <span class="o">+</span> <span class="n">i</span><span class="o">)</span>
<span class="nf">benchmark</span><span class="o">(</span><span class="s">"Brute force"</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">s</span> <span class="k"><-</span> <span class="n">surnames</span>
<span class="n">n</span> <span class="k"><-</span> <span class="n">names</span>
<span class="k">if</span> <span class="n">s</span> <span class="n">endsWith</span> <span class="n">n</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">benchmark</span><span class="o">(</span><span class="s">"Parallel"</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">s</span> <span class="k"><-</span> <span class="nv">surnames</span><span class="o">.</span><span class="py">par</span>
<span class="n">n</span> <span class="k"><-</span> <span class="nv">names</span><span class="o">.</span><span class="py">par</span>
<span class="k">if</span> <span class="n">s</span> <span class="n">endsWith</span> <span class="n">n</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">benchmark</span><span class="o">(</span><span class="s">"Smart"</span><span class="o">)</span> <span class="o">{</span>
<span class="k">val</span> <span class="nv">suffixMap</span> <span class="k">=</span> <span class="nv">collection</span><span class="o">.</span><span class="py">mutable</span><span class="o">.</span><span class="py">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]().</span><span class="py">withDefaultValue</span><span class="o">(</span><span class="nc">Nil</span><span class="o">)</span>
<span class="nf">for</span> <span class="o">(</span><span class="n">s</span> <span class="k"><-</span> <span class="n">surnames</span><span class="o">;</span> <span class="n">i</span> <span class="k"><-</span> <span class="mi">0</span> <span class="n">until</span> <span class="nv">s</span><span class="o">.</span><span class="py">length</span><span class="o">;</span> <span class="n">suffix</span> <span class="k">=</span> <span class="nv">s</span><span class="o">.</span><span class="py">substring</span><span class="o">(</span><span class="n">i</span><span class="o">))</span>
<span class="nf">suffixMap</span><span class="o">(</span><span class="n">suffix</span><span class="o">)</span> <span class="k">=</span> <span class="n">s</span> <span class="o">::</span> <span class="nf">suffixMap</span><span class="o">(</span><span class="n">suffix</span><span class="o">)</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">n</span> <span class="k"><-</span> <span class="n">names</span>
<span class="n">s</span> <span class="k"><-</span> <span class="nf">suffixMap</span><span class="o">(</span><span class="n">n</span><span class="o">)</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">}</span>
<span class="nf">benchmark</span><span class="o">(</span><span class="s">"Smart (amortized)"</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">n</span> <span class="k"><-</span> <span class="n">names</span>
<span class="n">s</span> <span class="k"><-</span> <span class="nf">suffixMap</span><span class="o">(</span><span class="n">n</span><span class="o">)</span>
<span class="o">}</span> <span class="nf">yield</span> <span class="o">(</span><span class="n">n</span> <span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>There are four implementations:</p>
<ul>
<li>Brute Force: the original implementation</li>
<li>Parallel: same as before, but using parallel collections.</li>
<li>Smart: Using the prefix map (and measuring the map construction)</li>
<li>Smart (amortized): same as before, but with the prefix map cost amortized.</li>
</ul>
<h1 id="benchmark-results">Benchmark Results</h1>
<p>Running this script in a four core machine, I get the following results:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Run #1
Brute force: 2158
Parallel: 1355
Smart: 153
Smart (amortized): 27
Run #2
Brute force: 1985
Parallel: 899
Smart: 82
Smart (amortized): 7
Run #3
Brute force: 1947
Parallel: 716
Smart: 69
Smart (amortized): 5
Run #4
Brute force: 1932
Parallel: 714
Smart: 67
Smart (amortized): 6
Run #5
Brute force: 1933
Parallel: 713
Smart: 68
Smart (amortized): 5
</code></pre></div></div>
<p>As expected, the parallel version runs 3.5 times as fast as the naive one, but the implementation using the “Smart” approach runs more than <strong>30 times faster</strong> than the naive one. If we were able to amortize the cost of building the suffix map, the speed-up is even more staggering, at a whooping 380 times faster (although this is not always possible)!</p>
<p>What we can conclude from this, paraphrasing Fred Brooks<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, is that regarding performance there is <a href="http://en.wikipedia.org/wiki/No_Silver_Bullet">no silver bullet</a>. The basics matter, maybe now they even matter more than ever.</p>
<p>Analyzing algorithmic order, in its most basic form (making huge approximations) is a very practical tool to solving hard performance problems. Still, the simple approach used here to optimize the problem is also parallelizable, which for a large enough problem, it might gain some speedup using parallel collections.</p>
<p>Don’t get me wrong, I love Scala parallel collections and parallel algorithms are increasingly important, but they are by no means a magical solution that can be used for any problem (and I think Aleksandar is with me in this one).</p>
<h1 id="footnotes">Footnotes</h1>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>misquoting is probably more accurate. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>UPDATE: There was a bug in the way the suffix map was built in the original post that lolplz found. I updated this post with that bug fixed, however, the content of the post has not changed significantly.File locks in bash2011-05-17T21:06:00+00:002011-05-17T21:06:00+00:00https://juancn.github.io/post/2011/05/17/file-locks-in-bash<p>For quite a while I’ve been looking for a portable utility that mimics <a href="http://www.linuxmanpages.com/man1/lockfile.1.php">Procmail’s “lockfile” command</a>. I didn’t need all the functionality, just for it to lock a single file and support a retry limit and sleep parameters.
I finally implemented one using Bash’s “noclobber” option. I don’t know if it will work correctly on NFS, but it should work fine on most filesystems. Hopefully it will be useful to some of you.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nb">set</span> <span class="nt">-e</span>
<span class="nb">declare </span><span class="nv">SCRIPT_NAME</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">basename</span> <span class="nv">$0</span><span class="si">)</span><span class="s2">"</span>
<span class="k">function </span>usage <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"Usage: </span><span class="nv">$SCRIPT_NAME</span><span class="s2"> [options] <lock file>"</span>
<span class="nb">echo</span> <span class="s2">"Options"</span>
<span class="nb">echo</span> <span class="s2">" -r, --retries"</span>
<span class="nb">echo</span> <span class="s2">" limit the number of retries before giving up the lock."</span>
<span class="nb">echo</span> <span class="s2">" -s, --sleeptime, -<seconds>"</span>
<span class="nb">echo</span> <span class="s2">" number of seconds between each attempt. Defaults to 8 seconds."</span>
<span class="nb">exit </span>1
<span class="o">}</span>
<span class="c">#Check that at least one argument is provided</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-lt</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then </span>usage<span class="p">;</span> <span class="k">fi
</span><span class="nb">declare </span><span class="nv">RETRIES</span><span class="o">=</span><span class="nt">-1</span>
<span class="nb">declare </span><span class="nv">SLEEPTIME</span><span class="o">=</span>8 <span class="c">#in seconds</span>
<span class="c">#Parse options</span>
<span class="k">for </span>arg<span class="p">;</span> <span class="k">do
case</span> <span class="s2">"</span><span class="nv">$arg</span><span class="s2">"</span> <span class="k">in</span>
<span class="nt">-r</span><span class="p">|</span><span class="nt">--retries</span><span class="p">)</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-lt</span> 2 <span class="o">]</span><span class="p">;</span> <span class="k">then </span>usage<span class="p">;</span> <span class="k">fi</span><span class="p">;</span>
<span class="nv">RETRIES</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="p">;</span> <span class="nb">shift
echo</span> <span class="s2">"</span><span class="nv">$RETRIES</span><span class="s2">"</span> | egrep <span class="nt">-q</span> <span class="s1">'^-?[0-9]+$'</span> <span class="o">||</span> usage <span class="c">#check that it's a number</span>
<span class="p">;;</span>
<span class="nt">-s</span><span class="p">|</span><span class="nt">--sleeptime</span><span class="p">)</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-lt</span> 2 <span class="o">]</span><span class="p">;</span> <span class="k">then </span>usage<span class="p">;</span> <span class="k">fi</span><span class="p">;</span>
<span class="nv">SLEEPTIME</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span><span class="p">;</span> <span class="nb">shift
echo</span> <span class="s2">"</span><span class="nv">$SLEEPTIME</span><span class="s2">"</span> | egrep <span class="nt">-q</span> <span class="s1">'^[0-9]+$'</span> <span class="o">||</span> usage <span class="c">#check that it's a number</span>
<span class="p">;;</span>
<span class="nt">--</span><span class="p">)</span> <span class="nb">shift</span> <span class="p">;</span> <span class="nb">break</span> <span class="p">;;</span>
-[[:digit:]]<span class="k">*</span><span class="p">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-lt</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then </span>usage<span class="p">;</span> <span class="k">fi</span><span class="p">;</span>
<span class="nv">SLEEPTIME</span><span class="o">=</span><span class="k">${</span><span class="nv">1</span>:1<span class="k">}</span><span class="p">;</span> <span class="nb">shift
echo</span> <span class="s2">"</span><span class="nv">$SLEEPTIME</span><span class="s2">"</span> | egrep <span class="nt">-q</span> <span class="s1">'^[0-9]+$'</span> <span class="o">||</span> usage <span class="c">#check that it's a number</span>
<span class="p">;;</span>
<span class="nt">--</span><span class="k">*</span><span class="p">)</span> usage<span class="p">;;</span> <span class="c">#fail on other options</span>
<span class="k">esac</span>
<span class="k">done</span>
<span class="c">#Check that only one argument is left</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$# </span><span class="nt">-ne</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then </span>usage<span class="p">;</span> <span class="k">fi
</span><span class="nb">declare </span><span class="nv">lockfile</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="k">for</span> <span class="o">((</span> <span class="nv">i</span><span class="o">=</span>0<span class="p">;</span> <span class="nv">$RETRIES</span> < 0 <span class="o">||</span> i < <span class="nv">$RETRIES</span><span class="p">;</span> i++ <span class="o">))</span><span class="p">;</span> <span class="k">do
if</span> <span class="o">(</span> <span class="nb">set</span> <span class="nt">-o</span> noclobber<span class="p">;</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$$</span><span class="s2">"</span> <span class="o">></span> <span class="s2">"</span><span class="nv">$lockfile</span><span class="s2">"</span><span class="o">)</span> 2> /dev/null<span class="p">;</span>
<span class="k">then
</span><span class="nb">exit </span>0
<span class="k">fi</span>
<span class="c">#Wait a bit</span>
<span class="nb">sleep</span> <span class="nv">$SLEEPTIME</span>
<span class="k">done</span>
<span class="c">#Failed</span>
<span class="nb">cat</span> <span class="nv">$lockfile</span>
<span class="nb">exit </span>1
</code></pre></div></div>For quite a while I’ve been looking for a portable utility that mimics Procmail’s “lockfile” command. I didn’t need all the functionality, just for it to lock a single file and support a retry limit and sleep parameters. I finally implemented one using Bash’s “noclobber” option. I don’t know if it will work correctly on NFS, but it should work fine on most filesystems. Hopefully it will be useful to some of you.Content-aware image resizing2011-04-14T20:20:00+00:002011-04-14T20:20:00+00:00https://juancn.github.io/post/2011/04/14/content-aware-image-resizing<p>Today I’m going to discuss a technique called <a href="http://www.faculty.idc.ac.il/arik/SCWeb/imret/index.html">Seam Carving</a>, originally presented in Siggraph 2007. This algorithm at it’s core it’s fairly simple but produces impressive results.</p>
<p>We will start from this image:</p>
<p><img src="/images/2011-04-14-content-aware-image-resizing/seam1.jpg" alt="Photo of a bridge and lake" /></p>
<p>And take 200 pixels from its width, and turn it into this one:</p>
<p><img src="/images/2011-04-14-content-aware-image-resizing/seam1-out.png" alt="Photo of a bridge and lake reduced" /></p>
<p>Note that the image wasn’t just resized, but most of the detail is still there. The size reduction is rather aggressive so there are some artifacts. But the results are quite good.</p>
<p>This algorithm works by repeatedly finding vertical seams of pixels and removing them. It chooses which one to remove by finding the seam with the minimal amount of energy.</p>
<p>The whole algorithm revolves around an energy function. In this case, I’m using a function suggested in the original paper which is based on the luminance of the image. What we do is compute the vertical and horizontal derivatives of the image, take the absolute value of each, and add both. The derivative is approximated by a simple subtraction.</p>
<p>The following code computes the energy of the image. The intensities image is basically the grayscale version of the image, normalized between 0 and 1.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kd">static</span> <span class="nc">FloatImage</span> <span class="nf">computeEnergy</span><span class="o">(</span><span class="nc">FloatImage</span> <span class="n">intensities</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">w</span> <span class="o">=</span> <span class="n">intensities</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">h</span> <span class="o">=</span> <span class="n">intensities</span><span class="o">.</span><span class="na">getHeight</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">energy</span> <span class="o">=</span> <span class="nc">FloatImage</span><span class="o">.</span><span class="na">createSameSize</span><span class="o">(</span><span class="n">intensities</span><span class="o">);</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="c1">//I'm aproximating the derivatives by subtraction</span>
<span class="kt">float</span> <span class="n">e</span> <span class="o">=</span> <span class="n">abs</span><span class="o">(</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">)-</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o">,</span><span class="n">y</span><span class="o">))</span>
<span class="o">+</span> <span class="n">abs</span><span class="o">(</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">)-</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="o">));</span>
<span class="n">energy</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">energy</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>After applying this function to our image, we get the following:</p>
<p><img src="/images/2011-04-14-content-aware-image-resizing/seam1-energy.png" alt="Energy map of the original photo" /></p>
<p>You can observe that the edges are highlighted (i.e. have more energy). That is caused by our choice of an energy function. Since we’re taking the derivatives and adding its absolute value, abrupt changes in luminance are highlighted (i.e. edges).
The next step is where things start to get interesting. To find the minimal energy seam, we build an image with the accumulated minimal energy. We do so by computing an image where the value of each pixel is the value of the minimum of the three above it, plus the energy of that pixel:</p>
<p><img src="/images/2011-04-14-content-aware-image-resizing/seam minimal.png" alt="Diagram showing how three pixels in the row above are combined, see following code for details" /></p>
<p>We do so with the following code:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">energy</span> <span class="o">=</span> <span class="n">computeEnergy</span><span class="o">(</span><span class="n">intensities</span><span class="o">);</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">minima</span> <span class="o">=</span> <span class="nc">FloatImage</span><span class="o">.</span><span class="na">createSameSize</span><span class="o">(</span><span class="n">energy</span><span class="o">);</span>
<span class="c1">//First row is equal to the energy</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="mi">0</span><span class="o">,</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="mi">0</span><span class="o">));</span>
<span class="o">}</span>
<span class="c1">//I assume that the rightmost pixel column in the energy image is garbage</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">h</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span><span class="n">y</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">(</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">),</span>
<span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)));</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">float</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">(</span><span class="n">min</span><span class="o">(</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span> <span class="o">-</span> <span class="mi">1</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">),</span>
<span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)),</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">));</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">sum</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span><span class="n">y</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">(</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">),</span>
<span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">3</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)));</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Once we do this, the last row contains the sum of all the potential minimal seams.</p>
<p><img src="/images/2011-04-14-content-aware-image-resizing/seam1-minima.png" alt="Graphic showing the seam minimas in grayscale" /></p>
<p>With this, we search the last row for the one with the minimum total value:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//We find the minimum seam</span>
<span class="kt">float</span> <span class="n">minSum</span> <span class="o">=</span> <span class="nc">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">seamTip</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="o">;</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">float</span> <span class="n">v</span> <span class="o">=</span> <span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="k">if</span><span class="o">(</span><span class="n">v</span> <span class="o"><</span> <span class="n">minSum</span><span class="o">)</span> <span class="o">{</span>
<span class="n">minSum</span><span class="o">=</span><span class="n">v</span><span class="o">;</span>
<span class="n">seamTip</span><span class="o">=</span><span class="n">x</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And backtrace the seam:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//Backtrace the seam</span>
<span class="kd">final</span> <span class="kt">int</span><span class="o">[]</span> <span class="n">seam</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">int</span><span class="o">[</span><span class="n">h</span><span class="o">];</span>
<span class="n">seam</span><span class="o">[</span><span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">]=</span><span class="n">seamTip</span><span class="o">;</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="n">seamTip</span><span class="o">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">y</span> <span class="o">></span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span><span class="o">--)</span> <span class="o">{</span>
<span class="kt">float</span> <span class="n">left</span> <span class="o">=</span> <span class="n">x</span><span class="o">></span><span class="mi">0</span><span class="o">?</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">):</span><span class="nc">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">up</span> <span class="o">=</span> <span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="kt">float</span> <span class="n">right</span> <span class="o">=</span> <span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o"><</span><span class="n">w</span><span class="o">?</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">):</span><span class="nc">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">left</span> <span class="o"><</span> <span class="n">up</span> <span class="o">&&</span> <span class="n">left</span> <span class="o"><</span> <span class="n">right</span><span class="o">)</span> <span class="n">x</span><span class="o">=</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span>
<span class="k">else</span> <span class="nf">if</span><span class="o">(</span><span class="n">right</span> <span class="o"><</span> <span class="n">up</span> <span class="o">&&</span> <span class="n">right</span> <span class="o"><</span> <span class="n">left</span><span class="o">)</span> <span class="n">x</span><span class="o">=</span> <span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o">;</span>
<span class="n">seam</span><span class="o">[</span><span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">]=</span><span class="n">x</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Having the minimum energy seam, all is left to do is remove it.</p>
<p>If we repeat this process several times, removing one seam at a time, we end up with a smaller image. Check the following video to see this algorithm in action:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/Elpdgxi1Ytw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p>If you want to reduce an image vertically, you have to find horizontal seams. If you want to do it vertically and horizontally you have to find which seam has the least energy (either the vertical or the horizontal one) and remove that one.
This implementation is quick & dirty and very simplistic. Many optimization can be done to make it work faster. It is also quite incomplete. By priming the energy image, you can influence the algorithm to avoid distorting certain objects in the image or to particularly pick one.
It is also possible to use it to enlarge an image (although I haven’t implemented it), and by a combination of both methods one can selectively remove objects from an image.
The full source code for this demo follows. Have fun!</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">javax.imageio.ImageIO</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.io.File</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.io.IOException</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.awt.image.BufferedImage</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.awt.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">static</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Math</span><span class="o">.</span><span class="na">abs</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">static</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Math</span><span class="o">.</span><span class="na">min</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">SeamCarving</span>
<span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">IOException</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">input</span> <span class="o">=</span> <span class="nc">ImageIO</span><span class="o">.</span><span class="na">read</span><span class="o">(</span><span class="k">new</span> <span class="nc">File</span><span class="o">(</span><span class="n">args</span><span class="o">[</span><span class="mi">0</span><span class="o">]));</span>
<span class="kd">final</span> <span class="nc">BufferedImage</span><span class="o">[]</span> <span class="n">toPaint</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedImage</span><span class="o">[]{</span><span class="n">input</span><span class="o">};</span>
<span class="kd">final</span> <span class="nc">Frame</span> <span class="n">frame</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Frame</span><span class="o">(</span><span class="s">"Seams"</span><span class="o">)</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">update</span><span class="o">(</span><span class="nc">Graphics</span> <span class="n">g</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">im</span> <span class="o">=</span> <span class="n">toPaint</span><span class="o">[</span><span class="mi">0</span><span class="o">];</span>
<span class="k">if</span> <span class="o">(</span><span class="n">im</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="n">g</span><span class="o">.</span><span class="na">clearRect</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span><span class="mi">0</span><span class="o">,</span><span class="n">getWidth</span><span class="o">(),</span> <span class="n">getHeight</span><span class="o">());</span>
<span class="n">g</span><span class="o">.</span><span class="na">drawImage</span><span class="o">(</span><span class="n">im</span><span class="o">,</span><span class="mi">0</span><span class="o">,</span><span class="mi">0</span><span class="o">,</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="n">frame</span><span class="o">.</span><span class="na">setSize</span><span class="o">(</span><span class="n">input</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">input</span><span class="o">.</span><span class="na">getHeight</span><span class="o">());</span>
<span class="n">frame</span><span class="o">.</span><span class="na">setVisible</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
<span class="nc">BufferedImage</span> <span class="n">out</span> <span class="o">=</span> <span class="n">input</span><span class="o">;</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">200</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">deleteVerticalSeam</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
<span class="n">toPaint</span><span class="o">[</span><span class="mi">0</span><span class="o">]=</span><span class="n">out</span><span class="o">;</span>
<span class="n">frame</span><span class="o">.</span><span class="na">repaint</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">BufferedImage</span> <span class="nf">deleteVerticalSeam</span><span class="o">(</span><span class="nc">BufferedImage</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">deleteVerticalSeam</span><span class="o">(</span><span class="n">input</span><span class="o">,</span> <span class="n">findVerticalSeam</span><span class="o">(</span><span class="n">input</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">BufferedImage</span> <span class="nf">deleteVerticalSeam</span><span class="o">(</span><span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">input</span><span class="o">,</span> <span class="kd">final</span> <span class="kt">int</span><span class="o">[]</span> <span class="n">seam</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">w</span> <span class="o">=</span> <span class="n">input</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">h</span> <span class="o">=</span> <span class="n">input</span><span class="o">.</span><span class="na">getHeight</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">out</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedImage</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span><span class="n">h</span><span class="o">,</span> <span class="nc">BufferedImage</span><span class="o">.</span><span class="na">TYPE_INT_ARGB</span><span class="o">);</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">h</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">seam</span><span class="o">[</span><span class="n">y</span><span class="o">];</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">out</span><span class="o">.</span><span class="na">setRGB</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span><span class="n">input</span><span class="o">.</span><span class="na">getRGB</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">));</span>
<span class="o">}</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="n">seam</span><span class="o">[</span><span class="n">y</span><span class="o">]+</span><span class="mi">1</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">out</span><span class="o">.</span><span class="na">setRGB</span><span class="o">(</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span><span class="n">y</span><span class="o">,</span><span class="n">input</span><span class="o">.</span><span class="na">getRGB</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">out</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">int</span><span class="o">[]</span> <span class="nf">findVerticalSeam</span><span class="o">(</span><span class="nc">BufferedImage</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">w</span> <span class="o">=</span> <span class="n">input</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">h</span> <span class="o">=</span> <span class="n">input</span><span class="o">.</span><span class="na">getHeight</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">intensities</span> <span class="o">=</span> <span class="nc">FloatImage</span><span class="o">.</span><span class="na">fromBufferedImage</span><span class="o">(</span><span class="n">input</span><span class="o">);</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">energy</span> <span class="o">=</span> <span class="n">computeEnergy</span><span class="o">(</span><span class="n">intensities</span><span class="o">);</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">minima</span> <span class="o">=</span> <span class="nc">FloatImage</span><span class="o">.</span><span class="na">createSameSize</span><span class="o">(</span><span class="n">energy</span><span class="o">);</span>
<span class="c1">//First row is equal to the energy</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="mi">0</span><span class="o">,</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="mi">0</span><span class="o">));</span>
<span class="o">}</span>
<span class="c1">//I assume that the rightmost pixel column in the energy image is garbage</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">h</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span><span class="n">y</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">(</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">),</span>
<span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)));</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">float</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">(</span><span class="n">min</span><span class="o">(</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span> <span class="o">-</span> <span class="mi">1</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">),</span>
<span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)),</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">));</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">sum</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">minima</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">energy</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span><span class="n">y</span><span class="o">)</span> <span class="o">+</span> <span class="n">min</span><span class="o">(</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">2</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">),</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">w</span><span class="o">-</span><span class="mi">3</span><span class="o">,</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)));</span>
<span class="o">}</span>
<span class="c1">//We find the minimum seam</span>
<span class="kt">float</span> <span class="n">minSum</span> <span class="o">=</span> <span class="nc">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">seamTip</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="o">;</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">float</span> <span class="n">v</span> <span class="o">=</span> <span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="k">if</span><span class="o">(</span><span class="n">v</span> <span class="o"><</span> <span class="n">minSum</span><span class="o">)</span> <span class="o">{</span>
<span class="n">minSum</span><span class="o">=</span><span class="n">v</span><span class="o">;</span>
<span class="n">seamTip</span><span class="o">=</span><span class="n">x</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="c1">//Backtrace the seam</span>
<span class="kd">final</span> <span class="kt">int</span><span class="o">[]</span> <span class="n">seam</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">int</span><span class="o">[</span><span class="n">h</span><span class="o">];</span>
<span class="n">seam</span><span class="o">[</span><span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">]=</span><span class="n">seamTip</span><span class="o">;</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="n">seamTip</span><span class="o">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">y</span> <span class="o">></span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span><span class="o">--)</span> <span class="o">{</span>
<span class="kt">float</span> <span class="n">left</span> <span class="o">=</span> <span class="n">x</span><span class="o">></span><span class="mi">0</span><span class="o">?</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">):</span><span class="nc">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="kt">float</span> <span class="n">up</span> <span class="o">=</span> <span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="kt">float</span> <span class="n">right</span> <span class="o">=</span> <span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o"><</span><span class="n">w</span><span class="o">?</span><span class="n">minima</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">):</span><span class="nc">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">left</span> <span class="o"><</span> <span class="n">up</span> <span class="o">&&</span> <span class="n">left</span> <span class="o"><</span> <span class="n">right</span><span class="o">)</span> <span class="n">x</span><span class="o">=</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span>
<span class="k">else</span> <span class="nf">if</span><span class="o">(</span><span class="n">right</span> <span class="o"><</span> <span class="n">up</span> <span class="o">&&</span> <span class="n">right</span> <span class="o"><</span> <span class="n">left</span><span class="o">)</span> <span class="n">x</span><span class="o">=</span> <span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o">;</span>
<span class="n">seam</span><span class="o">[</span><span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="o">]=</span><span class="n">x</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">seam</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">FloatImage</span> <span class="nf">computeEnergy</span><span class="o">(</span><span class="nc">FloatImage</span> <span class="n">intensities</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">w</span> <span class="o">=</span> <span class="n">intensities</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">h</span> <span class="o">=</span> <span class="n">intensities</span><span class="o">.</span><span class="na">getHeight</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">energy</span> <span class="o">=</span> <span class="nc">FloatImage</span><span class="o">.</span><span class="na">createSameSize</span><span class="o">(</span><span class="n">intensities</span><span class="o">);</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">w</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="c1">//I'm approximating the derivatives by subtraction</span>
<span class="kt">float</span> <span class="n">e</span> <span class="o">=</span> <span class="n">abs</span><span class="o">(</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">)-</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="o">,</span><span class="n">y</span><span class="o">))</span>
<span class="o">+</span> <span class="n">abs</span><span class="o">(</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">)-</span><span class="n">intensities</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="o">));</span>
<span class="n">energy</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">energy</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kn">import</span> <span class="nn">java.awt.image.BufferedImage</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">FloatImage</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">width</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">height</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">float</span><span class="o">[]</span> <span class="n">data</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">FloatImage</span><span class="o">(</span><span class="kt">int</span> <span class="n">width</span><span class="o">,</span> <span class="kt">int</span> <span class="n">height</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">width</span> <span class="o">=</span> <span class="n">width</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">height</span> <span class="o">=</span> <span class="n">height</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">data</span> <span class="o">=</span> <span class="k">new</span> <span class="kt">float</span><span class="o">[</span><span class="n">width</span><span class="o">*</span><span class="n">height</span><span class="o">];</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="nf">getWidth</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">width</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="nf">getHeight</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">height</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">float</span> <span class="nf">get</span><span class="o">(</span><span class="kd">final</span> <span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">y</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span><span class="o">(</span><span class="n">x</span> <span class="o"><</span> <span class="mi">0</span> <span class="o">||</span> <span class="n">x</span> <span class="o">>=</span> <span class="n">width</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"x: "</span> <span class="o">+</span> <span class="n">x</span><span class="o">);</span>
<span class="k">if</span><span class="o">(</span><span class="n">y</span> <span class="o"><</span> <span class="mi">0</span> <span class="o">||</span> <span class="n">y</span> <span class="o">>=</span> <span class="n">height</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"y: "</span> <span class="o">+</span> <span class="n">y</span><span class="o">);</span>
<span class="k">return</span> <span class="n">data</span><span class="o">[</span><span class="n">x</span><span class="o">+</span><span class="n">y</span><span class="o">*</span><span class="n">width</span><span class="o">];</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">set</span><span class="o">(</span><span class="kd">final</span> <span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">y</span><span class="o">,</span> <span class="kt">float</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span><span class="o">(</span><span class="n">x</span> <span class="o"><</span> <span class="mi">0</span> <span class="o">||</span> <span class="n">x</span> <span class="o">>=</span> <span class="n">width</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"x: "</span> <span class="o">+</span> <span class="n">x</span><span class="o">);</span>
<span class="k">if</span><span class="o">(</span><span class="n">y</span> <span class="o"><</span> <span class="mi">0</span> <span class="o">||</span> <span class="n">y</span> <span class="o">>=</span> <span class="n">height</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nc">IllegalArgumentException</span><span class="o">(</span><span class="s">"y: "</span> <span class="o">+</span> <span class="n">y</span><span class="o">);</span>
<span class="n">data</span><span class="o">[</span><span class="n">x</span><span class="o">+</span><span class="n">y</span><span class="o">*</span><span class="n">width</span><span class="o">]</span> <span class="o">=</span> <span class="n">value</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">FloatImage</span> <span class="nf">createSameSize</span><span class="o">(</span><span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">sample</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">FloatImage</span><span class="o">(</span><span class="n">sample</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">sample</span><span class="o">.</span><span class="na">getHeight</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">FloatImage</span> <span class="nf">createSameSize</span><span class="o">(</span><span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">sample</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">FloatImage</span><span class="o">(</span><span class="n">sample</span><span class="o">.</span><span class="na">getWidth</span><span class="o">(),</span> <span class="n">sample</span><span class="o">.</span><span class="na">getHeight</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">FloatImage</span> <span class="nf">fromBufferedImage</span><span class="o">(</span><span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">src</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">width</span> <span class="o">=</span> <span class="n">src</span><span class="o">.</span><span class="na">getWidth</span><span class="o">();</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">height</span> <span class="o">=</span> <span class="n">src</span><span class="o">.</span><span class="na">getHeight</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">FloatImage</span> <span class="n">result</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FloatImage</span><span class="o">(</span><span class="n">width</span><span class="o">,</span> <span class="n">height</span><span class="o">);</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">width</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">height</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">argb</span> <span class="o">=</span> <span class="n">src</span><span class="o">.</span><span class="na">getRGB</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">);</span>
<span class="kt">int</span> <span class="n">r</span> <span class="o">=</span> <span class="o">(</span><span class="n">argb</span> <span class="o">>>></span> <span class="mi">16</span><span class="o">)</span> <span class="o">&</span> <span class="mh">0xFF</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">g</span> <span class="o">=</span> <span class="o">(</span><span class="n">argb</span> <span class="o">>>></span> <span class="mi">8</span><span class="o">)</span> <span class="o">&</span> <span class="mh">0xFF</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">b</span> <span class="o">=</span> <span class="n">argb</span> <span class="o">&</span> <span class="mh">0xFF</span><span class="o">;</span>
<span class="n">result</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span> <span class="o">(</span><span class="n">r</span><span class="o">*</span><span class="mf">0.3f</span><span class="o">+</span><span class="n">g</span><span class="o">*</span><span class="mf">0.59f</span><span class="o">+</span><span class="n">b</span><span class="o">*</span><span class="mf">0.11f</span><span class="o">)/</span><span class="mi">255</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">result</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">BufferedImage</span> <span class="nf">toBufferedImage</span><span class="o">(</span><span class="kt">float</span> <span class="n">scale</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">BufferedImage</span> <span class="n">result</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BufferedImage</span><span class="o">(</span><span class="n">width</span><span class="o">,</span> <span class="n">height</span><span class="o">,</span> <span class="nc">BufferedImage</span><span class="o">.</span><span class="na">TYPE_INT_ARGB</span><span class="o">);</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">x</span> <span class="o"><</span> <span class="n">width</span><span class="o">;</span> <span class="n">x</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span><span class="o">(</span><span class="kt">int</span> <span class="n">y</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">y</span> <span class="o"><</span> <span class="n">height</span><span class="o">;</span> <span class="n">y</span><span class="o">++)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">intensity</span> <span class="o">=</span> <span class="o">((</span><span class="kt">int</span><span class="o">)</span> <span class="o">(</span><span class="n">get</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">)</span> <span class="o">*</span> <span class="n">scale</span><span class="o">))</span> <span class="o">&</span> <span class="mh">0xFF</span><span class="o">;</span>
<span class="n">result</span><span class="o">.</span><span class="na">setRGB</span><span class="o">(</span><span class="n">x</span><span class="o">,</span><span class="n">y</span><span class="o">,</span><span class="mh">0xFF000000</span> <span class="o">|</span> <span class="n">intensity</span> <span class="o">|</span> <span class="n">intensity</span> <span class="o"><<</span> <span class="mi">8</span> <span class="o">|</span> <span class="n">intensity</span> <span class="o"><<</span> <span class="mi">16</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">result</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>Today I’m going to discuss a technique called Seam Carving, originally presented in Siggraph 2007. This algorithm at it’s core it’s fairly simple but produces impressive results.Nerding with the Y-combinator2011-04-08T19:51:00+00:002011-04-08T19:51:00+00:00https://juancn.github.io/post/2011/04/08/nerding-with-y-combinator<p>What follows is a pointless exercise. Hereby I present you the <a href="http://kestas.kuliukas.com/YCombinatorExplained/">Y-combinator</a> in Java with generics:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Combinators</span> <span class="o">{</span>
<span class="kd">interface</span> <span class="nc">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span><span class="no">B</span><span class="o">></span> <span class="o">{</span>
<span class="no">B</span> <span class="nf">apply</span><span class="o">(</span><span class="no">A</span> <span class="n">x</span><span class="o">);</span>
<span class="o">}</span>
<span class="c1">//Used for proper type checking</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">interface</span> <span class="nc">FF</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="kd">extends</span> <span class="no">F</span><span class="o"><</span><span class="no">FF</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">>,</span> <span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">>></span> <span class="o">{}</span>
<span class="c1">//The Y-combinator</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="nf">Y</span><span class="o">(</span><span class="kd">final</span> <span class="no">F</span><span class="o"><</span><span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">>,</span><span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">>></span> <span class="n">f</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">U</span><span class="o">(</span><span class="k">new</span> <span class="no">FF</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">>()</span> <span class="o">{</span>
<span class="kd">public</span> <span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="nf">apply</span><span class="o">(</span><span class="kd">final</span> <span class="no">FF</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="n">x</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">f</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">>()</span> <span class="o">{</span>
<span class="kd">public</span> <span class="no">B</span> <span class="nf">apply</span><span class="o">(</span><span class="no">A</span> <span class="n">y</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">U</span><span class="o">(</span><span class="n">x</span><span class="o">).</span><span class="na">apply</span><span class="o">(</span><span class="n">y</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">});</span>
<span class="o">}</span>
<span class="o">});</span>
<span class="o">}</span>
<span class="c1">//The U-combinator</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="o"><</span><span class="no">A</span><span class="o">,</span><span class="no">B</span><span class="o">></span> <span class="no">F</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="nf">U</span><span class="o">(</span><span class="no">FF</span><span class="o"><</span><span class="no">A</span><span class="o">,</span> <span class="no">B</span><span class="o">></span> <span class="n">a</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">a</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">static</span> <span class="no">F</span><span class="o"><</span><span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">>,</span> <span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">>></span> <span class="nf">factorialGenerator</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="no">F</span><span class="o"><</span><span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">>,</span> <span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">>>()</span> <span class="o">{</span>
<span class="kd">public</span> <span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">></span> <span class="nf">apply</span><span class="o">(</span><span class="kd">final</span> <span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">></span> <span class="n">fact</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">>()</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">Integer</span> <span class="nf">apply</span><span class="o">(</span><span class="nc">Integer</span> <span class="n">n</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">?</span> <span class="mi">1</span> <span class="o">:</span> <span class="n">n</span> <span class="o">*</span> <span class="n">fact</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="no">F</span><span class="o"><</span><span class="nc">Integer</span><span class="o">,</span> <span class="nc">Integer</span><span class="o">></span> <span class="n">fact</span> <span class="o">=</span> <span class="no">Y</span><span class="o">(</span><span class="n">factorialGenerator</span><span class="o">());</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">fact</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="mi">6</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Having the Y-combinator implemented in Java, actually serves no purpose (Java supports recursion) but it was interesting to see if it could be done with proper generics.</p>What follows is a pointless exercise. Hereby I present you the Y-combinator in Java with generics:Paper Programming2011-04-04T15:46:00+00:002011-04-04T15:46:00+00:00https://juancn.github.io/post/2011/04/04/paper-programming<p>When I was a kid, all I wanted was a computer. Finally when I was twelve I made a bargain with my dad. I would give up the graduation trip in exchange for a Commodore 64 (graduation trips are customary in Argentina when you finish primary and secondary school).</p>
<p><img src="/images/2011-04-04-paper-programming/c64.jpg" alt="A Commodore 64" /></p>
<p>We bought a “Segundamano” (lit. second hand) magazine and found a used one for U$S 200. My dad contacted the seller and we went to pick it up.</p>
<p>You have to keep in mind that this was 1989 and the social and economic landscape in <a href="http://en.wikipedia.org/wiki/1989_riots_in_Argentina">Argentina was a mess</a>. That year the <a href="http://www.wolframalpha.com/input/?i=argentina+inflation+1989">inflation rate was over 3000%</a> (it is not a typo) and those 200 dollars were a lot of money, so my dad really made an effort.</p>
<p>If you are still paying attention, you might have noticed that I never mentioned a <a href="http://en.wikipedia.org/wiki/Commodore_Datasette">Datasette</a> nor a disk drive. All I got was a Commodore 64, plain and simple. But this was not going to stop me.</p>
<p>After we got it, my dad was concerned that I might get too obsessed with the computer, so he placed some additional constraints on when I could use it (the fact that we had only one TV might have also been a factor in his decision). I was allowed to use it only on Saturday mornings before noon.</p>
<p>In retrospective, I think that this two factors made me a better programmer.</p>
<p>At that time I had some experience programming in Logo mostly. It was a Logo dialect in Spanish that we used at school (“HAL Logo en Español”, do not confuse with HAL Laboratories). We had a Commodore-128 laboratory at school (about eight machines with disk drives and monitors). I started learning Logo when I was 8, by the time I was twelve I could program a bit of BASIC also, but not much since literature was scarce.</p>
<p>One great thing about the Commodore 64 was that it came with BASIC, but most importantly, it came with a manual! The Commodore’s Basic manual was my first programming book.</p>
<p>What happened was that I was forced to work as if I had punched cards. I would spend most of my spare time during the week writing programs in a notebook I had. Thinking about ways to solve problems and reading and re-reading the C64 manual.</p>
<p>On saturday mornings I would wake up at 6 am and hook-up the computer to the TV and start typing whatever program I was working on that week. Run it and debug it, improve it and around noon my mom would start reminding me that time was up. So at that point I began listing the BASIC source code and copying it back to my notebook.</p>
<p>It was during this time that I rediscovered things like an optimized Bubble-sort, although I didn’t know its name then (and I wouldn’t learn it for many more years). I still vividly remember the moment. It was one afternoon that I was trying to figure out a way to sort an array, so I started playing with a <a href="http://en.wikipedia.org/wiki/Baraja_(playing_cards)">deck of cards</a>. I finally figured out that I could do several passes of comparison and exchanges on adjacent cards. And if I did exactly as many passes as the number of elements the array would be sorted. I also noticed that the largest element would be at the end of the array after the first pass, and the second largest would be in place after the second pass and so on. This allowed me to save about half the number of comparisons.</p>
<p>The most valuable thing that I learned during this time is that when it comes to programming, thinking hard about the problem at hand before doing anything pays off, big time!</p>
<p>I eventually was able to save enough for a Datasette (it costed 40 dollars, my dad payed half of it) and I was able to build much larger programs.</p>
<p>The biggest one I wrote was an interpreter with a multitasking runtime. It was very basic. I had no notion of a lexer, let alone a parser. Commands were composed of a single letter (the first one of the line) and several arguments. You could declare procedures with names and call them. It was like pidgin-basic (since my exposure was to basic) but the procedure structure resembled Logo.</p>
<p>Programs were stored in memory in several arrays. There was a main one that contained statements, and couple to hold procedure names and offsets into the main array.</p>
<p>The runtime divided the screen in 4 areas (this was a 40x25 text screen, so not much room was left for each), and each window could run a separate program. The runtime would do a round robin on each running program, executing a single statement of each on each pass. For this it had to maintain a separate program counter and variables for each. I even planned to add windowing calls to create custom windows but I never got to finish it.</p>
<p>It was at this time that I also got interested in electronics, so I built a a few contraptions controlled by the C64, but that’s a tale for another post.</p>When I was a kid, all I wanted was a computer. Finally when I was twelve I made a bargain with my dad. I would give up the graduation trip in exchange for a Commodore 64 (graduation trips are customary in Argentina when you finish primary and secondary school).Matching Regular Expressions using its Derivatives2011-03-30T23:42:00+00:002011-03-30T23:42:00+00:00https://juancn.github.io/post/2011/03/30/matching-regular-expressions-using-its<h2 id="introduction">Introduction</h2>
<p>Regular expressions are expressions that describe a set of strings over a particular alphabet. We will begin with a crash course on simple regular expressions. You can assume that we’re talking about text and characters but in fact this can be generalized to any (finite) alphabet.</p>
<p>The definition of regular expressions is quite simple<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, there are three basic (i.e. terminal) regular expressions:</p>
<ul>
<li>The null expression (denoted as: ∅) which never matches anything</li>
<li>The empty expression, that only matches the empty string (I will use ε to represent this expression since it’s customary<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>)</li>
<li>The character literal expression (usually called ‘c’), that matches a single character</li>
</ul>
<p>These three basic building blocks can be combined using some operators to form more complex expressions:</p>
<ul>
<li>sequencing of regular expressions matches two regular expressions in sequence</li>
<li>alternation matches one of the two sub-expressions (usually represented by a ‘|’ symbol)</li>
<li>the repetition operator (aka. Kleene’s star) matches zero or more repetitions of the specified subexpression</li>
</ul>
<p>Some examples will make this clearer:</p>
<ul>
<li>The expression ‘a’ will only match the character ‘a’. Similarly ‘b’ will only match ‘b’. If we combine them by sequencing ‘ab’ will match ‘ab’.</li>
<li>The expression ‘a|b’ will match either ‘a’ or ‘b’.</li>
<li>If we combine sequencing with alternation as in ‘(a|b)(a|b)’ (the parenthesis are clarifying), it will match: ‘aa’, ‘ab’, ‘ba’ or ‘bb’.</li>
<li>Kleene’s star as mentioned before matches zero or more of the preceding subexpression. So the expression ‘a*’ will match: ‘’, ‘a’, ‘aa’, ‘aaa’, ‘aaaa’, …</li>
<li>We can do more complex combinations, such as ‘ab*(c|ε)’ that will match things like: ‘a’, ‘ab’, ‘ac’, ‘abc’, ‘abb’, ‘abbc’, … that is any string starting with an ‘a’ followed by zero or more ‘b’’s and optionally ending in a ‘c’.</li>
</ul>
<p>Typical implementations of regular expression matchers convert the regular expression to an NFA or a DFA (which are a kind of <a href="http://en.wikipedia.org/wiki/Finite_state_machine">finite state machine</a>).</p>
<p>Anyway, a few weeks ago I ran into <a href="http://matt.might.net/articles/implementation-of-regular-expression-matching-in-scheme-with-derivatives/">a post about using the derivative of a regular expression for matching</a>.</p>
<p>It is a quite intriguing concept and worth exploring. The original post gives an implementation in Scheme<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>. But leaves out some details that make it a bit tricky to implement. I’ll try to walk you through the concept, up to a working implementation in Java.</p>
<h2 id="derivative-of-a-regular-expression">Derivative of a Regular Expression</h2>
<p>So, first question: What’s the derivative of a regular expression?</p>
<blockquote>
<p>The derivative of a regular expression with respect to a character ‘c’ computes a new regular expression that matches what the original expression would match, assuming it had just matched the character ‘c’.</p>
</blockquote>
<p>As usual, some examples will (hopefully) help clarify things:</p>
<ul>
<li>The expression ‘foo’ derived with respect to ‘f’ yields the expression: ‘oo’ (which is what’s left to match).</li>
<li>The expression ‘ab|ba’ derived with respect to ‘a’, yields the expression: ‘b’
Similarly, the expression ‘ab|ba’ derived with respect to ‘b’, yields the expression: ‘a’</li>
<li>The expression ‘(ab|ba)*’ derived with respect to ‘a’, yields the expression: ‘b(ab|ba)*’</li>
</ul>
<p>As we explore this notion, we will work a RegEx class. The skeleton of this class looks like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">RegEx</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">abstract</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">);</span>
<span class="kd">public</span> <span class="kd">abstract</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">();</span>
<span class="c1">//...</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">unmatchable</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RegEx</span><span class="o">()</span> <span class="o">{</span> <span class="cm">/* ... */</span> <span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">empty</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RegEx</span><span class="o">()</span> <span class="o">{</span> <span class="cm">/* ... */</span> <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>It includes constants for the unmatchable (null) and empty expressions, and a derive and simplify methods wich we will cover in detail (but not just now).</p>
<p>Before we go in detail about the rules of regular expression derivation, let’s take a small -but necessary- detour and cover some details that will help us get a working implementation.</p>
<p>The formalization of the derivative of a regular expression depends on a set of simplifying constructors that are necessary for a correct implementation. These will be defined a bit more formally and we will build the skeleton of its implementation at this point.</p>
<p>Let’s begin with the sequencing operation, we define the following constructor (ignore spaces):</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">seq( ∅, _ ) = ∅</code></li>
<li><code class="language-plaintext highlighter-rouge">seq( _, ∅ ) = ∅</code></li>
<li><code class="language-plaintext highlighter-rouge">seq( ε, r2 ) = r2</code></li>
<li><code class="language-plaintext highlighter-rouge">seq( r1, ε ) = r1</code></li>
<li><code class="language-plaintext highlighter-rouge">seq( r1, r2 ) = r1 r2</code></li>
</ol>
<p>The first two definitions state that if you have a sequence with the null expression (∅, which is unmatchable) and any other expression, it’s the same than having the null expression (i.e. it will not match anything).</p>
<p>The third and fourth definitions state that if you have a sequence of the empty expression (ε, matches only the empty string) and any other expression, is the same than just having the other expression (the empty expression is the identity with respect to the sequence operator).</p>
<p>The fifth and last definition just builds a regular sequence.</p>
<p>With this, we can draft a first implementation of a sequence constructor (in the gang-of-four’s parlance it’s a factory method):</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">seq</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r2</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r1</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">unmatchable</span> <span class="o">||</span> <span class="n">r2</span> <span class="o">==</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="k">return</span> <span class="n">unmatchable</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">r2</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r2</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">r1</span><span class="o">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// ....</span>
<span class="o">};</span>
<span class="o">}</span>
</code></pre></div></div>
<p>I’m leaving out the details of the RegEx for the time being, we will come back to them soon enough.</p>
<p>The alternation operator also has simplifying constructor that is analogous to the sequence operator:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>alt( ε, _ ) = ε
alt( _, ε ) = ε
alt( ∅, r2 ) = r2
alt( r1, ∅ ) = r1
alt( r1, r2 ) = r1 | r2
</code></pre></div></div>
<p>If you look closely, the first two definitions are rather odd. They basically reduce an alternation with the empty expression to the empty expression (ε). This is because the simplifying constructors are used as part of a simplification function that reduces a regular expression to the empty expression if it matches the empty expression. We’ll see how this works with the rest of it in a while.</p>
<p>The third and fourth definitions are fairly logical, an alternation with an unmatchable expression is the same than the alternative (the unmatchable expression is the identity with respect to the alternation operator).</p>
<p>The last one is the constructor.</p>
<p>Taking these details into account, we can build two factory methods, one internal and one external:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">private</span> <span class="nc">RegEx</span> <span class="nf">alt0</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r2</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r1</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">empty</span> <span class="o">||</span> <span class="n">r2</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">empty</span><span class="o">;</span>
<span class="k">return</span> <span class="nf">alt</span><span class="o">(</span><span class="n">r2</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">alt</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r2</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r1</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="k">return</span> <span class="n">r2</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r2</span> <span class="o">==</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="k">return</span> <span class="n">r1</span><span class="o">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">//.....</span>
<span class="o">};</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The internal one alt0 includes the first two simplification rules, the public one is user-facing. That is, it has to let you build something like: ‘ab*(c|ε)’.</p>
<p>Finally, the repetition operator (Kleene’s star) has the following simplification rules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rep( ∅ ) = ε
rep( ε ) = ε
rep( re ) = re*
</code></pre></div></div>
<p>The first definition states that a repetition of the unmatchable expression, matches at least the empty string.</p>
<p>The second definition states that a repetition of the empty expression is the same than matching the empty expression.</p>
<p>And as usual, the last one is the constructor for all other cases.</p>
<p>A skeleton for the rep constructor is rather simple:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">rep</span><span class="o">()</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">re</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">re</span> <span class="o">==</span> <span class="n">unmatchable</span> <span class="o">||</span> <span class="n">re</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">empty</span><span class="o">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">// ....</span>
<span class="o">};</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="simplify--derive">Simplify & Derive</h2>
<p>As hinted earlier on, derivation is based on a simplification function. This simplification function reduces a regular expression to the empty regular expression (ε epsilon) if it matches the empty string or the unmatchable expression (∅) if it does not.
The simplification function is defined as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>s(∅) = ∅
s(ε) = ε
s(c) = ∅
s(re1 re2) = seq(s(re1), s(re2))
s(re1 | re2) = alt(s(re1), s(re2))
s(re*) = ε
</code></pre></div></div>
<p>Note that this function depends on the simplifying constructors we described earlier on.
Suppose that we want to check if the expression ‘ab*(c|ε)’ matches the empty expression, if we do all the substitutions:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">seq(s(ab*),s(c|ε))</code></li>
<li><code class="language-plaintext highlighter-rouge">seq(s(seq(s(a), s(b*))),s(alt(s(c), s(ε))))</code></li>
<li><code class="language-plaintext highlighter-rouge">seq(s(seq(∅, s(ε))),s(alt(∅, ε)))</code></li>
<li><code class="language-plaintext highlighter-rouge">seq(s(seq(∅, ε)),s(ε))</code></li>
<li><code class="language-plaintext highlighter-rouge">seq(s(∅),ε)</code></li>
<li><code class="language-plaintext highlighter-rouge">seq(∅,ε)</code></li>
<li><code class="language-plaintext highlighter-rouge">∅</code></li>
</ol>
<p>We get the null/unmatchable expression as a result. This means that the expression ‘ab*(c|ε)’ does not match the empty string.
If on the other hand we apply the reduction on <strong>‘a*|b’</strong>:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">alt(s(a*), s(b))</code></li>
<li><code class="language-plaintext highlighter-rouge">alt(ε, ∅)</code></li>
<li><code class="language-plaintext highlighter-rouge">ε</code></li>
</ol>
<p>We get the empty expression, hence the regular expression ‘a*|b’ will match the empty string.
The derivation function given a regular expression and a character ‘x’ derives a new regular expression as if having matched ‘x’.</p>
<p>Derivation is defined by the following set of rules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>D( ∅, _ ) = ∅
D( ε, _ ) = ∅
D( c, x ) = if c == x then ε else ∅
D(re1 re2, x) = alt(
seq( s(re1) , D(re2, x) ),
seq( D(re1, x), re2 )
)
D(re1 | re2, x) = alt( D(re1, x) , D(re2, x) )
D(re*, x) = seq( D(re, x) , rep(re) )
</code></pre></div></div>
<p>The first two definitions define the derivative of the unmatchable and empty expressions regarding any character, wich yields the unmatchable expression.
The third definition states that if a character matcher (for example ‘a’) is derived with respect to the same character yields the empty expression otherwise yields the unmatchable expression.</p>
<p>The fourth rule is a bit more involved, but trust me, it works.</p>
<p>The fifth rule states that the derivative of an alternation is the alternation of the derivatives (suitably simplified).</p>
<p>And the last one, describes how to derive a repetition. For example D(‘(ba)*’, ‘b’) yields ‘a(ba)*’.</p>
<p>We now have enough information to implement the simplify and derive.</p>
<h2 id="matching">Matching</h2>
<p>If you haven’t figured it out by now, matching works by walking the string we’re checking character by character and successively deriving the regular expression until we either run out of characters, at wich point we simplify the derived expression and see if it matches the empty string. Or we end up getting the unmatchable expression, at wich point it is impossible that the rest of the string will match.
A iterative implementation of a match method is as follows:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">matches</span><span class="o">(</span><span class="kd">final</span> <span class="nc">String</span> <span class="n">text</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">RegEx</span> <span class="n">d</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">s</span> <span class="o">=</span> <span class="n">text</span><span class="o">;</span>
<span class="c1">//The 'unmatchable' test is not strictly necessary, but avoids unnecessary derivations</span>
<span class="k">while</span><span class="o">(!</span><span class="n">s</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">()</span> <span class="o">&&</span> <span class="n">d</span> <span class="o">!=</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="o">{</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">d</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="na">charAt</span><span class="o">(</span><span class="mi">0</span><span class="o">));</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">d</span><span class="o">.</span><span class="na">simplify</span><span class="o">()</span> <span class="o">==</span> <span class="n">empty</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If we match <strong>‘ab*(c|ε)’</strong> against the text “abbc”, we get the following derivatives:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">D(re, a) = ab*(c|ε) , rest: "bbc"</code></li>
<li><code class="language-plaintext highlighter-rouge">D(re, b) = b*(c|ε) , rest: "bc"</code></li>
<li><code class="language-plaintext highlighter-rouge">D(re, b) = b*(c|ε) , rest: "c"</code></li>
<li><code class="language-plaintext highlighter-rouge">D(re, c) = b*(c|ε) , rest: ""</code></li>
</ol>
<p>And if we simplify the last derivative we get the empty expression, therefore we have a match.
One interesting fact of this matching strategy is that it is fairly easy to implement a non-blocking matcher. That is, doing incremental matching as we receive characters.</p>
<h2 id="implementation">Implementation</h2>
<p>The following is the complete class with all methods implemented. I provide a basic implementation of the toString method (which is nice for debugging), and a helper text method which is a shortcut to build an expression for a sequence of characters. This class is fairly easy to modify to match over a different alphabet, such as arbitrary objects and Iterables instead of Strings (it can be easily generified).</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">RegEx</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">abstract</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">);</span>
<span class="kd">public</span> <span class="kd">abstract</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">();</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">seq</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r2</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r1</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">unmatchable</span> <span class="o">||</span> <span class="n">r2</span> <span class="o">==</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="k">return</span> <span class="n">unmatchable</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">r2</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r2</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">r1</span><span class="o">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">r1</span><span class="o">.</span><span class="na">simplify</span><span class="o">().</span><span class="na">seq</span><span class="o">(</span><span class="n">r2</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">c</span><span class="o">))</span>
<span class="o">.</span><span class="na">alt0</span><span class="o">(</span><span class="n">r1</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">c</span><span class="o">).</span><span class="na">seq</span><span class="o">(</span><span class="n">r2</span><span class="o">));</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">r1</span><span class="o">.</span><span class="na">simplify</span><span class="o">().</span><span class="na">seq</span><span class="o">(</span><span class="n">r2</span><span class="o">.</span><span class="na">simplify</span><span class="o">());</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">r1</span> <span class="o">+</span> <span class="s">""</span> <span class="o">+</span> <span class="n">r2</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nc">RegEx</span> <span class="nf">alt0</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r2</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r1</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">empty</span> <span class="o">||</span> <span class="n">r2</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">empty</span><span class="o">;</span>
<span class="k">return</span> <span class="nf">alt</span><span class="o">(</span><span class="n">r2</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">alt</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r2</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">r1</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r1</span> <span class="o">==</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="k">return</span> <span class="n">r2</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">r2</span> <span class="o">==</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="k">return</span> <span class="n">r1</span><span class="o">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">r1</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">c</span><span class="o">).</span><span class="na">alt0</span><span class="o">(</span><span class="n">r2</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">c</span><span class="o">));</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">r1</span><span class="o">.</span><span class="na">simplify</span><span class="o">().</span><span class="na">alt0</span><span class="o">(</span><span class="n">r2</span><span class="o">.</span><span class="na">simplify</span><span class="o">());</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"("</span> <span class="o">+</span> <span class="n">r1</span> <span class="o">+</span> <span class="s">"|"</span> <span class="o">+</span> <span class="n">r2</span> <span class="o">+</span> <span class="s">")"</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">rep</span><span class="o">()</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">re</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">re</span> <span class="o">==</span> <span class="n">unmatchable</span> <span class="o">||</span> <span class="n">re</span> <span class="o">==</span> <span class="n">empty</span><span class="o">)</span> <span class="k">return</span> <span class="n">empty</span><span class="o">;</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">c</span><span class="o">).</span><span class="na">seq</span><span class="o">(</span><span class="n">re</span><span class="o">.</span><span class="na">rep</span><span class="o">());</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">empty</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">s</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span>
<span class="k">return</span> <span class="n">s</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="s">"("</span><span class="o">)</span>
<span class="o">?</span> <span class="n">s</span> <span class="o">+</span> <span class="s">"*"</span>
<span class="o">:</span><span class="s">"("</span> <span class="o">+</span> <span class="n">s</span> <span class="o">+</span> <span class="s">")*"</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">RegEx</span> <span class="nf">character</span><span class="o">(</span><span class="kd">final</span> <span class="kt">char</span> <span class="n">exp</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RegEx</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">exp</span> <span class="o">==</span> <span class="n">c</span><span class="o">?</span><span class="nl">empty:</span><span class="n">unmatchable</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">unmatchable</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">""</span><span class="o">+</span> <span class="n">exp</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">};</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">RegEx</span> <span class="nf">text</span><span class="o">(</span><span class="kd">final</span> <span class="nc">String</span> <span class="n">text</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">RegEx</span> <span class="n">result</span><span class="o">;</span>
<span class="k">if</span><span class="o">(</span><span class="n">text</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">empty</span><span class="o">;</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">character</span><span class="o">(</span><span class="n">text</span><span class="o">.</span><span class="na">charAt</span><span class="o">(</span><span class="mi">0</span><span class="o">));</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">text</span><span class="o">.</span><span class="na">length</span><span class="o">();</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">result</span><span class="o">.</span><span class="na">seq</span><span class="o">(</span><span class="n">character</span><span class="o">(</span><span class="n">text</span><span class="o">.</span><span class="na">charAt</span><span class="o">(</span><span class="n">i</span><span class="o">)));</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">result</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">matches</span><span class="o">(</span><span class="kd">final</span> <span class="nc">String</span> <span class="n">text</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">RegEx</span> <span class="n">d</span> <span class="o">=</span> <span class="k">this</span><span class="o">;</span>
<span class="nc">String</span> <span class="n">s</span> <span class="o">=</span> <span class="n">text</span><span class="o">;</span>
<span class="c1">//The 'unmatchable' test is not strictly necessary, but avoids unnecessary derivations</span>
<span class="k">while</span><span class="o">(!</span><span class="n">s</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">()</span> <span class="o">&&</span> <span class="n">d</span> <span class="o">!=</span> <span class="n">unmatchable</span><span class="o">)</span> <span class="o">{</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">d</span><span class="o">.</span><span class="na">derive</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="na">charAt</span><span class="o">(</span><span class="mi">0</span><span class="o">));</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">d</span><span class="o">.</span><span class="na">simplify</span><span class="o">()</span> <span class="o">==</span> <span class="n">empty</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ConstantRegEx</span> <span class="kd">extends</span> <span class="nc">RegEx</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">name</span><span class="o">;</span>
<span class="nc">ConstantRegEx</span><span class="o">(</span><span class="nc">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">name</span> <span class="o">=</span> <span class="n">name</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">derive</span><span class="o">(</span><span class="kt">char</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">unmatchable</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">RegEx</span> <span class="nf">simplify</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">this</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">toString</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">name</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">unmatchable</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ConstantRegEx</span><span class="o">(</span><span class="s">"<null>"</span><span class="o">);</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">empty</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ConstantRegEx</span><span class="o">(</span><span class="s">"<empty>"</span><span class="o">);</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">RegEx</span> <span class="n">regEx</span> <span class="o">=</span> <span class="n">character</span><span class="o">(</span><span class="sc">'a'</span><span class="o">)</span>
<span class="o">.</span><span class="na">seq</span><span class="o">(</span><span class="n">character</span><span class="o">(</span><span class="sc">'b'</span><span class="o">).</span><span class="na">rep</span><span class="o">())</span>
<span class="o">.</span><span class="na">seq</span><span class="o">(</span><span class="n">character</span><span class="o">(</span><span class="sc">'c'</span><span class="o">).</span><span class="na">alt</span><span class="o">(</span><span class="n">empty</span><span class="o">));</span>
<span class="k">if</span><span class="o">(</span><span class="n">regEx</span><span class="o">.</span><span class="na">matches</span><span class="o">(</span><span class="s">"abbc"</span><span class="o">))</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Matches!!!"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Disclaimer: Any bugs/misconceptions regarding this are my errors, so take everything with a grain of salt. Feel free to use the code portrayed here for any purpose whatsoever, if you do something cool with it I’d like to know, but no pressure.</p>
<p>Footnotes</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Sometimes the simpler something is, the harder it is to understand. See lambda calculus for example. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>I will not use ε (epsilon) to also represent the empty string since I think it is confusing, even though it is also customary. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>I think that the Scheme implementation in that article won’t work if you use the repetition operator, but I haven’t tested it. It might just as well be that my Scheme-foo is a bit rusty. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>IntroductionPratt Parsers2011-03-14T22:10:00+00:002011-03-14T22:10:00+00:00https://juancn.github.io/post/2011/03/14/pratt-parsers<p>Some time ago I came across <a href="http://crockford.com/javascript/tdop/tdop.html">Pratt parsers</a>. I had never seen them before, and I found them quite elegant.</p>
<p>They were first described by Vaughan Pratt in the 1973 paper “Top down operator precedence”. From a theoretical perspective they are not particularly interesting, but from an engineering point of view they are fantastic.</p>
<p>Let’s start with a real-world example. This is the grammar from the expression language for my <a href="https://github.com/juancn/performance">Performance Invariants agent</a>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* omitted */</span>
<span class="kn">import</span> <span class="nn">static</span> <span class="n">performance</span><span class="o">.</span><span class="na">compiler</span><span class="o">.</span><span class="na">TokenType</span><span class="o">.*;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">SimpleGrammar</span>
<span class="kd">extends</span> <span class="nc">Grammar</span><span class="o"><</span><span class="nc">TokenType</span><span class="o">></span> <span class="o">{</span>
<span class="kd">private</span> <span class="nf">SimpleGrammar</span><span class="o">()</span> <span class="o">{</span>
<span class="n">infix</span><span class="o">(</span><span class="no">LAND</span><span class="o">,</span> <span class="mi">30</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">LOR</span><span class="o">,</span> <span class="mi">30</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">LT</span><span class="o">,</span> <span class="mi">40</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">GT</span><span class="o">,</span> <span class="mi">40</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">LE</span><span class="o">,</span> <span class="mi">40</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">GE</span><span class="o">,</span> <span class="mi">40</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">EQ</span><span class="o">,</span> <span class="mi">40</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">NEQ</span><span class="o">,</span> <span class="mi">40</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">PLUS</span><span class="o">,</span> <span class="mi">50</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">MINUS</span><span class="o">,</span> <span class="mi">50</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">MUL</span><span class="o">,</span> <span class="mi">60</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">DIV</span><span class="o">,</span> <span class="mi">60</span><span class="o">);</span>
<span class="n">unary</span><span class="o">(</span><span class="no">MINUS</span><span class="o">,</span> <span class="mi">70</span><span class="o">);</span>
<span class="n">unary</span><span class="o">(</span><span class="no">NOT</span><span class="o">,</span> <span class="mi">70</span><span class="o">);</span>
<span class="n">infix</span><span class="o">(</span><span class="no">DOT</span><span class="o">,</span> <span class="mi">80</span><span class="o">);</span>
<span class="n">clarifying</span><span class="o">(</span><span class="no">LPAREN</span><span class="o">,</span> <span class="no">RPAREN</span><span class="o">,</span> <span class="mi">0</span><span class="o">);</span>
<span class="n">delimited</span><span class="o">(</span><span class="no">DOLLAR_LCURLY</span><span class="o">,</span> <span class="no">RCURLY</span><span class="o">,</span> <span class="mi">70</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">INT_LITERAL</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">LONG_LITERAL</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">FLOAT_LITERAL</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">DOUBLE_LITERAL</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">ID</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">THIS</span><span class="o">);</span>
<span class="n">literal</span><span class="o">(</span><span class="no">STATIC</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">Expr</span><span class="o"><</span><span class="nc">TokenType</span><span class="o">></span> <span class="nf">parse</span><span class="o">(</span><span class="kd">final</span> <span class="nc">String</span> <span class="n">text</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">ParseException</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">Lexer</span><span class="o"><</span><span class="nc">TokenType</span><span class="o">></span> <span class="n">lexer</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">JavaLexer</span><span class="o">(</span><span class="n">text</span><span class="o">,</span> <span class="mi">0</span> <span class="o">,</span> <span class="n">text</span><span class="o">.</span><span class="na">length</span><span class="o">());</span>
<span class="kd">final</span> <span class="nc">PrattParser</span><span class="o"><</span><span class="nc">TokenType</span><span class="o">></span> <span class="n">prattParser</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PrattParser</span><span class="o"><</span><span class="nc">TokenType</span><span class="o">>(</span><span class="no">INSTANCE</span><span class="o">,</span> <span class="n">lexer</span><span class="o">);</span>
<span class="kd">final</span> <span class="nc">Expr</span><span class="o"><</span><span class="nc">TokenType</span><span class="o">></span> <span class="n">expr</span> <span class="o">=</span> <span class="n">prattParser</span><span class="o">.</span><span class="na">parseExpression</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span>
<span class="k">if</span><span class="o">(</span><span class="n">prattParser</span><span class="o">.</span><span class="na">current</span><span class="o">().</span><span class="na">getType</span><span class="o">()</span> <span class="o">!=</span> <span class="no">EOF</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">ParseException</span><span class="o">(</span><span class="s">"Unexpected token: "</span> <span class="o">+</span> <span class="n">prattParser</span><span class="o">.</span><span class="na">current</span><span class="o">());</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">expr</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">SimpleGrammar</span> <span class="no">INSTANCE</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">SimpleGrammar</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Pretty, isn’t it?</p>
<p>The number represents a precedence, for infix operators is quite obvious (it’s basically a precedence table), but for clarifying and delimited expressions it sets the lower bound for the subexpression. In the grammar above, the delimited expression only accepts dot expressions and literals, parenthesis on the other hand, accept anything.</p>
<p>So, how does the parser work? The PrattParser itself is rather elegant also:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* omitted */</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">PrattParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">Grammar</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">grammar</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">Lexer</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">lexer</span><span class="o">;</span>
<span class="kd">private</span> <span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">current</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">PrattParser</span><span class="o">(</span><span class="nc">Grammar</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">grammar</span><span class="o">,</span> <span class="nc">Lexer</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">lexer</span><span class="o">)</span>
<span class="kd">throws</span> <span class="nc">ParseException</span>
<span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">grammar</span> <span class="o">=</span> <span class="n">grammar</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">lexer</span> <span class="o">=</span> <span class="n">lexer</span><span class="o">;</span>
<span class="n">current</span> <span class="o">=</span> <span class="n">lexer</span><span class="o">.</span><span class="na">next</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Expr</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">parseExpression</span><span class="o">(</span><span class="kt">int</span> <span class="n">stickiness</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">ParseException</span> <span class="o">{</span>
<span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span> <span class="o">=</span> <span class="n">consume</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">prefix</span> <span class="o">=</span> <span class="n">grammar</span><span class="o">.</span><span class="na">getPrefixParser</span><span class="o">(</span><span class="n">token</span><span class="o">);</span>
<span class="k">if</span><span class="o">(</span><span class="n">prefix</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">ParseException</span><span class="o">(</span><span class="s">"Unexpected token: "</span> <span class="o">+</span> <span class="n">token</span><span class="o">);</span>
<span class="o">}</span>
<span class="nc">Expr</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">left</span> <span class="o">=</span> <span class="n">prefix</span><span class="o">.</span><span class="na">parse</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">token</span><span class="o">);</span>
<span class="k">while</span> <span class="o">(</span><span class="n">stickiness</span> <span class="o"><</span> <span class="n">grammar</span><span class="o">.</span><span class="na">getStickiness</span><span class="o">(</span><span class="n">current</span><span class="o">()))</span> <span class="o">{</span>
<span class="n">token</span> <span class="o">=</span> <span class="n">consume</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">infix</span> <span class="o">=</span> <span class="n">grammar</span><span class="o">.</span><span class="na">getInfixParser</span><span class="o">(</span><span class="n">token</span><span class="o">);</span>
<span class="n">left</span> <span class="o">=</span> <span class="n">infix</span><span class="o">.</span><span class="na">parse</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">left</span><span class="o">,</span> <span class="n">token</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">left</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">current</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">current</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">consume</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">ParseException</span> <span class="o">{</span>
<span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">result</span> <span class="o">=</span> <span class="n">current</span><span class="o">;</span>
<span class="n">current</span> <span class="o">=</span> <span class="n">lexer</span><span class="o">.</span><span class="na">next</span><span class="o">();</span>
<span class="k">return</span> <span class="n">result</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>All the magic happens in the parseExpression method.</p>
<p>Given the current token, it fetches an appropriate prefix parser. Prefix parsers recognize simple expressions (such as literals, unary operators, delimited expressions, etc.). Then it goes to process infix parsers according to precedence (stickiness).</p>
<p>Pratt parsers are a variation of <a href="http://en.wikipedia.org/wiki/Recursive_descent_parser">recursive descent parsers</a>. The parseExpression methods represents a generalized rule in the grammar.</p>
<p>At this point you’re thinking there must be more to this. The trick must be in the Grammar class:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* omitted */</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Grammar</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">Map</span><span class="o"><</span><span class="no">T</span><span class="o">,</span> <span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">>></span> <span class="n">prefixParsers</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o"><</span><span class="no">T</span><span class="o">,</span> <span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">>>();</span>
<span class="kd">private</span> <span class="nc">Map</span><span class="o"><</span><span class="no">T</span><span class="o">,</span> <span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">>></span> <span class="n">infixParsers</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o"><</span><span class="no">T</span><span class="o">,</span> <span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">>>();</span>
<span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">getPrefixParser</span><span class="o">(</span><span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">prefixParsers</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">token</span><span class="o">.</span><span class="na">getType</span><span class="o">());</span>
<span class="o">}</span>
<span class="kt">int</span> <span class="nf">getStickiness</span><span class="o">(</span><span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">InfixParser</span> <span class="n">infixParser</span> <span class="o">=</span> <span class="n">getInfixParser</span><span class="o">(</span><span class="n">token</span><span class="o">);</span>
<span class="k">return</span> <span class="n">infixParser</span> <span class="o">==</span> <span class="kc">null</span><span class="o">?</span><span class="nc">Integer</span><span class="o">.</span><span class="na">MIN_VALUE</span><span class="o">:</span><span class="n">infixParser</span><span class="o">.</span><span class="na">getStickiness</span><span class="o">();</span>
<span class="o">}</span>
<span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">getInfixParser</span><span class="o">(</span><span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">infixParsers</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">token</span><span class="o">.</span><span class="na">getType</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">infix</span><span class="o">(</span><span class="no">T</span> <span class="n">ttype</span><span class="o">,</span> <span class="kt">int</span> <span class="n">stickiness</span><span class="o">)</span>
<span class="o">{</span>
<span class="n">infix</span><span class="o">(</span><span class="n">ttype</span><span class="o">,</span> <span class="k">new</span> <span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">stickiness</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">infix</span><span class="o">(</span><span class="no">T</span> <span class="n">ttype</span><span class="o">,</span> <span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="n">infixParsers</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">ttype</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">unary</span><span class="o">(</span><span class="no">T</span> <span class="n">ttype</span><span class="o">,</span> <span class="kt">int</span> <span class="n">stickiness</span><span class="o">)</span>
<span class="o">{</span>
<span class="n">prefixParsers</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">ttype</span><span class="o">,</span> <span class="k">new</span> <span class="nc">UnaryParser</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">stickiness</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">literal</span><span class="o">(</span><span class="no">T</span> <span class="n">ttype</span><span class="o">)</span>
<span class="o">{</span>
<span class="n">prefix</span><span class="o">(</span><span class="n">ttype</span><span class="o">,</span> <span class="k">new</span> <span class="nc">LiteralParser</span><span class="o"><</span><span class="no">T</span><span class="o">>());</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">prefix</span><span class="o">(</span><span class="no">T</span> <span class="n">ttype</span><span class="o">,</span> <span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
<span class="n">prefixParsers</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">ttype</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">delimited</span><span class="o">(</span><span class="no">T</span> <span class="n">left</span><span class="o">,</span> <span class="no">T</span> <span class="n">right</span><span class="o">,</span> <span class="kt">int</span> <span class="n">subExpStickiness</span><span class="o">)</span> <span class="o">{</span>
<span class="n">prefixParsers</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="k">new</span> <span class="nc">DelimitedParser</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">right</span><span class="o">,</span> <span class="n">subExpStickiness</span><span class="o">,</span> <span class="kc">true</span><span class="o">));</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">clarifying</span><span class="o">(</span><span class="no">T</span> <span class="n">left</span><span class="o">,</span> <span class="no">T</span> <span class="n">right</span><span class="o">,</span> <span class="kt">int</span> <span class="n">subExpStickiness</span><span class="o">)</span> <span class="o">{</span>
<span class="n">prefixParsers</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="k">new</span> <span class="nc">DelimitedParser</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">right</span><span class="o">,</span> <span class="n">subExpStickiness</span><span class="o">,</span> <span class="kc">false</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Nope. Just a couple of maps and some factory methods.</p>
<p>Even the infix and prefix parsers are rather simple:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">InfixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">stickiness</span><span class="o">;</span>
<span class="kd">protected</span> <span class="nf">InfixParser</span><span class="o">(</span><span class="kt">int</span> <span class="n">stickiness</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">stickiness</span> <span class="o">=</span> <span class="n">stickiness</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Expr</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">parse</span><span class="o">(</span><span class="nc">PrattParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">prattParser</span><span class="o">,</span> <span class="nc">Expr</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">left</span><span class="o">,</span> <span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span><span class="o">)</span>
<span class="kd">throws</span> <span class="nc">ParseException</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">BinaryExpr</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">token</span><span class="o">,</span> <span class="n">left</span><span class="o">,</span> <span class="n">prattParser</span><span class="o">.</span><span class="na">parseExpression</span><span class="o">(</span><span class="n">getStickiness</span><span class="o">()));</span>
<span class="o">}</span>
<span class="kd">protected</span> <span class="kt">int</span> <span class="nf">getStickiness</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">stickiness</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">LiteralParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span>
<span class="kd">extends</span> <span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">Expr</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">parse</span><span class="o">(</span><span class="nc">PrattParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">prattParser</span><span class="o">,</span> <span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span><span class="o">)</span>
<span class="kd">throws</span> <span class="nc">ParseException</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">ConstantExpr</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">token</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">UnaryParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span>
<span class="kd">extends</span> <span class="nc">PrefixParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">stickiness</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">UnaryParser</span><span class="o">(</span><span class="kt">int</span> <span class="n">stickiness</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">stickiness</span> <span class="o">=</span> <span class="n">stickiness</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Expr</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="nf">parse</span><span class="o">(</span><span class="nc">PrattParser</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">prattParser</span><span class="o">,</span> <span class="nc">Token</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">token</span><span class="o">)</span>
<span class="kd">throws</span> <span class="nc">ParseException</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">UnaryExpr</span><span class="o"><</span><span class="no">T</span><span class="o">>(</span><span class="n">token</span><span class="o">,</span> <span class="n">prattParser</span><span class="o">.</span><span class="na">parseExpression</span><span class="o">(</span><span class="n">stickiness</span><span class="o">));</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The infix and prefix parsers, just build an AST node. They recursively parse sub-expressions if necessary. If you want to check how delimited expressions work, you can <a href="https://github.com/juancn/performance">browse the code in github</a>.</p>
<p>These parsers have several interesting characteristics. One one of them is that the grammar can be modified at runtime (even though it’s not shown here) by adding/removing parsers, even while parsing. You can also easily add conditional grammars for sub-languages (think embedded SQL for example).</p>
<p>The code shown here only supports an LL(1) grammar (if I’m not mistaken), but adding additional lookahead should allow for LL(k) grammars.</p>
<p>Another interesting fact is that they way the parser is extended (by adding infix/prefix parsers) naturally yields grammars without left recursion.</p>
<p>One thing to note is that in my simple expression language, I’m not syntactically restricting the types of sub-expressions that infix operators receive, so that has to be checked in a later stage.</p>
<p>The only downside I can think of (besides the LL(k)-ness), is that these parsers are heavily geared towards expressions (everything is an expressions), but with some creativity statements could be added. For example, you could treat the semicolon in Java/C/C++/etc. as an infix operator.</p>
<p>Feel free to take all this code as yours for any purpose whatsoever. Happy hacking!</p>Some time ago I came across Pratt parsers. I had never seen them before, and I found them quite elegant.Performance Invariants - Part II2011-02-25T23:04:00+00:002011-02-25T23:04:00+00:00https://juancn.github.io/post/2011/02/25/performance-invariants-part-ii<p>A few days ago I wrote <a href="/post/2011/02/22/performance-invariants">a post about performance invariants</a>. The basic idea behind them, is that there should be an easy way to declare performance constraints at the source code level, and that you should be able to check them every time you run your unit tests. To make a long story short, I have been a busy little bee for the last few days and managed to build <a href="http://github.com/juancn/performance">a reasonable proof-of-concept</a>.</p>
<p>Let’s start with simple example:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">performance.annotation.Expect</span><span class="o">;</span>
<span class="o">...</span>
<span class="kd">class</span> <span class="nc">Test</span> <span class="o">{</span>
<span class="nd">@Expect</span><span class="o">(</span><span class="s">"InputStream.read == 0"</span><span class="o">)</span>
<span class="kd">static</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">list</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">//...</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>What we’re asserting here is that we want to make sure that the number of calls to methods called read defined in classes named InputStream should be exactly zero.</p>
<p>If we want to exclude basically all IO, we can change the expectation to:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">performance.annotation.Expect</span><span class="o">;</span>
<span class="o">...</span>
<span class="kd">class</span> <span class="nc">Test</span> <span class="o">{</span>
<span class="nd">@Expect</span><span class="o">(</span><span class="s">"InputStream.read == 0 && OutputStream.write == 0"</span><span class="o">)</span>
<span class="kd">static</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">list</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">//...</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Note that these are checked even for code that is called indirectly by the method process.</p>
<p>If we add an innocent looking <code class="language-plaintext highlighter-rouge">println</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nd">@Expect</span><span class="o">(</span><span class="s">"InputStream.read == 0 && OutputStream.write == 0"</span><span class="o">)</span>
<span class="kd">static</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">list</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hi!"</span><span class="o">);</span>
<span class="c1">//...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And run it with the agent enabled by using:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~>java -javaagent:./performance-1.0-SNAPSHOT-jar-with-dependencies.jar \
-Xbootclasspath/a:./performance-1.0-SNAPSHOT-jar-with-dependencies.jar Test
You should get something like the following output:
Hi!
Exception in thread "main" java.lang.AssertionError: Method 'Test.process' did not fulfil: InputStream.read == 0 && OutputStream.write == 0
Matched: [#OutputStream.write=7, #InputStream.read=0]
Dynamic: []
at performance.runtime.PerformanceExpectation.validate(PerformanceExpectation.java:69)
at performance.runtime.ThreadHelper.endExpectation(ThreadHelper.java:52)
at performance.runtime.Helper.endExpectation(Helper.java:61)
at Test.process(Test.java:17)
at Test.main(Test.java:39)
</code></pre></div></div>
<p>This is witchraft, I say! … well kind of.</p>
<p>Let’s stop a moment and consider what’s going on here. Notice the first line of the output. It contains the text “Hi!” that we printed. This happens because the check is performed after the method process finishes. In the fourth line, you can see how many times each method matched during the execution of the process method. Ignore the “Dynamic” list for just a second.</p>
<p>Let’s try something a bit more interesting:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">class</span> <span class="nc">Customer</span> <span class="o">{</span> <span class="cm">/*... */</span><span class="o">}</span>
<span class="c1">//...</span>
<span class="nd">@Expect</span><span class="o">(</span><span class="s">"Statement.executeUpdate < ${customers.size}"</span><span class="o">)</span>
<span class="kt">void</span> <span class="nf">storeCustomers</span><span class="o">(</span><span class="nc">List</span><span class="o"><</span><span class="nc">Customer</span><span class="o">></span> <span class="n">customers</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">//...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Note the <code class="language-plaintext highlighter-rouge">${customers.size}</code> in the expression, what this intuitively mean is that we want to take the size of the list as an upper bound. It’s like the poor programmer’s big-O notation. If we were to run this, but assuming that we execute two updates for each customer (instead of one as asserted), we would get:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception in thread "main" java.lang.AssertionError: Method 'Test.storeCustomers' did not fulfil: Statement.executeUpdate < ${customers.size}
Matched: [#Statement.executeUpdate=50]
Dynamic: [customers.size=25.0]
at performance.runtime.PerformanceExpectation.validate(PerformanceExpectation.java:69)
at performance.runtime.ThreadHelper.endExpectation(ThreadHelper.java:52)
at performance.runtime.Helper.endExpectation(Helper.java:61)
at Test.storeCustomers(Test.java:19)
at Test.main(Test.java:42)
</code></pre></div></div>
<p>Check the third line, this time, the “Dynamic” list contains the length of the list. In general, expressions of the form <code class="language-plaintext highlighter-rouge">${a.b.c.d}</code> are called dynamic values. They refer to arguments, instance variables or static variables. For example:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">${static.CONSTANT}</code> refers to a variable named <code class="language-plaintext highlighter-rouge">CONSTANT</code> in the current class.</li>
<li><code class="language-plaintext highlighter-rouge">${this.instance}</code> refers to a variable named <code class="language-plaintext highlighter-rouge">instance</code> in the current object (only valid for instance methods).</li>
<li><code class="language-plaintext highlighter-rouge">${n}</code> refers to an argument named <code class="language-plaintext highlighter-rouge">n</code> (this only works if the class has debug information)</li>
<li><code class="language-plaintext highlighter-rouge">${3}</code> refers to the fourth argument from the left (zero based indexing)</li>
</ul>
<p>All dynamic values MUST yield a numeric value, otherwise a failure will be reported at runtime. Currently the library will complain if any dynamic value is null.</p>
<p>Although this is an early implementation, it is enough to start implementing performance invariants that can be checked every time you run your unit tests.
Enough for today, in a followup post I’ll go into the internals of the agent. If you want to browse the source code or try it out, <a href="http://github.com/juancn/performance">go and grab a copy from github</a>.</p>A few days ago I wrote a post about performance invariants. The basic idea behind them, is that there should be an easy way to declare performance constraints at the source code level, and that you should be able to check them every time you run your unit tests. To make a long story short, I have been a busy little bee for the last few days and managed to build a reasonable proof-of-concept.Performance Invariants2011-02-22T15:30:00+00:002011-02-22T15:30:00+00:00https://juancn.github.io/post/2011/02/22/performance-invariants<p><strong>UPDATE</strong>: A newer post on this subject <a href="/post/2011/02/25/performance-invariants-part-ii">can be found here</a></p>
<p>Let’s start with a problem: How do you make unit tests that test for performance?</p>
<p>It might seem simple, but consider that:</p>
<ul>
<li>Test must be stable across hardware/software configurations</li>
<li>Machine workload should not affect results (at least on normal situations)</li>
</ul>
<p>A friend of mine (Fernando Rodriguez-Olivera if you must know) thought of the following (among many other things):
For each test run, record interesting metrics, such as:</p>
<ul>
<li>specific method calls</li>
<li>number of queries executed</li>
<li>number of I/O operations</li>
<li>etc.</li>
</ul>
<p>And after the test run, assert that these values are under a certain threshold. If they’re not, fail the test.
He even implemented a proof-of-concept using <a href="https://beanshell.github.io/">BeanShell</a> to record these stats to a file during the test run, and it would check the constraints after the fact.</p>
<p>Yesterday I was going over these ideas while preparing a presentation on code quality and something just clicked: annotate methods with performance invariants.
The concept is similar to pre/post conditions. Each annotation is basically a post condition on the method call that states which performance “promises” the method makes.</p>
<p>For example you should be able to do something like:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Ensure</span><span class="o">(</span><span class="s">"queryCount <= 1"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">CustomerInfo</span> <span class="nf">loadCustomerInfo</span><span class="o">()</span> <span class="o">{...}</span>
</code></pre></div></div>
<p>Or maybe something like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Ensure</span><span class="o">(</span><span class="s">"count(java.lang.Comparable.compareTo) < ceil(log(this.customers.size()))"</span><span class="o">)</span>
<span class="kd">public</span> <span class="nc">CustomerInfo</span> <span class="nf">findById</span><span class="o">(</span><span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{...}</span>
</code></pre></div></div>
<p>These promises are enabled only during testing since checking for them might be a bit expensive for a production system.</p>
<p>As this is quite recent I don’t have anything working (yet), but I think it’s worth exploring.
If I manage to find some time to try and build it, I’ll post some updates here.</p>UPDATE: A newer post on this subject can be found here