Implementing Reagent Test Case for JS Framework Benchmark

As part of preparation for Reagent 0.8 release, and inspired by this comparison, one thing I did was to implement Reagent test case for Stefan Krause's immensely popular JS web framework benchmark project.

This post documents the work that was required to add Reagent to the project, hopefully helping others to add other ClojureScript libraries there. I'll go through some of the choices I made writing the test code, and the open questions I have about Reagent performance.

Building ClojureScriptLink to Building ClojureScript

As this was the first ClojureScript library added to the project, some work was required to get Lein working with the Node -based build pipeline.

I decided to just use Leiningen and Cljsbuild, as this is what Reagent also uses. Because the benchmark is JavaScript -based, the scripts use npm to download deps and run the build. To make downloading Lein as simple as possible, I created lein-bin, taking example from sbt-bin, which provides few lines of JS wrapper code which will download correct Lein script depending on the operating system. Once the app is built, Webdriver-ts is used to run and benchmark the code.

Reagent Test ApplicationLink to Reagent Test Application

The test application needs just needs to render a table with rows with ID and random label, and provide a few buttons to update the data on the table. I based the test application on the React version. Similar to that, I wrote one file which provides functions to manage the data:

(ns demo.utils)

(def adjectives ["pretty", "large", ...])
(def colours ["red", "yellow", ...])
(def nouns ["table", "chair", ...])

(defrecord Data [id label])

(defn build-data [id-atom count]
  (repeatedly count (fn []
                      (->Data (swap! id-atom inc)
                              (str (rand-nth adjectives) " "
                                   (rand-nth colours) " "
                                   (rand-nth nouns))))))

The first function generates provided number of new rows. The ID for each row is retrieved from an atom, that keeps track of the last used number. Data is stored as records, which should be a bit faster than hash maps, to compare and to retrieve properties.

(defn add [data id-atom]
  (into data (build-data id-atom 1000)))

The second function is for adding 1000 new rows to existing list. The important bit here is that build-data returns a lazy seq, instead of vector, so it isn't realized unnecessarily before into.

(defn update-some [data]
  (reduce
    (fn [data index]
      (let [row (get data index)
            s (str (:label row) " !!!")]
        (assoc data index (assoc row :label s))))
    data
    (range 0 (count data) 10)))

One test case is updating every 10th element in the list. The data is stored in a vector, and the vector elements can be updated using assoc. To only modify the necessary elements, reduce can be used with range. Instead of using update or update-in, using assoc directly is faster.

(defn swap-rows [data]
  (if (> (count data) 998)
    (-> data
        (assoc 1 (get data 998))
        (assoc 998 (get data 1)))
    data))

Another test case is swapping two rows in the list. This is easy to do with Cljs, with persistent data structures we don't even need the temporary value used in JS.

(defn delete-row [data id]
  (vec (remove #(identical? id (:id %)) data)))

To remove a single item from the list, I tested several approaches, but it looks like just simple remove and casting back to vector is the fastest, even though it has to always go through the whole list. Using identical? should be a tiny bit faster than =, which calls some additional checks before calling identical?.

On the component side, I implemented two components, main which holds the state on two atoms, and row which is used to render each table row:

(defn row [data selected? on-click on-delete]
  [:tr
   {:class (if selected? "danger")}
   [:td.col-md-1 (:id data)]
   [:td.col-md-4
    [:a {:on-click (fn [e] (on-click (:id data)))}
     (:label data)]]
   [:td.col-md-1
    [:a {:on-click (fn [e] (on-delete (:id data)))}
     [:span.glyphicon.glyphicon-remove
      {:aria-hidden "true"}]]]
   [:td.col-md-6]])

Compared to React implementation, I chose to use multiple function parameters, instead of single properties map containing all the values. This should save some time on map construction and destructuring, but I don't have any proper stats about this. The precision of this benchmark wasn't quite enough to see differences between such changes, so this would need to be investigated further using Benchmark.js and/or Chrome performance profiler.

Another thing I'd like to know, is how expensive it is to create click event handler functions each time this function is called.

The main component is quite long, as it renders all the buttons for test cases, and implements click handlers for those, but the important part is the local state:

data (r/atom [])
selected (r/atom nil)

And the loop to render the table rows:

(let [s @selected]
  (for [d @data]
    ^{:key (:id d)}
    [row
     d
     (identical? (:id d) s)
     select
     delete]))

Similar to remove row function, this also uses identical? to check if the row is active.

ResultsLink to Results

The results are now online. I have included results for Reagent along with comparison to vanilla JS and React here:

The result for each performance test is quite similar to React, with small overhead, which is excepted. Comparison to React with Immutable.js would be interesting, to see how much of the overhead is due to ClojureScript data structures, and which is due to JS emitted by ClojureScript compiler.

One can also note that the standard deviation in some test cases is quite large, even much larger than with React, especially with partial update case. One guess is that persistent data structures and garbage collection might affect this.

On startup speed and output size, there is a bit bigger difference to React. One thing that should improve this a bit in the future, is when React can be used as Node module, which will allow Closure to optimize it.

Memory allocation results look... normal?

SummaryLink to Summary

I think the results are quite good. This simple benchmark mostly tests the rendering speed, where ClojureScript overhead will be obvious. Real world applications should benefit from persistent data structures, which should negate the overhead.

I haven't really given much thought to Reagent performance with some of the recent Reagent changes. Based on what I learned with this benchmark, I have some ideas on how to test Reagent performance in the future.

In addition to improving Reagent performance by optimizing the implementation, it should be also possible to improve the render performance by using alternative Hiccup compiler, like Hicada, which compiles the Hiccup forms into React calls during macro compilation, instead of interpreting them on runtime.

Reagent 0.8 will be released soon when everything is ready. I'm waiting for ClojureScript release with a fix which will prevent regression when using foreign-libs on Node target.