Spec Transformers

clojure.spec is a data specification library for Clojure, introduced almost two years ago. It's still in alpha.

I hope you find spec useful and powerful. - Rich Hickey

This is my fourth post about spec, describing a new way to transform data based on spec definitions. Earlier posts were about differences to Schema, spec coercion and using specs with ring & swagger.

Revisiting conformingLink to Revisiting conforming

clojure.spec allows us to conform a value to a Spec:

(require '[clojure.spec.alpha :as s])

(s/conform int? 1)
; => 1

(s/conform int? "1")
; => ::s/invalid

If the world was built only with Clojure/Script, EDN and Transit, this would be all the coercion we would ever need. But in real life, data is represented in many different formats such as JSON and XML and we need to transform the values between the formats. For example, JSON doesn't support keywords.

Runtime transformations are out of scope of clojure.spec, and the current best practice is to use normal Clojure functions first to transform the data from external formats into correct format and then validate it using the Spec. This is really a bad idea, as the structure of the data needs to be copied from specs into custom transformation functions.

Spec-tools addresses this issue by separating values, specs and conforming. If the data is in String format, one can transform it using string-conforming, which uses Spec :type definitions to do the required transformations. To support this, specs need to be wrapped:

(require '[spec-tools.core :as st])

;; :type is automatically resolved
(st/conform (st/spec int?) "1")
; => ::s/invalid

(st/conform (st/spec int?) "1" st/string-conforming)
; => 1

It's still just one-way transformation (decoding) and adding a new transformation required a new :type to be registered into all conforming instances. The new :type vocabulary is even thought to be a bad idea.

BijectionsLink to Bijections

I chatted with some smart people like Gary Fredericks and Miikka Koskinen about two-way transformations using the Spec. Gary has a Schema Bijections library, which already allows two-way transformations for Plumatic Schema. As usual, I had no clue, so I had to google what Bijection actually meant.

In short, it's a symmetric 1-to-1 value transformation between domains:

X => Y => X

Mapping numbers between string-domain (e.g. properties files) and Clojure:

"1" => 1 => "1"

In some cases there are multiple valid representation for a single a value in a domain, so the properties of bijections do not hold. For example, dates in the JSON have multiple valid formats:

"2014-02-18T18:25:37Z"
"2014-02-18T18:25:37.000+0000"

Bijections are cool, but two-way transformations are what we want.

Welcome Spec Transformers!

TransformersLink to Transformers

In the just released [metosin/spec-tools "0.7.0"], the transformations have been rewritten, supporting now both two-way and spec-driven transformations.

The higher order function conforming is replaced with a Transformer Protocol, supporting both value encoding and decoding:

(defprotocol Transformer
  (-name [this])
  (-encoder [this spec value])
  (-decoder [this spec value]))

There are two new functions in spec-tools.core: encode and decode. They flow the value through both conform and unform effectively doing value transformation. In addition, all conforming functions now automatically coerce the spec argument into Spec using a IntoSpec protocol, so plain specs can be used too:

(st/decode int? "1")
; => ::s/invalid

(st/decode int? "1" st/string-transformer)
; => 1

IntoSpec is not recursive, so nested specs need still to be wrapped. CLJ-2116 and CLJ-2251 would offer solutions to fix this.

Round-tripping ISO-8601 date-times from JSON domain (x => Y => X => Y):

(as-> "2014-02-18T18:25:37Z" $
      (doto $ prn)
      (st/decode inst? $ st/json-transformer)
      (doto $ prn)
      (st/encode inst? $ st/json-transformer)
      (doto $ prn)
      (st/decode inst? $ st/json-transformer)
      (prn $))
; "2014-02-18T18:25:37Z"
; #inst "2014-02-18T18:25:37.000-00:00"
; "2014-02-18T18:25:37.000+0000"
; #inst "2014-02-18T18:25:37.000-00:00"

There are encoder and decoder functions for many basic types in spec-tools.transform, for both Clojure & ClojureScript.

Spec-driven transformationsLink to Spec-driven transformations

Another new feature is that we can use the Spec meta-data to define how values should be transformed in different domains. For this, there are two new key namespaces: encode and decode. Keys in those namespaces with the domain as a name should contain 2-arity transformer function of type spec value => value.

(require '[clojure.string :as str])

(defn lower-case? [x]
  (-> x name str/lower-case keyword (= x)))

(s/def ::spec
  (st/spec
    {:spec #(and (simple-keyword? %) (lower-case? %))
     :description "a lowercase keyword, encoded upper in string"
     ;; decoder for the string domain
     :decode/string #(-> %2 name str/lower-case keyword)
     ;; encoder for the string domain
     :encode/string #(-> %2 name str/upper-case)}))

(st/decode ::spec :kikka)
; :kikka

(as-> "KiKka" $
      (st/decode ::spec $))
; :clojure.spec.alpha/invalid

(as-> "KiKka" $
      (st/decode ::spec $ st/string-transformer))
; :kikka

(as-> "KiKka" $
      (st/decode ::spec $ st/string-transformer)
      (st/encode ::spec $ st/string-transformer))
; "KIKKA"

The default Transformer implementation first looks for a spec-level transformer, then a :type-level transformer.

When creating new specs, one can also use :type to get encoders, decoders and documentation for free, like with Data.Unjson (thanks to Fabrizio for the pointer).

(s/def ::kw
  (st/spec
    {:spec #(keyword %) ;; anonymous function
     :type :keyword}))  ;; encode & decode like a keyword

(st/decode ::kw "kikka" st/string-transformer)
;; :kikka

(st/decode ::kw "kikka" st/json-transformer)
;; :kikka

Final WordsLink to Final Words

Transforming specced values is not in scope of clojure.spec, but it should.

Meanwhile, Spec-tools provides now both spec- and type-driven transformations and two-way transformations as a proof-of-concept. Like clojure.spec, it's alpha and partially built on non-documented apis, which will most likely to be broken later. My 2 cents are that spec-tools will break, it will be easy to fix and we'll fix it. The best case would be if spec-tools was not needed for this and clojure.spec could do these things on its own.

Comments and feedback welcome in Clojureverse.

Tommi