Schema & Clojure Spec for the Web Developer

This post will walk through the main differences of Schema and Spec from the viewpoint of a Clojure(Script) web developer. There is also some thinking aloud how we could achieve best of both worlds and peek into some evolving libraries. This is a first part of a blog series.

Clojure SpecLink to Clojure Spec

Clojure Spec is a new Clojure(Script) core library for specifying Clojure applications & data for communication, validation, parsing and generative testing. It is similar to Plumatic Schema but also has some cool new features like spec destructuring, multispecs and a inbuilt serialization format. Spec is still in alpha, and will ship with Clojure 1.9.0. There is a great introduction talk by Arne Brasseur from ClojuTRE 2016. Carin Meier's talk Genetic programming with clojure.spec on EuroClojure 2016 was a mind-blower, sadly the video is not on Internet.

SchemaLink to Schema

We at Metosin are big fans of Schema. For the last three years, it has enabled us to build robust and beautifully documented apps both for both Clojure & ClojureScript. Many of our open source libs have been built on top of Schema. These include ring-swagger, compojure-api and kekkonen. There is also others like pedestal-api and yada using Schema currently.

DifferencesLink to Differences

This post is not a complete comparison of the two, but instead highlights some of the key differences that a normal Clojure Web Developer (like me!) would see in the daily work: how to define and transform models (both at design- and runtime), validating/transforming values from external sources, api-docs and getting human-readable error messages for the end users. Things like function specs/schemas and generative testing are left out.

Defining the modelsLink to Defining the models

SchemaLink to Schema

With Schema, models are defined as Clojure data structures and Schema predicates or Java Classes. Schema maps are closed by default, allowing no extra keys. Schemas are easy to reason about as they are defined in the same form as the values it represents. Errors are presented in a hybrid of human/machine -readable format.

(ns user.schema)

(require '[schema.core :as s])

(def age
  (s/constrained s/Int #(> % 18) 'over-18))

(s/defschema Address
  {:street s/Str
   :zip s/Str})

(s/defschema Person
  {::id s/Int
   :age age
   :name s/Str
   :likes {s/Str s/Bool}
   (s/optional-key :languages) #{s/Keyword}
   :address Address})

(def liisa
  {::id 1
   :age 63
   :name "Liisa"
   :likes {"coffee" true
           "maksapihvi" false}
   :languages #{:clj :cljs}
   :address {:street "Amurinkatu 2"
             :zip "33210"}})

(s/check Person liisa) ; => nil

(s/check Person {:age "17", :bogus "kikka"})
; {:user.schema/id missing-required-key,
;  :age (not (integer? "17")),
;  :name missing-required-key,
;  :likes missing-required-key,
;  :address missing-required-key,
;  :bogus disallowed-key}

Reusing schemas is done either by predefining common parts of it (like the age and address above) or by transforming existing schemas. As the schemas as just data, transformations can also be done at runtime. For more complex transformations, there are external libraries like Schema Tools and Schema-bijections.

;; reuse at compile-time
(s/defschema PersonView
  (select-keys
    Person
    [::id :likes :address]))

(s/check
  PersonView
  (select-keys
    liisa
    [::id :likes :address])) ; => nil

;; reuse at runtime!
(let [keys [::id :likes :address]]
  (s/check
    (select-keys Person keys)
    (select-keys liisa keys))) ; => nil

Above Schemas visualized with schema-viz:

SpecLink to Spec

With Spec, models are defined using clojure.spec macros and function predicates. Maps are defined using keysets instead of key-value pairs. All map keys need to be globally registered. Calling s/form on any given spec returns the original source code for it, and should later enable spec serialization. Errors are reported in machine-readable format.

(ns user.spec)

(require '[clojure.spec :as s])

(s/def ::id integer?)
(s/def ::age (s/and integer? #(> % 18)))
(s/def ::name string?)
(s/def ::likes (s/map-of string? boolean?))
(s/def :user.address/street string?)
(s/def :user.address/zip string?)

(s/def ::languages
  (s/coll-of keyword? :into #{}))

(s/def ::address
  (s/keys :req-un [:user.address/street
                   :user.address/zip]))

(s/def ::person
  (s/keys :req [::id]
          :req-un [::age
                   ::name
                   ::likes
                   ::address]
          :opt-un [::languages]))

(def liisa
  {::id 1
   :age 63
   :name "Liisa"
   :likes {"coffee" true
           "maksapihvi" false}
   :languages #{:clj :cljs}
   :address {:street "Amurinkatu 2"
             :zip "33210"}})

(s/valid? ::person liisa) ; => true

(s/explain-data
  ::person {:age "17", :bogus "kikka"})
; {:clojure.spec/problems
;  ({:in [], :path [],
;    :pred (contains? % :user.spec/id),
;    :val {:age "17",
;          :bogus "kikka"},
;    :via [:user.spec/person]}
;    {:in [], :path [],
;     :pred (contains? % :name),
;     :val {:age "17",
;           :bogus "kikka"},
;     :via [:user.spec/person]}
;    {:in [], :path [],
;     :pred (contains? % :likes),
;     :val {:age "17",
;           :bogus "kikka"},
;     :via [:user.spec/person]}
;    {:in [], :path [],
;     :pred (contains? % :address),
;     :val {:age "17",
;           :bogus "kikka"},
;     :via [:user.spec/person]}
;    {:in [:age], :path [:age],
;     :pred integer?, :val "17",
;     :via [:user.spec/person
;           :user.spec/age]})}

Spec promotes application level reuse as all the specs are found in the registry. New specs can be composed with the clojure.spec macros like and, or and merge. Due to use of macros, creating specs at runtime is not easy - and would pollute the global spec registry. Spec is still young but there is already many evolving utility libs for it. We are doing the Spec Tools and there is at least Schpec and Spectrum out there.

;; reuse specs at compile-time
(s/def ::person-view
  (s/keys :req [::id]
          :req-un [::likes :address]))

(s/valid?
  ::person-view
  (select-keys
    liisa
    [::id :likes :address])) ; => true

;; runtime (bad idea but works)
(let [req-keys [::id]
      req-un-keys [::likes ::address]
      value-keys [::id :likes :address]]
  (s/valid?
    (eval
      `(s/keys :req ~req-keys
               :req-un ~req-un-keys))
    (select-keys
      liisa value-keys))) ; => true

Transforming valuesLink to Transforming values

For web app runtime, it's important to be able to both validate/conform values from external sources. Different wire-formats have different capabilities for presenting types. In string-based formats (like ring :query-params & :path-params) all values have to be represented and parsed from Strings. JSON supports maps, vectors, numbers, strings, booleans and null, but not for example Dates or Keywords. Both EDN and Transit can be extended to support any kind of values.

SchemaLink to Schema

In Schema, there is coercion. Given a Schema and a separate matcher at runtime, one can validate and transform values from different formats into Clojure data. Schema ships with matchers for both string and json formats. Matchers can be easily extended.

(require '[schema.coerce :as sc])

;; define a transformation function
(def json->Person
  (sc/coercer
    Person
    sc/json-coercion-matcher))

;; :languages from [s/Str] => #{s/Keyword}
(json->Person
  {::id 1
   :age 63
   :name "Liisa"
   :likes {"coffee" true
           "maksapihvi" false}
   :languages ["clj" "cljs"]
   :address {:street "Amurinkatu 2"
             :zip "33210"}})
; {:user.schema/id 1,
;  :age 63,
;  :name "Liisa",
;  :likes {"coffee" true,
;          "maksapihvi" false},
;  :languages #{:clj :cljs},
;  :address {:street "Amurinkatu 2"
;            :zip "33210"}}

SpecLink to Spec

Spec has a conform, which works like coercion but the transforming function is directly attached to the Spec instead of passed in at runtime. Because of this, it's not suitable for runtime-driven transformations.

(s/def ::str-keyword
  (s/and
    (s/conformer
      (fn [x]
        (if (string? x)
          (keyword x)
          x)))
    keyword?))

(s/conform ::str-keyword "clj") ; => :clj
(s/conform ::str-keyword :clj) ; => :clj

To support more Schema-like runtime conformations, we have the following options:

1. Better clojure.spec/conformLink to 1. Better clojure.spec/conform

Current conform takes only the spec and a value as arguments (s/conform spec x). It could have a 3-arity version where we could pass in a runtime provided callback function to selectively conform based on the spec value (s/conform spec x spec->conformer-fn). I have been mumbling about this in the Clojure Slack & in Google Groups. Would be simple, but not likely going to happen.

2. Dynamic conformingLink to 2. Dynamic conforming

Clojure has the Dynamic Scope, which could be used to pass conforming callback to the conformer function at runtime. The Conformer could read this value and conform accordingly. This requires a special "dynamic conformer" to be attached to all specs. Default operation would be no-op. There could also be set of predefined "Type Predicates" which would have a dynamic conformer attached. There could be a special dynamic-conform to set set the variable and call vanilla conform.

There is an implementation of this in Spec-tools, more about that in the next part of this blog.

3. Generate differently conforming SpecsLink to 3. Generate differently conforming Specs

Specs could be walked with clojure.spec/form generating (and registering) differently conforming specs for all conforming modes. All the new specs would have to have new generated names, e.g. :user/id => :user$JSON/id. Seems easy and elegant, but there are few challenges on the way:

• Due to the current implementation of s/keys, fully qualified spec keys can't be exposed this way. In Spec, by design, the keys and the values are not separate and thus a qualified spec key can't be mapped to multiple, differently conforming versions. I tried to create a modified version of s/keys for this but ended up copying most of the clojure.spec to properly support it. Maybe later.

• the s/form has a nasty bug in alpha-14, some specs still emit non-qualified forms. This should be fixed soon.

4. Create an extra layer of "Easy Data Specs"Link to 4. Create an extra layer of "Easy Data Specs"

One could invent a new and more data-driven format having it's own mechanisms for runtime conforming. But - adding a "easy" abstraction layer comes with a cost and most likely will backfire eventually. For now at least, it's good to work with Specs directly, as we are all still learning.

5. Generating Schemas from SpecsLink to 5. Generating Schemas from Specs

Tried this too, was a bad idea: there would be two sets of errors messages depending on where it was raised. It's better to have Specs (or Schema) all the way down.

Api-docsLink to Api-docs

This is important. With Schema, we have tools like ring-swagger, which transforms nested Schemas into Swagger JSON Schema enabling beautiful api-docs.

;; [metosin/ring-swagger "0.22.12"]
(require '[ring.swagger.swagger2 :as rs])

(rs/swagger-json
  {:paths
   {"/echo-person"
    {:post
     {:summary "Echoes a person"
      :parameters {:body Person}
      :responses {200 {:schema Person}}}}}})
; ... valid swagger spec returned

For Spec, there aren't any finalized solution for this yet. Andrew Mcveigh is working on something and we have a Spec -> JSON Schema transformer in Spec-tools, but it's not complete yet and has some hacks while waiting for the core s/form - bug to be fixed. The Swagger transformations can be used separately and plan is for it to be eventually merged into ring-swagger for easy transition. Something like (but with qualified keys?):

(require '[spec-tools.swagger :as swagger])

(swagger/swagger-object
  {:paths
   {"/echo-person"
    {:post
     {:summary "Echoes a person"
      :parameters {:body ::person}
      :responses {200 {:schema ::person}}}}}})

Human-readable error messagesLink to Human-readable error messages

Neither of the two libraries has solved this one for good. There are some promising experiments out there, looking forward to seeing them mature and get integrated into tooling.

ConclusionLink to Conclusion

As per today, Schema is a proven solution for building robust runtime-validating web apps and is not going away. There is good set of existing web libs using it already providing both runtime coercion & api-docs. Schema can be used in the ClojureScript for things like dynamic form and server request validation. We have been using Schema in most of our projects and will continue to use and support it in our libs.

Spec is awesome and without a doubt will be de facto data description library for Clojure. For now, the runtime conforming & api-docs story is not on par with Schema but it will be, eventually. Not all web apps need currently the runtime conforming feature, end2end Clojure(Script) apps can transfer data in Transit using just runtime validation instead of conforming. Also, for Spec, we have to remember that it's still in Alpha, so things might change.

Road aheadLink to Road ahead

Spec is still under development and there are lot of community libs evolving around it. We too are building tools to help adopting Spec for web & api development, more on spec-tools & friends on Part2. Our web-libs will support spec as soon as.

Exciting times to be a Clojure(Script) web developer :)

Tommi (@ikitommi)