Tommi Reiman
Technology

Clojure.spec as a Runtime Transformation Engine

This is the second post in the blog series about clojure.spec for web development and introduces spec-tools - a new Clojure(Script) library that adds some batteries to clojure.spec: extendable spec records, dynamic conforming, spec transformations and more.

Clojure.specLink to Clojure.spec

Clojure.spec is a new modeling and validation library for Clojure(Script) developers. Its main target is design and development time, but it can also be used for runtime validation. Runtime value transformations are not in it's scope and this is something we need for the web apps as described in the previous post.

We would like to use clojure.spec both for application core models and runtime boundary validation & transformations. Let's try to solve this.

GoalsLink to Goals

  1. Dynamic runtime validation & transformation
  2. Spec transformations, to JSON Schema & OpenAPI
  3. (Simple data-driven syntax for specs)

SolutionLink to Solution

Spec-tools is a small new library aiming to achieve the goals. It takes ideas from Plumatic Schema, including the clean separation of specs from conformers. Spec-tools targets both Clojure & ClojureScript and the plan is to make it compatible with Self-hosted ClojureScript as well. The README covers most of the features and options, while this post walks through the core concepts and the reasoning behind them.

Spec RecordsLink to Spec Records

As per today (alpha-16), Specs in clojure.spec are implemented using reified Protocols and because of that they are non-trivial to extend. To allow extensions, spec-tools introduces Spec Records. They wrap the vanilla specs and can act as both as specs and predicate functions. They also enable features like dynamic runtime conforming and extra spec documentation. Spec Records have a set of special keys, which include :spec, :form, :type, :name, :description, :gen, :keys and :reason. Any qualified keys can be added for own purposes.

Simplest way to create Spec Records is to use spec-tools-core/spec macro.

(require '[spec-tools.core :as st])
(require '[clojure.spec :as s])

(def x-int? (st/spec int?))

x-int?
; #Spec{:type :long
;       :form clojure.core/int?}

(x-int? 10)
; true

(s/valid? x-int? 10)
; true

(assoc x-int? :description "It's an int")
; #Spec{:type :long,
;       :form clojure.core/int?,
;       :description "It's an int"}

Optionally there is a map-syntax:

;; simple predicate
(s/def ::name (st/spec string?))

;; map-syntax with extra info
(s/def ::age
  (st/spec
    {:spec integer?
     :description "Age on a person"
     :json-schema/default 20}))

(s/def ::person
  (st/spec
    {:spec (s/keys :req-un [::name ::age])))
     :description "a Person"}))

(s/valid? ::person {:name "Tommi", :age 42})
; true

Instead of using the spec macro, one can also use the underlying create-spec function. With it, you need to pass also the :form for the spec. For most clojure.core predicates, spec-tools can infer the form using resolve-form multi-method.

(let [data {:name "bool"}]
  (st/create-spec
    (assoc data :spec boolean?))
; #Spec{:type :boolean,
;       :form clojure.core/boolean?
;       :name "bool"}

To save on typing, spec-tools.spec contains most clojure.core predicates wrapped as Spec Record instances:

(require '[spec-tools.spec :as spec])

spec/boolean?
; #Spec{:type :boolean
;       :form clojure.core/boolean?}

(spec/boolean? true)
; true

(s/valid? spec/boolean? false)
; true

(assoc spec/boolean? :name "truth")
; #Spec{:type :boolean
;       :form clojure.core/boolean?
;       :name "truth"}

Dynamic conformingLink to Dynamic conforming

The primary goal is to support dynamic runtime validation & value transformations. Same data read from JSON, Transit or string-based formats (:query-params etc.) should conform differently. For example Transit supports Keywords, while with JSON we have to conform keywords from strings. In clojure.spec, conformers are attached to spec instances so we would have to rewrite differently conforming specs for all different formats.

Spec-tools separates specs from conforming. spec-tools.core has own versions of explain, explain-data, conform and conform! which take a extra argument, a conforming callback. It is a function of type spec => spec value => (or conformed-value ::s/invalid) and is passed to Spec Record's s/conform* via a private dynamic Var. If the conforming is bound, it is called with the Spec Record enabling arbitrary transformations based on the Spec Record's data. There is CLJ-2116 to allow same without the dynamic binding. If you like the idea, go vote it up.

Example of conforming that increments all int? values:

(defn inc-ints [_]
  (fn [_ value]
    (if (int? value)
      (inc value)
      value)))

(st/conform spec/int? 1 nil)
; 1

(st/conform spec/int? 1 inc-ints)
; 2

Type-conformingLink to Type-conforming

Spec-tools ships with type-based conforming implementation, which selects the conform function based on the :type of the Spec. Just like :form, :type is mostly auto-resolved with help of spec-tools.type/resolve-type multimethod.

The following predefined type-conforming instances are found in spec-tools.core:

  • string-conforming - Conforms specs from strings.
  • json-conforming - JSON Conforming (numbers and booleans not conformed).
  • strip-extra-keys-conforming - Strips out extra keys of s/keys Specs.
  • fail-on-extra-keys-conforming - Fails if s/keys Specs have extra keys.

Example:

(s/def ::age (s/and spec/int? #(> % 18)))

;; no conforming
(s/conform ::age "20")
(st/conform ::age "20")
(st/conform ::age "20" nil)
; ::s/invalid

;; json-conforming
(st/conform ::age "20" st/json-conforming)
; ::s/invalid

;; string-conforming
(st/conform ::age "20" st/string-conforming)
; 20

type-conforming-mappings are just data so it's easy to extend and combine them.

(require '[spec-tools.conform :as conform])

(def strip-extra-keys-json-conforming
  (st/type-conforming
    (merge
      conform/json-type-conforming
      conform/strip-extra-keys-type-conforming)))

Map-conformingLink to Map-conforming

s/keys are open by design: there can be extra keys in the map and all keys are validated. This is not good for runtime boundaries: JSON clients might send extra data we don't want to enter the system and writing extra keys to database might cause a runtime exception. We don't want to manually pre-validate the data before validating it with spec.

When Spec Record is created, the wrapped spec is analyzed via the spec-tools.core/collect-info multimethod. For s/keys specs, the keyset is extracted as :keys and thus is available for the :map type-conformer which can strip the extra keys efficiently.

(s/def ::street string?)
(s/def ::address (st/spec (s/keys :req-un [::street])))
(s/def ::user (st/spec (s/keys :req-un [::name ::street])))

(def inkeri
  {:name "Inkeri"
   :age 102
   :address {:street "Satamakatu"
             :city "Tampere"}})

(st/conform
  ::user
  inkeri
  st/strip-extra-keys-conforming)
; {:name "Inkeri"
;  :address {:street "Satamakatu"}}

Inspired by select-schema of Schema-tools, there are also a select-spec to achieve the same:

(st/select-spec ::user inkeri)
; {:name "Inkeri"
;  :address {:street "Satamakatu"}}

The actual underlying conform function is dead simple:

(defn strip-extra-keys [{:keys [keys]} x]
  (if (map? x)
    (select-keys x keys)
    x))

Data macrosLink to Data macros

One use case for conforming is to expand intermediate (and potentially invalid) data to conform specs, kind of like data macros. Let's walk through an example.

A spec describing entities in an imaginary database:

(s/def :db/ident qualified-keyword?)
(s/def :db/valueType #{:uuid :string})
(s/def :db/unique #{:identity :value})
(s/def :db/cardinality #{:one :many})
(s/def :db/doc string?)

(s/def :db/field
  (st/spec
    {:spec (s/keys
             :req [:db/ident
                   :db/valueType
                   :db/cardinality]
             :opt [:db/unique
                   :db/doc])
     ;; custom key for conforming
     ::type :db/field}))

(s/def :db/entity (s/+ :db/field))

It accepts values like this:

(def entity
  [{:db/ident :product/id
    :db/valueType :uuid
    :db/cardinality :one
    :db/unique :identity
    :db/doc "id"}
   {:db/ident :product/name
    :db/valueType :string
    :db/cardinality :one
    :db/doc "name"}])

(s/valid? :db/entity entity)
; true

We would like to have an alternative, simpler syntax for common case. Like this:

(def simple-entity
  [[:product/id :uuid :one :identity "id"]
   [:product/name :string "name"]])

A spec for the new format:

(s/def :simple/field
  (s/cat
    :db/ident :db/ident
    :db/valueType :db/valueType
    :db/cardinality (s/? :db/cardinality)
    :db/unique (s/? :db/unique)
    :db/doc (s/? :db/doc)))

(s/def :simple/entity
  (s/+ (s/spec :simple/field)))

All good:

(s/valid? :simple/entity simple-entity)
; true

But the database doesn't understand the new syntax. We need to transform values to conform the database spec. Let's write a custom conforming for it:

(defn db-conforming [{:keys [::type]}]
  (fn [_ value]
    (or
      ;; only specs with ::type :db/field
      (if (= type :db/field)
        ;; conform from :simple/field format
        (let [conformed (s/conform :simple/field value)]
          (if-not (= conformed ::s/invalid)
            ;; custom transformations
            (merge {:db/cardinality :one} conformed))))
      ;; defaulting to no-op
      value)))

(defn db-conform [x]
  (st/conform! :db/entity x db-conforming))

That's it. We now have a function, that accepts data in both formats and conforms it to database spec:

(db-conform entity)
; [#:db{:ident :product/id
;       :valueType :uuid
;       :cardinality :one
;       :unique :identity
;       :doc "id"}
;  #:db{:ident :product/name
;       :valueType :string
;       :cardinality :one
;       :doc "name"}]

(db-conform simple-entity)
; [#:db{:ident :product/id
;       :valueType :uuid
;       :cardinality :one
;       :unique :identity
;       :doc "id"}
;  #:db{:ident :product/name
;       :valueType :string
;       :cardinality :one
;       :doc "name"}]

(= (db-conform entity)
   (db-conform simple-entity))
; true

Dynamic conforming is a powerful tool and generic implementations like type-conforming give extra leverage for the runtime, especially for web development. conforming should be a first-class citizen, so we have to ensure that they easy to extend and to compose. Many things have already been solved in Schema and Schema-tools, so we'll pull more stuff as we go, for example a way to conform the default values.

Domain-specific data-macros are cool, but add some complexity due to the inversion of control. For just this reason, we have pulled out Schema-based domain coercions from some of our client projects. Use them wisely.

Spec transformationsLink to Spec transformations

Second goal is to be able to transform specs themselves, especially to convert specs to JSON Schema & Swagger/OpenAPI formats. First, we need to be able to parse the specs and there is s/form just for that. Spec Records also produce valid forms so all the extra data is persisted too.

(s/def ::age
  (st/spec
    {:spec integer?
     :description "Age on a person"
     :json-schema/default 20}))

(s/form ::age)
; (spec-tools.core/spec
;  clojure.core/integer?
;  {:type :long
;   :description "Age on a person"
;   :json-schema/default 20})

(eval (s/form ::age))
; #Spec{:type :long
;       :form clojure.core/integer?
;       :description "Age on a person"
;       :json-schema/default 20}

VisitorsLink to Visitors

spec-tools has an implementation of the Visitor Pattern for recursively walking over spec forms. A multimethod spec-tools.visitor/visit takes a spec and a 3-arity function which gets called for all the nested specs with a dispatch key, spec and vector of visited children as arguments.

Here's a simple visitor that collects all registered spec forms linked to a spec:

(require '[spec-tools.visitor :as visitor])

(let [specs (atom {})]
  (visitor/visit
    :db/entity
    (fn [_ spec _]
      (if-let [s (s/get-spec spec)]
        (swap! specs assoc spec (s/form s))
        @specs))))
; #:db{:ident clojure.core/qualified-keyword?
;      :valueType #{:string :uuid}
;      :cardinality #{:one :many}
;      :unique #{:identity :value}
;      :doc clojure.core/string?
;      :field (spec-tools.core/spec
;              (clojure.spec/keys
;                :req [:db/ident :db/valueType :db/cardinality]
;                :opt [:db/unique :db/doc])
;              {:type :map
;               :user/type :db/field
;               :keys #{:db/unique :db/valueType :db/cardinality :db/doc :db/ident}})
;      :entity (clojure.spec/+ :db/field)}

Currently, s/& and s/keys* specs can't be visited due to a bug the spec forms.

JSON SchemaLink to JSON Schema

Specs can be transformed into JSON Schema using the spec-tools.json-schema/transform. Internally it uses the visitor and spec-tools.json-schema/accept-spec multimethod to do the transformations.

(require '[spec-tools.json-schema :as json-schema])

(json-schema/transform :db/entity)
; {:type "array",
;  :items {:type "object",
;          :properties {"db/ident" {:type "string"},
;                       "db/valueType" {:enum [:string :uuid]},
;                       "db/cardinality" {:enum [:one :many]},
;                       "db/unique" {:enum [:identity :value]},
;                       "db/doc" {:type "string"}},
;          :required ["db/ident" "db/valueType" "db/cardinality"]},
;  :minItems 1}

With Spec Records, :name gets translated into :title, :description is copied as-is and all qualified keys with namespace json-schema will be added as unqualified into generated JSON Schemas.

(json-schema/transform
  (st/spec
    {:spec integer?
     :name "integer"
     :description "it's an int"
     :json-schema/default 42
     :json-schema/readOnly true}))
; {:type "integer"
;  :title "integer"
;  :description "it's an int"
;  :default 42
;  :readOnly true}

Swagger/OpenAPILink to Swagger/OpenAPI

There is almost a year old issue for supporting clojure.spec aside Plumatic Schema in ring-swagger. Now as the dynamic conforming and JSON Schema transformation works, it should be easy to finalize. Plan is to have a separate spec-swagger just for clojure.spec with identical contract for the actual web/routing libs. This would allow easy transition between Schema & Spec while keeping the dependencies on the spec-side on minimum. Some ideas from spec-tools will also flow back to schema-tools, some thoughts on a gist.

FutureLink to Future

As clojure.spec is more powerful than the OpenAPI spec, we lose some data in the transformation. For end-to-end Clojure(Script) systems, we could build a totally new api and spec documentation system with "spec-ui" ClojureScript components on top. We have been building CQRS-based apps for years and having a generic embeddable ui for actions is a valuable feature. Also, if specs could be read back from their s/form without eval, we could build dynamic systems where the specs could be loaded over the wire from database to the browser. For development time, there should be a Graphviz-based visualization for Specs, just like there is the Schema-viz.

ConclusionLink to Conclusion

By separating specs (what) and conforming (how) we can make clojure.spec a real runtime transformation engine. Spec-tools is a library on top of clojure.spec adding Spec Records, runtime value transformations via dynamic conforming and Spec transformations (including to JSON Schema) with the Spec visitor. It's designed to be extendable via data and multimethods. There are more features like the data-specs, more on those on the upcoming posts. As a final note, as clojure.spec is still in alpha, so is spec-tools.

Give it a spin and tell us what you think.

Tommi Reiman

Related posts

More

Contact