Clojure.spec with Ring (& Swagger)
This is the third post in a blog series about clojure.spec, showing how it can be used with Ring to produce input & output validation & api-docs via Swagger. We use Compojure-api as an example.
Other posts:
Validating client inputLink to Validating client input
In Ring, the client input parameters are located in the Request Map. Depending on the middleware used, parameters are found under keys like :query-params, :header-params, :path-params and :body-params. By default, Ring doesn't provide any parameter coercion, so most non-body parameters are just Strings. :body-params can have richer types, depending on the format and decoder being used (e.g. JSON vs EDN & Transit).
Here is a naive Ring handler that adds two numbers together via :query-params:
(require '[ring.http-response :refer [ok]])
(defn handler
"Throws if parameters can't be coerced to numbers"
[{{:keys [x y]} :query-params :as req}]
(if (and (= :get (:request-method req))
(= "/plus" (:uri req)))
(ok {:total (+ (Long/parseLong x)
(Long/parseLong y))})))
Same using Compojure with custom coercion:
(require '[compojure.core :refer [GET]])
(require '[compojure.coercions :refer [as-int]])
(GET "/plus" [x :<< as-int, y :<< as-int]
(ok {:total (+ x y)}))
Compojure-api with Schema coercion:
(require '[compojure.api.core :refer [GET]])
(GET "/plus" []
:query-params [x :- Long, y :- Long]
:return {:total Long}
(ok {:total (+ x y)}))
With compojure-api we can also apply response Schema validation and instead of having just opaque coercion functions, all defined Schemas are available at runtime as data. Evaluating the compojure-api route at REPL yields the internal Route definition:
#Route{:path "/plus",
:method :get,
:info {:public
{:parameters
{:query {:x java.lang.Long
:y java.lang.Long
Keyword Any}
:responses
{200 {:schema
{:total java.lang.Long}}}}}}}
The request & response coercion flow with Schema is roughly the following:
- Decode the incoming request and populate request parameters
- When a Schema-enforced route is matched, coerce the defined request parameters against the defined Schemas with a suitable
coercion-matcher- based on parameter type & request body format.string-coercion-matcheris used for all string-based formats,json-coercion-matcherfor JSON,just-validate-matcherfor EDN and Transit. - invoke the actual request handler with coerced types
- Check response route Schema requirements for the route and coerce them too if requested.
If the request or response coercion fails, compojure-api returns HTTP status 400 or 500 is returned with a descriptive body.
{
"schema": "{Keyword Any, :x Int, :y Int}",
"errors": {
"y": "(not (integer? \"a\"))"
},
"type": "compojure.api.exception/request-validation",
"coercion": "schema",
"value": {
"x": "1",
"y": "a"
},
"in": [
"request",
"query-params"
]
}
Abstracting the CoercionLink to Abstracting the Coercion
To support clojure.spec, coercion was redesigned to be pluggable. There is a new Protocol, Coercion, responsible for all all the work related to the defined models, including request & response coercion, error-formatting and api-doc generation. Coercion implementation can be bound for the whole api, context or endpoints. Coercion implementation include SchemaCoercion and SpecCoercion.
(defprotocol Coercion
(get-name [this])
(get-apidocs [this model data])
(make-open [this model])
(encode-error [this error])
(coerce-request [this model value type format request])
(coerce-response [this model value type format request]))
By default, compojure-api uses SchemaCoercion, so apps look exactly the same as before:
(ns c2.schema
(:require [compojure.api.sweet :refer [context GET resource]]
[ring.util.http-response :refer [ok]]
[schema.core :as s]))
(s/defschema Total
{:total s/Int})
(def routes
(context "/schema" []
:tags ["schema"]
(GET "/plus" []
:summary "plus with schema"
:query-params [x :- s/Int, {y :- s/Int 0}]
:return Total
(ok {:total (+ x y)}))
(context "/data-plus" []
(resource
{:post
{:summary "data-driven plus with schema"
:parameters {:body-params {:x s/Str, :y s/Str}}
:responses {200 {:schema Total}}
:handler (fn [{{:keys [x y]} :body-params}]
(ok {:total (+ x y)}))}}))))
Invoking the api with HTTPie we see that the coercion works:
> http :3000/schema/plus x==1 y==2
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Sun, 02 Jul 2017 08:43:02 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked
{
"total": 3
}
Enter SpecLink to Enter Spec
Thanks to Spec-tools, we can apply runtime spec coercion mostly just like with Schema. The new 0.3.0 version also supports transforming specs into Swagger2 specs, so we get the api-docs too. The swagger-code is merged from spec-swagger, as the whole transformation code ended up being less than 150 lines of code on top of the already existing JSON Schema transformation. Spec-tools will later be integrated as module to ring-swagger.
Like Schema, Spec is more powerful than Swagger Schema, so we are losing some data in translation, but we retain all data using Swagger Vendor Extensions.
(require '[spec-tools.swagger.core :as swagger])
(require '[clojure.spec.alpha :as s])
(swagger/transform (s/alt :int int? :str string?))
; {:type "integer"
; :format "int64"
; :x-anyOf [{:type "integer"
; :format "int64"}
; {:type "string"}]}
For coercion, there is a catch: before Spec supports directly selective runtime conforming (go vote it up), we need to wrap all Specs into Spec Records in order to use the runtime coercion. Without wrapping, we only get spec validation, but not coercion.
Here's the same app with clojure.spec:
(ns c2.spec
(:require [compojure.api.sweet :refer [context GET resource]]
[ring.util.http-response :refer [ok]]
[clojure.spec.alpha :as s]
[spec-tools.spec :as spec]))
;; wrap as Spec Records
(s/def ::x spec/int?)
(s/def ::y spec/int?)
(s/def ::total spec/int?)
(s/def ::total-map (s/keys :req-un [::total]))
(def routes
(context "/spec" []
:tags ["spec"]
;; bind SpecCoercion for all subroutes
;; we can use just :spec here thanks to
;; compojure.api.coercion.core/named-coercion
;; multimethod
:coercion :spec
(GET "/plus" []
:summary "plus with clojure.spec"
;; both x & y are coerced from string->long
:query-params [x :- ::x, {y :- ::y 0}]
:return ::total-map
(ok {:total (+ x y)}))
(context "/data-plus" []
(resource
{:post
{:summary "data-driven plus with clojure.spec"
;; no coercion done as all formats can send numbers
:parameters {:body-params (s/keys :req-un [::x ::y])}
:responses {200 {:schema ::total-map}}
:handler (fn [{{:keys [x y]} :body-params}]
(ok {:total (+ x y)}))}}))))
The produced swagger-ui for the routes:
From command line:
> http :3000/spec/plus x==1 y==2
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Sun, 02 Jul 2017 08:44:02 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked
{
"total": 3
}
Coercion failures are reported just like Schema ones:
> http POST :3000/spec/data-plus x:=1 y=2
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Date: Sun, 02 Jul 2017 08:45:12 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked
{
"coercion": "spec",
"in": [
"request",
"body-params"
],
"problems": [
{
"in": [
"y"
],
"path": [
"y"
],
"pred": "clojure.core/int?",
"val": "2",
"via": [
"c2.spec/y"
]
}
],
"spec": "(spec-tools.core/spec {:spec (clojure.spec.alpha/keys :req-un [:c2.spec/x :c2.spec/y]), :type :map, :keys #{:y :x}})",
"type": "compojure.api.exception/request-validation",
"value": {
"x": 1,
"y": "2"
}
}
Closed spec keysetsLink to Closed spec keysets
s/keys are open by design and all qualified keys are validated, even if they are not defined in the Spec.
(s/def ::int int?)
(s/def ::kw keyword?)
(s/valid?
(s/keys :req [::int])
{::int 1, ::kw "kikka6"})
; false <-- eh.
This is not what we want and because of that, the SpecCoercion is using spec-tools.conform/strip-extra-keys-type-conforming to strip efficiently keys that are not part of s/keys Specs. To enable this, s/keys need to be wrapped into Spec Records. All the top-level specs are automatically wrapped by SpecCoercion.
> http POST :3000/spec/data-plus x:=1 y:=2 c2/total=INVALID
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Sat, 01 Jul 2017 22:56:52 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked
{
"total": 3
}
As opposed to Spec, Schemas are closed by default:
> http POST :3000/schema/data-plus x:=1 y:=2 c2/total=INVALID
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Date: Sun, 02 Jul 2017 05:49:57 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked
{
"coercion": "schema",
"errors": {
"c2/total": "disallowed-key"
},
"in": [
"request",
"body-params"
],
"schema": "{:x Int, :y Int}",
"type": "compojure.api.exception/request-validation",
"value": {
"c2/total": "INVALID",
"x": 1,
"y": 2
}
}
Configuring coercionLink to Configuring coercion
Coercion implementations are designed for easy customization. For example, to disable response coercion with SpecCoercion just dissociate the :response key from the options.
(require '[compojure.api.coercion.spec :as cs])
(def no-response-coercion
(cs/create-coercion
(dissoc
cs/default-options
:response)))
(context "/spec" []
:coercion no-response-coercion
...)
The SpecCoercion default options look like this:
{:body {:default cs/default-conforming
:formats {"application/json" json-conforming
"application/msgpack" json-conforming
"application/x-yaml" json-conforming}}
:string {:default string-conforming}
:response {:default default-conforming}})
ConclusionLink to Conclusion
With a help of Spec-tools, clojure.spec can be used as a generic coercion & api-doc solution for Ring/HTTP just like Schema is used today. Compojure-api (2.0.0-alpha5) ships with a new Coercion abstraction providing a the needed lifecycle hooks to integrate clojure.spec (or any other lib) into request/response processing pipeline. That also could be extracted out into a separate lib if other web libs find it usable. And as clojure.spec is still in alpha, so are these libs, and subject to change.
The Schema & Spec apis in the post are found in an example repository on Github.
Off to vacation.
Tommi
