kamera
Visual testing tools for Clojure with figwheel-main and devcards integration.
Give kamera some reference images and your devcards build and get automatic screenshots and comparisons to your references of all your devcards. If you don't use figwheel or devcards, kamera can accept a list of urls for you to roll your own.
Why?
When data is represented visually for a human to view you must take to present it intuitively, accessibly and beautifully. This requires skill, time and above all human judgement.
Once achieved you want to ensure it does not suffer regressions. kamera is a library designed to help you capture and compare screenshots of your application, failing if there is too much divergence between an expected reference image and the current display and creating a difference image to show you where.
The best way to test visual representation is to create devcards which you can use to display components in as many states as possible. If you ensure you separate rendering from business logic you can ensure that refactoring will not affect them and prevent them becoming brittle - I outlined this approach in a blog post for JUXT.
Prerequesites
kamera uses ImageMagick for image processing and Chrome to capture screenshots. By default it looks for them on the path, but you can supply paths if they reside somewhere else - see the options. kamera lets you choose the metric for image comparison - read more about the choices here.
Usage
figwheel + devcards
See the example project for a full working example.
The following assumes a figwheel-main build called dev which has devcards that you view at /devcards.html.
If you have the following devcards files:
test
└── example
├── another_core_test.cljs
└── core_test.cljs... and a directory populated with reference images, named in the same way:
test-resources
└── kamera
├── example.another_core_test.png
└── example.core_test.png
You can generate these images initially by running kamera and copying the 'actual' files from the target directory into your reference directory
... you can get kamera to screenshot the devcards and compare with the corresponding reference images with the following:
(ns example.devcards-test
(:require [kamera.devcards :as kd]
[clojure.test :refer [deftest testing is]]))
(deftest devcards-test
(kd/test-devcards "dev" kd/default-opts))The output will look like this:
Results
example.kamera-test
1 non-passing tests:
Fail in devcards-test
#!/example.another_core_test
example.another_core_test.png has diverged from reference by 0.020624, please compare
Expected: test-resources/kamera/example.another_core_test.png
Actual: target/kamera/example.another_core_test.png
Difference: target/kamera/example.another_core_test-difference.png
expected: (< metric metric-threshold)
actual: (not (< 0.020624 0.01))The target directory will contain an expected, actual and difference image for every devcard. It will also contain an html report which presents the juxtaposed images, the normalisation steps and the difference compared to the threshold.
Core API
If you don't use figwheel or devcards you can still use kamera to take screenshots and compare them to reference images.
You will have to provide a list of "targets" for kamera to test.
Each target must provide a :url and :reference-file and can override any setting from the :default-target options.
(require '[kamera.core :as k])
(k/run-tests [{:url "http://localhost:9500/"
:reference-file "home.png"}
{:url "http://localhost:9500/preferences"
:reference-file "preferences.png"
:metric-threshold 0.2}
{:url "http://localhost:9500/help"
:reference-file "help.png"
:metric "RMSE"
:normalisations [:trim]}])Options
{:default-target ;; default options for each image comparison
{:metric "mae" ;; the imagemagick metric to use for comparison
;; see https://imagemagick.org/script/command-line-options.php#metric
:metric-threshold 0.01 ;; difference metric above which comparison fails
:reference-directory "test-resources/kamera" ;; directory where reference images are store
:screenshot-directory "target/kamera" ;; directory where screenshots and diffs should be saved
:ready? (fn [session] ... ) ;; predicate that should return true when screenshot can be taken
;; see element-exists? as an example
:normalisations [:trim :crop] ;; normalisations to apply to images before comparison, in order of application
:assert? true ;; runs a clojure.test assert on the expected/actual when true, makes no assertions when false
:resize-to-contents {:height? true ;; resize browser window dimensions to fit contents before screenshot - true for both is legacy behaviour
:width? false
:dom-selector "body"} ;; which dom element dimensions are used for the resize
:normalisation-fns ;; normalisation functions, add your own if desired
{:trim trim-images
:crop crop-images}
:imagemagick-options
{:path nil ;; directory where binaries reside on linux, or executable on windows
:timeout 2000} ;; kill imagemagick calls that exceed this time, in ms
:chrome-options ;; options passed to chrome, letting you turn headless on/off etc
;; see https://github.com/tatut/clj-chrome-devtools/blob/master/src/clj_chrome_devtools/automation/launcher.clj#L52
{:chrome-binary "/opt/bin/google-chrome-stable"
:headless? true
:extra-chrome-args ["--window-size=1600,900"
"--hide-scrollbars"]}
}devcards options
A few additional options exist if you are using the kamera.devcards namespace:
{:devcards-options
{:path "devcards.html" ;; the relative path to the page where the devcards are hosted
:init-hook (fn [session] ... ) ;; function run before attempting to scrape targets
:on-targets (fn [targets] ... )} ;; function called to allow changing the targets before the test is run
}Normalisation
When comparing images ImageMagick requires both input images to be the same dimensions. They can easily differ when changes are made to your application, across operating systems or browser versions. Normalisation is the process of resizing both the expected and the actual images in a way that keeps the images lined up with one another for the best comparison.
The built-in normalisations are trim and crop.
The former cuts out whitespace around image content and the latter crops the image canvas.
They are run sequentially and each stage is output to the target directory, giving a set of images as follows:
kamera
├── example.core_test.actual.png
├── example.core_test.actual.trimmed.cropped.png
├── example.core_test.actual.trimmed.png
├── example.core_test.expected.difference.png
├── example.core_test.expected.png
├── example.core_test.expected.trimmed.cropped.png
└── example.core_test.expected.trimmed.pngYou can override the normalisations to each image, perhaps adding a resize:
{:normalisations [:trim :resize :crop]}And provide the resize function in the options map:
{:normalisation-fns {:trim trim-fn
:crop crop-fn
:resize resize-fn}}The signature of resize should look like this:
(defn resize-images [^File expected ^File actual opts])And it should return [expected actual]. See the existing trim and crop functions for inspiration.
Example use cases
Here are some example use cases you may wish to consider in addition to the standard ones given above:
Desktop / tablet / mobile testing
(ns example.devcards-test
(:require [kamera.devcards :as kd]
[clojure.test :refer [deftest testing is]]))
(deftest desktop-test
(kd/test-devcards
"dev"
(-> kd/default-opts
(update :default-target merge
{:reference-directory "test-resources/kamera/desktop"}))))
(deftest tablet-test
(kd/test-devcards
"dev"
(-> kd/default-opts
(update :default-target merge
{:reference-directory "test-resources/kamera/tablet"})
(assoc-in [:chrome-options :chrome-args] ["--headless" "--window-size=1024,768"]))))
(deftest mobile-test
(kd/test-devcards
"dev"
(-> kd/default-opts
(update :default-target merge
{:reference-directory "test-resources/kamera/mobile"})
(assoc-in [:chrome-options :chrome-args] ["--headless" "--window-size=800,600"]))))Spot comparison during a webdriver test
(ns example.devcards-test
(:require [kamera.core :as k]
[clojure.test :refer [deftest testing is]]))
(deftest my-user-acceptance-test
(let [driver (init-driver {:host "localhost" :port 9500})]
;;; webdriver stuff happens ...
(navigate! driver "/")
(k/run-test {:url (.getUrl driver)
:reference-file "homepage-with-cookies-banner.png"}
k/default-opts)
(click! driver "#accept-cookies")
(k/run-test {:url (.getUrl driver)
:reference-file "homepage-cookies-accepted.png"}
k/default-opts)
(.quit driver)))
Development
Start a normal clj&cljs repl.
You will need sassc for building the sass via lein sass auto.
cljs tests: http://localhost:9500/figwheel-extra-main/auto-testing devcards: http://localhost:9500/cards.html
License
Copyright © 2018 oliyh
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.