Felipeand I have been busy again, read all aboutwhat we’ve been up to in this month’s Lambda Island Open Source update.
We currently have two major projects we are working to get out: Regal andKaocha-cljs2. They are both still a work in progress, but we’ve again made somegood progress this month. Regal is not officially out yet but is at the stagewhere people can safely start incorporating it into their projects.Kaocha-cljs2 is a big undertaking, but we’re splitting this work into smallerself-contained pieces, and two of those saw their first release this month: Chuiand Funnel.
Regalis a regular expression Swissarmy knife, built around “regal syntax”, a way of representing regexes asClojure data structures, similar to what Hiccup syntax does for HTML.
For instance the regex#"(\w+)=(\w+)"would look like this:
[:cat [:capture [:+ :word]] "=" [:capture [:+ :word]]]
Regal can convert regal forms to regexes, but also parse regexes back to regal.It can create generators for use withtest.check, and integrates withclojure.spec orMalli.
Regal also knows about different regex flavors, it currently supportsJavaScript, Java <= version 8, or >= version 9, and can parse and generateregexes for each flavor. This means the semantics of a given regal form areconstant, it will always match the same strings, no matter the platform. It alsomeans you can do things like convert JavaScript regexes so you can use them inClojure.
All the major pieces of Regal of there, and we don’t expect the syntax of Regalforms to change going forward. The tricky part is that we have multiple piecesof code that encode the semantics of these regal forms. The regex generator, theparser, the test.check generators, as well as specs for validating, and possiblygenerating regal forms. These all need to match up.
Most of the recent changes have been about fine tuning these differentimplementations. We might come across a regex that trips up the parser, or caseswhere generating and then parsing a regex doesn’t yield the same result.
So far we’ve dealt with this by having data-driventest-casesthat cover all the regex features we support. These are used in multiple ways totest the different parts of Regal, and we still have some more cases we need tocheck for to make sure all our bases are covered.
But the best way to check that multiple implementations match is throughgenerative testing. We were unable to do this so far because we weren’t able toreliably generate regal expressions. Their recursive nature means thatgenerators easily overflow the stack. However we’vemanaged to work aroundthis,and so the road to more property-based testing is now open.
Kaocha seeks to be a complete solution to all your Clojure testing needs, and ofcourse that includes ClojureScript. When we came out with Kaocha-cljs thisdrastically lowered the bar for people to start writing tests in ClojureScript.All you needed was a single line in yourtests.edn, and things just worked …
#kaocha/v1{:tests [{:type :kaocha.type/cljs}]}… unless they didn’t. Dealing with ClojureScript is inherintly complex. Youcan’t just look for tests on the filesystem and run them. The ClojureScriptcompiler needs to be involved to compile your tests to JavaScript, and then aJavaScript runtime is needed to execute them, and meanwhile Kaocha needs to stayin control, deciding which tests to run, and collecting their results.
Kaocha-cljs managed to push that complexity away by building on top ofClojureScript’s repl-env abstraction, in particular the REPL implementations forthe browser and for Node.js. These repl-envs take care of launching a JavaScriptruntime and communicating with it. They also handle the ClojureScriptcompilation, so that we can just send snippets of code their way, and they willcompile them, send them on to the browser or node, and notify us of the result.
In a way this worked better than we had expected, and many people are quitecontent with what kaocha-cljs currently offers, but we had some persistentcomplaints.
People who used Shadow-cljs were out of luck, shadow has its own compilationpipeline and in particular differs from standard ClojureScript in how it dealswith npm packages. That means that code written for shadow-cljs may not becompatible with the built-in repl-envs.
There’s a wide spectrum of how people set up their ClojureScript compilation. Wedid allow people to specify their own compiler settings, but they had to becompatible with what the repl-envs expected. You can’t for instance use advancedcompilation. Given how many problems only show up when using:advancedthis isa legitimate ask, and something we were unable to support.
Finally the repl-envs internals are opaque and hard to debug. People routinelyhad trouble getting them to work headless on CI, without any meaningful feedbackor error message to diagnose the issue.
So we went back to the drawing board. We started by creating a new ClojureScripttest runner,Chui. Chui lets you runspecific tests on demand, dynamically at runtime. This is different from therunner that ClojureScript ships with, which determines which tests to run atcompile time through macros.
chui-coreprovides the test runner API. On top of that we builtchui-ui, abrowser based UI for running tests and inspecting their results.
We also builtchui-remote, a small components which allows controllingchui-coreover a websocket. This means that by includingchui-remoteintoyour build it will be able to talk to Kaocha.
This gets us out of the business of invoking the ClojureScript compiler, or ofstarting a JavaScript runtime. People can use whatever tool they like(cljs.main, Figwheel, Shadow-cljs), whatever compiler settings they like, andwhatever runtime they like.
To facilitate the communication between Kaocha and Chui, and to make it easierto do things like reuse existing browser tabs, we createdFunnel, you can read all about Funnelin itsrelease announcement(ClojureVerse).
This is all starting to come together. Kaocha-cljs2 is not ready yet forprimetime, but the main parts are implemented.
All of this does mean that using Kaocha-cljs2 will be a bit more involved interms of setup, at least in the beginning. Once the dust settles we may lookinto providing conveniences for common scenarios. The big win however is that byseparating these different responsibilities we can cater for many more usecases.
Last but not least this month also saw a small release of