Linköpings universitet
Linköping University | Department of Computer and Information Science
Master thesis, 30 ECTS | Computer Science
2017 | LIU-IDA/LITH-EX-A--17/043--SE
Evalua ng Clojure Spec
Utvärdering av Clojure Spec
Chris an Luckey
Supervisor : Bernhard Thiele Examiner : Christoph Kessler
De a dokument hålls llgängligt på Internet – eller dess fram da ersä are – under 25 år från pub-liceringsdatum under förutsä ning a inga extraordinära omständigheter uppstår. Tillgång ll doku-mentet innebär llstånd för var och en a läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och a använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrä en vid en senare dpunkt kan inte upphäva de a llstånd. All annan användning av doku-mentet kräver upphovsmannens medgivande. För a garantera äktheten, säkerheten och llgäng-ligheten finns lösningar av teknisk och administra v art. Upphovsmannens ideella rä innefa ar rä a bli nämnd som upphovsman i den omfa ning som god sed kräver vid användning av dokumentet på ovan beskrivna sä samt skydd mot a dokumentet ändras eller presenteras i sådan form eller i så-dant sammanhang som är kränkande för upphovsmannensli erära eller konstnärliga anseende eller egenart. För y erligare informa on om Linköping University Electronic Press se förlagets hemsida h p://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years star ng from the date of publica on barring excep onal circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educa onal purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are condi onal upon the consent of the copyright owner. The publisher has taken technical and administra ve measures to assure authen city, security and accessibility. According to intellectual property law the author has the right to be men oned when his/her work is accessed as described above and to be protected against infringement. For addi onal informa on about the Linköping University Electronic Press and its procedures for publica on and for assurance of document integrity, please refer to its www home page: h p://www.ep.liu.se/.
ABSTRACT
The objective of this thesis is to evaluate whether or not Clojure Spec meets the goals it sets out to meet with regards to easy data validation, performance and automatically generated tests in comparison to existing specification systems in the Clojure ecosystem.
A specification for a real-world data format was implemented in the three currently popular spec-ification systems used in Clojure. They were then compared on merits in terms of performance, code size and additional capabilities.
The results show that Spec shines with complex data, both in expressivity and validation perfor-mance, but has an API more complex than its competitors. For complex enough use cases where expressing regular data structures and generative testing is desired the time investment of learn-ing Spec pays off, in simpler situations an assertions library like Truss can be recommended.
Contents
Abstract iii
Acknowledgments v
Contents vi
List of Figures viii
List of Tables ix
1 Introduction 3
1.1 The objectives of Spec . . . 3
1.2 Aim . . . 4
1.3 Questions . . . 4
1.4 Scope . . . 5
2 Background 7 2.1 Definitions . . . 7
2.2 Code complexity and quality . . . 9
2.3 Related work . . . 11
2.4 Introduction to Clojure . . . 12
2.5 Introduction to Spec, Schema and Truss . . . 15
3 Method 23 3.1 Pre-study . . . 23
3.2 Data selection . . . 25
3.3 Filtering test data . . . 27
3.4 Writing specifications . . . 27 3.5 Benchmarking . . . 27 3.6 Effort reduction . . . 28 3.7 Edge cases . . . 28 4 Results 29 4.1 Effort reduction . . . 29 4.2 Performance . . . 30
4.5 Generating data from Clojure Spec . . . 34 5 Discussion 37 5.1 Method . . . 37 5.2 Results . . . 39 5.3 Writing specifications . . . 41 5.4 In a wider context . . . 52 6 Conclusion 53 Bibliography 56
A Statistical results from generating data with Spec 57
B Validation time broken down by keyword per system 61
List of Figures
4.1 Project file validation time summary . . . 31
4.2 Validation time broken down by whether the tested data is valid or not. . . 32
5.1 Validation time comparison for boolean value in a map. . . 40
5.2 Validation time for a dependency vector. . . 40
5.3 Validation time for a value with multiple options. . . 40
B.1 Validation time broken down by keyword for Truss . . . 62
B.2 Validation time broken down by keyword for Spec . . . 63
B.3 Validation time broken down by keyword for Schema . . . 64
B.4 Validation time broken down by keyword for plain Clojure validation . . . 65
C.1 Validation time by all systems for keys :aliases to :exclusions . . . 68
C.2 Validation time by all systems for keys :filespecs to :main . . . 69
C.3 Validation time by all systems for keys :managed-dependencies to :release-tasks . . 70
3.1 Downloads from Clojars per library. . . 25
3.2 Number of projects on Clojars that depend on the given library. . . 25
4.1 SLOC per specification implementation. . . 29
4.2 Statistical measures in ms for validation of project files. . . 31
4.3 Criteria comparison: Xmeans full, d partial and7 no support. . . . 34
4.4 Generation time in milliseconds from spec. . . 35
1
Introduction
Clojure Spec is an upcoming standard library of the programming language Clojure [48] for specifying the functionality of programs and the nature of data.
It allows the programmer to describe the structure and contents of data held in any combination of Closure’s data structures as well as that given to and returned from functions and macros. It also allows for the relation between the data given to and returned from a function or macro to be expressed. These specifications can then be used for data validation, higher level parsing, generative testing as well as to provide improved documentation and error messaging.
This work seeks to evaluate Clojure Spec in comparison to other competing specification systems as well as plain, normal Clojure code.
1.1
The objectives of Spec
Introduced in May 2016, but as of July 2017 yet to see a stable release, Spec seeks [10] to fill a lot of gaps in the Clojure ecosystem by:
• Documenting functions, macros, keywords, lists, arrays, maps and sets for both program-matic and human consumption.
• Reporting errors1 on the parsing and destructuring of data. • Providing run time data validation.
• Automatic destructuring and parsing. • Generating property based tests.
1Error messages being hard to understand has been the prime reason given by potential users of Clojure as to why they are not currently using Clojure, according to the Clojure survey. [44] [45]
• Generating test data for these tests.
Some of the terms used above are described further in the Definitions section.
The specifications themselves are available at application run time, including test time. They are not intended as mathematical proofs like those which type systems provide [10], although such applications exist2; instead the intention is to provide an environment where arbitrary validation of run time data is not only possible but easy to perform.
1.2
Aim
The aim of this thesis is to try to evaluate whether Clojure Spec succeeds in achieving some of the goals outlined in section 1.1. We do this partly through a measurement of time consumed in validating a real-world data set but also through measures of code quality and criteria based evaluation.
1.2.1
Classifying this study
Stol and Fitzgerald write in their “Holistic Overview of Software Engineering Research Strategies” [46] that “terminology is a challenge in research methodology, that there is no commonly adopted taxonomy to describe such”. Nevertheless, some key words as defined in their paper describes the approach of this study.
This thesis describes primary, quantitative and qualitative, desk research. It is a field study, an
exploratory case study. The target of the study is the common ways of implementing specifications
in the Clojure ecosystem.
1.3
Questions
The question asked in this paper is: To what degree and at what cost are the goals of Spec achieved? Specifically:
(1) How does the code of Spec specifications compare to equivalent code in competing systems, both in terms of plain SLOC and convenience of expression?
(2) How does Spec perform in real-world benchmarks compared to competing systems? (3) To which degree does Spec expose issues in data or functions not found by competing
systems, if at all?
The competing systems to Clojure Spec were deemed to be the existing data specification or assertion systems Schema [39] and Truss [5], and to not use any library at all.
A specification for the project declaration file of the highly popular build, project and depen-dency management tool Leiningen was produced using each system and compared to each other with regards to the research questions. A comparison was also made using the criteria of related work.
1.4. Scope
1.3.1
Formulating research questions
As Kitchenham et al. [28] write, a well-formulated research question should have three parts, with the focus on the first two.
1. A study factor, i.e. the technology. In this case Clojure Spec. 2. The population, i.e. the samples of code.
3. The outcomes.
They express how the technological abstraction should neither be to high nor to low, and that “For some questions it may be necessary to be even more precise e.g. Contract-based specifications
. . . ” which is precisely what this paper seeks to study.
1.4
Scope
With so many aspects to Spec there were many different approaches which were explored but not delved further into during the process of the thesis project.
1. By how much do the improved3 error messages given by Spec increase the speed at which a developer is able to fix the erring code?
If there are two versions of a function, one annotated with Spec and one not; which function’s error message helps in correcting the incorrect usage of the function faster? Does this change between different developer groups, for example newcomers vs. experienced clojurists? Does the complexity of the function or arguments have an impact?
Preliminary research was done on the subject, a method and theory chapter formulated, and the task of collecting test subjects was initiated. Though it was soon terminated based on advice from core Clojure developers [38] who relayed that the error messaging of Spec was still subject to large changes; that there was a good chance any results being invalidated by a new release.
2. To which degree does Spec expose edge cases not found by previous defensive programming, if at all?
If a library is annotated with Spec and the generated tests run for the library, how many new bugs are found in the library?
3Improved especially in terms of exactness. While a common Clojure error message reads ClassCastException java.lang.String cannot be cast to clojure.lang.IFn
an error given by Spec for the same issue could read
Call to #'ns/bar did not conform to spec: val: ("Some string") fails at: [:args :function] predicate: fn? :clojure.spec/args (1) :clojure.spec/failure :instrument :clojure.spec.test/caller {:file "lekplats.clj", :line 20, :var-scope ns/fn1}
This question, while interesting, would reasonably end up either being a case-study or too large time-wise considering how deeply a developer has to go in understanding how a program works in order to correctly specify the different kinds of data flowing through it. 3. How well does Spec perform as a parser?
Is it possible to express any regular language in Spec? How performant is it compared to a hand written parser?
With background in the fact that Clojure Spec has yet to be performance tuned [37], but foremost that language parsing in the sense of parsing a string is rather the work of a parser such as Instaparse [27], it made sense to not delve further into this question. 4. How much specification code is needed to cover the entire input of a function? Is there
a limit to how large a specification can be in order for it to stay performant? Is there a sweet spot between these two?
The answer to this question, it was reasoned, was going to vary so widely between cases that it would be hard to formulate any scientific method which would yield a result other than: “It depends.”
2
Background
This chapter will go over the theoretical underpinnings of this thesis starting with a short dictionary of terms and concepts. Following that, there are two short sections on how this paper fits into the world of computer science and on that follows theory or background sections on code quality and finding edge cases after which the reader will find a section on related work. Finally this chapter will bring the reader an introduction to Clojure and the specification systems dealt with in this paper.
2.1
Definitions
This section defines field specific nomenclature that the reader may find useful knowing about when reading this paper.
• Application Programming Interface (API), used in reference to the functions exposed by a library and the arguments they take together with the values they return.
• Artifact, a Maven [30] concept for a Java archive which is uniquely identified by a sequence of a group-id and a name followed by a version string. In Clojure these are expressed as [group-id/name "1.0.0"].
• Derivative, the derivative of a language L with respect to a character c is a new language that has been:
1. Filtered to only contain words with the character c. 2. Had the letter c cut out from every word. [7] [33]
• Destructuring, to destructure means to break down some structure into smaller compo-nents. Examples can be found in the Clojure Guide on Destructuring [15].
To parse a text in a regular language into a data structure can also be seen as a kind of destructuring.
• Domain Specific Language (DSL), a language defined for the specific purpose of solving problems in one domain.
• Example based testing, testing where the tests including input and expected output of a function are written by hand by a programmer.
• Keyword, a specific data type in Clojure which only ever represents itself,:apple is only ever evaluated to:apple. They are commonly used as keys in maps and may be used as a function that given a map returns the value held under itself in that map.
(:k {:k 5}) ; => 5
A Java programmer may see them as interned1strings with a function attached.
• Macros are function which transform a piece of code. [3] These are run at compile time as opposed to normal functions which run at run time.
• Plain Clojure, used to signify code written using only the standard libraries of Clojure. • Predicate function, a function which takes one or more arguments and returns a boolean value. Predicate functions are usually given names ending with a question mark such as seq? in Clojure.
• Property based test, testing where the tests are generated based on specifications as opposed to example based testing. Sometimes called generative testing.
Using property based testing it is possible to find edge cases in code that the author either did not account for or that comes out of mistakes [17].
• Regular expression (regex), consist of constants and operator symbols that denote sets of strings and certain operations over these sets, respectively. In this thesis the strict definition of the term regex will be used, i.e. that there is no notion of recollection. [24] About all modern implementations2 of regexes allow recollection with parentheses like (.+)\1 where \1 recalls what was read in the parenthesis. This is not allowed in the strict definition of regular expressions.
• Schema with capital S is used to refer to the library formerly called Prismatic Schema, nowadays Plumatic Schema [39]. Written with a lowercase s schema is used to refer to a specific schema for X.
1https://docs.oracle.com/javase/7/docs/api/java/lang/String.html#intern()
2In Clojure and this thesis the Java [35] flavour of string-based regex syntax will be used. In it ‘(’ starts a capture group, ‘.’ means any non-newline character, ‘+’ means one or more of the previous character class.
This thesis will deal with two syntaxes of regular expressions: The string-based expressions most programmers will find familiar and those formed by Spec’s regex functions.
2.2. Code complexity and quality
• Source lines of code (SLOC), the number of lines of code that the program consists of not counting comments.
• Spec with capital S is used to refer to the core library Clojure Spec itself. Written with a lowercase s, spec is used to refer to a specific specification: a spec for X.
• Specification, any definition on the characteristics of data written in code with either with Spec, Schema, Truss or Plain Clojure.
• Truss with capital T is used when referring to Peter Taoussanis library Truss [5]. A truss with lowercase s is used to refer to a specific piece of code asserting that Y is true for X using Truss.
2.2
Code complexity and quality
A survey of the field of software metrics was performed in search of a metric beyond the obviously simplistic measure of SLOC. The two most cited sets of such metrics comes from two separate authors; Maurice H. Halstead [22] and Thomas J. McCabe [31], both claiming a large following at their time.
2.2.1
Halstead complexity measures
Often referred to as Halstead’s metrics Maurice Halstead defined what he would later call “Software Science” starting with the length measure which correlated the number of operators and operands in a program with the number of bugs in his paper [23]. Continuing in his book Elements of Software Science [22] he defines four basic measures:
n1 the number of distinct operators in a program,
n2 the number of distinct operands in a program,
N1 the total number of occurrences of operators in a program,
N2 the total number of occurrences of operands in a program;
and the vocabulary of a program as n=n1+n2 and the length of the program as N =N1+N2. He also stipulated that the length can be estimated by ˆN =n1log2n1+N2log2n2, but only for polished programs [18]. In short this means a higher quality program has a N and ˆN which
are closer to one another.
But for our purposes the most important metric is that of effort:
E = n1N2(N1+N2)log2(N1+n2)
2n2
.
In Curtis, Sheppard and Millimans study [14] relating Halstead metrics of complexity to psychological complexity it was concluded that this metric correlates well with the speed at which a programmer is able to locate and fix a bug in some preexisting piece of code. They also observed that:
“Many small-sized programs can be grasped by the typical programmer as a cognitive gestalt. The psychological complexity of such programs is adequately represented by the volume of the program as indexed by the number of lines. When the code
grows beyond a subroutine or module, its complexity to the programmer is better assessed by measuring constructs other then [sic] the number of lines of code.”
In other words, metrics such as those Halstead proposes he argues are needed to measure the cognitive complexity of real world programs; something which is confirmed by Banker et al. who found significant correlation between software complexity and maintenance cost [6] of large projects.
Coppeick and Cheatham [12] then applied the Halstead metrics on Lisp defining a function to be equivalent to an operator and an argument to be the equivalent of an operand, which means that the code snippet (+ (/ a 2) (inc a) 2) yields:
n1 = 3. +, / and inc. n2 = 2. a and 2. N1 = 3. +, / and inc. N2 = 4. a, 2, a and 2. E = 3˚4(3+4) log2(3+2) 2˚2 « 49
But when reading modern literature on the subject of evaluating metrics of software complexity, namely Alain Abran’s book Software Metrics and Software Metrology [1], we find it discounts Halstead’s metrics as both ill defined and imprecise to the level that different researches following different interpretations of his metrics come to differing conclusions for the same code; perhaps precisely as Coppeick and Cheatham have done above.
In other words: The results of Curtis, Sheppard and Millimans are not necessarily valid for the interpretation of Halstead’s metrics that Coppeick and Cheatham apply; that is because Curtis et al. use a different interpretation of the metrics than Coppeick and Cheatham and hence it does not follow that we can apply Coppeick and Chathams interpretations of the metrics and draw conclusions from them according to the results of Curtis et al.
This leaves us having to repeat the trials of Curtis et al. using Coppeick and Cheathams definitions before we could in a scientifically honest way proceed in using their definitions for the subject of this thesis.
2.2.2
Cyclomatic complexity measures
McCabe proposes a different measure of complexity which he names cyclomatic complexity [31], a measure of how many independant paths there are through a program. There exists a Leiningen plugin named Uncomplexor [2] which calculates cyclomatic complexity for Clojure projects that uses Leiningen, though it is not the result of scientific research.
But without delving further on the subject we note that Stuart Halloway, author of the book Programming Clojure [21], write that complexity in Clojure code does not come from structurally complex code but rather from using the wrong things [32].
To back up that reasoning Abran notes that the transposition of cyclomatic complexity from graph theory into the field of software seems almost arbitrary and that practitioners build their own interpretation of what the resulting number from applying cyclomatic complexity on a software module actually means [1]. This being the result of McCabe never explicitly defining how to interpret the relation between the cyclomatic complexity measurement and software complexity itself.
2.3. Related work
2.2.3
Conclusions
Based on the background presented in in section 2.2.1 and section 2.2.2 it can be concluded that the field of code quality measurement is lackluster at best, that perhaps this thesis is better off sticking with the simple metric of SLOC and leaving it up to the reader how to interpret that number.
2.3
Related work
Nothing has thus far been written about Clojure Spec in a scientific publication. This thesis remedies that and draws upon previous works both in- and outside the scientific community.
2.3.1
On Spec
When it comes to evaluating Spec there exists non-scientific comparisons [19] [40] of Spec and Schema, but none that base their discussion on the specification of a real world data, instead making cases based on theoretical examples.
One performance evaluation of Spec compared to some of its competitors can be found on the web [49], but it is based on the concept of validating the same small amount of artificial data thousands of times. This thesis contrasts itself against these on the basis that it takes a complex real world data set, produces specifications for validating this data and then compares them around the three research questions.
2.3.2
On benchmarking
Kraus and Kestlers “Multi-core parallelization in Clojure: a case study” [29] contains a comparison of two implementations of the same algorithm; one pre-existing implementation in R and one original in Clojure. The algorithms are then run with varying sizes of generated artificial data sets ranging from 20 to 100 thousand samples. They then visualize the execution time using box plots with whiskers, something this thesis will take after as it conveys the most important statistical measures in a manner which is easy to understand.
2.3.3
On evaluating contract systems
In Plösch’s work on evaluating contract for Java he defines a number of criteria that can be used to describe and rank different such libraries [36]. He devises four different levels of support which the author can boil down to the following when translating it to functional rather than object-oriented programming:
BAS Basic Assertion Support3:
1. Is there support for assertions in the body of a function?
3The paper also speaks about class invariants, a concept which does not apply as there are no classes in Clojure.
2. Are there pre- and post-conditions on functions? AAS Advanced Assertion Support:
1. Can one define a relation between the input and output of a function?
2. Does the system support assertion expressions on collections? Are these guaranteed to not return a mutated4 collection?
3. Are the assertions guaranteed to be side effect free? SBS Support for Behavioral Subtyping:
1. Is it possible to specify contracts for interfaces? RMA Runtime Monitoring of Assertions:
1. Are there mechanisms available for handling broken contracts? 2. Is it possible to disable these checks in production?
3. Is it possible to do some checks selectively?
Except for SBS-1 these criteria will be used to provide additional evaluation of the specification systems. SBS-1 will not be used since inheritance is not a widely used concept in the Clojure world. The lack of support for this concept in any contract system for Clojure should not come as a surprise to a Clojurist.
2.4
Introduction to Clojure
Clojure is a modern functional Lisp on the JVM5, but also has official implementations that target CLR6 and JS7 called Clojure CLR and ClojureScript respectively, the latter often referred to as cljs.
Clojure is a modern Lisp both in that it implements first class literals for not only lists but also vectors, maps, and sets. In that its core data types, called persistent data structures8, are immutable but do not require the creation of complete copies in order to facilitate permutation; instead structures common between the original and mutated data are shared. And in that it has built in primitives for concurrent execution and management of shared state.
4As long as the Clojure collections are used, which are immutable and persistent, it is impossible for any code to mutate a collection; though its functions may return a different collection than that passed in.
5Java Virtual Machine, a virtual computer specification and with multiple implementations for the execution of Java programs.
6Common Language Runtime, a virtual machine by Microsoft for execution of its .net languages.
7JavaScript, standardized in the ECMAScript language specification, implemented by a multitude of VM’s including Mozilla SpiderMonkey and Google V8.
8The book Purely Functional Data Structures [34] can be recommended for further reading on the subject.
2.4. Introduction to Clojure
2.4.1
Numbers
Comments in Clojure begin with semicolons. Along with the usual number types Clojure also supports rational numbers as a first class data type.
1 ; Integer
50322143214123443210 ; Arbitrary precision integer
5.1 ; Floating point number
5E52 ; Floating point number
5/2 ; Ratio
2.4.2
Symbols, namespaces and keywords
A symbol in Clojure is a name or identifier which may resolve to something, perhaps a variable or a function. The symbol itself holds no value but is just an interned string that can be resolved by the Clojure compiler to a value. A symbol is written without any special notation using alphanumeric characters as well as *, +, !, -, _, ’, and ?.
A namespace is similar to a module in Python or package in Java in that they are groupings of names, named symbol tables. A symbol containing either a. or / are said to be namespace qualified, they refer to something in a namespace other than the current one.
name-of-something ; Dashes are allowed characters in symbols.
java.lang.Double ; Symbol referencing the fully qualified Java Double type.
lein/something ; The symbol something in namespace lein, or a namespace ; referred to as lein in the current namespace.
In order to prevent a symbol to be resolved into the value it references it is possible to quote the symbol using a single quote.
(def thing 5)
(str thing 'thing) ; => "5thing"
In this thesis we adopt a convention common in the Clojure ecosystem, that the return value of a function call or expression is indicated with an arrow like=>.
A keyword is a type in Clojure which only ever references itself, a constant commonly used for programmatic lookup in maps. These may also be qualified, but only in the Clojure fashion using /.
:key ; The unqualified keyword key.
:spacious/key ; The keyword key in the namespace spacious. ::key ; The keyword key in the current namespace. ::lein/key ; The keyword key in the namespace referred to
2.4.3
Data structures
Clojure is a homoiconic9language with built in literals for data structures like lists, vectors, maps and sets:
(1 2 "a") ; List
[1 "a"] ; Vector
{1 "a", :b 2} ; Map
#{1 2 "hat"} ; Set
All data structures are heterogeneous10 and as previously mentioned are immutable but do not require copies to be made upon “modification”, i.e. when a new value is added, removed or something is updated in a new version of the same data structure.
2.4.4
Additional literals
There are boolean, string and character literals as one would expect, but also string regex11 literals and an analogue of Python’sNone and Java’s null: nil.
true ; Boolean truth
nil ; Nothing
"What a day!" ; String
\c ; The character c
#"ab." ; Regular expression matching ab followed by any character.
It is worth noting that any value except for false and nil is considered true in Clojure’s equivalent of if-statements12.
2.4.5
Evaluation
Clojure, being a Lisp dialect, is drastically different from languages in the C family in that it puts the parenthesis before the name of the function being called.
# Python
print(1, 2)
;; Clojure
(print 1, 2) ; Commas are optional in Clojure,
(print 1 2) ; thus this and the line above are equivalent.
9Homoiconicity is a term used to describe languages where the structure of the program is be expressed using the data structures of the language. The data structure expressed in source code is then similar to how the program will be laid out in memory when executed.
10Heterogeneous data structures can contain data of multiple different types at the same time. 11Java string regexes are regular expression in the wider sense of the word.
2.5. Introduction to Spec, Schema and Truss
As can be seen above, function calls are simply lists starting with a function name. It follows that one should be able to perform operations on these lists like with any other list, something enabled by Clojure’s macro system.
(+ 1 2) ; => 3
(defmacro infix [[first-operand operator second-operand]] (list operator first-operand second-operand))
(infix (1 + 2)) ; => 3
The above infix macro takes three arguments and returns a list containing the operation in prefix notation for Clojure to evaluate. Worth noting for discussions later in this thesis is that macros are evaluated at compile13time, as opposed to the programs run time.
2.5
Introduction to Spec, Schema and Truss
This section of the background chapter will try to explain the basic need of a contract system and how the more complex specification systems are motivated. To start off we will look at a functionabs which returns the absolute value of a number.
Named functions are in Clojure defined withdefn which is a combination of def and fn. The first argument of defn is the function name, the second the function’s argument list and the third is the body of the function.
(defn abs [num] (if (pos? num)
num
(- num)))
(abs 1) ; => 1
(abs -1) ; => 1
(abs nil) ; java.lang.NullPointerException (NPE): No message
If passed a number, the function returns the expected value, but when passed something which is not a number it throws an exception. If the programmer either does not want an exception as a part of the normal application flow, or simply considers NPE to be a user hostile error message exposing intricacies of the implementation they may chose to check the data before attempting to perform number-specific operations on it. Depending on the situation they may either throw an exception with a more useful message or return a value symbolizing failure.
(defn abs-nil [num]
(when (number? num) (if (pos? num)
num
13On the JVM there is no interpretation, ever, of Clojure code; it is always compiled before execution. With ClojureScript macros are run in a separate compilation stage before being sent to the JS VM.
(- num))))
(abs-nil nil) ; => nil
Instead of throwing an NPE the functionabs-nil returns nil which can flow through the application, something a programmer may see as preferable if the result of the function is non-essential and its more important for the process as a whole to finish than it being exactly correct.
But if the result of the function is essential and invalid arguments are exceptional the pro-grammer may try to provide the caller with a more useful exception instead. In Clojure there is a built in concept of pre-and post-conditions for functions that, while they never became popular, are worth mentioning.
(defn abs-pre [num]
{:pre (number? num)} (if (pos? num)
num
(- num)))
(abs-pre nil) ; java.lang.AssertionError "Assert failed: num"
When passingabs-pre the value nil the error message thrown is quite meagre. No information about what was asserted or what the value being passed as num was is given in the exception message is given. Though "Assert failed: num" is still an improvement over the complete lack of error message given with the NPE. But this is where Truss comes in, when the programmer wishes to pass on more information about the failing data in the error message of the exception.
2.5.1
Truss
Truss is an assertions library by Peter Taoussanis [5] which first and foremost packs relevant data into the exceptions thrown when incorrect data is found. It is by far the simplest of the three libraries examined in this thesis. In the below function abs-truss the assertion is placed in the body of the function, but it might as well have been placed in the pre-condition of the function.
(defn abs-truss [num]
(if (pos? (truss/have number? num))
num
(- num))) (abs-truss "hat")
;; Invariant violation in `user.clj:561`. Test form ;; `(number? num)` failed against input val `"hat"`. ;; {:dt #inst "2017-06-27T13:54:40.402-00:00", ;; :val "hat",
;; :ns-str "user",
;; :val-type java.lang.String, ;; :?line 561,
2.5. Introduction to Spec, Schema and Truss
Truss’s only14 functionhave asserts that there should be a number? in num and returns the value of num if that is true. If that turns out to be false Truss will throw an exception much like the Clojure’s pre-conditions, but with additional information attached to help the programmer debug the problem. As seen above Truss both lets us know which piece of code deemed the data to be invalid, the invalid value itself and other relevant information such as the time of the failure and the run time type of the value. The user may also attach arbitrary data to be included with each thrown exception.
Beyond allowing us to assert thatnum is a number? Truss also provides some handy shorthands for expressing basic boolean logic.
(truss/have [:or string? integer?] 1) ; => 1
(truss/have [:or string? integer?] "Tux") ; => "Tux"
(truss/have [:or string? integer?] 'sym) ; Invariant violation in ...
(truss/have [:and integer? even?] 2) ; => 2
(truss/have [:and integer? even?] 1) ; Invariant violation in ...
It is also easy to assert that all the elements in a collection should be valid according to some predicate, i.e. that it is a homogeneous collection.
(truss/have integer? :in [1 2 3]) ; => [1 2 3]
(truss/have integer? :in [1 'sym]) ; Invariant violation in ...
Though it is notable that have only returns vectors, no matter the type of collection it is given.
(truss/have integer? :in '(1 2 3)) ; => [1 2 3]
(truss/have integer? :in #{1 2 3}) ; => [1 3 2] (yes, that really is the output)
Truss also provides a way to assert that an item should be a member of a set.
(truss/have [:el #{1 3}] 1) ; => 1
(truss/have [:el #{1 3}] 2) ; Invariant violation in ...
Beyond that it provides custom syntax for “set X is a superset of Y” and “map has exactly keys x, y and z” and “map has keys x, y and z but not less/more”, but no shorthand way to express what the values of these keys should be. Such functionality has be added by the user providing their own function or macro if desired.
2.5.2
Spec
Both Spec and Schema are usually included in a project when the programmer wishes to express more complex data structures, perhaps with nested structures or heterogeneous sequences. Or perhaps value validation in maps is desired. But this introduction will start with simple predicate validation.
Clojure Spec has three entry points for validating that some data is valid to a spec:
14Almost true: It also provideshave? and have! which are slight variations on the same function.
• valid?, a predicate returning true if the value is valid to the spec, else false.
• conform, returns either the conformed value or :clojure.spec/invalid. The conformed value can be different from the original value depending on the spec. This is what provides Spec’s capabilities as a high level parser.
• assert, returns the given value if valid to the spec, else thows an exception. The checking of assertions has to explicitly be enabled with a call to check-asserts.
Spec is capable of using any predicate function as a specification and below the built in predicatenumber? is used as such.
(spec/valid? number? 5) ; => true
(spec/valid? number? nil) ; => false
(spec/conform number? 5) ; => 5
(spec/conform number? nil) ; => :clojure.spec/invalid
(spec/conform (spec/or :name string? :id int?) 5) ; => [:id 5]
(spec/check-asserts true)
(spec/assert number? 5) ; => 5
(spec/assert number? nil)
;; Spec assertion failed val: nil fails predicate: :clojure.spec/unknown ;; :clojure.spec/failure :assertion-failed
Though, as seen in the above demonstration, in order for Spec to be able to name what failed the value in its error messages the specification has to be formalized in a spec.
(spec/def ::number number?) (spec/assert ::number nil)
;; Spec assertion failed val: nil fails predicate: number? ;; :clojure.spec/failure :assertion-failed
But considering thatassert is a macro and thus knows the name used by the caller of the macro in referring to the predicate function, this behaviour seems strange and a potential solution is proposed in section 5.3.6.
Either way, the error messages Spec does produce can also be given as a string.
(spec/explain-str ::number nil)
;; => "val: nil fails spec: :user/number predicate: number?"
The below implementation has exactly the same functionality asabs-nil.
(defn abs-nil-spec [num]
(when (spec/valid? ::number num) (if (pos? num)
num
(- num))))
If the programmer instead wishes an exception to be thrown when invalid data is given to abs it is normally done through a function spec which is then attached to the function using clojure.spec.test/instrument.
2.5. Introduction to Spec, Schema and Truss
(spec/fdef abs
:args (spec/cat :num ::number)
:ret ::number)
(require '[clojure.spec.test :as spec-test]) (spec-test/instrument `abs)
(abs nil)
;; Call to #'leiningen.core.spec.project/abs did not conform to spec: ;; In: [0] val: not-a-number fails spec:
;; :user/number at: [:args :num] predicate:
;; number? :clojure.spec.alpha/args (not-a-number) ;; :clojure.spec/failure :instrument
;; :clojure.spec.test/caller {:file "form-init1202700168622381044.clj", :line 631}
While the above example may seem inane, remember that::number can be replaced with a spec of much higher complexity, one which could describe any of the different collections Clojure provides. To start off: maps are expressed in terms of sets of keys:
(spec/valid? (spec/keys :req #{::id ::name}) {::id 'apple ::name "Peter"}) ; => true
(spec/valid? (spec/keys :req #{::id ::name}) {::id 'apple}) ; => false
And validation of the values held by these keys is performed when specs for individual keys are entered into the central registry withdef:
(spec/def ::id integer?)
(spec/valid? (spec/keys :req #{::id ::name}) {::id 'apple ::name "Peter"}) ; => false
(spec/valid? (spec/keys :req #{::id ::name}) {::id 5 ::name "Peter"}) ; => true
The simplest subset of collections, excluding maps, are the homogeneous collections which are expressed usingcoll-of:
(spec/valid? (spec/coll-of number?) #{1 2 3}) ; => true
(spec/valid? (spec/coll-of number?) [1 2 3]) ; => true
(spec/valid? (spec/coll-of number?) '(1 2 3)) ; => true
(spec/valid? (spec/coll-of number?) '(1 'a 3)) ; => false
Spec also has a dedicated concept for tuples, i.e. sequences of fixed length:
(spec/valid? (spec/tuple double? string? symbol?) [1.0 "bear" 'pine]) ; => true
For more complex sequences Spec has a set of functions which it calls regex ops: cat, alt, *, +, ? and &. These can be combined to match any regular language in the strict sense of the term. For example a sequence which begins with an id number followed by either one keyword and one float or two strings.
(spec/def ::db-row
(spec/cat :id integer?
:pairs (spec/alt :kw-float (spec/cat :key keyword? :value float?)
:name-desc (spec/cat :name string? :description string?)))) (spec/valid? ::db-row [1 :height 5.5]) ; => true
(spec/valid? ::db-row [1 "hat" "black"]) ; => true
(spec/valid? ::db-row [1 "hat" 5.5]) ; => false
Spec’s functionconform can be used to destructure this sequence into a map using the defined spec. This essentially allows for parsing the implicit semantics of syntax into maps with explicit names.
(spec/conform ::db-row [1 "hat" "black"])
;; => {:id 1, :pairs [:name-desc {:name "hat", :description "black"}]}
Spec can also generate data from a given spec, something which becomes useful when sample data is needed or for property based testing:
(require '[clojure.spec.gen :as spec-gen]) (spec-gen/generate (spec/gen ::db-row))
;; => (-342030845 "EfcznnD2apFlceNG34n5" "P")
One may also use this as a tool to reflect on whether the spec is correct or not, e.g. was negative identifiers really what we wanted or should we restrict the spec further?
2.5.3
Schema
Plumatic Schema is a specification system much like Spec and has two entry points for validating that some data is valid to a schema, check and validate. The first returns nil if the data matches the schema or an error message in a string if the data is invalid, the latter returns the data if valid or else throws an error.
(schema/check schema/Num 5) ; => nil
(schema/check schema/Num nil) ; => "(not (instance? java.lang.Number nil))"
(schema/validate schema/Num 5) ; => 5
(schema/validate schema/Num nil)
;; Value does not match schema: (not (instance? java.lang.Number nil))
;; {:type :schema.core/error, :schema java.lang.Number, :value nil, :error ...
One difference between Schema and its brother Spec is that the preferred leaf15in a schema is a type declaration rather than a predicate function. In the above code the type isschema/Num 15A leaf node in a specification is one which has no children and is directly matched against a value.
2.5. Introduction to Spec, Schema and Truss
which gets translated to the host specific type java.lang.Number. This does not preclude the usage of arbitrary predicates in schemas, though they have to be wrapped in thepred function for some of the same reasons that Spec requires predicates to be wrapped in a spec for it to be able to name them.
(schema/check (schema/pred number?) 5) ; => nil
(schema/check (schema/pred number?) nil) ; => "(not (number? nil))"
The below implementation has exactly the same functionality asabs-nil.
(defn abs-nil-schema [num]
(when (nil? (schema/check schema/Num num)) (if (pos? num)
num
(- num))))
But as with Spec the more common way of enforcing a contract on the arguments and return value of a function is through a function specification. In Schema’s case these are declared together with the function and in order to enforce the contract the call to the function is wrapped with with-schema-validation.
(schema/defn abs-schema :- schema/Num [num :- schema/Num]
(if (pos? num)
num
(- num)))
(schema/with-fn-validation (abs-schema "not-number"))
;; Input to abs-schema does not match schema:
;; [(named (not (instance? java.lang.Number "not-number")) num)]code
It is possible to construct schemas for collections using the data DSL Schema provides. It uses the Clojure data types to represent what the schema should match. Schemas for homogeneous collections, including maps, are showcased below:
(schema/check #{schema/Num} #{1 2 3}) ; => nil ;; Sequences are expressed using array literals.
(schema/check [schema/Num] [1 2 3]) ; => nil
(schema/check [schema/Num] '(1 2 3)) ; => nil
(schema/check [schema/Num] '(1 2 "a")) ; => [nil nil (not (instance? java.lang.Number "a"))]
(schema/check {schema/Str schema/Str} {"key" "value", "a" "b"}) ; => nil
Tuples such as a sequence of a number followed by either a string or a number can be expressed:
(schema/defschema a-row
[(schema/one schema/Num "id") (schema/one (schema/cond-pre schema/Str schema/Num) "value")]) (schema/check a-row [1]) ; => [nil (not (present? "value"))]
(schema/check a-row [1 "str"]) ; => nil
But expressing the previously given spec::db-row is not as easy. The closest we come using Schema’s own constructs is:
(schema/defschema db-row
(schema/conditional #(keyword? (second %))
[(schema/one schema/Num "id") (schema/one schema/Keyword "key") (schema/one schema/Num "value")] #(string? (second %))
[(schema/one schema/Num "id") (schema/one schema/Str "name")
(schema/one schema/Str "description")])) (schema/check db-row [1 :degreees 5.5])
(schema/check db-row [1 :degreees "hat"])
;; => [nil nil (named (not (instance? java.lang.Number "hat")) "value")]
Unlike the spec for the same piece of data the schema has to repeat the complete sequence for each permutation due to the lack of concatenation, there is simply no way to express “put these two parts together to form a sequence”. The selection between the two alternatives also has to be manually specified by providing the predicates#(keyword? (first %)) and #(string? (first %)).
Further observations on the differences between Spec, Schema and Truss can be found in section 5.3. Among other things we describe how the constrains of Schema make it unsuitable for describing named arguments.
Finally, it is worth mentioning that it is possible to generate data from a given schema using theschema-generators library, just like with Spec.
(require '[schema-generators.generators :as schema-gen]) (schema-gen/generate db-row)
3
Method
In order to study Spec the author first had to figure out how it was being used in the real world and identify its competitors. Seeing that it was commonly being used for validation the author searched for and found a large set of real world Clojure data which was then specified using Spec and its competitors. Measurements of validation time were then performed as well as other forms of inspection on the specifications and the results of the validation.
3.1
Pre-study
The first goal of the pre-study was to find existing code bases using Spec. The community driven website clojure-toolbox.com provides a categorised list of more or less popular libraries and tools a developer may use when writing Clojure applications. The source code for all projects was fetched and searched for usage of Clojure Spec, and subsequently found that 8 in total did so.1
The author also found a work in progress2 for the popular Leiningen build tool which specified what a project declaration can and cannot look like.
1alia, cljs-oops, core.typed, graphql-clj, java.jdbc, onyx, sablono, spandex. 2https://github.com/technomancy/leiningen/pull/2223
A dependency tree for the entirety of Clojars3was found on crossclj.info from which it was possible to extract a much longer list4 of projects using Clojure 1.9-alphaXX5 and specific functions of Spec.67
In addition some scripts were written to download the latest release of every artifact on Clojars,8 numbering 13995 in total, in order to search through their contents.
The two main interaction points of Clojure Spec when it comes to using specifications, as opposed to defining them, is valid?, which checks whether a datum is valid according to a specification, andconform, which in addition of checking the datums validity destructures the datum if possible.
Inspecting the use of Spec in the gathered data revealed equal use of conform and valid?. Inspecting these uses further with git blame it was found that:
1. Alia only uses Spec for testing.
2. Oops was written from the start withconform and valid?. 3. Untangled-web rewrote parts of its internals to useconform.
4. GraphQL-clj used to usevalid?. The specs remain but its uses are gone. 5. java.jdbc only uses Spec for tests.
6. Onyx uses Schema and has only a toy spec. It uses Schema extensively throughout its code.
7. Sablono only uses Spec for tests.
8. Spandex only uses Spec for testing and documentation.
9. Net uses onlyvalid? and its use was added on top of existing defensive programming. 10. Zprint usesvalid? and has moved to Spec from Schema.
The conclusion that Spec is foremost used as an addon to existing code was drawn; that, as of yet, it has only had limited use as a replacement for existing code and if it replaced anything it was a Schema.
In other words it made sense to make the study itself centre around how Spec compares to other specification or contract systems. The ecosystem was surveyed and five such systems were found: Spec, Schema, Truss, Herbert [25] and Annotate [4]. All but Spec are delivered to end users through Clojars which meant download numbers were publicly available and are presented in table 3.1.
3Repository of Clojure code where anyone can contribute, at clojars.org. Like Maven central but without review process.
4systems-toolbox, spex, clj-edocu-users, mistakes-were-made, utangled-client, clj-edocu-help, net, clj-element-type, gadjett, datomic-spec, tappit, tick, deploy-static, tick, spectrum, sails-forth, ring-spec, doh, tongue, diglett, rp-query-clj, pedestal.vase, clj-jumanpp, om-html, crucible, ctim, crucible, wheel, merlion, turbovote-admin-specs, odin, pc-message, active-status, rp-json-clj, curd, rp-util-clj, swagger-spec, invariant, functional-vaadin, untangled-client, flanders, uniontypes, dspec, schpeck, macros, replique, specific.
5https://crossclj.info/ns/org.clojure/clojure/1.9.0-alpha14/project.clj.html 6Defining data specs: https://crossclj.info/fun/clojure.spec/def
7Defining function specs: https://crossclj.info/fun/clojure.spec/fdef 8https://github.com/Rovanion/all-the-clojars
3.2. Data selection
Table 3.1: Downloads from Clojars per library.
name downloads downloads per year Spec not available9
Schema 1 250 727 320 699
Truss 257 800 161 125
Herbert 7 182 1 795
Annotate 2 074 1 037
The number of uses in published code on Clojars was also available on crossclj.info which gave us an additional set of data gauge popularity from, presented in table 3.2
Table 3.2: Number of projects on Clojars that depend on the given library.
name uses uses per year
Spec 60 55
Schema 498 54
Truss 213 135
Herbert 5 1
Annotate 0 0
If the age of each project is taken into account it stands clear that there are only two real competitor libraries to Spec: Schema and Truss.
3.2
Data selection
The internet was scoured for large amounts of production data typical for what a Clojure program would deal with. The goal was to find a data source where the data was produced by humans so to have a large amount of variance with many different types of errors in them. And the best thing would be if there existed a parser to bring the data into the Clojure data types that Clojure programs normally deal with.
In the ecosystems of modern programming languages there are often repositories of liberally licensed code available, such is also the case with Clojure. The lion share of such code is shared in the repository named Clojars and it was found that most of these had some version of the very same file located within their archives: The Leiningen project file. The contents vary from project to project, here is an example from the popular routing library Compojure [11]:
(defproject compojure "1.6.0"
:description "A concise routing library for Ring"
:url "https://github.com/weavejester/compojure"
9Spec has been delivered alongside Clojure 1.9-alphaXX meaning that it has been delivered with all programs using 1.9 thus far. Worth noting is that it recently was split out into its own artifact.
:license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.7.0"] [org.clojure/tools.macro "0.1.5"] [clout "2.1.2"] [medley "1.0.0"] [ring/ring-core "1.6.0"] [ring/ring-codec "1.0.1"]] :plugins [[lein-codox "0.10.3"]] :codox {:output-path "codox"
:metadata {:doc/format :markdown}
:source-uri "http://github.com/weavejester/compojure/blob/{filepath}#L{line}"}
:aliases
{"test-all" ["with-profile" "default:+1.8" "test"]}
:profiles
{:dev {:jvm-opts ^:replace []
:dependencies [[ring/ring-mock "0.3.0"] [criterium "0.4.4"]
[javax.servlet/servlet-api "2.5"]]}
:1.8 {:dependencies [[org.clojure/clojure "1.8.0"]]}})
These files are read by the Clojure reader into Clojure data types and are very typical for just Clojure programs as they turn into maps which is the preferred structure for Clojurists to put their data in. This comes from the Clojurists preference to hold explicit semantics in names instead of implicit semantics in syntax.
The values then held by these keys are of many different degrees of complexity and structure. Some hold simple singular values such as:url which holds a string representing just that, an URL. Others are of intermediate but still strictly bounded complexity such as:mailing-list which holds a map with the possible keys:name, :archive, :other-archives, :post, :subscribe and :unsubscribe each with their own valid values, but none more complex than a vector of URL:s. Then there are the complex keys such as:dependencies which can vary in complexity from
[org.clojure/clojure "1.8.0"]
to
[log4j "1.2.15" :exclusions [[javax.mail/mail :extension "jar"] [javax.jms/jms :classifier "*"] com.sun.jdmk/jmxtools
com.sun.jmx/jmxri]
:native-prefix ""]
or:profiles which holds as its value an entirely new project-map.
3.3. Filtering test data
3.3
Filtering test data
The test data was acquired using a software named All the Clojars10, developed as part of the thesis work. It uses Clojars public API to acquire a list of every single artifact in their repository to then feed that into Leiningen which pulls down its code and the code of all its dependencies, which could be located in any repository.
Out of the 15089 project files collected from Clojars and Maven Central, 21 were removed since they were made for Leiningen 1, whereas this thesis project is for version 2. 121 files were removed because they could not be read by the Clojure reader, the most common cause being duplicate keys. On the other hand well over 200 files were repaired semi-manually, most of them broken due to being taken out of their original file system context.
In this data there were a few top level keys that were never present: :plugin-repositories, :offline?, :implicit-hooks, :implicit-middleware, :jar-inclusions and :install-releases?. It follows that the specifications written for these keys will not be part of the evaluation.
3.4
Writing specifications
All measurements and conclusions thereof are drawn from the specifications written or co-written by the author of this thesis.
The author of Figwheel[16] Bruce Hauman wrote a library for specifying configuration files he called strictly specking [47] which adds additional functionality on top of Clojure Spec. He then applied this library to the project Leiningen and produced a partial spec for its project files, this is the previously mentioned work in progress spec for Leiningen.
The main part of the study consisted of removing strictly specking from Bruce’s partial spec, and then completing the spec. Additional specifications were then implemented using Schema, Truss and Plain Clojure for the same data. Issues encountered during the process are documented in section 5.3.
3.5
Benchmarking
All the acquired data was read into memory and each specification was used to validate both the project files in their original form and sorted into per keyword sets for per keyword benchmarks. The resulting timings were sorted on whether the data was valid or not.
The benchmarks themselves consisted of timing each library validating project maps or sets of values per some specification. Each run was performed on the same JVM and the time measured was that of the execution of the validation function. The JVM had been warmed up by running the benchmarks twice before extracting the results of a third run. An overview of the benchmarking procedure follows:
1. Record the system time right before starting the validator function.
2. Run the validator function on a project file or value from one of keys depending on whether we are producing results for the graphs separated by keyword or not.
3. Record the system time right after the function is completed.
4. Calculate delta and put it together with the result of the function into the list of of results. 5. Go to 1 if there is yet more data to validate. Else calculate statistical results and output
them to file.
When running across the original project files each validator was run across the entire data set of 14947 files.
When benchmarking the validation of values separated out by keyword each set of values was validated ten times over except for those keys which there were11 fewer than ten values present, in which case the values were validated a 100 times to reach statistical significance.
The benchmarks presented was run on a consumer-grade Intel Core i7-2640M with 4GB RAM on 32-bit Ubuntu 16.04. The source code of the benchmarking software can be found on GitHub12 along with instructions for running benchmark.
3.6
Effort reduction
The programcloc[9] will be used to count the number of lines of code in each implementation. The source code of the different validator implementations can also be found on GitHub13.
3.7
Edge cases
Two separate inquiries into finding edge cases in data not found by previous programming was performed during the study.
The first method was applied during the pre-study where Hauman’s existing partial implemen-tation of a spec for Leiningen project files was run across the project-files from clojure-toolbox.com. We then manually inspected the reported failures and recounted our findings.
The second inquiry was made during the study’s main section and is based on running the written validators across the entire data set in order to find any discrepancies there.
11https://gist.github.com/Rovanion/ec72b5092b4737763a377b7e616f6a06 12https://github.com/Rovanion/leiningen-validation-benchmark
13 https://github.com/Rovanion/leiningen/tree/{spec,schema,truss,pred-validation}/leiningen-core/src/leiningen/core/{spec,schema,truss,pred_validation}
4
Results
This chapter contains answers for the three research questions of the paper: That there is no real difference in code size, that all three libraries add overhead with different characteristics and that Spec is not inherently better at finding edge cases in existing code.
4.1
Effort reduction
This section seeks to answer research question (1): How does the code of Spec specifications compare to equivalent code in competing systems, both in terms of plain SLOC and convenience of expression?
As is seen in table 4.1 there is no major difference between length of the code using the different systems.
Table 4.1: SLOC per specification implementation.
Spec Schema Truss Plain Clojure
423 401 426 413
Though one may observe the number of utility functions and macros needed to support the specifications as an indicator of the deficiencies in each library. One could also see it as the amount of code needed to unify the capabilities of all the libraries.
For Spec two macros named vcat and stregex were written to express concatenation of vectors and strings which matches a specific string regular expression. These were macros and not functions since most of the Spec API is also macro-based; and it follows that if one wishes to manipulate data before is passed on to a macro that manipulator has to also be a macro.
For Schema a total of four functions were added: • stregex!, in the same vein as stregex for Spec.
• key-val-seq?, emulates spec/keys* and is treated in greater detail in the Named argu-ments section of the Discussion.
• first-rest-cat-fn and pair-rest-cat-fn which both emulate parts of the functionality of spec/cat.
For Truss one function and four macros were implemented. These macros were macros instead of functions since Truss uses the source code in error messages and these would become hard to understand had they referenced anonymous functions:
• key-val-seq?, the very same as for Schema.
• opt-key and req-key to emulate Schema and Spec’s validation of values held by keys. • stregex-matches, a macro likelike stregex.
• >>, a convenience macro to apply its first argument to the end of each of the lists given as the rest of its arguments and lastly return the first argument. In practice:
(macroexpand '(util/>> [1 2 3]
(truss/have vector)
(truss/have integer? :in)))
;; => (do (truss/have vector [1 2 3]) ;; (truss/have integer? :in [1 2 3])
;; [1 2 3])
As perhaps is clear from the above text, the simpler the library the more complex the hand written utility code had to become. Discussion around the specificities of these utility functions and macros can be found in section 5.3.
Haumans’s incomplete spec for Leiningen project files, as introduced in section 3.4 and studied in the pre-study, was 804 SLOC and 80 lines of comments; though a large part of those SLOC were docstrings1. Hauman’s code did not replace any existing code but added the feature of being able to identify and suggest corrections for common error in such files [20]. Since it replaces no code no comparison in terms of SLOC could be drawn to a previous implementation.
4.2
Performance
This section seeks to answer research question (2): How does Spec perform compared to competing techniques?
The benchmarks were run across all Leiningen project files published on Clojars or depended on by any project on Clojars as described in the Filtering test data section of the Method.
The graphs in this chapter are box plots where the colored box represents the second Quartile2, the single black line across the coloured box is the median and the whiskers represent the whole range of the data. As an example: In fig. 4.1 this means that for Spec the fastest validation of 1The documentation bundled with a variable or function that is available at run and develop-ment time for the programmer to read.
4.2. Performance
0.00010 0.00100 0.01000 0.10000 1.00000 10.00000Spec Schema Truss Plain
m ill is ec on ds
Validation time for all data
Figure 4.1: Project file validation time summary
a project file completed in 0.0005ms, that 25% of the validation completed in 0.03ms, 75% in 0.14ms and 100% in 6.6ms with a median of 0.07ms.
The term validation time is used to denote the time it takes to validate some piece of data using a specification system. In fig. 4.1 and table 4.2 this piece of data is one whole project file. In the per keyword breakdowns presented in Appendix B the piece of data being validated is that held under a keyword.
As is obvious in fig. 4.1 Schema is by far the slowest of the three with a median evaluation time almost two magnitudes higher than its competitors. Spec has a median validation time 0.04ms slower than Truss and 0.05ms slower than Plain Clojure, but 1.58ms faster than Schema.
Though it may seem from fig. 4.1 that Schema is also the most stable of the three one has to remember that the y-axis is in logarithmic scale3. And by looking at the numbers in table 4.2 we can confirm that such is not the case.
Table 4.2: Statistical measures in ms for validation of project files.
min 1st quart median 3rd quart max std-dev name 0.00042 0.02618 0.06388 0.12772 1.59931 0.11643 Spec 1.48286 1.57328 1.64274 1.77755 21.27848 0.61164 Schema 0.00216 0.01287 0.02371 0.08320 41.78816 0.79916 Truss 0.00008 0.00893 0.01280 0.02000 0.49467 0.01294 Plain
Schema is in fact second in standard deviation to Spec, with Truss trailing behind by almost a whole order of magnitude. But then again 75% of the data is validated an order of magnitude faster by Spec and Truss than by Schema. There seem to be some specific cases that drive up the standard deviation of Truss. Breaking down the results by whether or not the test data was valid, as is displayed in fig 4.2, perhaps lays bare one of the causes.
