Design and Implementation of Semantic Patch Support for the Spoon Java Transformation Engine
MIKAEL FORSBERG
KTH ROYAL INSTITUTE OF TECHNOLOGY
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE
Semantic Patch Support for the Spoon Java
Transformation Engine
MIKAEL FORSBERG
Master in Computer Science Date: January 26, 2021
Supervisor: Nicolas Yves Maurice Harrand Examiner: Martin Monperrus
School of Electrical Engineering and Computer Science
Swedish title: Design och implementering av stöd för semantiska patchar för Javatransformeringsmotorn Spoon
Abstract
Software development is more often than not a collaborative process, creating a need for tools and file formats that enable developers to create and share suc- cinct representations of changes to source code in order to facilitate efficient communication. Standard POSIX diffs and patches have long been important parts of the toolkit, but their lack of support for the syntax and semantics of spe- cific programming languages results in limited expressiveness. The Semantic Patch Language (SmPL), introduced in 2006 together with the tool Coccinelle, increases the expressiveness of POSIX-style patches for the C programming language by leveraging support for the syntax and semantics of C. For exam- ple, an SmPL patch can specify changes to source code using metavariables that bind arbitrary program variable names, allowing for the specification of transformations involving variable references regardless of what specific vari- able names appear in programs targeted by the patch. A recent development is Coccinelle4J, a prototype modification of Coccinelle targeting the Java pro- gramming language. Coccinelle4J remains based on a toolkit designed for the parsing and modeling of C, adapted to operate on Java source code. The language mismatch of the base toolkit gives rise to limitations. Despite this, Coccinelle4J remains the state of the art for an SmPL targeting Java. In this thesis we lay the foundations for an SmPL for Java based on Spoon, a robust Java metaprogramming toolkit. We qualitatively investigate to which extent the features of SmPL and Coccinelle are generalizable to a Java context, and we implement and evaluate SPOON-SMPL, a prototype SmPL tool for Java based on Spoon. We base the core design of SPOON-SMPL on temporal logic and model checking, heavily inspired by the design of Coccinelle. We find the majority of identified SmPL features to generalize for Java. We quantitatively evaluate SPOON-SMPL by comparing the running time performance to that of Coccinelle4J over a set of six semantic patches with associated real-world project code bases used in an API migration case study originally performed by the authors of Coccinelle4J. Additionally, we compare the running times of SPOON-SMPL to the average build time of each associated project. We find that SPOON-SMPL performs worse than Coccinelle4J, but that the performance remains in a range acceptable for a single developer using inexpensive hard- ware. Finally, we provide two proposed designs for extensions to SPOON-SMPL along with a set of suggestions for future work. The proposals show that our prototype offers a strong potential to leverage the capabilities of the Spoon li- brary, particularly in providing improved and robust support for certain aspects of Java for which Coccinelle4J provides only limited support.
Sammanfattning
Mjukvaruutveckling är ofta en kollaborativ process med behov av effektiv kommunikation. Ett centralt inslag i denna kommunikation är möjligheten för utvecklare att skapa och sinsemellan dela kortfattade sammanfattningar över källkodsändringar. De POSIX-standardiserade verktygendiffochpatchhar länge utgjort en viktig del av verktygslådan, men deras avsaknad av stöd för syntax och semantik hos specifika programspråk ger upphov till en begrän- sad uttrycksfullhet. Semantic Patch Language (SmPL), introducerat år 2006 tillsammans med verktyget Coccinelle, erbjuder ökad uttrycksfullhet i POSIX- liknande patchar för programspråket C. En SmPL-patch kan bland annat an- vända metavariabler, logiska variabelnamn som binder godtyckliga program- variabler, för att specificera transformationer som berör variabelreferenser oav- sett vilka variabelnamn som förekommer i målprogrammet. Coccinelle4J, en modifikation av Coccinelle, är en nyligen framtagen prototyp på ett SmPL- verktyg för programspråket Java. Coccinelle4J baseras på en teknisk grund designad för tolkning och bearbetning av C som anpassats till att bearbeta Ja- va. Språkskillnader gör det svårt att få en heltäckande anpassning, vilket leder till ett begränsat stöd för vissa av Javas egenskaper. Trots detta är Coccinelle4J i dagsläget den främsta lösningen för SmPL för Java. I denna avhandling tar vi de första stegen mot ett SmPL för Java baserat på Spoon, ett robust metapro- grammeringsbibliotek för Java. Vi undersöker kvalitativt vilka egenskaper hos SmPL och Coccinelle som kan generaliseras till Java, samt implementerar och utvärderar SPOON-SMPL, en prototyp på ett SmPL-verktyg för Java baserat på Spoon. Designen av SPOON-SMPL är kraftigt inspirerad av Coccinelle, och ba- seras på temporallogik och modellprövning. Vi finner att en klar majoritet av de egenskaper vi identifierat hos SmPL och Coccinelle låter sig generaliseras till Java. Vi utvärderar kvantitativt SPOON-SMPL genom att jämföra körtidspre- standan mot Coccinelle4J över sex semantiska patchar med tillhörande pro- jektkodbaser som ursprungligen användes i en fallstudie kring API-migrering utförd av teamet bakom Coccinelle4J. Vi jämför även körtidsprestandan mot byggnadstiden för vardera projekt. Vi finner att körtidsprestandan hos SPOON- SMPL är sämre än Coccinelle4J, men att den trots det befinner sig inom ett område som är acceptabelt för en enskild mjukvaruutvecklare med en enkel persondator. Slutligen presenterar vi två detaljerade förslag till utökningar av SPOON-SMPL tillsammans med en uppsättning förslag för framtida arbete. Vi visar genom detta att vår prototyp har en kraftfull potential för utökningar som drar nytta av de funktioner som finns i Spoon, i synnerhet kring ett förbättrat och robust stöd för vissa egenskaper hos Java där Coccinelle4J endast erbjuder ett begränsat stöd.
Acknowledgements
I would like to thank:
• Prof. Martin Monperrus, my examiner.
Martin suggested the project and gave me the opportunity to pursue it.
Martin also helped establish the research methodology and formulate the formal research questions, gave regular feedback on the structure of the thesis and my approaches to various aspects of the work, suggested many papers on related works, and provided tips on the use of Spoon.
• Nicolas Yves Maurice Harrand, my supervisor.
Like Martin, Nicolas provided feedback on the methodology and the structure of the thesis, and also helped me with a couple of difficult choices in the implementation. Nicolas also provided detailed feedback on the full text, introduced me to a set of useful tools and ideas for im- proving the text, provided papers on the subtleties involved in bench- marking the performance of Java programs, and helped eliminate a for- mal research question for which the results were overly speculative.
• Ann Bengtsson, degree project coordinator at KTH EECS.
Ann greatly helped me solve the complications surrounding my formal admittance to the degree project course.
Finally, I would like to jointly thank Martin and Nicolas for their sympathy and patience throughout the project in general, and in particular surrounding the passing of my father.
Thank you for everything.
1 Introduction 1
1.1 Problem statement . . . 1
1.2 Research questions . . . 2
1.3 Contributions . . . 2
1.4 Intended audience . . . 3
1.5 Outline of the thesis . . . 3
2 Background 4 2.1 Text file differencing . . . 4
2.1.1 diff . . . 4
2.1.2 patch . . . 6
2.2 Formal logics for the modeling of computer programs . . . 7
2.2.1 Computation Tree Logic . . . 7
2.2.2 CTL with free variables . . . 13
2.2.3 CTL with quantified variables . . . 16
2.2.4 CTL with variables and witnesses . . . 17
2.3 Program analysis and transformation . . . 19
2.3.1 Spoon . . . 20
2.3.2 Semantic Patch Language . . . 22
3 Related work 28 3.1 Semantic patching . . . 28
3.1.1 Coccinelle . . . 28
3.1.2 Coccinelle4J . . . 30
3.2 Program transformation using temporal logic . . . 31
3.3 Other approaches to Java source code transformation . . . 32
3.4 API migration . . . 32
vii
4 Design of spoon-smpl 34
4.1 Design goals . . . 34
4.2 Core engine . . . 36
4.3 Parsing SmPL . . . 37
4.4 Formula language . . . 41
4.5 Formula compilation . . . 43
4.6 Batch processing . . . 50
4.7 Use of Spoon . . . 50
5 Evaluation methodology 52 5.1 Analytical methodology . . . 52
5.1.1 RQ1: Generalizable features . . . 52
5.1.2 RQ2: Non-generalizable features . . . 53
5.2 Experimental methodology . . . 53
5.2.1 RQ3: Patch application performance . . . 54
5.2.2 RQ4: Project build times . . . 64
6 Evaluation results 66 6.1 Analytical results . . . 66
6.1.1 Coccinelle feature catalog . . . 66
6.1.2 RQ1: Generalizable features . . . 81
6.1.3 RQ2: Non-generalizable features . . . 81
6.2 Experimental results . . . 82
6.2.1 Hardware and software . . . 83
6.2.2 RQ3: Patch application performance . . . 83
6.2.3 RQ4: Project build times . . . 89
7 Discussion 91 7.1 Limitations . . . 91
7.2 Threats to validity . . . 92
7.3 Extension proposals . . . 93
7.3.1 Improving name resolution . . . 94
7.3.2 Improving sub-typing . . . 95
7.4 Future work . . . 96
7.4.1 Support for more simple Java constructs . . . 97
7.4.2 Support for looping constructs . . . 98
7.4.3 Support for isomorphisms . . . 98
7.4.4 Model checker optimizations . . . 99
7.4.5 Using Spoon sniper mode . . . 100
7.4.6 Using spoon.pattern . . . 100
7.4.7 Target-embedded parsing of the semantic patch . . . . 101
7.5 Ethical considerations . . . 102
8 Conclusions 104 Bibliography 105 A Full semantic patches 109 A.1 Semantic patch 4: should_vibrate . . . 110
A.1.1 Original version . . . 110
A.1.2 Modified version . . . 111
A.2 Semantic patch 5: get_height . . . 112
A.2.1 Original version . . . 112
A.2.2 Modified version . . . 113
Introduction
1.1 Problem statement
Automated code transformation over a full code base is an important tool in software engineering. Useful transformations range from simple renaming of variables to making adaptations for complex changes in library APIs which require corresponding modifications in client code.
The POSIX (originally Unix) programs diffandpatchand their varia- tions have long since offered a popular platform for declaratively specifying code transformations, but the platform is limited due to a lack of support for the syntax and semantics of the programming languages in which the transfor- mations are to take place. As an example of this limitation, it is generally not possible to write a POSIX patch describing a transformation that rearranges arbitrary expressions or includes constraints over arbitrary control flow paths.
The Semantic Patch Language (SmPL) was introduced [1] in the context of Linux kernel development as an extension to the familiar POSIX patch format, featuring support for the syntax and semantics of the C programming language.
SmPL resolves several of the limitations of POSIX patches, albeit limited to the C programming language.
The Java programming language is widely used and the subject of ongoing research. An example of research is the Spoon project [2], a Java metapro- gramming library developed under the helm of the French research institute Inria and designed to facilitate the implementation of bespoke Java source code analyses and transformations. However, as opposed to the declarative style of POSIX patches and SmPL, implementing a code transformation using Spoon generally requires writing an imperative program. Viewed as a specification for a code transformation, an imperative (as opposed to declarative) program
1
is often harder to understand and maintain.
The state of the art of SmPL for Java is the work on Coccinelle4J [3], a pro- totype based on Coccinelle [1], the original SmPL tool for C. Coccinelle4J suf- fers from a number of limitations regarding the syntax and semantics of Java, such as having limited support for sub-typing and requiring special rules for matching fully-qualified names to simply-qualified names. Limitations such as these could potentially be alleviated by using a more robust, Java-centric base such as Spoon.
The main objective of this thesis is to design and implement SPOON-SMPL;
an SmPL transformation engine for Java based on the Spoon library and trans- formation engine, and to evaluate its performance and potential. As part of this objective we also investigate which features of SmPL and Coccinelle that can be generalized to a Java context.
1.2 Research questions
• RQ1: Which features of Coccinelle/SmPL for C can be generalized to Java?
• RQ2: Which features of Coccinelle/SmPL for C cannot be generalized to Java? Why not?
• RQ3: Is the running time performance of an SmPL implementation for Java based on Spoon better or worse than that of Coccinelle4J?
• RQ4: Is the running time performance of an SmPL implementation for Java based on Spoon acceptable for an individual developer?
1.3 Contributions
The primary contribution of this work takes the form of Java source code con- tributed1to the Spoon open-source project2. Additionally, we make contribu- tions to the general body of work on SmPL.
• An implementation of a subset of SmPL for a subset of Java, written in Java, based on the Spoon Java transformation engine.
• A minor contribution to the spoon-control-flowsub-module of the Spoon project, consisting of the implementation of limited support for
1https://github.com/mkforsb/spoon/tree/smpl 2https://github.com/INRIA/spoon
the control flow of exceptions and the associated statementsthrow,try, catchandfinally.
• An alternative3 treatment of the formal logic CTL-VW and the precur- sors CTL-FV and CTL-V.
• An implementation of a CTL-VW model checking algorithm in Java.
To our knowledge, all of the implementation-related contributions are novel.
1.4 Intended audience
This text assumes the reader is familiar with elementary set theory, elemen- tary formal logic (propositional and first-order logic), context-free grammars presented in Backus-Naur form (BNF), the notions of Abstract Syntax Trees (ASTs) and Control Flow Graphs (CFGs), as well as the basic syntax and se- mantics of C-like programming languages.
1.5 Outline of the thesis
The general structure of the thesis is as follows. Chapter 2 introduces the fun- damental ideas and tools used in the implementation of SPOON-SMPL. This in- cludes topics such as text file differencing and Computation Tree Logic (CTL), as well as introductions for the Semantic Patch Language (SmPL) and the ba- sics of the Spoon library. Chapter 3 covers related works in general, and the works of Coccinelle and Coccinelle4J in particular. Chapter 4 presents the de- sign of SPOON-SMPL, giving an overview of the main design goals and require- ments as well as providing further details on the source code transformation engine, the parsing of SmPL and the compilation of CTL formulas. Chapter 5 presents the methods used to evaluate our formal research questions, and Chapter 6 presents the corresponding results. Chapter 7 discusses potential problems and limitations with our work, and provides suggestions for future work. Finally, Chapter 8 presents our conclusions.
3Alternative to Julien Brunel et al. ”A Foundation for Flow-Based Program Matching:
Using Temporal Logic and Model Checking.” [4]
Background
This chapter covers the fundamental ideas and tools relevant to the implemen- tation of SPOON-SMPL. Section 2.1 introduces the notions ofdiffandpatch, both of which have greatly influenced the original design of SmPL. Section 2.2 provides a semi-formal overview of Computation Tree Logic (CTL) and the series of extensions CTL-FV, CTL-V and CTL-VW, which are of central importance to SPOON-SMPL. Section 2.3 gives a brief overview on the topic of automated program analysis and transformation, followed by an introduction to API migration as an application of automated program transformation, be- fore ending with introductions to the Spoon metaprogramming library and the SmPL source code transformation language.
2.1 Text file differencing
This section introduces thediffprogram that generates summaries of differ- ences between files, and thepatchprogram that can use such a summary to apply changes to files.
2.1.1
diffThe original diff program emerged in the 1970s during the early develop- ments of the UNIX operating system at Bell Labs [5]. In typical operation, diff takes as input the contents of two files and produces an output (”the diff”) containing some representation of the minimal set of changes needed to turn one of the files into the other (or vice versa). The algorithm used to find the minimal set of changes is based around solving the longest common subsequence (LCS) problem, which results in finding the longest list of anchor
4
points common to (and matching in sequence) both input files around which the instructions for changes should be placed. An example of the fundamental ideas of diffis shown in Table 2.1.
Input A: a b c d e f g Input B: w a b x y z e
LCS: (a,b,e) Offsets in A: (0,1,4) Offsets in B: (1,2,6)
Diff (A→ B): Offset a: prepend(w)
Offset b: delete_successors(c,d); append(x,y,z)
Offset e: delete_successors(f,g) Table 2.1: Diff fundamentals.
To reverse the direction of the changes one simply swaps each prepend instruction for an instruction todelete_predecessorsand eachappendin- struction for an instruction to delete_successors (and vice versa, respec- tively). The resulting set of instructions is then applied to the LCS offsets corresponding to the second file.
Applying the set of instructions can be done manually or automatically and the act of doing so is often referred to as patching, in which case the diff itself is often also referred to as the patch.
There exists several syntax formats for representing diffs. Earlier formats tend to omit any elements common to both input files, storing only the anchor offsets and the changes to be made at said offsets. A later development was the context diff format, which included the anchor elements and added the optional inclusion of a limited number of elements common to both input files and adjacent to (immediately preceding or following) the body of elements targeted by the diff. These additions were introduced as a way of representing the context in which the changes were to take place. Having context available not only made the diffs easier for humans to read, but also allowed for the implementation of patching algorithms that made use of the context in order to apply patches to files for which the diff anchor points did not match exactly.
In modern use the most common format is the unified context diff in which additions and removals are simply lines annotated with + and-respectively,
included intermixed with context lines. Anchor offsets are included in the form of hunk headers@@[...]@@, which also allow for a single diff file to include multiple change sets. The header@@ -142,5 +142,7 @@denotes a change in the block of lines 142-146 of the original file, producing the block of lines 142- 148 in the updated file. Listing 1 contains an example of the unified context diff format.
Listing 1. Example of unified context diff.
1 @@ -142,5 +142,7 @@
2 end
3 local args, callback, ret
4 - table.insert(history, text)
5 + if active_command == false then
6 + table.insert(history, text)
7 + end
8 props['neode.command.argv'] = text:gsub('^%S+%s+', '')
9 if active_command then
2.1.2
patchThe originalpatchprogram was developed by Larry Wall [6] as a companion todiffthat, given a diff in the context diff format, applied the changes to a file or a set of files. Modern versions of patch are able to use the unified context format. By pattern matching against the context elements, changes can be applied even to files that differ slightly from the file used to produce the diff. For example, a new line of text introduced preceding a block targeted for patching can cause the line numbers specified in a patch to no longer match the line numbers of the block in the file. By scanning for the context elements, patchis able to recover from such mismatches to a limited degree. Bothdiff andpatchwere eventually made part of the POSIX standard [7].
The combination of diffand patchhas been useful for many software projects. In particular, the ability to share succinct representations of changes to source code has been important for projects with a geographically dispersed set of loosely coupled participants such as the Linux kernel and other free soft- ware projects. However, the lack of support for the semantics of the program- ming languages in which the changes are to be applied give rise to a number of limitations. For example, it is common for a change in library code to induce corresponding required updates in a large number of files containing client code using the library API. It is likely for client code to pass variables as ar- guments to library functions, but unlikely for all client code to use the same
variable names. As such, if an API change is in the parameters to a library function it may not be possible to construct a single succinct diff that can up- date all of the client code.
2.2 Formal logics for the modeling of computer programs
A formal logic is a formal language consisting of a strict, mathematically- defined vocabulary and grammar, coupled with a formal semantics that define how to judge the truth of a given statement in the language. This section in- troduces the formal logics Computation Tree Logic (CTL) and the series of extensions CTL-FV (CTL with Free Variables), CTL-V (CTL with quantified Variables) and CTL-VW (CTL with Variables and Witnesses).
2.2.1 Computation Tree Logic
Computation Tree Logic (CTL) [8] is categorized as a branching-time tempo- ral logic, making it well suited for reasoning about systems that change over time in a branching structure. Such systems are commonly modeled using a directed graph, where each individual node represents a possible state of the system and edges represent possible state transitions.
CTL consists of the elements of propositional logic extended with a set of connectives that allows one to express properties of paths in a directed graph.
A typical variant of CTL is shown in Table 2.2, where ϕ denotes a formula (a syntactically legal statement in the language), > and ⊥ denote trueand falserespectively and p denotes an atomic proposition.
ϕ ::=> | ⊥ | p | ¬ϕ | ϕ ∨ ϕ | ϕ ∧ ϕ | ϕ → ϕ | AX ϕ | EX ϕ | AF ϕ | EF ϕ | AG ϕ | EG ϕ | A[ϕ U ϕ] | E[ϕ U ϕ]
For all paths There exists a path
ϕholds in the neXt state: AX ϕ EX ϕ
ϕholds now or in the Future: AF ϕ EF ϕ ϕholds now and forever (Globally): AG ϕ EG ϕ ϕholds Until ψ holds: A[ϕ U ψ] E[ϕ U ψ]
Table 2.2: Typical variant of CTL.
A simple branching system is shown in Figure 2.1, in which we have that, for example, AG(p∨ q), EX ¬q and AF(AG q) all hold in state s1.
p
p, q
p
q
s1
s2
s3
s4
Figure 2.1: A simple branching system example.
The formal semantics of CTL are defined in terms of a model which cap- tures the properties of the system under inspection.
Definition 1. A CTL modelM is a triple (S, T, L) where S is the set of states, T is a binary relation over S such that (s1, s2) ∈ T if the system contains a directed transition from state s1 to state s2, and L is a function (the ”labeling function”) from S to the powerset of atomic propositions relevant to the system.
For technical reasons we also require that every state has at least one successor:
∀s ∈ S. ∃s0 ∈ S. (s, s0)∈ T . □
The modelM corresponding to the picture of Figure 2.1 is:
M = (S, T, L) where S = {s1, s2, s3, s4},
T ={(s1, s2), (s1, s3), (s2, s4), (s3, s4), (s4, s4)}, L(s1) ={p}, L(s2) ={p, q}
L(s3) ={p}, L(s4) ={q}
The semantics of CTL are recursively defined on the formM, s ⊨ ϕ. The notation is to be read as ”ϕ holds (is true) in state s of modelM”. The seman- tics for the variant of CTL of Table 2.2 is shown in Table 2.3.
It is common to treat the problem of performing judgments on CTL formu- lae as an instance of the more general notion of model checking [9], which is a generic term for algorithmic methods of verifying that a given specification holds on a given finite state graph model of a system.
For a modelM = (S, T, L) and a state s ∈ S :
M, s ⊨ > trueholds in any state of any model.
M, s 6⊨ ⊥ falsedoes not hold in any state of any model.
M, s ⊨ p ⇔ p ∈ L(s) pholds in states labeled with p M, s ⊨ ¬ϕ ⇔ M, s 6⊨ ϕ ¬ϕ holds in states where ϕ does not hold.
M, s ⊨ ϕ ∨ ψ ⇔ M, s ⊨ ϕ OR M, s ⊨ ψ M, s ⊨ ϕ ∧ ψ ⇔ M, s ⊨ ϕ AND M, s ⊨ ψ M, s ⊨ ϕ → ψ ⇔ M, s 6⊨ ϕ OR M, s ⊨ ψ M, s ⊨ AX ϕ ⇔ ∀(s, s0)∈ T. M, s0 ⊨ ϕ
AX ϕ holds in states where ϕ holds in all immediate successors.
M, s ⊨ EX ϕ ⇔ ∃(s, s0)∈ T. M, s0 ⊨ ϕ
EX ϕ holds in states where ϕ holds in some immediate successor.
M, s ⊨ AF ϕ ⇔ ∀P ∈ Path(s). (∃s0 ∈ P. M, s0 ⊨ ϕ)
AF ϕ holds in states where all paths contain a state where ϕ holds.
M, s ⊨ EF ϕ ⇔ ∃P ∈ Path(s). (∃s0 ∈ P. M, s0 ⊨ ϕ)
EF ϕ holds in states where some path contains a state where ϕ holds.
M, s ⊨ AG ϕ ⇔ ∀P ∈ Path(s). (∀s0 ∈ P. M, s0 ⊨ ϕ)
AG ϕ holds in states where ϕ holds in every state of all paths.
M, s ⊨ EG ϕ ⇔ ∃P ∈ Path(s). (∀s0 ∈ P. M, s0 ⊨ ϕ)
EG ϕ holds in states where ϕ holds in every state of some path.
M, s ⊨ A[ϕ U ψ] ⇔ ∀P ∈ Path(s).(∃si ∈ P.(
M, si ⊨ ψ ∧ ∀j ∈ [1, i − 1]. M, sj ⊨ ϕ)) A[ϕ U ψ] holds in states where all paths contain a state s0 where ψ holds with ϕ holding in every state preceding s0 on the path.
M, s ⊨ E[ϕ U ψ] ⇔ ∃P ∈ Path(s).(∃si ∈ P.(
M, si ⊨ ψ ∧ ∀j ∈ [1, i − 1]. M, sj ⊨ ϕ)) E[ϕ U ψ] holds in states where some path contains a state s0where ψ holds with ϕ holding in every state preceding s0 on the path.
Where Path(s) is the set of all successor chains originating in s, i.e all possible paths starting in s. Note that for all paths P ∈ Path(s) we have s∈ P , i.e all paths include the current state.
Path(s) ={(s1, s2, ...)| s1 = s∧ (si, si+1)∈ T ∧ i ∈ [1, ∞]}
Table 2.3: Semantics of typical CTL.
Listing 2. Pseudocode for the CTL model checking algorithm.
1 function FIX(f,X):
2 // Given a function f and an initial value X, returns the
3 // fixpoint f(f(...f(X)))
4 repeat:
5 Xold := X
6 X := f (X)
7 if X = Xold:
8 return X
9
10 function preAny(M,X):
11 // Given a model and a set of states X, returns the set of
12 // states which CAN transition into a state in X
13 return {s ∈ MS | ∃(s, s′)∈ MT. s′ ∈ X}
14
15 function preAll(M,X):
16 // Given a model and a set of states X, returns the set of
17 // states which can ONLY transition into a state in X
18 return {s ∈ MS | ∀(s, s′)∈ MT. s′ ∈ X}
19
20 function SAT(M,ϕ):
21 ϕ match:
22 case >: return MS
23 case atomic: return {s ∈ MS | ϕ ∈ ML(s)}
24 case ¬ϕ1: return MS−SAT(M,ϕ1)
25 case ϕ1∧ ϕ2: return SAT(M,ϕ1) ∩ SAT(M,ϕ2)
26 case EX ϕ1: return preAny(M,SAT(M,ϕ1))
27 case AX ϕ1: return preAll(M,SAT(M,ϕ1))
28 case E[ϕ1U ϕ2]:
29 X := SAT(M,ϕ2);
30 return FIX(λX7→ X ∪(SAT(M,ϕ1) ∩ preAny(M,X)), X)
31 case A[ϕ1U ϕ2]:
32 X := SAT(M,ϕ2);
33 return FIX(λX7→ X ∪(SAT(M,ϕ1) ∩ preAll(M,X)), X)
EX
∧
AF q p
Model checking the formula EX(p∧ AF(q)):
Given a modelM = (S, T, L), we begin by checking the for- mula leaf nodes p and q. We mark with p all states s ∈ S for which p∈ L(s) and with q all states s ∈ S for which q ∈ L(s).
Next, moving upwards in the formula parse tree, we mark with AF(q) all states marked with q, and then repeatedly apply the AF(q) mark to states for which all successors are marked with AF(q), until there is no change. Next, we mark with p∧ AF(q) all states that are marked with both p and AF(q), before finally marking with EX(p∧ AF(q)) all states that have at least one successor marked with p∧ AF(q). The result is the set of states marked with EX(p∧ AF(q)).
Figure 2.2: CTL model checking example.
The basic CTL model checking algorithm [8]SAT takes as input a model M = (S, T, L) and a CTL formula ϕ and outputs the set of states in which the formula holds:
SAT(M, ϕ)def={s ∈ S | M, s ⊨ ϕ}.
The algorithm takes the form of bottom-up propagation of results corre- sponding to the subformulas of the parse tree of a given formula, with filtering taking place when results of lower-level subformulas are collected by higher- level subformulas. An example of model checking the formula EX(p∧ AF(q)) is shown in Figure 2.2.
Several of the CTL connectives can be expressed in terms of the other con- nectives [8]. For example, AG ϕ is semantically equivalent to ¬EF(¬ϕ). By exploiting these equivalences it is possible to reduce the size of the semantics, or — perhaps more importantly, to reduce the size and complexity of a com- puter algorithm that performs model checking. It can be shown [8] that any set of connectives that includes EU, at least one of {AX, EX} and at least one of {EG, AF, AU} is able to derive the missing connectives.
Listing 2 represents a complete algorithm of SATfor the typical variant of CTL. For sake of brevity the formulas accepted by the algorithm have been restricted to the subset
ϕ ::=> | p | ¬ϕ | ϕ ∧ ϕ | EX ϕ | AX ϕ | E[ϕ U ϕ] | A[ϕ U ϕ]
Arbitrary input formulas would thus first need to be converted to this subset.
For the base connectives we have⊥ ≡ ¬>, ϕ ∨ ψ ≡ ¬(¬ϕ ∧ ¬ψ) and ϕ → ψ ≡ ¬ϕ ∨ ψ, while for the path connectives AG, EG etc. there are several alternative derivations [8].
The AU connective is useful for specifying global properties over arbitrary computation paths. For example, one could use the AU connective to specify that a certain variable is deleted exactly once:
A[¬(x-deleted) Ux-deleted]
However, the AU connective can become problematic in models that con- tain loops. Computer programs commonly contain loops that may in practice always terminate, but for which the possibility of an endless loop cannot be excluded by static analysis. From a CTL perspective, such a model may allow for infinite paths that endlessly loop over a sequence of states. The semantics of the AU connective requires the ψ sub-formula of the expression ϕ U ψ to eventually be satisfied on all paths, which means the existence of an unrealistic, infinite successor path on which ψ is never satisfied could cause a complete formula A[ϕ U ψ] to fail to be satisfied for certain states.
One approach to this issue is to replace, when appropriate, the use of the AU connective with what is known as the AW connective [4]. The AW con- nective can be viewed as a Weaker variant of AU, with the typical semantics of AW [8] being identical to that of AU with the single modification of lifting the requirement of the eventual satisfaction of ψ. Figure 2.3 presents the typical semantics for AW.
M, s ⊨ A[ϕ W ψ] ⇔ M, s ⊨ A[ϕ U ψ] ∨ AG ϕ
Figure 2.3: Typical semantics of AW.
A derivation of the AW connective using the CTL subset accepted by the pseudocode of Listing 2 is A[ϕ W ψ]≡ ¬E[¬ψ U ¬(ϕ ∨ ψ)] [8].
2.2.2 CTL with free variables
CTL with Free Variables (CTL-FV) [4] is an extension of CTL that:
1. Introduces an environment θ to all judgments: M, s ⊨θ ϕwhere θ is a function from (logical) variable names to a set of concrete values.
2. Replaces atomic propositions p with n-ary predicates p(x1, ..., xn).
The semantics of predicates involves the environment:
M, s ⊨θ p(x1, ..., xn) ⇔ p(θ(x1), ..., θ(xn))∈ ML(s)
In the definition of a modelM, the codomain of the labeling function Lchanges to the powerset of n-ary predicates.
3. Replaces theSATfunction with a functionSATFV that returns the set of state-environment pairs that satisfy a given formula:
SATFV(M, ϕ)def= {(s, θ) | M, s ⊨θ ϕ}
For model checking, when considering the expressionM, s ⊨θ p(x1, ..., xn) the environment θ can be treated as an unknown to be determined by resolv- ing the constraint p(θ(x1), ..., θ(xn)) ∈ ML(s). As an example, let some modelM0contain a state s0labeled with the predicate f (1, 2). The expression M0, s0 ⊨θ f (x1, x2)with θ unknown then implies θ(x1) = 1and θ(x2) = 2, so we would have
(s0, [x1 7→ 1, x2 7→ 2]) ∈SATFV(M0, f (x1, x2)).
Resolving the environment bindings in this way requires adaptations to the model checking algorithm:
• SATFV(M, ϕ ∧ ψ) must test for compatible merges between the environ- ments of results yielded from checking the two subformulas:
SATFV(M, ϕ ∧ ψ) = {(s, θ) | s ∈ MS∧ (s, θ1)∈SATFV(M, ϕ)
∧ (s, θ2)∈SATFV(M, ψ)
∧ θ1u θ2 = θ6= ⊥}
where θau θb =
(Merge of θaand θb, θa, θb compatible
⊥ θa, θb incompatible
The formal details of environment merges and compatibility will be left undefined at this point1. Informally, two environments are compatible if they do not contain contradictory information, and a merge of two environments is an environment that contains bindings for all logical variables jointly present in the two environments being merged while being compatible with each of them.
• SATFV(M, AX ϕ) must generate one result for every compatible merge of complete sets of successor environments found inSATFV(M, ϕ):
Definition 2. In the context of a modelM, a state s and a formula ϕ, a complete set of successor environments is a set E ={θ1, ..., θn} such that for every s0a successor to s there is at least one environment θi ∈ E
such thatM, s0 ⊨θi ϕ. □
Given a state s, a formula ϕ and some complete set of successor environ- ments{θ1, ..., θn} covering all successors to s, if it is possible to merge all the environments in the set into a single environment θ, then it must be the case that M, s ⊨θ AX ϕ. Similar reasoning on sets of succes- sor environments must be applied in the handling of AF ϕ, AG ϕ and A[ϕ U ψ]. TheSATfunction for AX ϕ takes the form
SATFV(M, AX ϕ) = {(s, θ) | s ∈ MS
∧ {s1, ..., sn} = {s0 | (s, s0)∈ MT}
∧ {(s1, θ1), ..., (sn, θn)} ⊆SATFV(M, ϕ)
∧ θ1u ... u θn = θ6= ⊥}
Note that there may exist many complete sets of successor environments formed according to Equation 2.1.
{(s1, θ1), ..., (sn, θn)} ⊆SATFV(M, ϕ). (2.1)
• SATFV(M, ¬ϕ) must include
1. All states for which ϕ does not hold under any environment.
2. All states for which ϕ holds under some environment(s), and for every such state-environment pair must specify the negation of its environment.
1Complete definitions are available in [4]
TheSATfunction for¬ϕ takes the form
SATFV(M, ¬ϕ) ={(s, ∅) | s ∈ MS∧ ∀(s0, θ)∈SATFV(M, ϕ). s 6= s0}
∪ {(s, ¬θ) | s ∈ MS∧ (s, θ) ∈SATFV(M, ϕ)}
where∅ is the empty environment.
The formal details of environment negation will be left undefined at this point2. Informally, the idea is that a negated environment is the environ- ment that is maximally contradictory to the original environment, with every binding mapped to the ”opposite” value of the original binding.
CTL-FV is an improvement over CTL in the context of reasoning over models of computer programs. The introduction of logical variables, environ- ments and predicates allow for the construction of generalized formulas able to match many programs, using logical variables to bind to arbitrary program elements such as program variable names. For example, one could express a formula checking for the existence of an unnecessary local variable as
ϕ=VarInit(x)∧ AX(A[¬VarUsed(x) UReturnVar(x)])
whereVarInit,VarUsedandReturnVarare predicates of a single logical variable. This formula would hold in model MA for the function shown in Listing 3 withSATFV(MA, ϕ) ={(s1, [x 7→result])}, but would not hold in modelMB for the function shown in Listing 4.
Listing 3. Example function A.
1 int fnA()
2 {
3 int result = 10;
4 bar();
5 return result;
6 }
MA= (S, T, L)where S = (s1, s2, s3)
T ={(s1, s2), (s2, s3), (s3, s3)} L(s1) ={VarInit(result)} L(s2) ={}
L(s3) ={ReturnVar(result)}
2Complete definitions are available in [4]
Listing 4. Example function B.
1 int fnB()
2 {
3 int result = 10;
4 bar(result);
5 return result;
6 }
MB = (S, T, L)where S = (s1, s2, s3)
T = {(s1, s2), (s2, s3), (s3, s3)} L(s1) = {VarInit(result)} L(s2) = {VarUsed(result)} L(s3) = {ReturnVar(result)}
2.2.3 CTL with quantified variables
CTL with quantified Variables (CTL-V) [4] is an extension of CTL-FV that introduces existentially quantified logical variables in formulas:
ϕ ::= ...| ∃x.ϕ
CTL-V extends the functionSATFV in the form of the functionSATV that additionally handles this new type of formula element. The semantics of a formula∃x.ϕ is defined over the codomain of the environment function (the set of concrete values that the logical variables can be mapped to), herein referred to as Val.
M, s ⊨θ ∃x.ϕ ⇔ ∃v ∈ Val. (M, s ⊨θ[x7→v] ϕ)
Note that this definition does not include the mapping [x 7→ v] in the environment associated with the left hand side expression, meaning the map- ping should not be included in results produced by SATV(M, ∃x.ϕ). Tak- ing some model M0 to contain a state s0 labeled with the predicate f (1, 2) and taking Val to be {1, 2}, the expression M0, s0 ⊨θ ∃x. f(x, y) holds by M0, s0 ⊨θ[x7→1]f (x, y)with the single requirement of θ(y) = 2.
As an example of why it would be incorrect for theSATV function to in- clude the mapping in the environment for the left hand side expression, con- sider model checking the formula∃x.f(x, 2)∧∃x.f(1, x) in the same example modelM0using aBADSATVfunction that did include the mapping:
(s0, [x7→ 1]) ∈BADSATV(M, ∃x.f(x, 2)) (s0, [x7→ 2]) ∈BADSATV(M, ∃x.f(1, x))
(s0, θ)6∈BADSATV(M, ∃x.f(x, 2) ∧ ∃x.f(1, x)) since [x7→ 1] u [x 7→ 2] = ⊥
CTL-FV requires that each logical variable can be bound to a single con- sistent concrete value. By the addition of existentially quantified logical vari- ables, CTL-V allows for expressing formulas where logical variables can bind to different values along different computation paths.
Looking at state 1 of the model in Figure 2.4 and taking Val ={1, 2, 3} we have f (x)∧ AX(∃y. g(y)), but we do not have f(x) ∧ AX(g(y)) which is the closest expression possible using CTL-FV.
f (1)
g(2)
h(1, 2)
g(3)
h(1, 3) 1
2
3
4
5
Figure 2.4: Example model.
2.2.4 CTL with variables and witnesses
CTL with Variables and Witnesses (CTL-VW) is an extension of CTL-V [4].
While CTL-V allows for reasoning, specification and model checking using parameterized propositions (predicates) and scoped logical variables (able to take on different values along different computation paths), it does not record the bindings of quantified logical variables involved in a satisfied formula. In the context of code transformation, having access to these bindings is useful.
CTL-VW was developed in part to address this shortcoming of CTL-V.
As an example of the need for variable bindings to be recorded, a transfor- mation may want to swap the order of two program variables that appear as parameters to a function call. Such a transformation rule consists of two parts:
a match rule and a production (replacement) rule. Specifying the match rule using logical variables avoids the need for writing separate rules for different cases of program variable names, but also creates the requirement for the pro- duction rule to be able to read the values bound to the logical variables in order to produce the correct replacement.
CTL-VW extends CTL-V by:
1. Introducing the notion of witnesses.
A witness is a 4-tuple (s, x, v∈ Val, {ω1, ..., ωn}), comprising a record of an environment mapping between a logical variable x and a value v that was involved in satisfying some formula in a specific state s, along with the set of other witnesses{ω1, ..., ωn} that were established prior to constructing the new record.
2. Changing judgments to be on the form M, s ⊨θ,Ω ϕ, where Ω is a set of witnesses. The SATVW function therefore computes a set of state- environment-witnesses triples:
SATVW(M, ϕ)def= {(s, θ, Ω) | M, s ⊨θ,Ω ϕ}
Importantly, the semantics of existentially quantified variables changes to:
M, s ⊨θ,{(s,x,v,Ω)} ∃x.ϕ ⇔ ∃v ∈ Val. M, s ⊨θ[x7→v],Ωϕ
Thus using witnesses to capture the bindings that are dropped from the environment.
In the model checking of CTL-VW, when results of subformulas are com- bined by their enclosing connectives (such as∧ or AX), sets of witnesses of results being combined are joined under regular set union. As an example, looking at state 1 of the model in Figure 2.4 and taking Val ={1, 2, 3} we have
(s1,∅, {(s1, x, 1,{(s2, y, 2,∅), (s4, y, 3,∅)})})
∈SATVW(M, ∃x.(f(x) ∧ AX(∃y. g(y))))
Witnesses never influence the matching process in the model checking al- gorithm. Therefore, they can be used to have theSATVWfunction produce an output that associates states with arbitrary information. Again using the exam- ple model of Figure 2.4 and taking Val ={1, 2, 3}, if we introduce the formula x := yto mean the unconditional assignment of environment variable x to the value y, we can construct the formula
∃x.(f(x) ∧ AX(∃y.(g(y) ∧ ∃op. op :=delete)))
Model checking this formula against the model in Figure 2.4 would pro- duce the result
(s1,∅, {(s1, x, 1,{(s2, y, 2,{(s2, op,delete,∅)}), (s4, y, 3,{(s4, op,delete,∅)})})})
Such a result can be seen as a tree encoding transformation operations, where operation instructions are found in the leaf nodes of the tree and any variable bindings available for use by a particular operation can be collected on the path to the leaf.
2.3 Program analysis and transformation
The term program analysis refers to the notion of algorithmically analysing the source code of a computer program in order to reason about its behavior without actually executing the program [10]. An alternative, often-used term is static analysis, which emphasizes that such analyses target a static, state- independent view of the program. Common approaches include Data Flow Analysis and Abstract Interpretation. In Data Flow Analysis, programs are usually modeled as directed graphs over which analyses are specified. One such analysis is Reaching Definition Analysis, in which each definition of the formx := exprproduces a list of subsequent graph nodes for which the particular definition still holds for the variable x. In Abstract Interpretation, some of the concrete types of values involved in the computations performed by the program are replaced with abstract values. For example, one could replace the notion of general integer values with the more abstract notion of signs equipped with arithmetical rules such asPos+Pos= PosandPos+Neg
=Unknown. Interpreting a block of code using the abstract values and compu- tations instead of the concrete ones may then produce an output that describes certain behavior, such as whether a certain function given a positive input will always produce a positive output.
Similarly, the term program transformation refers to the notion of algo- rithmically modifying the source code or object code of a computer program, typically based on the result of some analysis. For example, a Reaching Defi- nition Analysis may be used to find variable definitions that are never used and may therefore be removed. An algorithm taking the result of such an analysis and removing the unused variable definitions from the source code or object code of the analysed program would then fall under the notion of program transformation.
One application of program transformation is the automation of API migra- tion. The majority of modern software programs delegate part of their func- tionality to one or more libraries [11, 12, 13], with the most common form of delegation consisting of using library-provided functions to perform certain tasks. The functions provided by a library together with rules and policies for how the functions are to be used (e.g the ordering of a series of related function calls) form the API (Application Programming Interface) of the library.
The term API migration refers to the general notion of making changes to application code that uses a library API in order to adapt it to a different version of the API or to a different API that provides similar functionality, without fun- damentally changing the functionality of the application [14]. Adapting code to a different version of an API commonly occurs when software developers wish to upgrade to a newer version of a library. Switching to a different API can occur in adapting software to a different platform where a certain library may not be available.
In a more formal sense, an instance of an API migration between two spe- cific APIs can be seen as a mathematical mapping between elements (such as function calls) of the first API to elements of the second API. This model permits the useful characterization of API migrations by their relationship car- dinality3, such as being one-to-one or many-to-many. A one-to-one API mi- gration may consist of a single function call in the first API being replaced by a single corresponding call in the second API. A many-to-many API migration may prescribe that a certain order-sensitive series of function calls in the first API is to be replaced by a certain series of calls in the second API, which when taken together perform a corresponding task. The task of automating the ap- plication of many-to-many API migrations to programs remains challenging [12].
2.3.1 Spoon
Spoon [2] is a free and open-source metaprogramming library designed for the analysis and transformation of Java source code. Spoon is created and maintained under the helm of Inria, the French National Institute for Research in Digital Science and Technology. Spoon was first presented in 2005 [15] and remains under active development.
The central feature of Spoon is its ability to convert between Java source code and an extensive metamodel – a rich representation of the Abstract Syntax Tree (AST) corresponding to the source code. The metamodel supports effi-
3An alternative term is mapping multiplicity.
cient searching, introspection (self-examination) and intercession (self- modification). The current version of Spoon parses Java source code using the JDT4library from the Eclipse foundation.
One of the main ways of building code analyses and transformations using Spoon is by implementing a Spoon Processor. Listing 5 shows a simple Spoon processor that logs occurrences of emptycatchclauses. Theprocess method will be called on everyCtCatch metamodel element constructed by Spoon when parsing some given target source code, which will typically be a full project directory scanned recursively.
Listing 5. Example Spoon Processor for analysis.
1 class CatchProcessor extends AbstractProcessor<CtCatch> {
2 @Override
3 public void process(CtCatch element) {
4 if (element.getBody().getStatements().size() == 0) {
5 getFactory().getEnvironment()
6 .report(this, Level.WARN, element,
7 "empty catch clause");
8 }
9 }
10 }
Spoon processors are also suitable for implementing code transformations.
Listing 6 shows the processmethod of the processor from Listing 5 altered to instead modify the code by inserting into every emptycatchclause a call to some logging facility (assumed to exist in the target code base) providing the caught exception as the singular argument. The method loggerCall, of which no definition is provided in the example, is assumed to construct a valid Spoon metamodel object corresponding to the call.
Once processing is complete, Spoon will convert (pretty-print, unparse) the metamodel structure back into one or more Java source files. As the inter- cession API modifies the metamodel in-place, the produced source code will be affected by any modifications.
4https://wiki.eclipse.org/JDT_Core
Listing 6. Example Spoon Processor for transformation.
3 public void process(CtCatch element) {
4 if (element.getBody().getStatements().size() == 0) {
5 CtCatchVariable<?> exception = element.getParameter();
6 element.getBody().addStatement(loggerCall(exception));
7 }
8 }
2.3.2 Semantic Patch Language
The Semantic Patch Language (SmPL) was originally proposed [1] as part of a solution to the problem of managing collateral evolutions [16] in the Linux kernel source code. Collateral evolutions are modifications to code that uses some library API, made in direct response to changes made to the library API.
Such code is often referred to as client code, motivated by the idea that the library API provides a service to which the code forms a client.
SmPL mimics the familiar unified context diff format. Additions are taken as lines beginning with a plus character and deletions are taken as lines begin- ning with a minus character. Lines starting with a character that does not have a special meaning (such as plus and minus) are taken to be context lines that together with any deletions define the pattern to be matched in source code targeted for patch application. SmPL adds support for the syntax and seman- tics of the C programming language by operating on abstract syntax trees and control flow graphs rather than the plain text. By exploiting the support for the syntax and semantics of C, a number of powerful new features are introduced.
Although SmPL was developed specifically for C, the core concepts are applicable to other languages. For many C-like languages such as Java and PHP, the core concepts are likely to apply in a near-identical fashion. Moreover, other programming languages such as Python and Lisp can be fitted to a model of abstract syntax trees and control flow graphs that is similar in structure to a corresponding model for C. Mapping the core features of SmPL onto such languages would therefore be straightforward. In this thesis, the term SmPL will tend to refer to a generic notion of SmPL that is not coupled to any specific target programming language, and we will tend to write SmPL for X when referring to an SmPL that targets the specific language X.
A first core feature of SmPL is metavariables. Metavariables solve the problem of creating a generic patch for collateral evolutions where different instances of client code uses different variable names in calls to an API. For
example, the semantic patch shown in Listing 7 will match and transform any call to the functionfoousing three variables as arguments regardless of what their names are. The patch further shows how metavariables are available in transformations, as the call to foo will be replaced with a call to a function barwith the matched arguments arranged in a different order.
Listing 7. Metavariables in SmPL.
1 @@
2 identifier x,y,z;
3 @@
4 - foo(x,y,z)
5 + bar(z,x,y)
Metavariables are themselves typed, with the three metavariables shown in Listing 7 having typeidentifiermeaning they can only be bound to iden- tifiers such as variable names or function names. Many metavariable types are available, examples includeconstant,expressionandstatement, as well as explicit data types such asint.
A second core feature of SmPL is the wildcard operator ”...”, often re- ferred to as dots. The wildcard operator matches arbitrary sequences of state- ments over arbitrary computation paths in the control flow graph. The opera- tor also includes the ability to impose customized constraints on each instance.
In the semantic patch shown in Listing 8, the... on line 7 will match all se- quences of statements over all computation paths starting from a match of the preceding lineT ret = C;until there is a match forreturn ret;. Note that this means that all paths are required to eventually reach a match forreturn ret;in order for the operator to match. Additionally, the match will only be considered successful under the constraint specified by when != ret, which says that none of the statements found along any traversed path may involve the use of the variable name bound to the metavariableret. Furthermore, by default the wildcard operator will only match the shortest path between the el- ements that enclose it, meaning the preceding and succeeding statements are not allowed to appear on any intermediate step of any path. These and other behaviors can be altered with constraint modifiers specified on an operator in- stance or by redefining the default constraints for a single rule. Examples of constraint modifiers include the relaxationswhen existsthat lifts the require- ment that all paths need to satisfy all of the constraints, andwhen anythat lifts the shortest-path requirement.
