IN
DEGREE PROJECT INFORMATION AND COMMUNICATION TECHNOLOGY,
SECOND CYCLE, 30 CREDITS ,
STOCKHOLM SWEDEN 2016
An Embedded Multi-Core Platform
for Mixed-Criticality Systems
Study and Analysis of Virtualization Techniques
YOUSSEF ZAKI
KTH ROYAL INSTITUTE OF TECHNOLOGY
An Embedded Multi-Core Platform for Mixed-Criticality
Systems
Study and Analysis of Virtualization Techniques
Youssef Zaki
Master of Science Thesis
KTH Royal Institute of Technology
School of Information and Communication Technology Stockholm, Sweden
17 August 2016
Abstract
The common availability of multiple processors in modern CPU devices and the need to reduce cost of embedded systems has created a drive for integrating functionalities from different parts of a system into a single Multi-Processor System-on-Chip (MPSoC) device. As a result, system resources are shared amongst the critical and non-critical components of the system, which results in a mixed-criticality system (MCS). An example of a MCS is to combine an airbag control unit with the infotainment system of a car, in such a case, both components must be certified unless an isolation mechanism that can prevent the non-critical to interfere with the critical subsystems is implemented. This isolation can be achieved via spatial and temporal partitioning of system resources, such as static mapping of CPUs to critical tasks, memory and IO virtualization, and time domain multiplexing of applications.
System isolation is currently achievable through virtualization techniques, and is commonly used in data centers and personal computers. Recently, virtualization solutions have been emerging for embedded systems in order to cope with the increased design complexity, the stringent non-functional requirements, and to facilitate the certification process of MCS. The achieved performance, safety, security, and robustness in a virtualized system depends on the virtualization architecture and hardware platform.
This thesis work performs state-of the art research in the field of mixed-criticality embedded systems with a focus on virtualization of embedded systems. As a result, a deep study of virtualization architectures, and open-source virtualization solutions is conducted in order to understand the consequences of using this technology in MCS. The work is concluded with a design and implementation of mixed-criticality embedded system that leverages the hardware capabilities of the target device (Zynq-7000 all programmable SoC), and contributes to the Living Lab WP7 of the EMC2 project.
Keywords— Mixed-Criticality, EMC2, Safety, Security, Embedded Systems, Virtualization, Xilinx Zynq SoC
Acknowledgements
I would like to thank my advisors (Detlef Scholle and Johnny Öberg) and examiner (Ingo Sander) for their help and guidance throughout my master thesis work. I also thank my friend Mohamad Tagelsir (a.k.a Tage) and advisors for reviewing my thesis, the Department of Embedded Systems (ESY) at KTH Royal Institute of Technology for providing me with the development boards, and Alten Sverge AB for giving me the opportunity to be part of the EMC2 project.
Finally, I would like to thank my parents (Bouazza Zaki and Fatima Chakour) for their support and prayers, and my sister (Nada Zaki) for introducing me to KTH and for hosting me throughout my study period in Sweden.
Contents
1 Introduction 1 1.1 Background . . . 2 1.2 Problem Statement . . . 2 1.3 Purpose . . . 2 1.4 Goals . . . 3 1.4.1 Team Goal . . . 3 1.4.2 Individual Goal . . . 3 1.5 Initial Resources . . . 3 1.6 Method . . . 4 1.7 Thesis Structure . . . 4 2 Mixed-Criticality Systems 5 2.1 Mixed-criticality Systems. . . 5 2.1.1 Definition of Safety . . . 52.1.2 Safety in Mixed-Criticality Systems . . . 6
2.1.3 Security in Tradition Embedded Systems . . . 6
2.1.4 Security in Mixed-Criticality Systems . . . 7
2.2 Motivation for Using MPSoC . . . 7
2.2.1 Advantages of MPSoCs in MCSs . . . 8 2.2.2 Limitations of MPSoCs in MCS . . . 9 3 Hardware Platform 11 3.1 Zynq Overview . . . 11 3.1.1 Processing System . . . 11 3.1.2 Programmable Logic . . . 12
3.1.3 PS-to-PL Boundary Interfaces . . . 13
3.2 Application Processing Unit . . . 13
3.2.1 ARM . . . 13
3.2.2 ARM Instruction Set . . . 14
3.2.3 ARM Processor Modes . . . 14
vi Contents
3.2.5 Current Program Status Register . . . 16
3.2.6 Exception Handling in ARM . . . 16
3.2.7 ARM Coprocessors . . . 17
3.2.8 Virtual Memory System Architecture . . . 17
3.2.8.1 Memory Management Unit . . . 18
3.2.8.2 Page Tables . . . 18
3.2.8.3 Translation Lookaside Buffer . . . 18
3.2.9 ARM TrustZone Architecture . . . 19
4 System Virtualization 21 4.1 High-Level View of System Virtualization . . . 21
4.1.1 Type II Hypervisor . . . 22
4.1.2 Type I Hypervisor . . . 22
4.2 Virtualization Architectures for Embedded Systems . . . 23
4.2.1 Full Virtualization . . . 23
4.2.2 Paravirtualization . . . 24
4.2.3 Monolithic Hypervisor . . . 24
4.2.4 Console Guest Hypervisor . . . 24
4.2.5 Microkernel-Based Hypervisor . . . 25
4.3 Resource Management . . . 25
4.4 Hypervisor robustness . . . 26
4.5 Hardware Virtualization Acceleration . . . 26
4.5.1 Memory Virtualization . . . 28
4.5.2 Device and I/O Virtualization . . . 28
4.5.2.1 Emulation . . . 29
4.5.2.2 Pass-through . . . 29
4.5.2.3 Mediated Pass-through . . . 29
4.6 Virtualization Requirements for MCS . . . 30
5 Exploration of Available Hypervisor Solutions 31 5.1 Xen Hypervisor . . . 31 5.2 Xen Zynq . . . 32 5.3 SEL4 Microkernel . . . 32 5.4 TrustZone-based Hypervisor . . . 33 5.4.1 SierraVisor . . . 34 5.4.2 SafeG . . . 34
5.5 SICS Thin Hypervisor . . . 36
5.6 Hypervisor Solution Matrix . . . 36
Contents vii
6 System Implementation 41
6.1 Implementation Tools . . . 41
6.1.1 Xilinx Vivado . . . 41
6.1.2 Xilinx ARM Cross-Compiler . . . 41
6.2 System Architecture Overview . . . 42
6.3 Hardware Components . . . 42
6.3.1 Resource Planning . . . 42
6.3.2 Network-on Chip Subsystem . . . 44
6.3.3 Network-on Chip Integration . . . 45
6.4 Software Components . . . 45
6.4.1 SafeG Virtual Machine Monitor . . . 45
6.4.2 TOPPERS/FMP . . . 45
6.4.3 Linux OS . . . 45
6.4.3.1 Linux Kernel . . . 46
6.4.3.2 Root File System . . . 46
6.4.3.3 Device Tree Blob . . . 46
6.4.4 SHAPE . . . 46
6.4.5 SHAPE Services . . . 47
6.4.6 Inter OS Communication . . . 47
6.5 Project File Structure . . . 48
6.6 System Build Overview. . . 48
6.6.1 Xilinx Build . . . 49 6.6.2 OS Build . . . 51 6.6.3 VMM Build . . . 51 6.6.4 SOA Build . . . 51 6.7 Demo System . . . 52 6.7.1 System Boot . . . 52
6.7.2 Boot Sequence of Dual-OS System . . . 53
6.7.3 Hello-World Service . . . 53
6.7.4 Shared Memory Monitor Service . . . 55
6.8 System Test and Results . . . 55
6.8.1 Robustness . . . 55
6.8.2 Isolation Test . . . 56
6.8.3 Board-to-Board Communication. . . 56
6.8.4 Dual-OS Communication SHAPE Service . . . 56
7 Conclusion and Future Work 59 7.1 Conclusion. . . 59
7.2 Future work . . . 60
List of Figures
3.1 Summary of Zynq SoC Device . . . 12
4.1 Type II Hypervisor . . . 22
4.2 Type I Hypervisor . . . 23
4.3 OS Level Virtualization in 2-level Mode Hierarchy System . . 27
4.4 Type I Hypervisor Virtualization in 2-level Mode Hierarchy System . . . 27
4.5 Type I Hypervisor Virtualization in 3-level Mode Hierarchy System . . . 28
5.1 SafeG Architecture . . . 35
6.1 Hardware and Software Synopsis of Implemented System . . . 43
6.2 High-Level View of Project Directory . . . 48
6.3 System Build Overview Diagram . . . 50
6.4 Demo Setup . . . 52
6.5 Boot Sequence of the Dual-OS System . . . 54
List of Tables
3.1 CPSR Subfield Description. . . 16
3.2 ARMv7 Architecture Exception Vector Table . . . 17
3.3 ARM Page Table Details . . . 18
5.1 Hypervisor Solution Matrix . . . 37
6.1 Resource Planning . . . 44
6.2 Address Mapping . . . 53
6.3 Code Size Comparison of System Software Components . . . . 55
List of Acronyms
API Application Programming Interface
ASIC Application Specific Integrated Circuit
ASIL Automotive Safety Integrity Level
ARM Advanced Risk Machine
AXI Advanced eXtensible Interface
BSP Board Support Package
CPS Cyber Physical Systems
DTB Device Tree Blob
DTC Device Tree Compiler
DTS Device Tree System
DMA Direct Memory Access
ECU Electronic Control Unit
EMC2 Embedded Multi-Core systems for Mixed Criticality applications in dynamic and changeable real-time environments
FPGA Field Programmable Gate Array
FSBL First Stage boot Loader
GCC GNU Compiler Collection
GPL2 General Purpose License 2
xiv List of Acronyms
JTAG Joint Test Action Group
IDE Integrated Development Environment
IOMMU Input Output Memory Management Unit
IP Intellectual Property
MB MicroBlaze
MCS Mixed-Criticality System
MMU Memory Management Unit
MPSoC Multi-Processor System-on-Chip
NoC Network-on-Chip
NS Non-Secure
OCM On-Chip Memory
OS Operating System
PC Personal Computer
PL Programmable logic
POSIX Portable Operating System Interface
PS Processing System
PV Processing System
QoS Quality of Service
RTOS Real-Time Operating System
RTL Register Transfer Level
SDK Software Development Kit
SDSoC Software Defined System-on Chip
SHAPE Self-configurable High Availability and Policy based
platform for Embedded systems
List of Acronyms xv
SoC System-on Chip
SS SHAPE Service
SSBL Second Stage Boot Loader
TFTP Trivial File Transfer Protocol
TLB Translation Lookaside Buffer
UART Virtual Extension
VE Virtual Extension
VM Virtual Machine
VMM Virtual Machine Monitor
WCET Worst-Case Execution Time
Chapter 1
Introduction
Stringent non-functional requirements [1] of complex embedded systems together with the common availability of Multi-Processor System-on-Chip (MPSoC) devices have created a drive for integrating system functionalities that potentially have different criticality levels into a single computing platform [2]. As a result, these mixed-criticality systems must incorporate spatial and temporal partitioning mechanisms in order to avoid unwanted interactions between the critical and non-critical components, increase the security assurance level, and enable the certification of partitions independent of the other components in the system [3].
Virtualization is a technique that is commonly used in computers and servers to provide isolated execution environments and to support the execution of heterogeneous operating systems on the same hardware platform [4]. Virtualization of embedded systems has recently been a growing trend, mainly because it provides a mechanism to isolate execution environments. This approach provides safety and security measures, and facilitates the certification of safety-critical systems.
This thesis will investigate virtualization solutions for embedded systems, motivate the drive behind applying virtualization techniques in order to facilitate design and development of mixed-criticality systems, and conclude with the design and implementation of a Mixed-Criticality System (MCS) that demonstrate the mixed-criticality concept while remaining within the scope of the Living Lab Work Package 7 (WP7) of European research project ”Embedded Multi-Core systems for Mixed Criticality applications in dynamic and changeable real-time environments (EMC2)” [5].
2 Chapter 1. Introduction
1.1 Background
Currently, modern automotive systems contain a large number of Electronic Control Units that collectively constitute many heterogeneous single-core systems [6]. Each Electronic Control Unit (ECU) is optimized to execute an application with a specific criticality level such as safety-critical anti-lock brake system or non-critical entertainment systems [7]. This approach provides isolation for the numerous critical and non-critical applications in the collective system, and a simple mechanism to qualify an individual ECU. However, it yields an inefficient and expensive system implementation. In order to lower the cost of the system and increase performance, mixed-criticality applications can be integrated into a single multicore platform. This solution will reduce the number of ECUs in the system, which in turn lowers the manufacturing and maintenance costs [6].
Combining applications into a single multicore platform can greatly increase performance and reduce cost. However, this approach increases system complexity, and hinders the certification of safety-critical systems [3]. In order to facilitate the design, test, and certification of such systems, spatial and temporal partitioning can be used in the architecture of the system.
1.2 Problem Statement
Modern embedded systems are following the trend of integrating mixed-criticality applications into a single computing platform. In the automotive industry, for example, this integration would reduce the number of ECUs in a vehicle, which in turn reduces the manufacturing cost and increase the reliability of the system. However, this approach makes the certification of systems very tedious. This limitation can be overcome by providing spatial and temporal isolation to applications.
Virtualization techniques have been ported to embedded systems, and have demonstrated satisfactory results regarding the isolation of subsystems and system security. While many virtualization solutions exist, they all have different capabilities. Therefore, it is important to understand the basics of virtualization techniques in order to identify which virtualization solution is best suited for the EMC2 project?
1.3 Purpose
1.4. Goals 3 systems for commercial vehicles, as described in the EMC2 Living Lab WP7 [5].
1.4 Goals
This thesis is part of a larger team project that aims to develop the next generation of electronic systems suitable for Cyber Physical Systems (CPS) in the automotive industry. The platform will aim to exploit the potential of heterogeneous multi-cores for MCSs, and create an adaptive system able to adjust to changes in real-time environments.
The heterogeneous multi-core embedded platform should be suitable to serve as a computing base for CPS in the automotive industry. This CPS will include Service Oriented Architecture (SOA) to create an adaptive system that can respond to changes in real-time environments, Network-on-Chip (NoC) component to provide system scalability, and system virtualization to support concurrent execution of heterogeneous operating systems and achieve isolation between the critical and non-critical tasks of the system.
1.4.1 Team Goal
The team consists of three master thesis students, each working on a different layer of the system:
• Service Oriented Architecture layer enables a system to adapt to changing environment conditions
• Mixed-criticality Architecture enables applications with different criticality levels to concurrently execute in a single computing platform • Network-on-Chip enables efficient system scalability
1.4.2 Individual Goal
The individual goal is to investigate virtualization technologies as they relate to MCSs, establish design requirements, and develop a MCS prototype that includes the individual contribution of this thesis and integrates the work of the other team members.
1.5 Initial Resources
4 Chapter 1. Introduction will also target the same SoC device. In this case, the Zedboard, which is a development platform by Digilent, which contains the Zynq-7000 SoC at its core, is available (provided by KTH – Royal Institute of Technology) and will be used during the development process. Furthermore, Alten Sweden AB, the company where this thesis work is being conducted, is providing a proprietary software (SHAPE) that represents the SOA aspects of the system.
1.6 Method
This thesis will follow the applied research methodology, where knowledge is derived from well-known and accepted theories and principles, and is applied to solve specific problems [8]. This implies that the project will commence with State-of-the-Art research in the field of MCS with a focus on system safety, security, and certification. This investigation will help identify system requirements and guide the development direction for the remainder of the project.
1.7 Thesis Structure
In order for the reader to understand the work done in this thesis, it is important to gain an understanding of MCS, the Zynq-7000 SoC with a deep understanding of the ARMv7 processor architecture, and system virtualization. However, the reader can skip to Chapter 5 if he or she is already familiar with these topics. The following is a brief description of each subsequent chapter:
• Chapter 2 presents a brief background regarding mixed-criticality systems, and addresses the safety, security, and certification challenges associated with such systems. Therefore, this chapter is important for readers that do not have a background on MCS.
• Chapter 3 introduces the Zynq-based hardware platform with a focus on the ARMv7 architecture.
• Chapter 4 introduces the fundamentals of system virtualization. • Chapter 5 explores the available open-source virtualization solutions. • Chapter 6 presents the design, implementation, and test of the system
prototype.
Chapter 2
Mixed-Criticality Systems
This chapter gives an introduction to mixed-criticality systems, motivates the drive for pursuing such systems, and describes design challenges.
2.1 Mixed-criticality Systems
A current trend in embedded systems is to take advantage of the availability of multicore processor chips in order to consolidate subsystems, and achieve a higher CPU utilization. Naturally, the embedded systems that are present in CPSs, such as automotive vehicles, contain components (or ECUs) with different criticality levels. As an example, the task for checking airbag sensor and deploying the airbag when needed has a higher criticality level than controlling the volume of the infotainment system. Therefore, these components are integrated into a single computing platform, the response time of the airbag system should not be affected by non-critical functions of the infotainment system. As a result, scheduling the two functions into the same computing platform yields a mixed-criticality system.
The development of MCS must comply with safety and security regulations as dictated by each industry field (e.g. automotive, aerospace, railway) in order to certify products [2]. These industries have defined several criticality levels that depend on elements such as environment of operation and danger to human life.
2.1.1 Definition of Safety
”Safety is the absence of unacceptable risk, that is a system is safe if the risk associated with the system is acceptable” [9]. In industrial plants the safety constraint is often described as an average frequency of 10-3 large
6 Chapter 2. Mixed-Criticality Systems accidents per year [10]. As a result, all of computer systems, sensors, and other electronic components and subsystems of the plant must meet this constraint.
2.1.2 Safety in Mixed-Criticality Systems
The safety requirements in MCS are domain specific. Depending on the industry and application, the certification process will demand a specific integrity level, which is defined by industry standards such as RTCA DO-178B in aerospace [11], IEC 61508 in industrial control [12], EN 50129 in railway [13], and ISO 26262 for automotive [14]. The automotive industry, as an example, assigns Automotive Safety Integrity Level (ASIL)s, from ASIL A to ASIL D, to rank the level of protection required when creating a safety-critical system, where ASIL A indicates the lowest safety integrity level. Each level dictates guidelines that should be followed in order to achieve a safe and certifiable system within the targeted industry.
The characteristics of the required services for these industries also depends on the final application of the device. In the aerospace industry, the safety process requires both service integrity and availability because an airplane can not stop during flight if a service is no longer available. On the other hand, in railway signaling applications, only service integrity is sufficient to meet the safety criterion because a train can simply stop if a service is no longer available. As a result, availability is a secondary function for railway signaling systems [15].
2.1.3 Security in Tradition Embedded Systems
2.2. Motivation for Using MPSoC 7 However, the cryptographic module in this approach faces similar security robustness issues as the external hardware solution because it can only protect cryptographic key material. Moreover, the system monitor processor consumes precious silicon area, and is in general a low-performance processor, which increase the energy consumption of the system. Consequently, a new mechanism is required in order to produce a system that is secure and cost effective.
2.1.4 Security in Mixed-Criticality Systems
In addition to the traditional issues relating to securing information and protecting the system from external attacks, in MCSs tasks (or subsystems) of different criticality levels share system resources such as the processor, memory, and I/O devices. This implies that an isolation mechanism is required in order to prevent the non-critical components from affecting the execution of critical components.
One approach to remedy this problem is to virtualize the hardware system (section 4.1). In such a configuration, the hypervisor is the trusted element that governs the resources of the system. The hypervisor holds the highest privilege level in the system and controls the access of virtual machines to system resources. Furthermore, the hypervisor contains each virtual machine in an isolated environment that does not allow the propagation of errors. Therefore, if a guest machine is damaged, it will not contaminate the rest of the system. However, this implies that the security of the system is as robust as the hypervisor itself (section 4.4). As a result, it is recommended to minimize the size of the hypervisor in order to facilitate the verification and validation process. In some cases, the hypervisor can be formally verified, which yields a better guarantee that the system will behave as intended [17].
2.2 Motivation for Using MPSoC
8 Chapter 2. Mixed-Criticality Systems bring software and hardware design closer [19]. This approach has the potential of delivering systems with higher performance at a lower cost.
A recent trend in the design of embedded systems is to take advantage of the available computational power of multiprocessor system-on-chip (MPSoC) devices to integrate subsystems of different criticality levels into a single platform. This design approach reduces the number of individual physical components in the system, which can significantly reduce the costs associated with manufacturing and maintenance. Many industries, such as aerospace, railway, automotive, and industrial controls, are adopting this technique to integrate safety-critical, mission critical, and non-critical components into a single platform in order to reduce stringent non-functional requirements such as size, weight, power, and cost [7] [20]. While this approach offers many benefits, it also presents many design and certification challenges.
2.2.1 Advantages of MPSoCs in MCSs
MPSoCs have the potential to provide many benefits for embedded systems as opposed to increasing the micro-architecture complexity of a single core processor. According to Pollack’s Rule, within the same process technology, a leading single core processor can achieve a 40% increase in performance compared to the previous generation of microprocessor. However, when this rule is considered under the factors of power and area, it indicates that increasing the micro-architecture complexity to gain performance yields a diminishing return. On the contrary, implementing a multicore architecture can potentially result in a near linear increase in performance [21].
It could be argued that Amdahl’s Law is a limiting factor in increasing computational performance of MPSoC platforms. This argument would be correct if the target application is to accelerate a computation heavy algorithm via parallel processing. Then, the smallest amount of serial code would quickly saturate the achievable gain from multiple processors. However, in the case of MCSs, the goal is to integrate multiple independent applications, and most of which are inherently parallel as well [7].
2.2. Motivation for Using MPSoC 9 distribute the heat generated in the system for a fixed load. This integration strategy lowers the number of physical units in the system, which in turn reduces cost, and simplifies the hardware design of the system.
MPSoCs can also contain free logic components such as the FPGA fabric in Xilinx’s Zynq-7000 SoC device. The main processing system consist of two fast ARM Cortex-A9 cores, capable of operating at frequencies up to 1 GHz. The FPGA logic can provide custom blocks to accelerate the execution of specific tasks [22]. While current MPSoC platforms provide many benefit for embedded systems, they introduce some limitations with respect to safety in mixed-criticality systems.
2.2.2 Limitations of MPSoCs in MCS
Real-world electronic systems require certification in order to be used in safety-critical applications. Most of the current MPSoCs face many limitations in this aspect because they are considered as highly complex electronic devices. According to the classification and determination of COTS device characteristics section of the European Avionics Safety Agency (EASA) certification memorandum [23], a COTS micro-controller is classified as ”highly complex” if it contains one of the following features:
• ”more than one Central Processing Unit (CPU) are embedded and they use the same bus (which is not strictly separated or which uses the same single port memory)”
• ”several controllers of complex peripherals are dependent on each other and exchange data”
• ”several internal busses are integrated and are used in a dynamic way (for example, a dynamic bus switch matrix)”
These characteristics apply to most MPSoCs, which render system certification for safety-critical applications unattainable due to time and cost budget constraints.
10 Chapter 2. Mixed-Criticality Systems rather than WCET. Therefore, they lack temporal determinism, which is a critical element in safety-critical system with hard real-time constrains.
Chapter 3
Hardware Platform
Hardware imposes constraints during the design space exploration phase of the project. Therefore, in order to reach an optimal design solution, it is imperative to have a thorough understanding of the hardware architecture and its capabilities. As was previously stated, this project is restricted to the use of the Zedboard as a hardware platform. The core of this development board is a Zynq all programmable SoC* device. Therefore,
this section will cover some of the basics of Zynq’s system architecture such as the Processing System (PS), the Programmable Logic (PL), and their interconnect. Furthermore, this section will dive deeper into the Cortex-A9 architecture, which constitutes the central processing unit in Zynq.
3.1 Zynq Overview
Zynq combines the high performance of Application Specific Integrated Circuit (ASIC) devices and the flexibility of Field Programmable Gate Array (FPGA) fabric into a single die. The ASIC and FPGA components are represented by the PS and PL regions inFigure 3.1 respectively.
3.1.1 Processing System
The PS is a hardwired system that is composed of several third party Intellectual Property (IP)s from vendors such as ARM, Cadence, and Arasan [24]. The main processing unit is a dual-core Cortex-A9 ARM processor, which is a high performance processor that is commonly found in many commercial systems. Zynq is designed such that the PS processor always
* From this point forward, this device will simply be referred to as Zynq.
12 Chapter 3. Hardware Platform Processing System (PS) PS-to-PL Boundary 32 KB I-Cache ARM Cortex-A9 CPU MMU FPU/NEON Engine 32 KB D-Cache Snoop Controller, AWDT, Timer 512 KB L2 Cache M_AXI_GP x2 S_AXI_GP x2 S_AXI_HP x4 S_AXI_ACP x1 AXI Interfaces FCLKs IRQ, Event, Standby DMA Req/Ack DDR Arb, AXI Idle, SRAM Int FTMD Trace, FTMT Trigs Misc. PL Signals Programmable Logic (PL) EMIO JTAG MIO Boot Mode PS_CLK, POR_RST_N SRST_N 256 KB OCM DDR Controller TTC SWDT System Level Control Regs DMA 8 Channels GIC Application Processing Unit (APU) I/O Peripherals USB x2, GigE x2, SDIO x2, GPIO UART x2, CAN x2, I2C x2, SPI x2 DDR Memory User SelectIO XADC MGTX PL Signals
Figure 3.1: Summary of Zynq SoC Device
boots first, which enables a software centric approach to the PL system boot and configuration.
3.1.2 Programmable Logic
3.2. Application Processing Unit 13
3.1.3 PS-to-PL Boundary Interfaces
The PS and PL regions are tightly coupled via a number of communication ports that are visible as common regions in Figure 3.1. These interfaces can be placed into two categories:
1. Functional interfaces – include the Advanced eXtensible Interface (AXI) ports such as AXI_GP for general purpose master/slave device interface between PS and PL regions, extended MIO (EMIO) which enable PL IPs to access most I/O peripherals, interrupts, DMA flow control, clocks, and debug interfaces.
2. Configuration interfaces – these signals are connected to the configuration block of the PL, which allow the PS to control the configuration of the PL.
3.2 Application Processing Unit
The application processing unit (APU) constitutes the computing component of the PS. The APU contains a dual-core ARM Cortex-A9 processor system with a memory hierarchy of 32KB L1 instruction and data cache, and a shared 512KB L2 cache memory. Additionally, the APU also includes a 256KB On-Chip Memory (OCM), dedicated local timers for each core, and shared timers for the system. The PS also serves as the main unit to access I/O peripherals such as UART, USB, and Ethernet ports [24].
The remainder of this section will focus on the ARMv7-A architecture, which is the instruction set architecture (ISA) of the Cortex-A9 system. This version of the ARMv7 architecture is the application profile that supports Virtual Memory System Architecture (VMSA), which is based on the utilization of the Memory Management Unit (MMU) of the processor system [25].
3.2.1 ARM
ARM (Advance RISC Machine) ISA follows a Reduced Instruction Set Computer (RISC) architecture approach, which implies features such as:
• Large uniform register file.
14 Chapter 3. Hardware Platform • Simple addressing modes.
In addition to the core RISC architecture features, ARM includes additional instructions such as combined shift with an arithmetic or logical operation, automatic address increment/decrement operations to optimize program loops, load and store multiple instruction, and conditional executions. These extensions enrich the core ISA and lead to enhanced performance, small code size, and low power consumption.
3.2.2 ARM Instruction Set
The available ARMv7 architecture provides three instruction sets:
• ARM instruction set: 32-bit instructions that are four-bytes aligned. • Thumb instruction set: 16-bit and 32-bit instructions are available, and
can be use in the same program. This can reduce code size at the cost of reduced performance.
• Jazelle instruction set: 8-bit JavaTM byte codes.
3.2.3 ARM Processor Modes
ARMv7 architecture supports nine processor modes*, and each mode holds
a privilege level (PL0-to-PL2) in the system.
• User mode (USR): least privileged (or non-privileged) mode in the system (PL0). Operating systems run applications in User mode in order to restrict their access to system resources. Furthermore, software executing in non-privileged mode can not cause the processor to change modes except by creating an exception.
• System mode (SYS): processor mode with privilege level PL1. Exceptions can not cause the processor to move into this mode, but only direct modification of the mode bits in the CPSR.
• Supervisor mode (SVC): typically, the kernel executes at the supervisor mode with PL1. The processor enters this mode when a Supervisor call instruction is executed. This instruction is usually used by OS kernels to issue system calls.
3.2. Application Processing Unit 15 • Abort mode (ABT): privileged mode with PL1 that is entered when
Data Abort exception or Prefetch exception occurs.
• Undefined mode (UND): processor enters this privileged mode (PL1) when an exception occurs due to an undefined instruction.
• Fast Interrupt mode (FIQ): processor enters this privileged mode (PL1) when fast interrupt request (FIQ) is detected*
• Interrupt mode (IRQ): processor enters this privileged mode (PL1) when an interrupt request (IRQ) is detected†
• Monitor mode (MON): privileged mode (PL1) that allows the system to access secure and non-secure resources of the system, and supports the execution of TrustZone monitor software (see subsection 3.2.9 for more details).
• Hypervisor mode (HYP): highest privilege level in the system‡ (PL2)
that only exists in the non-secure zone of the system. Therefore, the Hypervisor mode can not be accessed while the processor is in the Secure state.
3.2.4 ARM Core Registers
From the application level view of the ARM core, 16 registers are available (R0 to R15), with the last 3 registers serve as special purpose registers:
• R13: Stack Pointer (SP) – each processor mode has its own SP register except for User and System modes.
• R14: Link Register (LR) – stores subroutine return address. Hypervisor, System, and User modes share the same LR register.
• R15: Program Counter (PC) – stores the address of the next instruction to be fetched. PC register is common for all modes.
These banked registers, which are part of a greater set of registers that may or may not be available depending on the presence of Security and Virtualization Extensions in the system. Typically, these registers follow a naming convention such as R0_usr to indicate the mode association of a particular register.
16 Chapter 3. Hardware Platform
3.2.5 Current Program Status Register
The Current Program Status Register (CPSR) is a 32-bit register that is used to control and monitor processor internal operations.
Table 3.1: CPSR Subfield Description Bit
Field Field Type Field Description
[31:28] Condition flags Indicate the status of arithmetic operations [27] Q Cumulative saturation bit.
[26:25] IT If-Then execution state bits for Thumb IT instruction.
[24] J Jazelle bit.
[23:20] N/A Reserved for future use.
[19:16] GE Status bits to indicate greater than or equal events.
[15:10] IT Together with bits[26:25] make the 8-bit status field (IT[7:0]) for If-Then Thumb instruction.
[9] E Sets the endianness for data access
[8:6] Mask bits Controls exception masking for abort, IRQ, and FIQ signals. Subfield requires PL1 or higher to change its state.
[5] Thumb execution state bit
Together with bit[24], the instruction set state of the processor is elected (ARM, Thumb, Jazelle, or ThumbEE)
[4:0] Mode field Used to control processor mode, and can only be written to at PL1 or higher.
3.2.6 Exception Handling in ARM
3.2. Application Processing Unit 17 Table 3.2: ARMv7 Architecture Exception Vector Table
Memory Offset Exception Type 0x00 Reset 0x04 Undefined Instruction 0x08 Supervisor Call* 0x0C Prefetch Abort 0x10 Data Abort 0x14 Hyp Trap† 0x18 IRQ Interrupt 0x1C FIQ Interrupt
3.2.7 ARM Coprocessors
The ARMv7 architecture extends the ARM CPU’s ISA via coprocessor support. These are not physical coprocessors, but are a means to abstract the fundamental ISA of ARM and to enhance features of the CPU system. ARMv7 architecture can provide access up to 16 coprocessors (CP0-to-CP15), where CP15 and PC14 are reserved and play an important role in the configuration, control, and debug of the CPU system.
• CP14 is reserved for the configuration and control of debug and trace features.
• CP15 is the System Control coprocessor. It provides configuration and control support for the ARM CPU system, such as TLB and cache management, and MMU control.
3.2.8 Virtual Memory System Architecture
The Virtual Memory System Architecture (VMSA) is the memory system architecture of the ARMv7-A implementation. VMSA allows memory virtualization by providing support for virtual-to-physical address translation, memory access permission control, and memory attribute validation. These features are enabled via the Memory Management Unit (MMU), Page Table, and Translation Lookaside Buffer (TLB).
18 Chapter 3. Hardware Platform
3.2.8.1 Memory Management Unit
Memory Management Unit (MMU) is a hardware block that controls access permissions, memory attributes, and address translations. The MMU is a critical component in system virtualization. It allows for efficient coexistence of applications by abstracting (or virtualizing) the physical memory. If the MMU is disabled, all virtual addresses (VA)s are mapped in a one-to-one fashion to physical addresses (flat address mapping). In this case, the address translation has to be managed in software, which lowers system performance and increases software design complexity.
The MMU features different address translation capabilities depending on the availability of Security and Virtualization Extensions. For example, with the presence of Security Extensions, the architecture provides two physical address spaces (Secure and Non-secure). The MMU provides, in this case, features to isolate these regions, and supporting registers to control access. In all versions of the ARMv7-A architecture, the MMU divides the physical memory into contiguous regions commonly known as pages, and stores virtual-to-physical translation information in Page Tables.
3.2.8.2 Page Tables
The VMSAv7 architecture provides two levels* of address lookup (or pages
tables). The page table mapping can be divided into Small Pages (4KB), Large Pages (64KB), and Sections (1MB) (Table3.3). Each partition scheme provides a different level of access granularity. Section partitioning only require a level-one lookup, which reduces the penalty of a full table walk. On the other hand, a page-based mapping requires both one and level-two lookups. The first level (L1) table is indexed by 12 bits, and therefore, contains 4096 entries, and the second level table is indexed by 8 bit, which yields 256 entries.
Table 3.3: ARM Page Table Details
Type Page Level Memory Size (KB) Number of Entries Sections L1 1024 4096
Pages L2 4, 64 256
3.2.8.3 Translation Lookaside Buffer
The Translation Lookaside Buffer (TLB) is a cache memory that serves the special purpose of storing recently translated virtual addresses. Before the
3.2. Application Processing Unit 19 MMU starts the lookup process for translating virtual addresses, it checks if the address is already present in the TLB. Similar to a regular processor cache system, the lookup operation can either result in a hit or a miss. In case of a hit, the TLB provides the translated entry of the target virtual address. Otherwise, if the TLB does not contain the translated entry, a TLB miss is generated, and the MMU continues with the address translation steps (table walk). The TLB is then updated (according to a round robin replacement policy) with the newly translated virtual address.
3.2.9 ARM TrustZone Architecture
TrustZone is a security extension available in modern ARM processors that creates a security infrastructure designers can use to protect critical system assets [25]. This infrastructure is achieved by enabling the partition of system components, both hardware and software, into either a Secure and a Normal world (or zone). Resources that are marked as normal are not permitted to access Secure zone components. This mechanism is enforced by the AMBA3 (Advanced Microcontroller Bus Architecture) AXI (Advanced eXtensible Interface) bus system. It contains an extra control signal for each of the read and write channels (Non-Secure bit or NS bits) that control the access rights of the Non-Secure bus masters to the Secure slaves.
Chapter 4
System Virtualization
System virtualization is the abstraction and management of system resources such as CPU, memory, and peripherals. Virtualization solutions implement spacial and temporal isolation techniques in order to facilitate the integration of mixed-criticality systems [3]. This approach results in independent virtual machine that are fully contained in an execution environment that can not infect the remainder system. In addition to isolation, virtual machine can use any operating system as long as the OS is compatible with the virtual machine. As a result, a single computing platform can enable the execution of heterogeneous operating systems. This section presents the background necessary to understand the fundamentals of system virtualization.
4.1 High-Level View of System Virtualization
In the computing world, virtualization is a method used to abstract hardware resources of underlying platforms [4]. The most common virtualization environment is the operating system found in Personal Computer (PC)s (i.e. Windows, Mac OS X, or Linux). These operating systems hide details about the CPU and GPU units, hard disks, Ethernet controller, and other hardware components found in PCs. As a result, software can be developed independent of the final hardware platform via the available Application Programming Interface (API). While operating systems can abstract hardware systems, and achieve a satisfactory performance level, they are not suitable for MCS. Essentially, a MCS requires deadline guarantees of hard real-time tasks while ensuring the Quality of Service (QoS) of general-purpose applications. At the same time, the system must be secure and robust such that it offers protection against external attacks and unintended interactions between the critical and non-critical components of the system.
22 Chapter 4. System Virtualization VM1 VM2 VMn Hypervisor Host GPOS (Windows/Linux) Hardware Platform Figure 4.1: Type II Hypervisor
Consequently, a thin software layer, hypervisor or Virtual Machine Monitor (VMM), is introduced that can abstract and manage system resources. Since the code size of the hypervisor is small, it is easier to validate. The hypervisor provides an isolation mechanism that can encapsulates an entire OS and applications into a Virtual Machine (VM). Furthermore, the hypervisor can support multiple virtual machines, which enables the concurrent execution of heterogeneous operating systems.
4.1.1 Type II Hypervisor
Type II hypervisor (Figure 4.1) is a software layer that runs on top of a General Purpose Operating System (GPOS) [4]. It takes advantage of Operating System (OS) services such as resource management (e.g. CPU allocation, scheduling, and memory management), and the hardware abstraction facility of the host OS, which enable the reuse of device drivers, communication stacks, and APIs. Furthermore, type II hypervisors enable execution of applications in a native as well as a virtual environment, which provides another level of flexibility in the system. However, the security of type II hypervisors is as robust as the underlying host GPOS. Therefore, the hypervisor can be subverted by one of the security gaps in the host GPOS, thereby corrupting the entire system. Additionally, the host OS layer increases system complexity and overall code size, which is an important factor for resource constrained embedded systems. As a result, type II hypervisors are not suited for most embedded systems.
4.1.2 Type I Hypervisor
4.2. Virtualization Architectures for Embedded Systems 23 VM1 VM2 VMn
Hypervisor Hardware Platform Figure 4.2: Type I Hypervisor
applications [4]. However, legacy type I hypervisors that do not take advantage of hardware virtualization support (section 4.5) must develop custom device drivers and hardware management services that optimize the performance and robustness of the virtualized system. Furthermore, some type I hypervisor solutions use a specialized guest to support I/O device sharing for the guest systems. With this type of solution, the code size of the trusted software becomes too large that the virtualization solution is no better than type II hypervisor. Without the availability of hardware virtualization support, these limitations become trade-offs that vary depending on the selected type I hypervisor architecture.
4.2 Virtualization Architectures for Embedded
Systems
Virtualization is a widely used technique in enterprise computing systems, and recently, it has been gaining strong interest in the embedded systems domain [4]. This is particularly relevant for multicore embedded systems that integrate mixed-criticality applications, where critical tasks include hard real-time deadlines, and the system requires a high level of safety and security standard.
4.2.1 Full Virtualization
24 Chapter 4. System Virtualization embedded systems [4].
4.2.2 Paravirtualization
The prefix ”Para” is an English affix that originated from Greek and means ”beside”, ”with”, or ”alongside”, which yields the meaning of ”alongside virtualization” [26]. Unlike full virtualization, in paravirtualization, guest operating systems are modified in order to improve the performance of the hypervisor. These modifications are applied specifically to the guest OS kernel in order to replace non-virtualizable instructions and critical kernel operations with hypercalls that can request services directly from the hypervisor. These services represent system calls that are part of the OS kernel, and they execute with the highest privilege level in the system. However, once the OS kernel is pushed into a virtual machine environment, the hypervisor gains the highest privilege level in the system. Consequently, the normal execution of system calls will cause system faults that must be trapped and emulated by the hypervisor. Paravirtualization remedies the need to trap-and-emulate sensitive instruction. However, this process comes with high development and maintenance costs. Detailed knowledge of the OS kernel is required in order to apply the necessary changes to the source code. Xen ARM, for example, requires the modification of approximately 4500 lines of code (LOC) [27]. Nevertheless, paravirtualization is the only viable solution for embedded platforms that do not provide any hardware virtualization support.
4.2.3 Monolithic Hypervisor
Similar to an operating system, a monolithic hypervisor contains all device drivers and middleware to enable execution of guest operating systems [4]. This hypervisor architecture results in a large software layer, which makes it difficult to verify and validate. Furthermore, the monolithic hypervisor uses a single instance of the virtual environment to run multiple guest systems. Therefore, a single defect in the hypervisor could corrupt the entire system, which contradicts the isolation characteristics of system virtualization.
4.2.4 Console Guest Hypervisor
4.3. Resource Management 25 control. The selection of the Dom0 OS is critical because a general purpose OS might dramatically increase the size of the abstraction layer, and as a result reduce the robustness of the system.
4.2.5 Microkernel-Based Hypervisor
In order to increase the robustness of the hypervisor, its size should be as small as possible. Microkernel-based hypervisors represent a thin software layer that runs as bare-metal, and can provide strong isolation between guest operating systems. This approach implements virtualization as a service on top of the trusted microkernel. Therefore, each separate instance is as robust as the guest environment itself. Damaged guest environments can not contaminate the rest of the system because only the microkernel executes in the highest privilege mode.
4.3 Resource Management
In a virtualized embedded system, the hypervisor is responsible for managing all system resources, including CPU units [4]. The hypervisor can use spatial and temporal partitioning techniques to distribute or consolidate system workloads in order to reach an optimization solution that satisfies specific operating conditions such as low power consumption or low heat generation. Spatial partitioning is only available in multicore systems, where only a single virtual machine is mapped to a specific set of CPUs. While this approach does not use system resources to their maximum capacity, it does provide availability of service guarantees for safety and security critical applications. Alternatively, the hypervisor can use dynamic partitioning, where resour-ces can be redistributed amongst the virtual machines in order to maximize system utilization. This architecture is more challenging to implement, but it unlocks many features such as load balancing, migration of virtual machines across cores, and temporal partitioning. Dynamic partitioning is a highly desirable feature in power efficient systems. As an example, in a multicore system where two VMs only require 50% utilization and each VM is mapped to separate processor, the hypervisor can determine that a single core is sufficient for both VMs, consolidate the workload, and turn off the other cores. As a result, the system will be able to save energy from static power consumption. The hypervisor can also use a hybrid approach that statically maps a safety-critical VM to a set of processors*, and use dynamic
partitioning for the rest of the cores in the system.
26 Chapter 4. System Virtualization
4.4 Hypervisor robustness
Virtualization techniques can increase system robustness against external malicious attacks [4] by isolating guest systems virtual environments. An attacker can only affect the guest system it penetrates, leaving the rest of the system unharmed. However, several studies were conducted to find vulnerabilities in the security of virtualized systems, such as SubVirt, Blue pill, Ormandy, Xen owning trilogy, and VMware’s security certification. In each case, the system was found susceptible to some type of attack that allowed the attacker to cause crashes, anomalous behavior, or run arbitrary code. This discovery led to the conclusion that using a hypervisor does not necessarily guarantee a robust isolation between virtual machines. Consequently, developers have to be aware of this fact when working in highly sensitive domains. Platform attestation is an approach used to increase the robustness of virtualized environments. It proposes that only known good firmware such as the hypervisor are allowed to boot and control the computing platform at any given time, which prevents hypervisors from being corrupted.
4.5 Hardware Virtualization Acceleration
4.5. Hardware Virtualization Acceleration 27
Kernel User Mode (PL0)
Supervisor Mode (PL1)
P1 P2 Pn
Figure 4.3: OS Level Virtualization in 2-level Mode Hierarchy System
Hypervisor Hypervisor Mode (PL1)
Kernel Guest User Mode (PL0)
Guest Supervisor Mode (PL0)
P1 P2 Pn
VM
Figure 4.4: Type I Hypervisor Virtualization in 2-level Mode Hierarchy System
28 Chapter 4. System Virtualization
Hypervisor Hypervisor Mode (PL2)
Kernel Guest User Mode (PL0)
Guest Supervisor Mode (PL1)
P1 P2 Pn
VM
Figure 4.5: Type I Hypervisor Virtualization in 3-level Mode Hierarchy System
4.5.1 Memory Virtualization
In addition to optimizing the virtualization of CPUs, it is important to find an efficient mechanism to manage the physical system memory in a virtualized MCS. Modern operating systems implement memory virtualization techniques in order to optimize the sharing and dynamic allocation of the physical system memory [26]. The key hardware components that enable efficient memory virtualization are the Memory Management Unit (MMU) and the Translation Lookaside Buffer (TLB). The MMU performs virtual-to-physical address translations and stores a copy of recently translated addresses in the TLB (a special cache dedicated to store virtual-to-physical address mapping). In order to support multiple virtual machines another level of address translation is required, which implies that the MMU itself has to be virtualized. The guest OS translates guest virtual addresses to guest physical addresses, and the hypervisor translates guest physical addresses to actual machine memory addresses. In order to accelerate this two-level address translation process, the hypervisor uses shadow page tables to directly map guest virtual memory addresses to machine memory addresses.
4.5.2 Device and I/O Virtualization
4.5. Hardware Virtualization Acceleration 29
4.5.2.1 Emulation
In the emulation architecture, the hypervisor intercepts and validates all I/O accesses of guest operating systems, and translated them into hypervisor-initiated operations [4]. The emulation method optimizes system reliability because all I/O accesses are handled by the trusted hypervisor. Furthermore, emulation ensures system availability independent of the state of the virtual machines. However, this approach produces an overhead in all I/O operations which may result in unsatisfactory performance. While these emulated systems provide flexibility and system reliability, they lack in performance, and require a large effort to maintain the custom device drivers.
4.5.2.2 Pass-through
The pass-through architecture allows the guest operating systems to bypass the hypervisor and gain direct access to I/O resources [4]. This model improves efficiency, but decreases system robustness. The pass-through approach reduces the level of isolation of virtual environments, which does not satisfy the safety and security requirement of MCS. In order to increase the robustness of the pass-through architecture, an Input Output Memory Management Unit (IOMMU) is required. Similar to how the MMU allows the hypervisor to manage memory accesses of virtual machines, the IOMMU manages access to I/O devices. This approach removes the risk of a virtual machine accessing a memory address that is beyond its allocated memory. The IOMMU is particularly important in the presence of DMA engines, where accesses does not necessarily originate from the CPU. While IOMMU can improve performance and robustness of a system that uses a pass-through model, it can not provide sufficient reliability for safety-critical systems. Mainly, because if the virtual machine that is mapped to a pass-through peripheral is corrupted, all other virtual machines lose access to that device as well.
4.5.2.3 Mediated Pass-through
30 Chapter 4. System Virtualization hypervisor can then allow, modify, or reject access based on system policy. Therefore, the mediated pass-through system trades an acceptable amount of performance for reliability.
4.6 Virtualization Requirements for MCS
From the literature review and the fundamental theory presented thus far, a set of requirements could be defined in order to find a suitable virtualization solution that leverages the available hardware resources, and enables the safe and secure integration of mixed-criticality systems.
• Req. 1: Hardware – hypervisor shall support ARM Cortex-A9 processor (support for Zynq is a plus).
• Req. 2: Hardware Assists – hypervisor shall leverage available hardware virtualization support.
• Req. 3: Robustness – hypervisor layer shall be as small as possible. • Req. 4: Multiple OS – hypervisor shall support at least one GPOS and
one Real-Time Operating System (RTOS) instances.
• Req. 5: Isolation – hypervisor shall provide strong isolation mechanism. • Req. 6: Communication – hypervisor shall provide a communication
mechanism between virtual machines.
• Req. 7: Multicore – hypervisor shall leverage all available CPUs in the hardware platform.
Chapter 5
Exploration of Available
Hypervisor Solutions
Many hypervisor solutions are available as either open-source or commercial products. This section will showcase some of the available implementations, compare their pros and cons, and conclude in the selection of a suitable virtualization solution that satisfies the system requirements formsection 4.6.
5.1 Xen Hypervisor
The Xen hypervisor is widely used in enterprise and is now making its way to embedded systems. It was developed in Cambridge University, and is available as open-source software under the the general public license (GNU). The Xen hypervisor is implemented as the guest console architecture, as discussed in subsection 4.2.4. The hypervisor layer is a thin software layer that resides above the hardware layer. It is the first program that runs after the bootloader, and is responsible for managing the CPU, Memory, and interrupts.
By default, the Xen hypervisor uses Credit as the CPU scheduler, which allows the user to allocate a percentage of the CPU time for each VM, or allow the hypervisor to automatically balance the workload across active CPUs in the system. Alternatively, the user can specify Simple Earliest Deadline First (SEDF) algorithm for the scheduler. However, the load-balancing feature will be unavailable [28].
The hypervisor is responsible for launches Dom0, which is a special virtual machine that has privileged access rights to the physical I/O resources. It handles I/O accesses and interacts with the other virtual machines. All other VM instances operate in Domain U (DomU), which runs in unprivileged
32 Chapter 5. Exploration of Available Hypervisor Solutions mode. The guest virtual machines can be either paravirtualized (PV) or fully virtualized {a.k.a. Hardware-assisted Virtual Machine (HVM)}. The PV guest are modified operating systems such as: Linux, Solaris, FreeBSD, or other UNIX operating systems. In order to facilitate I/O sharing, Xen uses split-driver architecture. This approach manges I/O accesses of DomU PV guests. The split-driver technique divides the driver into a front-end, located in the DomU PV guest, and a back-end, located in the Dom0 guest. DomU PV guests are aware that they do not have direct access to the hardware and that they are running alongside other virtual machines on the same hardware. However, DomU HVM guests are unaware of the presence of other VMs, and of the fact that they are sharing hardware resources. Instead of split-drivers, in the HVM architecture, a special daemon is started in Dom0 guest for each DomU HVM guest. The Xen hypervisor is available for both Intel and ARM devices. However, it is not recommended to use Xen with devices that do not contain IOMMU units because the hypervisor can be easily subverted by DMA capable devices [29].
5.2 Xen Zynq
The open-source Xen hypervisor has recently been ported to the new Xilinx Zynq Ultrascale+Multi-Processor System-on-Chip (MPSoC) device [30]. Xen Zynq Distribution is released under the GNU General Purpose License 2 (GPL2). The processing platform features a quad-core ARM Cortex-A53, a dual-core ARM Cortex-R5, a Mali-400MP2 GPU, and FPGA fabric that supports run-time reconfiguration. This device is the successor of Xilinx Zynq SoC, which features a dual-core Cortex-A9 processor and FPGA fabric.
5.3 SEL4 Microkernel
The sel4 microkernel is based on the L4 microkernel, which is one of the smallest kernels available today. Sel4 is the first formally verified microkernel, which implies that its specification is verified mathematically. Sel follows the ”minimality principle”, which dictates that the kernel shall only contain functionalities that can not be implemented at the user-level [31]. As a result, the microkernel is small, efficient, and robust. All device drivers are excluded from the microkernel level and execute in unprivileged mode, except for a timer driver and an interrupt controller driver.
5.4. TrustZone-based Hypervisor 33 to create and manage threads, virtual memory spaces, and interprocess communication (IPC). Furthermore, sel4 follows a ”capability-based access control model” in order to manage the access rights to all kernel services. Capabilities are unforgeable tokens that contain metadata about a specific kernel object, including its access rights. The use of capabilities as a control mechanism allows the system to maintain strong isolation between software components [32].
The sel4 microkernel implements a fixed-priority round-robin scheduler policy, mainly because its current ”time” abstraction method is under-developed and does not yield satisfactory results. As proposed in [31], reservations can be added to sel4 in order to provide a suitable temporal isolation solution for real-time systems.
Sel4 provides IOMMU support for Intel-based architectures (IA-32), which allows the safe integration of DMA enabled devices. Furthermore, Sel4 can support multicore systems via multikernel bootstrapping. However, this feature is only available for x86 machines; only uniprocessor is supported for ARM-based devices.
5.4 TrustZone-based Hypervisor
TrustZone technology refers to the security extensions available in most modern ARM systems. As discussed insubsection 3.2.9, TrustZone technology provides two working zones: ”Normal” zone and ”Secure” zone. Applications running in the Normal zone can not access resources from the Secure zone, but Secure zone software has full system access. This mechanism facilitates system partitioning, and the creation of a secure isolated environment that can host safety and security critical applications.
Most ARM processors offer two privilege levels (PL0 and PL2) for executing software. Typically, in operating systems, the kernel executes in the highest privilege level (PL1) and applications run in unprivileged mode (PL0). However, in order to virtualize a system, a third privilege level is needed to accommodate the hypervisor state (section 4.5). TrustZone provides a third level via the leveraging of the capabilities of the ”Monitor” mode (subsection 3.2.3). Therefore, ARM processor systems that include TrustZone technology can achieve an efficient virtualization implementation besides the use of paravirtualization.
ARM’s TrustZone security extensions can be utilized to virtualize a system in two ways:
34 Chapter 5. Exploration of Available Hypervisor Solutions 2. Use the efficient switching mechanism of the Secure zone Monitor to
host a dual-OS system (Secure zone OS and Normal zone OS)
5.4.1 SierraVisor
Sierraware offers a bare metal universal hypervisor (SierraVisor) that is available as open-source under the GNU GPL v2 license or with a commercial license [33]. It supports paravirtualization, TrustZone virtualization, and hardware assisted virtualization. SierraVisor is compatible with Cortex-A9/A15 and ARM11 based SoCs, but only Cortex-A15 supports the hardware assisted virtualization option*. The TrustZone virtualization
approach allows for the integration of guest operating systems without any kernel modifications. Each guest kernel and applications run in their usual privilege mode, supervisor and user mode respectively. Furthermore, each guest executes in an isolated container with low overhead.
5.4.2 SafeG
TOPPERS group of Nagoya University in Japan has developed an open-source dual-OS architecture (SafeG†– seeFigure 5.1) designed to concurrently
host a real-time operating system (RTOS) and a general purpose operating system (GPOS) on TrustZone enabled ARM SoC devices [34]. SafeG takes advantage of ARM’s TrustZone security extensions to efficiently partition the system into Trusted and Non-Trusted states, which provides full system access to trusted software, and limits the capabilities of software running in Non-Trusted state. SafeG includes the following features:
• Enables the concurrent execution of RTOS and GPOS on either single-core or multi-single-core ARM-based platforms.
• Devices and memory regions that are configured as Secure are protected against illegal GPOS accesses.
• Normal world devices can be accessed from both GPOS (Non-Trusted) and RTOS (Trusted) software.
• Real-time requirements are guaranteed in RTOS (Trusted) via the utilization of FIQ and IRQ interrupts, where FIQ interrupts are issued for RTOS and IRQ interrupts are issued for GPOS. While in the Trusted state, IRQ interrupts are disabled so that GPOS can not disturb the execution of the RTOS. Therefore, GPOS only
5.4. TrustZone-based Hypervisor 35 executes when the RTOS issues the Secure Monitor Call (SMC) instruction, which causes the SafeG monitor to switch from the Trusted world (RTOS) to the Non-Trusted world (GPOS). Furthermore, FIQ interrupts are active during the execution of GPOS, which enables RTOS to retake control of the system. For example, a cyclic execution of RTOS/GPOS can be controlled by an FIQ interrupt of a system timer.
• GPOS does not require any major changes, and can execute with minimal overhead.
• Includes an efficient guest-to-guest communication mechanism (i.e. referred to as SafeG COM)
ARM TrustZone® Core
Memory RTOS Data GPOS Data I/O RTOS Device GPOS Device RTOS FMP Trusted SafeG Monitor Non-Trusted GPOS Linux BUS (NS bit) Legend: Trusted Non-Trusted
Figure 5.1: SafeG Architecture [34]
36 Chapter 5. Exploration of Available Hypervisor Solutions and RTOS can gain access, which is essential for communication between the two regions.
5.5 SICS Thin Hypervisor
SICS Thin Hypervisor (STH) is a light-weight hypervisor designed for ARM-based devices [35]. STH runs directly on top of the hardware (bare metal), and achieves system virtualization through paravirtualization. As a result, guest systems require some modifications to the OS kernel, including the addition of a hypercall* inteface. STH strengthens the security of embedded
systems through the isolation capabilities of virtual machines, and allows for the existence of heterogeneous operating systems on the same platform. Current STH version supports ARMv5 (926EJ-S) and ARMv7 Cortex-A8 only. However, STH is a highly flexible and portable hypervisor that uses a hardware abstraction layer with minimal size.
5.6 Hypervisor Solution Matrix
There are many hypervisor solutions available in the market, some are open-source and others commercial. Table 5.1 lists hardware and features supported by selected open-source hypervisors. It is important to keep in mind that the target hardware platform for this project is the Zedboard, which contains Xilinx Zynq 7000 SoC. Furthermore, the Zynq 7000 SoC contains a dual-core Cortex-A9 processor, plus FPGA fabrique. Therefore, the solution matrix lists the Cortex-A9 and Zynq 7000 in the Hardware section in order to showcase the fact that most hypervisors are designed for the ARM processor and not necessarily for the Zynq 7000 SoC. Nevertheless, since the zynq 7000 SoC contains an ARM processor, the porting process is a little easier, or might even be assumed compatible. Therefore, all the information presented in the table is a representation of the literature review from (section 5.1, section 5.3, section 5.4,section 5.5).
Another key factor to keep in mind, is that Cortex-A8/A9/A15 are built on the ARMv-7 architecture, which indicates that they contain some level of compatibility. However, unlike the Cortex-A8/A9, Cortex-A15 includes Virtualization Extensions, which enables system virtualization without the need for paravirtualization or binary translation techniques. Table 5.1 also presents hypervisor solutions that leverage TrustZone technology (available
5.6. Hypervisor Solution Matrix 37
Table 5.1: Hypervisor Solution Matrix Xen ARM
PV VE Zynq STH SafeG SierraVisor Sel4 XtratuM
HARDWARE Cortex-A8 x - - x x x x -Cortex-A9 x - - - x x x -Cortex-A15 - x - - - x x -Zynq 7000 SoC - - - - x x2 x1 -Zynq Ultrascale + MPSoC - x x - - - - -x86 - - - x x FEATURES Multiple guests x x x x x3 x x4 x Paravirtualization (PV) x x5 x5 x x5 x x x TrustZone Security Extension - - - - x x - -Virtualization Extension (VE) - x x - - x x x Multiprocessor x x x - x x x4 x Documentation x6 x x x x x6 x x LEGEND: Not Available
-1 Support is visible in source code, but not in documentation. 2 Support is visible in documentation, but source coderequires modifications.
3 Dual-OS only.
4 Available on x86 only.
5 Lightweight PV of guests, or only needed for I/Os. Exceptions
