Redesign of readout driver using FPGA

Institutionen för systemteknik

Department of Electrical Engineering

Examensarbete

Redesign of readout driver using FPGA

Examensarbete utfört i Elektroniksystem vid Tekniska högskolan i Linköping

av

Per Klöfver

LITH-ISY-EX--08/4135--SE

Linköping 2008

Department of Electrical Engineering Linköpings tekniska högskola

Linköpings universitet Linköpings universitet

Redesign of readout driver using FPGA

Examensarbete utfört i Elektroniksystem

vid Tekniska högskolan i Linköping

av

Per Klöfver

LITH-ISY-EX--08/4135--SE

Handledare: Stefan Haas

cern

Examinator: Kent Palmkvist

isy, Linköpings universitet

Avdelning, Institution

Division, Department

Division of Electronics Systems Department of Electrical Engineering Linköpings universitet

SE-581 83 Linköping, Sweden

Datum Date 2008-06-09 Språk Language  Svenska/Swedish  Engelska/English   Rapporttyp Report category  Licentiatavhandling  Examensarbete  C-uppsats  D-uppsats  Övrig rapport  

URL för elektronisk version

http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-15157

ISBN

—

ISRN

LITH-ISY-EX--08/4135--SE

Serietitel och serienummer

Title of series, numbering

ISSN

—

Titel

Title

Modernisering av datautläsningsenhet mha FPGA Redesign of readout driver using FPGA

Författare

Author

Per Klöfver

Sammanfattning

Abstract

In the ATLAS experiment now being finished at CERN in Geneva, bunches of protons will collide at a rate of 40 million times per second. Over 40 TB of data will be generated every second. In order to reduce the amount of data to a more manageable level, a system of triggers is put in place. The trigger system must quickly evaluate if the data from a collision indicates that an interesting physical process took place, in which case the data are to be stored for further analysis.

ATLAS uses a trigger system with three steps. The first step, the First Level Trigger, is responsible of reducing the rate from 40MHz to 75KHz, and is done completely in hardware. It receives a new event every 25 ns, and must decide within 2.5 µs whether the event should be passed on to the next trigger level.

In this document is the redesign of two subsystems of the First Level Trigger described. When prototypes were made 5-10 years ago, both subsystems used 7 PLDs. Today, the same logic could be fitted in one FPGA, and because of the flexibility gained by having all logic in a single FPGA, both subsystems could be realized with the same PCB design.

Nyckelord

Abstract

In the ATLAS experiment now being finished at CERN in Geneva, bunches of protons will collide at a rate of 40 million times per second. Over 40 TB of data will be generated every second. In order to reduce the amount of data to a more manageable level, a system of triggers is put in place. The trigger system must quickly evaluate if the data from a collision indicates that an interesting physical process took place, in which case the data are to be stored for further analysis.

ATLAS uses a trigger system with three steps. The first step, the First Level Trigger, is responsible of reducing the rate from 40MHz to 75KHz, and is done completely in hardware. It receives a new event every 25 ns, and must decide within 2.5 µs whether the event should be passed on to the next trigger level.

In this document is the redesign of two subsystems of the First Level Trigger described. When prototypes were made 5-10 years ago, both subsystems used 7 PLDs. Today, the same logic could be fitted in one FPGA, and because of the flexibility gained by having all logic in a single FPGA, both subsystems could be realized with the same PCB design.

Acknowledgments

I would like to thank Stefan Haas, my supervisor at CERN, for all his help and support, as well as the rest of group PH-ATE – the coffee breaks at 9.30 are a fond memory, even though I had a hard time understanding the French language.

I would also like to thank Kent Palmkvist, my examiner, and Magnus Johans-son.

Contents

1 Introduction 1 1.1 CERN . . . 1 1.2 LHC . . . 2 1.2.1 Chain of accelerators . . . 3 1.2.2 Experiments on LHC . . . 4 2 Background 5 2.1 ATLAS . . . 5 2.1.1 Magnets . . . 5 2.1.2 Detectors . . . 5 2.2 Data Acquisition . . . 7

2.2.1 First Level Trigger . . . 8

2.2.2 MUCTPI . . . 11 3 Problem definition 13 4 Technology used 15 4.1 FPGA . . . 15 4.2 JTAG . . . 16 4.3 VHDL . . . 16 4.4 VME . . . 16 4.5 Signaling schemes . . . 16 5 Software Tools 19 5.1 Tools for FPGA . . . 19

5.2 Tools for Board . . . 19

6 Design 21 6.1 MIROD . . . 21 6.1.1 Old MIROD . . . 21 6.1.2 New MIROD . . . 22 6.1.3 Testbench . . . 24 6.1.4 Synthesis . . . 24 6.2 MICTP . . . 25 6.2.1 New MICTP . . . 25 ix

6.3 Board . . . 26

6.3.1 Both MIROD and MICTP . . . 27

6.3.2 Only in MIROD . . . 28

6.3.3 Only in MICTP . . . 28

7 Results and discussion 31 7.1 Benefits of using the same PCB design for both systems . . . 31

7.2 Disadvantages of using the same PCB design for both systems . . 31

7.3 Software tools . . . 32

Bibliography 33

Glossary 35

Chapter 1

Introduction

In this document I will describe my experiences working at CERN in Geneva, where I spent 14 months in the Technical Student Programme.

The main project was to redesign the MIROD, a component of the data ac-quisition system of the ATLAS experiment at CERN. In addition, when designing the PCB of the new MIROD, it was prepared so that it can also function as a

MICTP, which is another system located next to MIROD.

Content of chapters:

• Chapter 1, this chapter, gives an introduction to CERN and LHC.

• Chapter 2, Background, explains the context of the system, starting by

de-scribing ATLAS, and then “zooming in” closer and closer to the system being redesigned.

• Chapter 3, Problem, defines the task.

• Chapter 4, Technology, has information about the technologies used. • Chapter 5, Software tools, lists the software used to do the design.

• Chapter 6, Design, describes the system and how the (re)design was done. • Chapter 7, Results and discussion, discusses the result and experiences made.

1.1

CERN

CERN, European Organization for Nuclear Research, is the worlds largest particle physics laboratory. At CERN particle accelerators are used to study the properties of the smallest building blocks of matter.

It is situated northwest of Geneva, right on the border between Switzerland

and France. CERN was founded in 1954 as a cooperation between European

states. The acronym stands for “Conseil Européen pour la Recherche Nucléaire”. 1

This was actually the name of the council set up a few years earlier to plan the organization, but today the name is used for the organization itself.

The founding member states of CERN were

Bel-Figure 1.1. CERN logo gium, Denmark, France, Germany, Greece, Italy, The

Netherlands, Norway, Sweden, Switzerland, the United Kingdom and Yugoslavia (Yugoslavia left the organi-zation in 1961). Over the years, Austria, Bulgaria, the Czech Republic, Finland, Hungary, Poland, Por-tugal, the Slovak Republic and Spain have become members. In addition India, Israel, Japan, Russia Turkey and the USA have “Observer” status, so they can take part in everything except the decision mak-ing.

There are also people from a large number of non-member states involved in CERN programmes.

Outside the world of high energy physics, CERN is perhaps most well known for being the place where the World Wide Web (WWW) was invented. The web was invented in 1989 by Tim Berners-Lee, in an effort to facilitate information exchange in big projects as the upcoming LHC. .

1.2

LHC

The largest project at CERN today is the building of the LHC, Large Hadron Collider. When it is finished, which will hopefully happen in 2008, it will be the most powerful particle accelerator in the world. It is a circular accelerator with a circumference of 27 km, designed to create collisions of protons at very high speeds. At their maximum speed, each proton will have a kinetic energy of 7 TeV and be travelling at 0.999999991 times the speed of light (see appendix A on page 36). When two protons travelling with opposing directions collide a total energy of 14 TeV is released. While this is only 2 µJ, this energy is extremely concentrated and will generate a swarm of particles. One of the particles sometimes generated will be the Higgs boson – that is if the Higgs boson actually exists. To investigate the Higgs particle is one of the reasons for which LHC is created.

The accelerator consists of two tubes, with protons travelling in opposite di-rections, running next to each other in a 27 km long circular tunnel 100 meters below the surface. Around the tubes are supraconducting magnets to steer and accelerate the particles, and cryogenic equipment to cool the magnets. At a few spots along the tunnel, the two tubes are brought together to allow the particles to collide. These are the interaction points of the detectors, the main ones being ATLAS, CMS, ALICE and LHCb.

The protons are grouped in ”bunches“ of 1011_{protons. There are 2808 bunches}

around the LHC, separated by 25ns1_{. This means that bunches of opposing}

di-rections cross paths at the interaction points every 25ns, i.e. with a frequency of 40MHz. Collisions are rare. Most protons in a bunch will just pass the other

1.2 LHC 3

Figure 1.2. System of Accelerators at CERN[1]

bunch without any interaction. At each ”bunch crossing“ it is only estimated to be 20-25 proton-to-proton collisions.

1.2.1

Chain of accelerators

Before a proton enters the LHC, it is first accelerated in a chain of smaller accel-erators (see fig. 1.2). Protons are generated by removing electrons from hydrogen atoms. These protons are accelerated to 50 MeV in a linear accelerator (Linac2). After that, the protons are accelerated in the PS Booster (to 1.4 GeV), the Proton

Synchrotron (to 25 GeV), and the Super Proton Synchrotron (to 450 GeV) before they enter the LHC for the final acceleration to 7 TeV.

1.2.2

Experiments on LHC

The places where the beams collide and the detectors are situated, are called

experiments in CERN lingo. There are four main experiments on the LHC.

ATLAS (A Toroidal LHC ApparatuS) is the biggest experiment. It is a general

purpose detector which will described in section 2.1.

CMS (Compact Muon Solenoid) is the second largest experiment. It is also a

general purpose detector and has the similar objectives as ATLAS.

LHCb (Large Hadron Collider beauty experiment) will study asymmetry between

matter and antimatter.

ALICE (A Large Ion Collider Experiment) differs from the other experiments in

that it uses lead ions instead of protons. Because of this it can’t be run the same time as the other experiments.

Chapter 2

Background

2.1

ATLAS

ATLAS is the largest experiment on the LHC, both in terms of budget and in physical size. The structure (see fig. 2.1) is a 42 m long cylinder with a diameter of 22 m, weighting 7000 tonnes. The ATLAS detector has four main components; the inner detector (tracker), the calorimeter, the muon system and the magnet system.

2.1.1

Magnets

The Central solenoid is 5.3m long and 2.5m in diameter. It generates a 2T

uniform axial magnetic field at the interaction point and inner detector.

The Barrel toroid is 25m long and 20m in diameter. It generates a 4T toroidal

magnetic field.

The End cap toroids are 5m long and 11m in diameter and generate 4T toroidal

magnetic fields. They are located inside the Barrel toroid at the ends of the cylinder.

The magnets are supraconducting, and are operated a temperature of 4.8 K.

2.1.2

Detectors

Inner detector

In the center of ATLAS, closest to the interaction point, is the Inner detector. It tracks the trails of charged particles.

Calorimeter

Next is the Electromagnetic Calorimeter which measures the energy of electrons and photons, followed by the Hadronic Calorimeter which measures the energy

Muon Detectors

Solenoid

Electromagnetic Calorimeters

Forward Calorimeters

End Cap Toroid

Barrel Toroid

Inner Detector

Hadronic Calorimeters

Shielding

2.2 Data Acquisition 7

low p

_T

high p

_T 5 10 15 m 0

RPC 3

RPC 2

RPC 1

TGC 1

TGC 2

TGC 3

low p

_T

high p

_T BOS BMS BIS

Figure 2.2. Muon trigger chambers[4]

of protons, neutrons, pions and kaons. The Calorimeters measures the energy of these particles by absorbing them.

Muon Spectrometer

Most far away from the interaction point is the Muon system. Muons are one of few particles reaching this far, not having been absorbed by the earlier detectors.

The muon detector determines the transverse1_{momentum, p}

t, of passing muons

by measuring the curvature of their tracks in the magnetic field.

Four different kinds of detectors are used. Two slower kinds have high reso-lution and are used for precision measurement. Two faster kinds, Resitive Plate

Chambers (RPC) and Thin Gap Chambers (TGC) are used for the trigger. See

figure 2.2.

2.2

Data Acquisition

There are 40 million bunch crossings every second, each generating about 20-25 proton-to-proton collisions. A vast majority of these collisions will lead to well

LEVEL 2 TRIGGER LEVEL 1 TRIGGER

CALO MUON TRACKING

Event builder Pipeline memories Derandomizers Readout buffers (ROBs) EVENT FILTER Bunch crossing rate 40 MHz < 75 (100) kHz ~ 1 kHz ~ 100 Hz Interaction rate ~1 GHz

Regions of Interest Readout drivers

(RODs)

Full-event buffers and processor sub-farms

Data recording

Figure 2.3. Atlas Trigger levels[4]

known physical processes. To store all data produced by the sensors would not be worthwhile. It is necessary to be able to filter out the rare, interesting processes, which may only happen in the order of once in 1013_{collisions. To do this a system}

of triggers is put in place. The trigger system selects which bunch crossing data, which ”events“, to record permanently. Atlas uses a trigger system with three levels (see fig 2.3). The first trigger level(LVL1), described below, is implemented in hardware and reduces the rate from 40 MHz to 75 KHz. The second trigger level (LVL2) reduces the rate to 1 KHz, and the third level (EventFilter) finally reduces the rate to 100Hz, and that data (in the order of 100MB/s) is stored permanently. The system must also be able to identify which particular bunch crossing all pieces of data belong to. This is non-trivial, since when particles generated by a collision in a bunch crossing reach the outer detectors, the next bunch crossing has already occurred in the center of the detector.

2.2.1

First Level Trigger

“The primary function of the LVL1 trigger is to provide, for each bunch crossing, a signal specifying if the bunch crossing should be retained for further analysis, typically because it contains a potentially interesting physics signature”[4]

The First Level Trigger (LVL1) uses reduced granularity data from the Calorime-ter and the Muon SpectromeCalorime-ter to make its decision (see fig. 2.4). The maximum latency of LVL1 is 2 µs. During this time, all detector data is stored in buffers

2.2 Data Acquisition 9

Level 1 accept (L1A) Central Trigger Processor (CTP)

Calorimeter Trigger Muon Trigger

multiplicity

Figure 2.4. Level 1 Trigger

near the detector. The data is read out only when the First Level Trigger “trigs” and emits a Level 1 Accept signal (L1A). The maximum possible readout rate is 75KHz, which sets the limit of the L1A rate.

The First Level Trigger consists of the Calorimeter Trigger, the Muon Trigger and the Central Trigger Processor(CTP) .

Muon trigger

In the muon trigger (see fig. 2.5) RPC detector specific logic and TGC detector specific logic get hit patterns from the RPC muon detector and TGC muon detector (see section 2.1.2). Each of 208 sectors can output a maximum of two possible muon tracks, called muon candidates, to the MUCPTI. The muon candidate data contains information about the position and ptrange of the muon.

The MUCPTI sums up the total number of candidates in each of 6 pt ranges.

These sums are called the “multiplicity” values, and are fed to the Central Trigger Processor every bunch crossing.

In case of a L1A, the MUCTPI sends muon candidates data to the higher trigger levels.

Central Trigger Processor

The CTP uses the information from the Calorimeter Trigger and the Muon Trigger to create the overall Level 1 accept or reject decision (L1A).

Figure 2.5. Context of the MUCTPI[4]

2.2 Data Acquisition 11 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 000000000000000 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111 111111111111111

VME backplane dedicated backplane (MIBAK)

MIOCT MIROD MICTP MIOCT MIOCT MIOCT MIOCT MIOCT MIOCT MIOCT MIOCT

MIOCT

CHS07V01

MIOCT MIOCT MIOCT MIOCT MIOCT MIOCT

Figure 2.7. MUCTPI Crate[4]

2.2.2

MUCTPI

The role of the MUCTPI (Muon to Central Trigger Processor Interface) in the Muon Trigger is explained in the previous section (2.2.1).

The MUCPTI is made up of 16 MIOCTs, one MICTP and one MIROD in a VME crate with a custom backplane, MIBAK, in addition to the VME backplane and is equipped with a single board computer used for configuration. See fig. 2.7.

The MUCTPI works as follows (see fig. 2.6):

16 MIOCTs receives muon candidates from the RPC and TGC sector logic. The MIOCTs synchronizes the signals, and makes sure no muon candidate is counted twice, which could otherwise happen where there are overlaps in the detectors. The MIOCTs sums the multiplicity of muon candidates in each ptthreshold. The

total multiplicities for each threshold of all 16 MIOCTs are formed by adders on the MIBAK backplane. The total multiplicities are fed to the Central Trigger Processor (CTP) via the MICTP.

If the CTP decides on an accept (L1A), this signal is fed to the MICTP and then via MIBAK to the MIOCTs. This will initiate a readout of all muon candidates from that bunch crossing from the MIOCTs and the multiplicities from the MICTP to the MIROD via a data bus on the MIBAK. The MIROD then sends data of up to 16 candidates to the Level-2 Trigger to help it with where to look, and send information about all candidates to DAQ, which is the system responsible for storing all data when the trigger system accept an event.

Chapter 3

Problem definition

The job was to redesign the MIROD, the readout driver of the MUCTPI (see 2.2.2). The work consisted of creating a schematic and approximate layout of the board, and creating and simulating VHDL code.

The redesign was to be based on the original design of MIROD, which had been used in prototypes. The original design had all the required functions, but could not be used in the finished system since it required an adapter to use the S-LINK mezzanine cards used for optical output. The adapters could be used as an temporary solution during testing, but in a complete system they couldn’t be used because of space constraints.

Another advantage of doing a redesign is that more modern components can be used, which have better performance and availability. Some of the componets of the original design, made around year 2000, were hard to get hold of.

Later it was decided that while redesigning the MIROD, the PCB should be prepared so that it can also be used as a MICTP, which is another part of the MUCTPI, to save board manufacturing costs.

Chapter 4

Technology used

4.1

FPGA

An FPGA, Field Programmable Grid Array, is a programmable integrated circuit. Using an FPGA leads to much lower non-recurring engineering costs compared to fabricating a chip, and is therefore significantly cheaper for small series.[3]

It consists of an array of programmable logic blocks, tied together with a pro-grammable interconnect. A block is typically made up of look-up tables (LUTs), multiplexers and flip-flops.

The basic building block of the Altera Stratix II is called a “Adaptive Logic Module”(see fig 4.1). It can be configured in several different ways. The Adaptive LUT can either be configured as one 6-bit LUT or as two smaller LUTs.

Figure 4.1. Altera Stratix II Adaptive Logic Module

4.2

JTAG

A JTAG (Joint Test Action Group) port is an 4 or 5 pin interface which can be used for boundary scan testing, to test the connectivity on a board, or to program programmable chips.

4.3

VHDL

VHDL is an acronym for VHSIC1 _{Hardware Description Language. It is a}

lan-guage which can be used in all steps (design, verification, synthesis and testing) of hardware development. VHDL was originally developed in the early 1980s by the American Department of Defense and is heavily influenced by ADA, a program-ming language also developed by DoD. Today VHDL is standardized by IEEE2.

4.4

VME

VMEbus3 _{is a computer bus standard. It was introduced in 1981 and is a}

com-bination of Eurocard physical dimension, and the VERSAbus bus that Motorola had developed two years earlier for its 68000 processor.

VMEbus is a master/slave architecture with separate address and data buses. Two different sizes are defined for VME cards. A card can either have the height 3U4 and have one connector or be 6U and have two connectors. There are also systems with 9U cards, but those are not actually part of the standard. The original VMEbus specified addressing ranges between 16 and 32 bits and data width between 8 and 32 bits, and a bandwith up to 40MB/s. The newer VME64x standard from 1997 specifies data path width and address range up to 64 bits, and a up to 160MB/s bandwidth.

4.5

Signaling schemes

The card needs to be able to handle several different signaling levels for the inter-faces to other systems. These are the signalling standards used.

NIM

The card will have connections on the front panel working according the the NIM5

standard. The NIM standard was first created by the US Department of Energy in the 1960s to be used in the nuclear sector. It defines both mechanical and electrical properties of electronics modules. The part of the standard which will be implemented on the MIROD is a front panel input connector using NIM Fast

1_{Very High Speed Integrated Cirquits} 2_{IEEE Std 1076}

3_{VERSAmodule Eurocard bus} 4_{1U=1.75" =44.45mm}

4.5 Signaling schemes 17

Figure 4.2. LVDS receiver schematic

Logic Levels. This means that a logic 1 is represented by drawing 16mA and a logic 0 is represented by not drawing any current, with a 50 Ω resistor to VT T next

to the receiver.[2] The connector is a LEMO6 00.250 coaxial connector.

LVTTL

Many of the connections internal to the board and on the backplane connections use LVTTL7_{signaling levels. It is similar to TTL levels, but uses a supply voltage}

VCC of 3.3 Volts instead of 5 Volts.

LVPECL

Several signals on the backplane are LVPECL. LVPECL stands for Low Voltage Positive Emitter Coupled Logic. Emitter Coupled Logic, ECL, is a high perfor-mance family of logic introduced in the 1960s, designed to have lower propagation delay and less skew than CMOS or TTL logic. In ECL, the transistors are always in the active region, so there is always current flowing. This means that they consume power also when they are not switching (as opposed to CMOS, which have have much smaller power consumption when not in use).

ECL uses a negative voltage supply (-5.2 V), which makes it a bit complicated to use together with other types of logic. By connecting an ECL part to a positive voltage supply (connect VCC to +5V and VEE to GND, instead of VCC to GND

and VEE to -5.2V) you get PECL. In ECL and PECL, the output level is relative

to VCC. VOH = VCC - .9 V and VOL= VCC - 1.7 V, i.e. the swing is .8 V

LV PECL is similar to PECL, but with VCC = 3.3 V

When using (LV)PECL, one must remember that since the output is relative to

VCC, any noise on the voltage supply will be seen on the output.[8]

ECL signals can either be differential or single ended.

LVDS

LVDS8 _{is a differential signaling technology introduced in the 1990s. A 3.5 mA}

current, which direction depends on if the signals is a ’1’ or a ’0’, is led through a 100 Ω resistor at the receiver, leading to a 0.35 V difference. The common mode voltage is 1.2 V. See fig 4.2.

6_{LEMO is a Swiss manufacturer of electronic and fiber optic connectors.} 7_{Low Voltage Transistor Transistor Logic}

Figure 4.3. The S-LINK Concept[6]

S-LINK

S-LINK is a data-link interface specification created by CERN[6]. An overview of an S-LINK system can be seen in figure 4.3. The MIROD will have a link source interface, on which a Link Source Card will be fitted. How the physical link is implemented is not specified in the S-LINK specification. On the MIROD, a HOLA9 optical link, with a maximum data rate of 160MB/s[7] will be used.

Chapter 5

Software Tools

The following software was used.

5.1

Tools for FPGA

To create the top level of the design, the graphical tool Visual Elite from Summit Design1 was used. By using this tool, the design is first represented as a block diagram, which was then exported to VHDL code.

To get more control over the VHDL, the text editor XEmacs2 _{was used,}

to-gether with VHDL Mode3 _{created by Reto Zimmermann and Rod Whitby.}

Sim-ulation of the VHDL design was done using NC-VHDL from Cadence4_.

Synplify from Synplicity5_{was used to synthesize the VHDL to a netlist.}

Last, Quartus II from Altera6_{was used to do place and route, to get a binary}

file to load onto the FPGA.

5.2

Tools for Board

The schematic of the board was done using Cadence Allegro Design Entry HDL7

and the layout was done using Cadence Allegro PCB Editor. These are part of the Cadence SPB (Silicon Package Board) suite.

1_{Today owned by Mentor Graphics, www.mentor.com.} 2_{www.xemacs.org/}

3_{http://www.iis.ee.ethz.ch/˜zimmi/emacs/vhdl-mode.html} 4_{www.cadence.com}

5_{www.synplicity.com} 6_{www.altera.com}

7_{Previously called “ConceptHDL”}

Chapter 6

Design

This chapter first describes the design of the MIROD and then the design of the MICTP. These two system are both part of the MUCTPI (section 2.2.2) and share many features. In the end of the chapter is a design of a unified board described, which is possible to configure as either a MIROD or a MICTP.

6.1

MIROD

MIROD (Muon Interface Read Out Driver) is the read out driver of the MUCTPI (see Sec. 2.2.2). It is used for reading out trigger data to the Level 2 trigger (LVL2) and to the Data acquisition system (DAQ).

The interfaces to the MIROD can be seen in fig 6.1. On the right hand side you have the back panel connections. The VME bus (see section 4.4) is used for configuration and testing. A single board computer in the VME crate can issue read or write commands to the MIROD to read or set configuration bits or status bits, or to read or write test data to memories. This single board computer is accessed via ethernet.

MIBAK is a custom backplane only used in the MUCTPI. The boards in the MUCPTI are 9U in height. Since the VME standard defines connectors for 6U (or 3 U) boards), by using a 9U board there is some extra unused space available. This is where the MIBAK backplane is located(see fig. 2.7).

MIROD uses the MIBAK backplane for two things. It receives the 40 MHz sys-tem clock via MIBAK, and it reads out data from the MIOCTs and the MUCTPI via a 36 bits wide, token based data bus on the MIBAK.

On the front panel, MIROD has fiber optic S-LINK(see section 4.5) connections for data output.

6.1.1

Old MIROD

How the older version of MIROD was implemented can be seen in fig. 6.2. It is using Altera Flex 10K PLDs (Programmable Logic Device). Each of the yellow boxes in the diagram represents a PLD. An additional PLD not shown handles the

MIROD

VME S-LINK DAQ

Front panel _{Back plane}

S-LINK LVL2 MIBAK

Clock

Data bus

Figure 6.1. MIROD Interfaces

programming of other six, reading the programming data from a Flash memory. Each of the PLDs is mapped to the VME address space, allowing the PLD’s configuration bits to be written or status bits to be read via VME.

The “TestramController” and “Monitoring” PLDs shown in the diagram are used for testing. The TestramController can fill a memory with data via VME, which can then be used as input data for testing, instead of reading from MIBAK. The Monitoring PLD can be programmed to store output data in a memory, which can then later be read via VME.

In case of a L1A, the “Eventreader” handles reading out from the MICTP and MIOCTs via the MIBAK data bus. It does consistency check on the data, and changes the format of the RoI data, before it writes the data to a 40 bit wide FIFO memory.

If the FIFO isn’t empty, and both the DAQ S-LINK and the LVL2 S-LINK are ready to receive data, the “Extractor” reads from the FIFO. The words in the FIFO contains two muon candidates. The Extractor splits them up, so that each outgoing (32-bit) word only contains one candidate. Candidates that has a

ptvalue lower than a programmable value are flagged.

The “LVL2Processor” sorts out the 16 candidates with the highest ptand sends

them in order to the Level-2 Trigger, on condition that they are higher than the programmed value.

The "DAQProcessor“ sends all candidates to the DAQ system.

6.1.2

New MIROD

In the new version of the MIROD, it is desirable to fit all logic in a single FPGA. To see if this was possible, the VHDL from the six PLDs (the seventh PLD used to program the other six is not needed anymore) of the old MIROD was used. These six blocks were tied together by modelling the interconnects between them on the circuit board, as well as the external memories, in VHDL.

6.1 MIROD 23 M o n it o r in g T e s tr a m - C o n tr o ll e r L V L 2 P r o c e s s o r E x tr a c to r L V L 2 S -L IN K F IF O D A Q S -L IN K E v e n tR e a d e r D A Q P r o c e s s o r M IB A K M e m o r y M e m o r y

MIROD VME Model MIBAK Model S-LINK DAQ Model S-LINK LVL2 Model

Figure 6.3. MIROD Testbench

6.1.3

Testbench

To validate that this new VHDL design of the MIROD, with the content of all six PLDs and including the memories, was working properly, a testbench was created. This testbench instantiated the MIROD design together with a model of the VME bus, a simple model of the MIBAK, with a MICTP and 16 MIOCTS, and a model of the S-LINK Link Source Cards.

After starting the clock and doing the initial reset, the testbench issues write commands via VME to set up the MIROD. These commands are specified in a text file which is read by the VME model. In the actual system, these VME commands would have been sent by the single board computer in the VME crate.

The MIBAK model allows data specified in text files to be read out. In the actual system, this would be data read out from MICTP and the 16 MIOCTs via the MIBAK data bus.

The S-LINK models receive data and write it to text files, which can later be analyzed. In the actual system, this data would be received by the S-LINK LSC cards, and be transmitted to LVL2 and DAQ.

VME and S-LINK models were available ready to be used, but no MIBAK model was available so it had to be created.

6.1.4

Synthesis

This new VHDL design with the content of all six PLDs and including the mem-ories could then be synthesized. It was found that the design can be implemented using an Altera Stratix II EP2S60 FPGA, utilizing 10% of the available logic and 20% of the memory.

6.2 MICTP 25 Back plane VME MICTP MIBAK Clock and Multiplicity trigger signals Data bus Connectors

Clock and trig. signals Busy

Multiplicity Front panel

Figure 6.4. MICTP Interfaces

6.2

MICTP

The interfaces of MICTP can be seen in figure 6.4. MICTP is connected to the VME bus in the same way as MIROD. It is also connected to MIBAK, but has some additional connections on MIBAK compared to MIROD. MICTP both trans-mitts and receives the clock and the trigger signals L1A, ORBIT, ECR, and TST. MICTP receives these signals via connectors on the front panel and distributes them to the 16 MIOCTs and to itself, and in case of the clock also to MIROD, via MIBAK. The reason MICTP also receives these signals on MIBAK is to be in sync with the rest of the system.

MICTP receives Multiplicity values, which have been summed up by a adder tree on the MIBAK using data from the 16 MIOCTs (see fig. 2.6). These values are sent to the Central Trigger Processor on each clock via a connector on the front panel. These multiplicity sums are also stored in a buffer, to be read out to MIROD via the MIBAK data bus in case of a Level 1 Accept (L1A).

To adjust the timing of MUCTPI, MICTP is able add a delay to the Clock, L1A, Orbit and ECR signals received on the front panel. To know how much adjustment is needed, the MICTP measures the phase of these signals and of the Multiplicities signal.

6.2.1

New MICTP

The logic in the MICTP is relatively simple, and can easily be fitted in the kind of FPGA used in MIROD. The most critical part of the MICTP is the latency of the Multiplicity signals that MICTP leads from MIBAK to a front panel connector.

Used in both

Front panel connectors

Trigger type Status signals

Trigger and timing Multiplicity

Delay SCAN

BRIDGE

Buffers DAQ S-LINK Link Source Card

Timing Measure LVL2 S-LINK Link Source Card

ID Sys. mon. TMB TBC Buffers jtag_hptdc jtag_altera jtag_BusLVDS jtag_ram jtag_mibak RAM V M E FPGA i2c EPCS M I B A K timing Data bus Status Trig. and Multiplicity colors: MIROD Level Convert-ers MICTP

Figure 6.5. Block diagram with the combined designs of MIROD and MICTP.

To make the latency of when these signals enter the FPGA until they leave the FPGA as small as possible, they’re synchronized with a clock phase shifted by a a PLL.

6.3

Board

Since the MIROD and MICTP are similar in many ways, it is decided to design a single board which can be used either as a MIROD or a MICTP, depending on configuration.

A block diagram of the combined system can be seen in fig. 6.5, where the components used by both MIROD and MICTP are shown in green, components used only by MIROD in blue, and components used only by MICTP are in yellow. In case of the MIROD, no front panel connectors are mounted, to make room for the S-LINK cards. In case of MICTP, the front panel connectors are mounted, making it impossible to fit the S-LINK cards.

6.3 Board 27

6.3.1

Both MIROD and MICTP

The following components are used both by MIROD and MICTP.

FPGA

The central component of both MIROD and MICTP is an Altera Stratix II FPGA. The FPGA will be programmed in different ways depending on if the board is to function as a MIROD or as a MICTP. It is the flexibility of the FPGA that makes it possible to let the two designs share a board.

VME

The FPGA is connected to the VME connectors on the backplane using SN74VMEH22501 bus transceivers from Texas Instruments.

MIBAK

Both designs receives the clock from the MIBAK, and is connected to the MIBAK data bus. A difference is that MIROD only needs to be able to read the data bus, and MICTP only needs to be able to write to the bus. SCAN92LV090 Bus LVDS transceivers, made by National Instruments, are used for the data bus. They function both as receivers and transmitters, and can do boundary scan testing using JTAG.

FPGA configuration device

At power-up, the FPGA is programmed by a Altera EPCS serial configuration device, which has the configuration data stored in a flash memory. It is also possible to configure the FPGA using JTAG.

The configuration device can be loaded with programming data either by cable or via the FPGA.

Memory

A RAM chip is put on the board, in addition to the memory in the FPGA. This is not required for either of the designs to function, but is usefull to be able to store more data for testing purposes. The RAM chip is a 36-bit wide QDR. The memory will be run with a 160 MHz clock (four times the system clock), and is capable of two read cycles and two write cycles every cycle.

JTAG

Some of the components of the board can be accessed via JTAG. JTAG can be used either to configure the device, or to perform boundary scan testing to make sure that the connections work. The JTAG components on the board are sep-arated into five chains. Theoretically, all could belong to the same chain, but since some components might be incompatible with each other, separating makes

it more robust.

The components in each of the JTAG chains are:

• The FPGA.

• The Bus LVDS chips for the MIBAK data bus. • The QDR RAM.

• The adder chips of the MIBAK backplane. • The HPTDC

The chains are accessed via the scanbridge, which is a SCANSTA112 JTAG multiplexer by National Semiconductor.

The scanbridge can either be controlled by the FPGA, using the

testbuscon-troller (a 74LVT8980 from Texas Instruments), or by the TMB (Test and

Main-tenance Bus) on the VME, or by attaching a cable to the board.

6.3.2

Only in MIROD

S-LINK

In addition to the components shared with MICTP, the MIROD have two connec-tors to connect S-LINK cards

6.3.3

Only in MICTP

MIBAK

MICTP transmitts the clock and receives and transmitts the signals L1A, ORBIT, ECR, and TST.

MICTP receives the 18 bit wide multiplicity signal.

Front panel connections

MICTP has connectors on the front panel to receive and transmitt a number of signals.

Delay and timing

MUCTPI, the MICTP is equipped with a chip called delay25, which can delay the Clock, L1A, Orbit and ECR signals received on the front panel. The delay of each of these signal can be set independently by configuring the delay25 via i2_c.

To measure if timing adjustment is needed, the MICTP measures the timing of these signals, as well as the Multiplicities signal. This measurement is done by a chip called hptdc(High Performance Time to Digital Converter)

6.3 Board 29 N IM / L V T T L L V D S / L V T T L L V T T L / L V P E C L L V P E C L / L V T T L L V D S / L V P E C L H P T D C F R O N T P A N E L T S T L 1 A E C R O R B B C K X ta l N IM / P E C L d e la y 2 5 L V D S / L V T T L M A IN F P G A M IB A K M u lt . L V T T L / L V D S T T L / L V T T L

Chapter 7

Results and discussion

A board (see fig. 7.1) has been manufactured and been assembled as a MIROD. It is now being tested.

7.1

Benefits of using the same PCB design for

both systems

To bring all logic together in a single FPGA, which was made possible by increasing FPGA sizes, makes the hardware very flexible. The MIROD and the MICTP have different functions, but could still be implemented with the same board layout. This saves hardware costs. For a system like MIROD or MICTP, which is produced in very small numbers (only one MIROD and one MICTP is needed in the ATLAS experiment), the starting costs for manufacturing the PCB (creating films etc.) are very high compared to the total costs.

Another advantage is that it makes it easier to keep spares. Unassembled PCBs, can be assembled later when they are needed either to a MIROD or to a MICTP.

7.2

Disadvantages of using the same PCB design

for both systems

One disadvantage of using the same PCB design on both systems is that the component costs are higher. Some components are only needed in one of the designs, but are present in the other as well. But since the number of boards are so small, this is not significant.

Figure 7.1. The new MIROD

Another disadvantage is that it makes the design more complex. This is a huge disadvantage if you are an inexperienced designer. The difference in the content of the FPGA doesn’t really matter. It is easy to handle. But the differences in the board can be tricky to handle. In the case of the MIROD/MICTP, the MIROD has most of it’s complexity inside the FPGA, while the MICTP has most complexity in the handling of the front panel interfaces.

In retrospect, to let the two systems share PCB design may not have been the right design choice.

7.3

Software tools

In general, the quality of the tools have been poor. Especially the user interfaces of the Cadence programs are very hard to work with, with plenty of bugs and in-consistencies. The prime example of an inconsistency is screen panning in Cadence SPB Suite. One program in the suite uses “push background” panning, while the next uses “push camera” panning, so when switching back and forth between the programs you always pan in the wrong direction. This kind of issues make it very frustrating to work with the programs.

Bibliography

[1] FAQ: The LHC. http://public.web.cern.ch/Public/Content/Chapters/AskAnExpert/LHC-en.html, accessed 2007-10-03.

[2] Standard NIM instrumentation system. US. NIM Committee, Washington, DC, 1990.

[3] James R. Armstrong and F. Gail Gray. VHDL design representation and synthesis (2nd ed.). Prentice Hall PTR, Upper Saddle River, NJ, USA, 2000.

[4] ATLAS Collaboration. ATLAS first-level trigger: Technical Design Report, volume TDR-12 of Technical Design Report ATLAS. CERN, Geneva, 1998. [5] Vittorio Frigo. Atlas detector.. detecteur atlas. AC Collection. Legacy of AC.

Pictures from 1992 to 2002., Mar 1997.

[6] Robert McLaren and Erik Van der Bij. The s-link interface specification. Technical Report ALICE-INT-1995-35. CERN-ALICE-INT-1995-35, CERN, Geneva, 1995.

[7] Aurelio Ruiz, Erik Van der Bij, and Stefan Haas. HOLA S-LINK.

http://www94.web.cern.ch/HSI/s-link/devices/hola/, accessed 2007-10-18.

[8] Fred Zlotnick. A comparison of lvds, cmos, and ecl.

http://www.onsemi.com/pub/Collateral/AND8059-D.PDF, 2001.

Glossary

pt transverse momentum

ATLAS A Toroidal LHC ApparatuS

CTP Central Trigger Processor (of First Level Trigger)

DAQ Data Acquisition

FIFO First In, First Out

FPGA Field Programmable Gate Array

L1A Level-1 Accept

LHC Large Hadron Collider

MIBAK Muon Interface Backplane

MICTP board to interface the MUCTPI to the CTP

MIOCT Muon Interface Octant Board

MIROD Muon Interface ReadOut Driver

MUCTPI Muon to Central Trigger Processor Interface

PCB Printed Circuit Board

PLD Programmable Logic Device

RoI Region of Interest

VME VERSAmodule Eurocard

Appendix A

Speed of Protons

The speed of a proton can be calculated with the following equations. The calu-lation has to be done relativistically, since the speed is close to the speed of light

c. The resting mass mo of an proton is 1.007 u.

When a proton enters the LHC it has kinetic energy Ek= 450 GeV , which

means that the speed is 0.999997837 c. In the LHC the proton is accelerated to 7 TeV, i.e. 0.999999991 c. v = c s 1 − ( moc 2 Ek+ moc2 )2 _(A.1) 36

Upphovsrätt

Detta dokument hålls tillgängligt på Internet — eller dess framtida ersättare — under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för icke-kommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av doku-mentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerhe-ten och tillgänglighesäkerhe-ten finns det lösningar av teknisk och administrativ art.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan be-skrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se förla-gets hemsida http://www.ep.liu.se/

Copyright

The publishers will keep this document online on the Internet — or its possi-ble replacement — for a period of 25 years from the date of publication barring exceptional circumstances.

The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for his/her own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.

According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.

For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/

c