Qualitative assessment of the contribution of permanent highway observatories to socio-economic evaluations in France

(1)

STOCKHOLM SVERIGE 2021,

Qualitative assessment of the

contribution of permanent highway observatories to socio-economic evaluations in France

ROBIN GABORIT

(2)

First, I would like to thank Joel Franklin, professor at KTH, and Nicolas Coulombel, professor at the Ecole des Ponts ParisTech. They were my two academic tutors for this degree project. They were good advisers in defining the exact questions of this work, guide it and organize my report.

Secondly, I want to thank Arthur Sebes, Jean-Baptiste Gouesse, and Lionel Bertrand who supervised me at Setec International during this final year project. Thanks to them, the study carried out for the Highway A150 was very educative. They were able to give me autonomy while helping me when necessary and made my experience at Setec International a pleasant one.

I would particularly like to thank Didier Revillon and Caroline Cayzac, from the management of Setec International, for allowing me to carry out this degree project, trusting me for the second time after a first internship. These two experiences were very enriching and reinforced my desire to work in the field of transport studies. More generally, I would like to thank the entire team at Setec International Paris for their kindness, their sympathy, and their willingness to teach me.

I would also like to thank Aphrodite Michali and Florence Borde, respectively Academic Director and Inspector of Studies of the Civil Engineering and Construction Department of the Ecole des Ponts ParisTech, for their support throughout my studies in the Department. I am thinking in particular of the introduction of the cross-curriculum "Transport Infrastructures" and the possibility of this double degree with KTH, two options that I was lucky enough to have.

Finally, I want to thank Erika Charpentier, International Coordinator of the School of Architecture and the Built Environment at KTH for her availability and kindness to international students.

(3)

Sammanfattning

Denna mastersuppsats avslutar mina studier vid Ecole des Ponts ParisTech samt Kungliga Tekniska Högskolan. Den utfördes vid Setec Internationals institution för

transportdesign i Paris mellan augusti och december 2020. I studien var jag en del av ett team som skulle utvärdera förändringar på motorväg A150 i Frankrike.

Permanenta observatorier är organisationer som sammanför offentliga och privata aktörer för att i tidigt skede systematiskt samla data för att analysera effekterna av

transportinfrastruktur. De har satts upp sedan 80-talet, framförallt på franska motorvägar. I huvudsak bidrar de till framtagningen av utvärderingar kallade LOTI. Målet med projektet är att kvalitativt utvärdera bidraget av dessa observatorier på socioekonomiska utvärderingar, både klassiska och mindre klassiska effekter. I studiens metoder används den praktiska erfarenheten av utvärderingen av motorväg A150, litteraturstudier, samt analys av statliga rapporter om kvalitén på LOTI-utvärderingar.

Permanenta observatorier begränsar några av de fallgropar som utvärderingar som utfärdas i efterhand vanligtvis kan falla i, särskilt relaterat till de klassiska effekterna kring trafik och restider. Dessutom betonar de effekten av byggarbetsplatser och ekonomiska aktörers strategier. Det finns dock vissa begränsingar på tid- och rumsnivå, samt behov av data från deltagande aktörer. Tack vare tidig och regelbunden datainsamling har permanenta observatorier verkat bidra signifkant till analysen av de ekonomiska effekterna genom att begränsa risken för felaktiga analyser. De verkar dock inte helt nödvändiga för att kunna utföra goda studier och analyser.

Den mängd data som finns tillgänglig om kostnaden av permanenta observatorier indikerar att kostnaden är låg i jämförelse med dess effekter. På denna grund anser denna studie att permanenta observatorier har en positiv effekt på socioekonomiska analyser.

Studien är dock begränsad av den mängd data som finns tillgänglig då permanenta observatorier inte är obligatoriska.

Nyckelord: Permanenta observatorier, motorväg, socioekonomisk analys

(4)

Abstract

This thesis serves both as an end-of-studies project for the Ecole des Ponts ParisTech and as a Degree Project in the framework of a double-degree with the University of KTH (Sweden). It was carried out at the transport design office Setec International, in Paris, from August 3^rd to December 18^th, 2020. I was part of the team in charge of the post-evaluation of the A150 highway.

Permanent observatories are organizations that bring together public and private actors in order to collect early and systematically data to analyze the effects of a transport

infrastructure. They have been set up since the 1980s, mainly on French highways. In particular, they contribute to the production of a posteriori assessments known as "LOTI".

The objective of this project is to qualitatively evaluate the contribution of these observatories to socio-economic evaluations, both for "classic" and "wider economic" effects. The methods used are the "practical" experience of carrying out the LOTI assessment of the A150, the study of reports describing the results of the A39 and A71 observatories, and the analysis of reports from the State services on the quality of LOTI assessments.

The permanent observatories limit some of the pitfalls of ex-post assessments linked to the measurement of classic effects, especially traffic and journey times. They also highlight the effects of a construction site and the strategies of the economic actors. However, they retain certain limitations of ex-post assessments such as the spatial and temporal scale, or the concessionaire's willingness not to disclose certain data. Thanks to early and regular data collection, permanent observatories seem to contribute significantly to the analysis of classic and wider economic effects, they limit the risk of poor quality analyses. Nevertheless, they do not seem indispensable for the production of studies of satisfactory quality.

The rare data available on the cost of the observatories suggest that it is very low compared to the amounts invested in construction. Thus, in comparison with these costs, this end-of-study project gives a positive qualitative assessment of the contribution of permanent highway observatories to socio-economic studies. However, this study is limited by the amount of data available due to the non-mandatory nature of setting up permanent observatories.

Keywords: permanent observatory, highway, socio-economic evaluation

(5)

List of tables

Table 1: comparison of LOTI assessment procedures with and without a permanent observatory ... 19 Table 2: Extract from ALBEA database - vehicles going from the West to Barentin ... 42 Table 3: Extract from ALBEA database - vehicles going from Barentin to the West ... 42

(9)

List of figures

Figure 1: period of an observatory set up ... 21

Figure 2: the A150 highway - regional scale ... 34

Figure 3: the A150 highway - local scale ... 34

Figure 4: evaluation of LOTI assessments by the CGEDD - costs ... 39

Figure 5: A150 and A29 highway nodes ... 42

Figure 6: competition between A150 and D6015 ... 43

Figure 7: An example of the evolution of traffic over a day (source Département Seine-Maritime).... 44

Figure 8: traffic counts of a dysfunctional station (source: public agency DIRNO) ... 44

Figure 9: evaluation of LOTI assessments by the CGEDD - traffic ... 45

Figure 10: evaluation of LOTI assessments by the CGEDD - travel time ... 47

Figure 11: evolution of road safety on D6015 (A150 commissioned in 2015) ... 48

Figure 12: evaluation of LOTI assessments by the CGEDD - road safety ... 49

Figure 13: economic competition between Barentin and Yvetot ... 52

Figure 14: competition between A150, A151, and D6015 ... 55

Figure 15: Number of rooms (ch.) created between 1987 and 1995 – A71 was put into service in 1989 (source [22]) ... 59

Figure 16: evaluation of LOTI assessments by the CGEDD - territorial impacts ... 60

(10)

List of abbreviations

CETE: Centre d’Etudes Techniques de l’Equipement, technical studies center of equipement CGEDD: Conseil Général de l’Environnement et du Développement Durable, general council for the environment and sustainable development

LOTI: Loi d’Orientation sur les Transports Intérieurs, domestic transport planning law

ONISR: Observatoire National Interministériel de la Sécurité Routière, national interministerial road safety observatory

(11)

List of annexes

Annex 1: List of the highways included in the study of CGEDD reports ... 77 Annex 2: Results of the study ... 78

(12)

Qualitative assessment of the contribution of permanent highway observatories to socio-economic

evaluations in France

Introduction

Socio-economic appraisal is a crucial aspect of public decision-making. This is a matter of analyzing what are the benefits, disadvantages, and costs of a given project. The transport sector is an area in which appraisal is particularly important. The benefits include for instance reduction of travel time or increase in safety meanwhile air and noise pollutions are part of the disadvantages. Investment and maintenance are part of the costs. Policymakers aim to decide and justify transport policies and infrastructure constructions through socio-economic assessments. The assessments that aim at predicting the future effects of an infrastructure are called ex-ante assessments; those based on the observation of the effects of an existing infrastructure to evaluate it are ex-post assessments.

Conducting good quality ex-post evaluations is useful for two reasons. First, it helps to identify infrastructure effects that were poorly predicted at the time of the ex-ante evaluation.

The ex-post evaluation of a project thus highlights the precautions that need to be taken in the ex-ante evaluation of future projects of the same type. The qualities of ex-post and ex-ante evaluations are therefore deeply linked, the improvement of the practice of ex-post evaluation is in the first place at the service of better ex-ante evaluations. With more accurate, comprehensive, and less error-prone ex-ante evaluations, public decision-makers are better able to make the right decisions. It is said that improving the infrastructure assessment process

(13)

reduces selection bias, i.e., the risk of carrying out project A rather than project B when in fact project B would have been of greater benefit to the community.

Besides, an ex-post evaluation is useful for adjusting the project that is assessed after it is commissioned. For example, measures can be taken to try to reduce the observed environmental or noise nuisance of a highway or to increase the economic benefit by locating businesses in the most strategic locations.

Permanent observatories, the subject of this paper, are tools dedicated to improving the quality of ex-post socio-economic assessments of transport infrastructures. They are especially interested in the effects the infrastructure has on the territory development, notably those called wider economic benefits, see further. A permanent observatory is a collaboration of a concession company, researchers, public agencies, and stakeholders to systematically collect and analyze data about a specific infrastructure. They have been almost exclusively implemented in France; although a recent example is the Egnatia highway in Greece [22].

This thesis particularly focuses on highway observatories, the most widespread ones, although instances of high-speed railway lines are mentioned. It also concentrates on socio- economic permanent observatories; the subject of environmental observatories is not discussed.

The question that this thesis attempts to answer is the qualitative evaluation of the contribution of these highway observatories in France to socio-economic evaluations. This question is addressed both in the case of so-called "classic" effects such as costs or travel times and in the case of the indirect economic effects of infrastructure called “wider economic effects”.

First, the historical and technical context for the establishment of permanent observatories is set out. Then, a synthesis of the academic literature on which the establishment of this study is based is presented. This literature provides the context for developing the research questions and approaches that seem appropriate. Next, the methodology put in place to answer these questions is presented. This methodology is applied to the so-called "classic"

effects and then to the "wider" economic effects. Finally, the overall contribution of the

(14)

permanent observatories is discussed by comparing the results obtained with the costs and known limitations of the observatories.

This project is carried out in the framework of a Degree Project for Kungliga Tekniska högskolan (Stockholm) and a Projet de Fin d’Etudes (graduation project), for Ecole des Ponts ParisTech. It was conducted within the study office Setec International, in Paris. This thesis is notably based on the work in which I participated at Setec, the Highway A150 Barentin - Ecalles-Alix ex-post assessment.

(15)

I. Historical and technical context

This chapter presents background information on the ex-ante and ex-post socio- economic assessments, the elements present in these assessments, and the setting up of

permanent observatories. The academic literature base and research questions are described in the following chapter.

A. Ex-post LOTI audits

As mentioned above, ex-ante assessment is undertaken before project implementation.

Thus, it is based on prediction.

On the contrary, ex-post assessment is performed after the completion of a project. It is based on the observation of the actual effects related to the project. The ex-post assessment has several objectives:

- It evaluates the utility of the project for society.

- By comparing predicted and actual effects, it criticizes the quality of the ex-ante analysis of the project.

- It contributes to improving future ex-ante analysis by highlighting which effects should be better estimated.

- It enables to take adjustment measures to add value to the infrastructure or mitigate its damages.

In Europe, the first public studies for ex-post assessment were funded in the 1970s [5]. In France, the Loi d’Orientation sur les Transports Intérieurs LOTI), domestic transport planning law, was implemented in 1982 [5]. It legally binds the promoters of a transport project (maître d’ouvrage) to carry out an ex-post assessment if it is supported by public funding and has a cost

(16)

exceeding €83 million. This assessment is called “LOTI audit” or “LOTI assessment”. It must be performed between 3 and 5 years after the commissioning of the project. Like any ex-post assessment, a LOTI audit has for purpose to give feedback to judge the quality of the corresponding ex-ante assessment. Besides, it also aims to inform the public about the effects of the project and give an evaluation of the efficiency of public spending from socio-economic and environmental points of view; that is why LOTI assessments are public. They must follow the guidelines announced by Sétra public agency; the latest version dates from 2011 [20].

A LOTI audit consists of three parts. The first deals with the transport and economy of the project itself, i.e., costs, traffic, travel time, and road safety. These are the elements that directly affect the users of the infrastructure or its financiers. These elements are referred to as "classic effects" in the remainder of the thesis. The second part describes the indirect effects of the project on the actors of the territory in which it is carried out. This section is called

"Contribution to the development of the territory". A major source of information for this section is the undertaking of interviews with key actors of the territory. These two first sections are the subject of study of the permanent observatories of the socio-economic effects discussed in this thesis.

A third section describes the effects of the infrastructure on the environment. These effects can be observed by permanent observatories of environmental effects, often set up in conjunction with the observatories of socio-economic effects. Nevertheless, they are not part of the scope of this thesis.

(17)

B. The classic elements of an appraisal

Historically, ex-ante and ex-post assessments have focused on three main components.

- Financial costs

The main costs are investment costs. They are the fixed costs of all operations performed during the construction phase. In France, the mean cost of a highway kilometer is 6 million € [21]. Money is also spent on the maintenance of the highway after it is put into service. Finally, operating costs cover other expenses necessary to organize the infrastructure, such as the operation of offices belonging to the concessionaire company or toll stations.

- Travel time

Travel time savings are often the first benefit of a transport project. Permanent observatories are usually implemented for highways and high-speed railway lines, infrastructures that aim at reducing the travel time of their users by allowing high speeds. The construction of a highway can also reduce travel times on the secondary network if the new infrastructure attracts a part of its traffic and reduce its congestion.

- Road safety

The number of road fatalities and injuries is also a major element of road project assessments. For instance, the construction of a highway may improve traffic safety since it could attract heavy trucks from the secondary road network, where they are more dangerous.

Traffic is not itself a classic effect to which a monetary value or a qualitative assessment should be given; the amount of traffic does not directly affect social surplus. However, the

(18)

amounts of travel time savings, road accidents, depend on traffics. Thus, traffic is another

“classic element” that must be estimated or observe to deal with “classic effects”. The evolution of rail traffic may also be another interesting point in the case of potential modal competition.

C. Wider economic benefits

At the end of the 1960s, the European highway network began to be substantial [5].

Most major cities were connected with highways. Those highways were the ones to be constructed first because they could reduce congestion and decrease travel time where the need was the greatest. Thus, assessments were easily showing their usefulness, their cost-benefit ratios were very high. A lot of other highway projects seemed relevant to complete the highway network. However, those remaining projects could not have socio-economic cost-effectiveness as high as those already implemented. Thus, public authorities began to become interested in another kind of effect to include in the assessment: wider economic benefits [5].

A transport infrastructure does not only affect its users, it can also benefit businesses.

By reducing travel times, highways increase their area of influence. Companies are more connected to each other, facilitating exchanges with suppliers for instance. Accessibility to demographic centers is also essential for businesses to attract skilled labor and for shops to attract more customers.

Economy and employment are also stimulated during the construction phase. That is of course the case of the public work sector. The hotel sector and local shops may also benefit from the expenses of the construction site personnel.

All these aspects are known as the “wider economic effects”, the effects that concern actors that do not directly use the infrastructure. Some existing companies may have better accessibility and new ones are encouraged to set up in the vicinity, potentially creating conditions for economic growth.

(19)

Observing wider economic benefits is notably essential to deal with the “structuring effects” myth. In the 50s and 60s, France benefited from the economic boom. Economic growth was very high and the first highways were constructed to modernize the country. From this time a belief was born, the belief of structuring effects. Highways are seen as a major cause of this economic growth. The construction of a highway would necessarily trigger economic development in the region it crossed. Accessibility would necessarily lead to attractiveness [2].

This belief may have had a significant role in the multiplication of highways across the country.

However, since economic growth may have various causes, the role of infrastructure may be less important than supposed.

Wider economic effects are part of LOTI ex-post assessments, in a section called

“contribution of the infrastructure to the territory development”. But those effects are difficult to measure, much more than the classic elements of an appraisal. A lot of data is necessary;

project sponsors and public agencies do not own all of it.

D. Implementation of permanent observatories

A permanent observatory is the cooperation of several actors to systematically collect data about a specific infrastructure, its territory and produce analyses. Four types of actors are involved:

- The project owner of the infrastructure: a private concession company (common nowadays in France) or a public entity

- Academics who organize the process in cooperation with the project owner and produce analyses; they are often part of a university of the region crossed by the infrastructure - Public actors that can provide relevant data: local communities, State services, Chamber

of Commerce and Industry, Chamber of Agriculture

- Private companies are sometimes requested to collect data and produce analyses

(20)

Table 1 sums up the particularities of a LOTI ex-post study with a permanent observatory compared to a LOTI study without observatory. More details are then given.

Without a permanent observatory

With a permanent observatory

Studied effects

Classic effects and wider economic effects

Similar, but more resources put into wider economic effects

Beginning of data collection

Mostly as the beginning of ex-post assessment study, 3- 5 years after commissioning

Before construction

Frequency of data collection

Only for the ex-post assessment

Regular, for example, each year

Frequency of publication of studies

Only transmission of the LOTI assessment to the State services

Possibility to publish intermediary results

Organization

The project owner and the director of the study contact other actors to try to get data at the moment of the LOTI assessment

Cooperation and data transmission between actors decided in advance

Table 1: comparison of LOTI assessment procedures with and without a permanent observatory

(21)

Permanent observatories observe different kinds of data, such as the classic elements mentioned above, but more particularly wider economic effects. They collect and analyze data for 3 phases:

- Before construction starts

This is useful to observe the dynamics exogenous to the project, but also anticipations from different actors.

- Period of construction

Construction work itself impacts the territory.

- After commissioning

The infrastructure improves the mobility supply and modifies the flows within the transport network.

In 1982, shortly after the implementation of the first permanent observatories, the LOTI law obliged project sponsors to carry out an ex-post assessment between 3 and 5 years after commissioning. Thus, permanent observatories also became a potentially relevant tool to undertake LOTI ex-post assessments. Typically, a permanent observatory is implemented over 8 to 15 years; at least until the ex-post study is carried out, see Figure 1. This depends on the duration of construction and the will of the project owner to analyze the situation for a long time before construction and after commissioning. Contrary to a usual ex-post assessment, data collection is systematic. Thus, intermediary results can be regularly published before the final LOTI ex-post assessment, every few years for example. Additional studies out of the LOTI framework studies can also be published, for example, thematic studies dealing with certain aspects of wider economic effects.

(22)

Before construction Construction phase Operation phase

Figure 1: period of an observatory set up

At the beginning of the 1980s, Sétra public agency implemented the first permanent observatories in France to address the issue of wider economic benefits described earlier: A10, A40, A42, A51, A75, and A72 [5, 23]. From the end of the 1980s, all the new observatories are put in place by the private sector. Private highway companies like APRR and ASF set up in their turn permanent observatories in cooperation with Sétra and academics, mostly in the 1990s: A77, A71, A39, A20, A39, and A54. More recently, the concessionaries LISEA and ERE have set up observatories for high-speed railway lines.

This process of implementing permanent observatories has never been mandatory; this is a choice from concession companies. That is for them a tool to improve their knowledge of the effects of their infrastructure. Moreover, it is an argument to show their concern about the effect their infrastructure has on the region and win the call for bids.

3-5 years LOTI audit

Permanent observatory set up

(23)

II. Literature review

This chapter describes the academic literature on which this thesis is based to build research questions. The research questions are then presented, as well as methodological approaches that seem relevant to answer them.

To understand the context of the development of ex-post studies and highway observatories in France, it is relevant to bear in mind the evolution of socio-economic evaluation practices in France.

A. Changes in assessment practices in France

Emile Quinet gives an analysis of the evolution of the practice of socio-economic evaluation in France [19]. In particular, he insists on the transition from a single-criteria evaluation to a mixed evaluation between single-criteria and multi-criteria. The 2004 circular on the harmonization of methods for the evaluation of major transportation infrastructure projects [15] still placed considerable emphasis on cost-benefit analysis. The calculation of synthetic indicators based on social surplus theory and containing all the monetizable effects of the project, such as Net Present Value, was then presented as the only way to compare projects. Little importance is given to non-monetizable effects such as social and territorial inequalities or environmental effects.

The 2007 circular [16] marks a break. It insists on the project's capacity to meet the objectives announced ex-ante in the declaration of public utility, rather than simply achieving a level of performance expressed by the cost-benefit analysis. It was following this circular that separate elements of the cost-benefit analysis, such as the effects on the economy, the evolution of land tenure, or the feelings of local stakeholders, were integrated into the

evaluations. These elements can now be found in the "contribution to the development of the territory" section of the LOTI assessments. However, the cost-benefit analysis is still carried out on the monetizable effects and keeps a central place. Thus, Quinet describes the current

(24)

situation as being mixed between single-criteria and multi-criteria. The prediction and observation of both monetizable and non-monetizable effects are issues for the practice of socio-economic evaluation and are enshrined in law.

According to Quinet, the practices enshrined in the 2007 circular are educational. This multi-criteria procedure, which is more accessible to the public than an evaluation based solely on a cost-benefit assessment, should encourage exchanges during public debates. In this way, the project and its acceptance can be improved during its preparation.

Thus, the classic effects, which are generally monetizable, remain of major importance for the realization of socio-economic studies. If major progress has yet to be made in

predicting these effects, the establishment of tools such as permanent observatories could be of significant use in this regard.

B. Margin of progress in predicting the classic effects of highways

As Nicolaisen et al [17] and Flyvberg et al [3] point out, the literature contains few meta-analyses comparing observations from ex-post evaluations with predictions from ex-ante evaluations over large samples. Studies of only a few cases often consider differences

between ex-ante and ex-post evaluations to be the exception, due to specific causes and not to a problem affecting evaluation practice in general.

An extensive traffic study was carried out by Flyvberg et al [3]. It covers 210 transport projects, including 183 road projects, representing $59 billion of investment. These projects were carried out between 1969 and 1998 in 14 countries on five continents. The traffic taken into account is that one year after commissioning. The results are very different between rail and road projects. 50% of the road projects are subject to a difference of at least 20% between forecast and observed, 25% to a difference of at least 40%. This phenomenon includes both underestimates and overestimates; 21.3% of road projects have their demand overestimated

(25)

by at least 20%, 28.4% underestimated by at least 20%. The situation is very different from railways, where traffic is overestimated in 90% of cases, on average by 105.6%. In the case of road projects, the main causes of error are the estimate of total traffic demand in the project demand and the evolution of land use.

Flyvberg et al. also conducted a similar study dealing with costs [4]. They compared the projected and observed costs of 258 transport projects, located in 20 countries on 5 continents.

They were carried out between 1910 and 1998 and represent an investment of $90 billion. The costs considered ex-ante were those at the time of the construction decision. 86% of the projects studied are subject to underestimation of costs. On average, this underestimation is 28%. However, it is lower for road projects, 20%. Flyvberg et al. point out that this

phenomenon is less pronounced in Europe and the United States, although still real. The authors put forward political causes; forecasters tend to lie to satisfy actors in the public sphere. Besides, there is a competition between project promoters. The latter have an interest in lying about costs to win tenders.

In both studies, the authors also note that these differences have not decreased over time.

This is a sign that past errors may not be sufficiently considered to adjust forecasts in future studies.

A similar, smaller study was conducted by Céréma, a French government department dealing with transportation issues. The study compares the results of ex-post LOTI

evaluations of 32 road projects in France between 1987 and 1998 [6].

The traffic measured at the time of the LOTI assessment is taken into account, i.e. 3 to 5 years after entry into service, a longer period than in the Flyvberg study. Contrary to the Flyvberg study, there are more underestimated than overestimated projects. In the Céréma sample, traffic was on average underestimated by 20%. The error was greater than 20% for

(26)

75% of the projects. As in the Flyvberg study, the main cause of error is the misestimation of traffic growth in the region.

The same statistical study was carried out by Céréma on costs. 84% of projects are underestimated in terms of costs. The average variance is an underestimate of 20%.

The Flyvberg and Céréma studies point out the existence of significant errors in the cost and traffic forecasts of road projects. Costs are almost systematically overestimated.

Errors in traffic demand can be either over-or underestimates; projects in the Céréma study, which are more recent than the Flyvberg and exclusively French, are more likely to be underestimated. Thus, there is still significant room for improvement in traffic and cost forecasts, specific tools could be useful.

As mentioned above, the development of the multi-criteria character of LOTI audits reinforces the importance of the study of wider economic effects. Like classic effects, the implementation of tools for their observation would be all the more justified if these effects were poorly taken into account in ex-post studies.

C. Margin for progress in taking into account the wider economic effects

It seems that the margin for progress concerning current practices is even greater than for classic effects.

According to the International Transport Forum's report "Improving the Practice of Transport Project Appraisal" [24], wider economic effects may include effects on

productivity, agglomeration of economic actors, competition, or the labor market. Tracking all the impacts of a project on the economy is much more difficult than measuring the direct benefits to the users of the infrastructure. Therefore the first approach is generally less present in socio-economic assessments. De Jong [8] cites the case of Norway, for example, where the

(27)

ex-post evaluation program only includes costs, traffic, road safety, and local air pollution.

According to Nicolaisen [17], ex-post evaluations generally focus on travel time savings, construction costs, and road safety.

For Bonnafous [5], indirect economic effects are much less analyzed in the LOTI ex- post assessments than in the ex-ante evaluations. The approaches are not systematic, and the authors of the studies generally confine themselves to a sparse selection of data. This is particularly damaging because the consideration of wider economic effects has gained importance. These have become even more crucial in responding to local opposition to transport projects, see also Bérion [2]. To solve this problem, he calls for the establishment of highway observatories.

More generally, the contribution of permanent observatories to the study of classic and wider economic effects could be reflected in an improvement in the quality of ex-post audits.

D. Quality of ex-post audits

For Worsley [24], Nicolaisen [17], and Meunier [14], little financial means and effort are today attributed to ex-post evaluations, and in particular to data collection, compared to ex- ante evaluations.

This echoes De Jong [8], who indicates that the practice of ex-post evaluation of transport infrastructure is systematic in few countries. France is among the exceptions, along with Norway, the United Kingdom, and the United States.

According to Céréma [6], the quality of LOTI assessments in France has improved over the last ten years. However, Céréma insists above all on better considering the instructions of the Sétra methodological guide, a more systematic comparison between observations and forecasts in the inquiry before the declaration of public utility, and a better definition of the

(28)

reference scenario. The content of the LOTI assessments in terms of data is not put forward as a major cause of this improvement, nor as a margin of improvement for future studies. The exception is the case of costs; the data presented are not broken down by item of expenditure, both for investment and for maintenance and operating costs. In this situation, it is very difficult to understand the causes of the differences between forecasts and observations.

However, Céréma suggests that this is not due to a lack of data in the possession of the concessionaires but to their willingness to keep their costs, which are sensitive data, secret.

Bonnafous also supports this hypothesis [5].

The literature already mentions several advantages and limitations specific to permanent observatories.

E. Advantages and limitations of permanent observatories

According to Bonnafous [5], permanent observatories are particularly useful for dealing with the short duration of certain phenomena (such as site effects), loss of data, or oversights on the part of the infrastructure and territorial actors. According to Worsley [24], a difficulty in the practice of ex-post and ex-ante evaluations is the long period between these two

analyses; it can be several decades. With a permanent observatory, it is easier to detect all the anticipations and reactions to the project as they emerge.

Bonnafous [5] and Bérion [2] insist on the fact that the description of wider economic effects by permanent observatories is particularly precise, much more so than what is usually found in LOTI assessments. Thus, according to Bérion et al [2], permanent observatories have helped to show the absence of automatic causality between the construction of infrastructure and territorial effects such as economic growth. They also analyze the actors' strategies in the face of the arrival of the new infrastructure.

(29)

Bonnafous [5] emphasizes a particular advantage of the observatory: it allows certain trends to be identified. Thanks to ongoing and regular data collection that began even before the infrastructure was put into service, it is possible to distinguish between trends that are due to the infrastructure and those that are exogenous. Some of the observed economic effects could have been attributed to the highways, but others to dynamics already in action before construction. On the contrary, Bérion et al. [2] indicate that, like any ex-post process, observatories have difficulty isolating "gross effects from net effects"; in other words, observations are sometimes insufficient to determine which effects are due to the infrastructure and which are exogenous.

Joignaux [11] warns of some of the limitations of permanent observatories. First, they are confronted with a problem of spatial scale. Observatories are generally concentrated on the territory close to the infrastructure, 20 to 30 kilometers on either side, whereas highways can have effects beyond the local scale. Moreover, each territory being very specific, the author qualifies the possibility of extrapolating the results of an observatory to predict the effects of other infrastructures.

F. Research problems selected for this thesis

The literature above gives context for drawing research questions.

As Quinet explains, both monetizable and non-monetizable effects have an important place in the practice of socio-economic evaluation in France. This is particularly visible in the elaboration of ex-post LOTI assessments with the combination of cost-benefit analysis and wider economic effects, which are now rarely monetized.

The Flyvberg and Céréma studies show that significant errors exist between ex-ante forecasts and ex-post observations for classic effects. These errors can lead to selection biases with serious consequences for the community and public finances.

(30)

Good quality ex-post evaluations can allow lessons to be learned from projects that have already been carried out and thus reduce these errors in ex-ante evaluations. The means implemented to carry out these evaluations are currently weak, even if France is one of the countries that systematically implement them through LOTI evaluations. Céréma notes a recent improvement in the quality of these evaluations. Even if it is not mentioned by Céréma, the improvement in data collection is one of the possible explanations for this improvement.

Permanent observatories have advantages for data collection over the usual ex-post evaluation situation. They allow for systematic data collection and avoid data loss. However, their effective contribution to improving the quality of the analyses of classic effects in LOTI assessments is not addressed in the literature. The first question that this master thesis

attempts to answer is to evaluate the contribution of permanent observatories to the study of the "classic" effects of highways in France, such as costs, traffic, or travel times. The aim is to assess whether, in comparison with the carrying out of a usual ex-post evaluation, a

permanent observatory allows a more complete or more accurate data collection, and whether this possible improvement translates into an improvement in the quality of the ex-post

assessments, and therefore of the entire socio-economic evaluation process. It is also necessary to weigh this possible contribution of permanent observatories against the cost of setting them up.

Wider economic effects, which are more difficult to measure than direct effects, have gained interest in recent decades, particularly for the local acceptance of infrastructure. They are, however, much less analyzed in the LOTI ex-post audits than in the ex-ante audits.

Permanent observatories could be particularly well suited to the study of these effects.

They allow the strategies of economic actors to be observed as the project progresses, or even certain ephemeral effects such as the effects of the construction site. The second question of this thesis is to evaluate the contribution of permanent observatories of highways to the study of wider economic effects. The objective is the same as for classic effects. This thesis aims to

(31)

distinguish the relative contribution of observatories for collecting data on extended economic effects, analyzing them, and producing more complete and accurate ex-post assessments.

G. Relevant Research Approaches

Given the existing literature on the subject, several approaches seem relevant to answer these two questions.

To judge the quality of socio-economic evaluations, the literature focuses mainly on analyzing the results. However, it also seems instructive to look at the "practical" aspect of the studies. The usefulness of a permanent observatory is perceived by the authors of the ex-post study at the time when its data are exploited. Conversely, in the absence of an observatory, the writers of a LOTI report can see the contribution that an observatory could have made. Thus, the concrete experience of carrying out a LOTI assessment, even if it does not benefit from an observatory, seems essential for judging the contribution of observatories to LOTI

assessments, and therefore to socio-economic studies.

Then, listing examples of concrete observations of permanent observatories seems essential to report on their contribution to socio-economic assessments. Some of these elements may be difficult to observe in the context of an ex-post assessment without observatories, which highlights the usefulness of permanent observatories. The study of reports dealing with specific observations from observatories, therefore, seems particularly relevant for judging the usefulness of observatories.

Finally, the literature makes little assessment of the quality of LOTI assessments. Céréma compares the differences between ex-ante forecasts and observations of the LOTI assessments but does not analyze, by type of effect, the quality of the ex-post analyses and therefore the relative contribution of any permanent observatories. A third approach can therefore consist of carrying out a statistical study of the quality of the LOTI assessments based on the reports

(32)

of the Conseil Général de l’Environnement et du Développement Durable (CGEDD), General Commission for the Environment and Sustainable Development. The reports dealing with assessments that do not benefit from an observatory could be compared with those that do not.

(33)

III. Methodology

The general principle of this thesis is first of all to apply each of these three approaches to the two research questions, i.e. the analysis of the contribution of permanent observatories to socio-economic evaluations concerning classic and wider economic effects. The objective of each approach is to gather evidence of the added value of observatories compared to the case of ex-post analyses without observatories. Within each sub-section of the thesis, the possible teachings of each of the three approaches are presented one after the other. Then, in the discussion chapter, these advantages will be set against the known data on the costs of observatories and their limitations, some of which already mentioned earlier in the literature review chapter.

A. General qualitative approach

As noted earlier, there are two major objectives in improving the transport infrastructure appraisal process. First, it limits project selection bias by providing a more accurate prediction of the effects of each project considered in the ex-ante evaluation. Second, it optimizes the use of existing infrastructure by providing corrective measures through improved ex-post evaluation. For each of these two aspects, it would have been instructive to be able to observe quantitatively the impact of the contribution of permanent observatories to socio-economic studies.

However, for the limitation of selection bias, there is little or no data presenting the different options at the time of a public investment decision and the reason for the choice made.

Moreover, it is very difficult to isolate the influence of ex-post evaluations carried out through an observatory on future public decisions from the influence of those that do not. Thus, it is not possible to quantify the improvement in the usefulness of projects made possible by the reduction of selection bias by observatories.

(34)

Similarly, the effects of the corrective measures allowed by the observatories on the infrastructure they observe could have been measured to quantify their usefulness in this regard.

Nevertheless, too little data were available to make a satisfactory analysis.

Thus, this thesis will not analyze quantitatively the usefulness of setting up permanent observatories but will focus on qualitatively assessing their contribution to socio-economic studies.

B. Experience in the realization of the A150 LOTI audit

The first approach used to highlight evidence of the contribution of permanent

observatories is the "practical" study of carrying out a LOTI assessment without a permanent observatory. I had the opportunity to take part in the first phase of the A150 highway LOTI assessment at the Setec International study office in Paris between August and December 2020. This highway is in the Normandy region, in the North-West of France, see its location Figure 2 and Figure 3. The LOTI assessment concerns the most recent section, built between 2013 and 2015 between the communes of Barentin and Ecalles-Alix, near Yvetot, and conceded to the highway company ALBEA.

The approach consists of listing the aspects of the study that did not give rise to any difficulty, those that would have benefited from the existence of a permanent observatory, and those that would have been problematic even with an observatory.

(35)

Figure 2: the A150 highway - regional scale

Figure 3: the A150 highway - local scale

(36)

This method has the advantage of providing elements "in the heat of the moment"

contrary to most sources in the literature. Nevertheless, at the time of writing this thesis, the LOTI assessment was only in its first part. Thus, the analysis of the wider economic effects was not at a very advanced stage. The chapter "Analysis of the wider economic effects" will mainly exploit this approach for the experience of conducting interviews with the actors in the territory.

It would also have been very instructive to experience the realization of a highway LOTI audit benefiting from a permanent observatory, but only the A150 LOTI audit was in progress at the time of this degree project. Other lessons could also have been drawn from previous LOTI assessments carried out by Setec International, but these were several years old and therefore do not allow to observe "live" the realization of an assessment.

C. Reports by Bérion and Varlet on the results of the A39 and A71 observatories

To list the types of results that can be observed by the observatories, the second approach is the analysis of the Bérion [1] and Varlet [22] reports. These two researchers worked respectively on the implementation of the A39 and A71 highways. These two highways are conceded to the private company APRR, Autoroutes Paris-Rhin-Rhône, which financed the setting up of an observatory. The A39 was built in 1994 and put into service in 1998. It links Dole to Bourg-en-Bresse in eastern France. Its observatory was set up between 1993 and 2004. The A71 is located between Bourges and Clermont-Ferrand, in the center of the country. It was built in 1985 and put into service in 1989. Its observatory operated from 1985 to 1997.

Bérion and Varlet present the main results produced by their observatories. This second approach, therefore, consists of listing these effects and questioning the possibility of

(37)

collecting data of equivalent quality without a permanent observatory. Bérion and Varlet focus on the analysis of wider economic effects. Therefore, this approach is only present in the “Analysis of wider economic effects” chapter.

It would have been useful to be able to analyze the reports of other permanent

observatories, notably those set up by the State Service of the Sétra in the 1980s, to compare them with those of the A71 and A39. However, because of the restrictions due to the Covid- 19 pandemic, reports in paper format were not accessible at the time of this thesis. Thus, only the reports of the A71 and A39, accessible on the Internet in digital format, could be

exploited.

D. CGEDD’s criticisms of LOTI assessments

The third approach concerns the evaluation of the final LOTI reports, i.e. those transmitted to the State services for approval. The General Commission for the Environment and Sustainable Development (CGEDD) assesses the quality of the LOTI reports produced by the concessionaires, or rather the engineering firms they call upon, and makes recommendations. In particular, the CGEDD points out which items have not been properly analyzed for each type of effect. Some of these reports, the most recent, are available online on the CGEDD website [7]. Within the framework of this thesis, a statistical study of the CGEDD's opinions on the quality they attribute to the LOTI assessments was carried out for each type of effect: costs, traffic, travel time, road safety, wider economic effects. The quality of LOTI assessments with an observatory is then compared with that of assessments without an observatory. This enables to highlight the contribution of permanent observatories to the realization of LOTI assessments, and thus to economic studies in general.

The study covers 22 CGEDD reports published between 2007 and 2020 and dealing with French highways. Four of them benefit from an observatory: the A39, A20, A39, and A77.

For each type of effect, the 22 LOTI audits are classified in a category according to the

(38)

CGEDD's assessment. These results are presented in the form of charts. Then, more details are given on the CGEDD's explanations, in particular on the role that the establishment of a permanent observatory has had or could have had. The list of highways in the sample and the results by highway are in the appendices.

Nevertheless, this study has its limits.

The CGEDD reports have been collected on the CGEDD website. Making files available online is recent; before 2007, most reports were only in paper form. Due to the Covid-19 pandemic restrictions in France, older paper reports could not be consulted. That is why the study only covers 4 projects that benefited from a permanent observatory. Thus, these results are not statistically significant, but they still lead to interesting thoughts.

Moreover, it is sometimes difficult to estimate the cause of poor-quality analysis. It can be due to a lack of data collection, but also to misuse of available data or methodological errors on the part of the editors of the LOTI report. Thus, it may be difficult to assess whether the implementation of an observatory could have improved certain aspects of certain LOTI assessments that did not benefit from it.

Another method used for this approach could have been to study the LOTI assessments themselves directly. However, it seems more relevant to rely on the observations of a specialized entity such as the CGEDD, which knows what the strengths and weaknesses of the audits can be and can sometimes explain them. Moreover, although public, the LOTI assessments are sometimes more difficult to reach than the CGEDD reports, which are themselves gathered on the CGEDD website for the most recent ones.

Thus, the three approaches used seem complementary for estimating the contribution of permanent highway observatories to socio-economic studies. The "practical" experience of the LOTI A150 assessments indicates obstacles faced without an observatory, the analysis of the Bérion and Varlet reports describes the types of results produced by the observatories, and the

(39)

analysis of the CGEDD reports focuses on the quality of the assessments carried out according to whether they benefit from an observatory.

(40)

IV. Analysis of classic effects

This chapter is divided into four subsections corresponding to the four main "classic"

elements of the socio-economic evaluation of a highway. These are costs, traffic, travel times, and road safety. In each subsection, the lessons learned from the realization of the A150 LOTI assessment are described, followed by the lessons learned from the study of the CGEDD reports.

A. Costs

In the case of A150, obtaining the observed costs was not difficult. The concessionary company (ALBEA) and the construction company were exchanging about the advancement of construction work and expenses. ALBEA had access to detailed costs and could give us data.

However, the cost analysis was not advanced enough at the time of the redaction of this report to evaluate the need for more data.

Figure 4 represents how costs analyses are judged by the CGEDD.

Figure 4: evaluation of LOTI assessments by the CGEDD - costs

0 project

9 4

9

0%

50%

100%

With a Permanent Observatory (4 projects) Without (18 projects)

% CGEDD reports

Quality of costs analysis

satisfactory quality

improvement needed: detailed analysis to explain diffenrences ex ante / ex post

(41)

Contrary to what could be expected, all projects benefiting from a permanent observatory have been criticized on the cost analysis. That is only the case for half of the other projects. Of course, with a small sample, this is not statistically representative.

When an improvement is needed, critics expressed by the CGEDD are almost always the same. Planned and actual global costs of projects are always indicated in the LOTI audit.

However, there is a lack of details; costs are not expressed by item of expenditure. This makes it very difficult to perform a comparison between ex-ante and ex-post. A major interest in carrying out ex-post analysis is supposed to explain why costs were different than planned; that is impossible to do so only with global figures.

Almost all projects are conceded, including all those with a permanent observatory. This could explain why the detail of cost is a problem. As this data is often sensitive; these companies may prefer to keep cost data secret, even if that means being criticized by CGEDD. This hypothesis is put forward by the Céréma in the synthesis about LOTI assessments [6]; the Céréma also regrets the lack of details in cost sections and accuses concession companies. In that case, permanent observatories are not helpful; that is not a problem of data collection but a matter of willingness on the part of concession companies. Thus, the fact that each of the 4 projects benefiting from a permanent observatory is criticized for cost analysis is not so surprising.

(42)

B.

Traffic

At first sight, getting A150 traffics was not a problem. ALBEA counts passages at the toll station, hour by hour. However, the type of information ALBEA gets depends on the direction of traffic. When a vehicle is bound for Rouen, it leaves the charged section of the highway at Barentin tollbooth. That is where the driver must pay. ALBEA is in charge of the Barentin tollbooths, so it has the entire information available, notably the type of vehicles and means of payment, see Table 2.

On the contrary, a vehicle leaving Rouen and going to the North-West does not leave the highway network at Barentin but further, at Ecalles-Alix or Le Havre for instance, see Figure 5. These tollbooths are not controlled by ALBEA but by another concessionary company, SAPN. Thus, ALBEA only gets information when the vehicle enters the highway network, not when it leaves it, not when the toll is paid. Consequently, ALBEA is informed of the traffic count but not the type of vehicle or the means of payment, see Table 3. As a result, ALBEA could transmit to Setec a database of flows from Barentin to Ecalles-Alix but not the proportion of heavy goods vehicles or the number of subscribed users.

(43)

Figure 5: A150 and A29 highway nodes

Station Name Direction Day Time Lane Number Type of Lane Vehicle Class EuroClass Payment method Barentin Exit 01/01/2019 0:00-0:59 S26 VCM 1 (light vehicle) 0 Highway pass

Barentin Exit 01/01/2019 1:00-1:59 S21 VACM

4 (heavy goods

vehicle) 5 Highway pass

Barentin Exit 01/01/2019 1:00-1:59 S21 VACM 1 (light vehicle) 0 Cash

Table 2: Extract from ALBEA database - vehicles going from the West to Barentin

Station name Direction Day Time Lane Number Type of Lane Vehicle Class EuroClass Payment method

Barentin Entrance 01/01/2019 0:00-0:59 E13 IND ? ? ?

Barentin Entrance 01/01/2019 0:00-0:59 E14 IND ? ? ?

Table 3: Extract from ALBEA database - vehicles going from Barentin to the West

Agreements and cooperation exist between concessionary companies for communicating data. However, they do not cover everything. The concession company SAPN

(44)

communicated to ALBEA the traffics flows from their tollbooths to Barentin tollbooth, with the distinction between light vehicles and heavy goods vehicles but they were daily flows, not hourly ones. Having hourly flows would have been useful for Setec to distinguish house-to- work flows (peak hours) from other travel purposes. Thus, a permanent observatory would have been useful to get more precise data. An agreement could have been made in advance between SAPN and the permanent observatory to share this data for instance.

It was also necessary to observe traffic on competing roads. Collecting data about the D6015 was particularly important because one of the main objectives of the A150 was to attract heavy good vehicles from the D6015 where they are more dangerous and harmful, see Figure 6. Traffic counts on D6015 were supplied by the Direction interdépartementale des routes Nord-Ouest (DIRNO), a county agency, see Figure 7. We had to deal with the same kind of problems as for the A150 traffics. Data could have been more complete; sometimes we had traffic counts without differentiation between directions.

Figure 6: competition between A150 and D6015

(45)

Figure 7: An example of the evolution of traffic over a day (source Département Seine-Maritime)

Traffic counts on the non-conceded section of A150 were more problematic. The results provided by the counting station were very odd, see Figure 8. The station had had a dysfunction over several years. A systematic data collection with a permanent observatory would have limited this issue. After a few months of extraction, the malfunction would have been observed.

It may have been possible to repair it or to find another way to measure traffic flows.

Figure 8: traffic counts of a dysfunctional station (source: public agency DIRNO)

(46)

Figure 9 describes the results of the CGEDD reports for traffic analyses.

Figure 9: evaluation of LOTI assessments by the CGEDD - traffic

Almost half of the traffic analyses are satisfactory. Six of them are strongly criticized by the CGEDD and none of them benefited from a permanent observatory. The CGEDD emphasizes their lack of data collection. For instance, the data used for the A66 assessment is only based on a one-day observation. The A19 data was one year old after commissioning, no later, even though the LOTI assessment was performed 5 years after commissioning. The A75 Clermont-Ferrand - Sévérac-le-Château assessment lacks origin-destination studies. For each of these six assessments, a permanent observatory could have significantly increased the quality and quantity of data available.

Three of the assessments without permanent observatory lack data from the ex-ante assessment. Ex-ante assessments are sometimes of low quality; in that case, it is impossible to perform a high-quality ex-post assessment since the principle of an ex-post assessment is to compare planned and observed effects. For these three assessments, it is more difficult to judge the usefulness of a permanent observatory. An observatory collects data from a few years before the construction of the highway. It is not a priori supposed to have a direct impact on the execution of ex-ante studies many years before the start of construction. However, being aware

2 projects 2 8

0 1

3 0

6

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%