Swedish Institute for Social Research (SOFI)
Stockholm University
WORKING PAPER 2/2020
THE IMPACT OF ATTENDING AN INDEPENDENT UPPER SECONDARY SCHOOL:
EVIDENCE FROM SWEDEN USING SCHOOL RANKING DATA
by
Karin Edmark & Lovisa Persson
1
The impact of attending an independent upper secondary school:
Evidence from Sweden using school ranking data
♦March 24 2020 Karin Edmark Lovisa Persson
Abstract
This paper provides a comprehensive study on how attending a Swedish Independent upper secondary school affects students’ academic and short-term post-secondary outcomes. Beyond having access to population registers that measure school attendance and student outcomes, we are able to control for student preferences for independent provision, as stated in school application forms. The results from a CEM/VAM approach suggest a positive independent school effect on: final GPA, test results in English and Swedish, the likelihood of graduating on time, and attending post-secondary education. However, we also find a larger discrepancy between the final grade and the standardized test result among the independent school students, in a way that accords with more lenient grading practices among independent schools. Results from a difference- in-difference analysis around admission thresholds yielded no additional insights, due to imprecise estimates.
Key words: Private provision, mixed markets, voucher school reform, upper secondary education.
JEL-Classification: H44, I21, I26, I28.
♦ This project has benefited from funding from the Swedish Research Council, Project number: 2014-01783. We are grateful for comments from seminar and conference participants at the Fifth Lisbon Research Workshop on Economics, Statistics and Econometrics of Education; the Swedish Institute for Social Research at Stockholm University (SOFI); the Research Institute of Industrial Economics (IFN); Maastricht University; Cambridge University; and GRIP seminar participants at Kristianstad University. We are particularly thankful for comments from Jan Sauermann, Anders Stenberg, Anna Sjögren, and Jonas Vlachos.
Stockholm University, IFAU, IFN and CESIfo; http://orcid.org/0000-0002-5629-5499; karin.edmark@sofi.su.se
Research Institute of Industrial Economics (IFN) and Kristianstad University; https://orcid.org/0000-0002-2988-6226;
Lovisa.Persson@ifn.se
1. Introduction
1Upper secondary school is the last preparatory step before advancing to higher academic studies, entering the labor market, or commencing advanced vocational training. The effectiveness of upper secondary school consequently determines the quality of the academic abilities supplied to universities, the quality of the vocational abilities supplied to the labor market, as well as individual labor market prospects in general.
A widely discussed proposal to increase effectiveness in education is to allow entry for alternative providers with diverse approaches to learning and educational management, and/or let providers compete for students, hopefully in the dimension of educational quality (Friedman, 1955; Le Grand, 1991;
Shleifer, 1998; and Hoxby, 2003). In the early 1990s, Sweden introduced a set of reforms that made entry with full voucher-funding possible for private providers of primary and secondary school. The result was a large expansion of the private school sector, in particular at the upper secondary school level (grades 10–12), where currently a quarter of all students attend a privately provided school; or as henceforth referred to, an independent school.2
We contribute to a growing international research literature on school vouchers by estimating the impact of attending a Swedish independent upper secondary school on a broad range of academic and short-run labor market outcomes. Our data consists of several merged official registers for the full population of students entering upper secondary school. The data enable us to control for student background characteristics in several dimensions. Most importantly, through our access to school application data, we are able to control for stated student preferences for different types of providers. Considering Sweden’s journey from an almost complete public monopoly towards a very liberal school system by international standards, the case of Sweden should be of interest to the wider research community.
The previous literature evaluating the effects of the Swedish school voucher system is by large focused on primary and lower secondary education (grades 1–9). One exception is Hinnerich and Vlachos (2017), who show that upper secondary independent schools, on average, grade standardized tests 0.14 standard deviations more leniently than public schools: They show that the added value of independent schools is on average positive when regular teacher-graded test results are analyzed, but that it turns negative if externally re-graded test results are analyzed instead.3 Böhlmark and Lindahl (2015) study the primary and lower secondary school level, and they find a small positive effect on students’ academic achievements from an increasing independent market share within the municipality.4
1 A snapshot of this project, which was made prior to generating the results presented in this manuscript, is available at https://osf.io/u8r43. After the registration of the snapshot, we encountered and corrected a few data errors, which have been corrected in this version. See the Appendix B, section B.5, for details.
2 We use independent when referring to the Swedish version, and voucher as a catch-all term for international versions.
3 Another forthcoming working paper, Edmark et al (2020) studies the impact of regional variation in the supply of independent schools over time.
4 Other studies, that use the same type of municipality level variation, have found either no or positive effects on student’s educational attainment, see Hennerdahl et al (2018), Sandström and Bergström (2005), Ahlin (2003), Björklund et al (2005).
Our study contributes to the previous literature in the following respects:
- First, we make use of information on stated preferences for independent vis-a-vis public schools as indicated by students’ rankings of schools on official application forms. By controlling for student school rankings we reduce the risk that the results are biased due to differences in preferences for – or aversions against – either type of school.
- Second, we use a set of diverse estimation methods to paint a more robust empirical picture. That is, we estimate the effects of independent school attendance using i) a combination of coarsened exact matching (CEM) and value-added models (VAMs); and ii) a regression discontinuity inspired difference-in-differences analysis (RD/DID) around admission thresholds.
- Third, our analysis builds on data covering the entire student population, whereas Hinnerich and Vlachos (2017) are restricted to a subsample of roughly 10 percent of the student population for which there is information on externally regraded tests.
Our paper shares similarities with Kortelainen and Manninen (2019), who estimate a private school effect in Helsinki, Finland, using both RD around admissions thresholds and an added value approach. They find that the private school effect on matriculation exam scores is marginally positive but statistically insignificant. Unlike Kortelainen and Manninen (2019), we study a nationwide population of students, and we also have access to a broader range of student background variables. The Swedish case provides an interesting comparison to the Finnish case; the independent school sector in Sweden is larger, while at the same time, for-profits schools are allowed.
In general, the empirical evidence on the effects of voucher and charter schools on educational attainment is fairly inconclusive.5 Several studies on U.S. data have found positive educational effects from attending charter6 schools that adhere to the No Excuses approach (Dobbie and Fryer, 2019; Angrist et. al., 2013;
Dobbie and Fryer, 2013; and Abdulkadiroglu et. al., 2011). Dobbie and Fryer (2019) also find positive effects on four-year college enrolment, and Angrist et. al. (2016) find that charter high schools in Boston (where many adopt the No Excuses approach) boost college preparedness. In a meta-study, Chabrier et.
al. (2016) suggest that the “No-Excuses-effect” is driven by low performing fallback public schools in urban areas and intensive tutoring programs. Hahn et. al. (2018) show that high school students in private schools outperform high school students in public schools using data from Seoul, South Korea. On the other hand, Abdulkadiroglu et. al. (2018) suggest that participation in the Louisiana Scholarship (voucher) Program lowered student achievements. Studying the case of Chile, where a nationwide voucher system was implemented in 1981, Hsieh and Urquiola (2006) find no effects on educational achievement.
5 The literature review provided in our paper is by no means exhaustive; we focus on more recent studies, or studies that are more relevant to our paper in terms of method or content relating to Sweden. We refer to e.g. Epple et. al. (2015) and Epple et. al.
(2017) for reviews of earlier studies.
6 In the context of the U.S. educational system, charter schools are usually schools that operate within the public school system.
They are fully financed by public funds, and they are allowed to establish their own curriculum. However, they are usually more regulated than voucher schools, who are private schools that can be partly financed by a voucher given to all, or only some, families, depending on socio-economic status. Vouchers and charters share several core elements; student choice, public funding but private provision, curricular and organizational variety. Both voucher and charter schools are interesting comparison points for Swedish independent schools.
The results from our conditional-on-observables analysis using CEM/VAM suggest that attending an independent instead of a public upper secondary school has positive average effects on: students’ final GPA, standardized test scores in English and Swedish, and also the likelihood of graduating on time.
Analyses on subsamples suggest that the positive effects on final GPA are present in all parts of the ability distribution, and for students with varying socio-economic background. The positive effects on getting high test grades in English and Swedish are more pronounced in the upper part of the ability distribution, and the positive effect on graduating on time is more pronounced in the lower part. This is a reasonable pattern given that students with high abilities are on the margin of getting high grades, and students with low abilities are on the margin of graduating. Attending an independent school also has a positive effect on the probability of attending higher studies, including university studies, one year after graduation. Our results using CEM/VAM are robust to the use of different sampling and matching approaches, to bias-correction as suggested by Oster (2019), and to multiple hypothesis correction of p- values. However, our results from the RD/DID-analysis are overall too imprecise to be informative.
Our conditional-on-observables results mirror earlier results in Hinnerich and Vlachos (2017) who find a positive effect of independent school attendance on teacher assessed achievements. However, because of the potentially more generous grading standards in independent schools, documented in the same study, we cannot draw sharp conclusions about the actual educational added value of independent schools in Sweden, at least not based solely on our empirical results. Furthermore, we too find indications of more generous grading standards in independent schools. When we compare the standardized test grades with the final grades students get on the corresponding course, we find that students in independent schools are more likely to be “up-graded” on the course, compared to the test, but no more likely to be “down-graded.
In the conclusions to this paper, we discuss the results in more detail.
2. Institutional overview Swedish upper secondary education
7Swedish students enter a 3-year long upper secondary education at age sixteen, after ten years of compulsory schooling. Upper secondary school is divided into six academic and twelve vocational tracks, and there is a 1-2 year long preparatory track for students whose grades do not qualify them to enter directly into any of the regular tracks. Although Swedish upper secondary school (gymnasium) is completely voluntary, virtually everybody – 99 percent – commences upper secondary studies.89 Upper secondary education can be provided either by the local governments, the municipalities, or by private entities; independent schools. Publicly (or municipally) provided and independently provided schools are both fully funded via school vouchers supplied by the municipalities, whose primary source of financing is the local income tax. Additional tuition fees are not allowed.
7 For a more detailed review, see Appendix A.
8 This number relates to the cohort finishing compulsory school in 2011, see Statistics Sweden (2017a).
9 The graduation rate is however relatively low at 69 percent, compared to 85 percent in the US; 98 percent in Japan; and the OECD average at 86 percent (OECD, 2020). The low graduation rate is troubling, given that an upper secondary school diploma is correlated with significantly improved labor market prospects. In 2016, the unemployment rate among Swedish individuals aged 20–64 with an upper secondary degree was less than half of that of individuals who had not finished upper secondary education, even when restricting the comparison to individuals without post-secondary degrees (Statistics Sweden, 2017b).
As shown in Table 1, independent schools are on average smaller than their public counterparts, and they account for around one third of the market share in terms of school units. The academic track accounts for the larger share of students in both the independent and public schools and the vocational track share is roughly equal in both types of schools, but preparatory tracks are much less common in independent schools.10 The fact that the preparatory track share is so small in the independent sector is one motivation for why we exclude preparatory tracks from our analysis. Finally, the number of students per teacher is slightly higher in the independent schools, even after adjusting for differences in academic, vocational and preparatory track shares.
TABLE 1.SCHOOL CHARACTERISTICS – SCHOOL YEAR 2013/14
No. school unitsa
School size (No. students)
Academic tracks (student shares)
Vocational tracks (student shares)
Preparatory tracks (student shares)
Students per teacher, adjustedb
Independent 458 184 0.622 0.340 0.037 11.831
Public 882 273 0.562 0.320 0.118 11.368
a The definition of school units in the national School register changed in 2013. The new code is based on the division of headmaster responsibilities, rather than the physical school units. This has resulted in a large increase in administrative school units for the municipal schools: from 502 in school year 2011/12, to 766 in 2012/13 (after some schools had adopted the new system) and 882 in 2013/14 (when the new system was fully adopted. The number of independent schools was much less affected, and its numbers rather decreased over time; from 499 in 2011 to 484 in 2012 and 458 in 2013.
b The 0.5 percent top and bottom observations were excluded in order to eliminate the influence of extreme outliers, and the data was adjusted to account for the shares of students attending Academic, Vocational and Preparatory tracks, as these tend to have different student/teacher ratios. The raw data show a similar, but stronger, pattern of higher student/teacher ratios in independent schools.
Entering upper secondary education is associated with making two choices: a choice of school and a choice of educational track. Under the current system, students choose simultaneously the school and track as one package, and are allowed to rank varying track and school combinations in their application.11 Admission to a track and school combination is based on the grade sum, which is calculated as the sum of the grade credits of the’ 16 highest graded subjects from lower secondary school (GPS9). In case of ties – i.e. several students with the same grade sum as the admission threshold – the school provider can choose from a list of allowed criteria, such as: specific subject grades, the rank of the choice, or chance. Students can choose from all independent schools in the country, and from the public schools in their home admission region. They may also apply to public schools outside of their region, but home students are then given priority in the admission process.12
The current regulatory framework for Swedish independent schools stems from a set of reforms implemented in the early 1990s, which greatly expanded the possibilities for independent agents to start schools and obtain full public funding.13 As can be seen in Figure 1, the expansion of the independent market share following the school reforms was rapid, especially in the first decade of the 2000s, and a significant share of students opted for independent schools during our studied period of 2009–2013.
10 This might be due to the fact that independent schools have only been allowed to offer these tracks since 2006.
11 The first choice can be school A and a social science track, while the second choice could be school A and natural science track; or alternatively school B and a social science track.
12 See Appendix A for more detailed information on admission rules that is of relevance for our RD/DID-analysis.
13 The reforms are outlined in Propositions 1991/92:95 and 1992/93:23.
Specifically, the share of students attending an independent upper secondary school increased from 1.7 percent in 1992 to the highest share measured as of yet in 2013, at 28 percent.
FIGURE 1. INDEPENDENT UPPER SECONDARY SCHOOL MARKET SHARE
Source: Swedish National Agency for Education.
In 2013, the John Bauer group, which provided education to around 9000 students at the time, went bankrupt, which could be one of several explanations behind the stagnating independent share in the last five years.1415 The bankruptcy sparked a debate about financial misconduct, and contributed to increased financial monitoring of independent providers.16 Public oversight of the independent school sector had by then already developed from what was initially a relatively rudimentary system, to a system of more comprehensive and frequent monitoring. Since 2008, the Swedish Schools Inspectorate is responsible for the authorization of independent schools and for overseeing both public and independent schools.The Swedish Schools Inspectorate can close independent schools if severe violations are detected, but they are not authorized to close public schools.1718
The independent school reforms provided Sweden with a relatively liberal school system by international standards. For example, independent schools are allowed to be organized as for-profit organizations, such as the abovementioned JB-group. In fact, in 2013 – the year of the last cohort in our data – 85 percent of
14 In a robustness analysis that is reported in Appendix C we find that excluding students affected by the JB bankruptcy has no qualitative impact on the results.
15 See e.g. Sebhatu and Wennberg (2017) for an in-depth analysis of the JB-group.
16 In 2014, the regular school inspections expanded to include also oversight of the financial situation of school providers.
Starting from 2019 there are also stronger formal requirements on relevant experience and economic capability for private school providers, see https://www.skolinspektionen.se/sv/Tillstandsprovning/agar--och-ledningsprovning/.
17 The Swedish School Inspection can instead temporarily take over the running of a municipal school. A proposal to expand the possibility to close also municipal schools is currently being investigated.
https://www.regeringen.se/pressmeddelanden/2019/11/okade-mojligheter-for-skolinspektionen-att-stanga-skolor-utreds/
18 Chapter 3 in Angelov and Edmark (2016) describes the authorization of the independent schools in the early days of the reform, as well as the later developments. See also the National Agency for Education (Skolverket, 2004, pp. 21–22) for information on the government oversight.
upper secondary schools belonged to corporations. The remaining 15 percent were primarily organized as foundations or non-profit associations.
The government regulation concerning teaching- and instruction-related activities applies to independent and public providers alike: both are obliged to follow the same curriculum; meet the same educational goals; and use the same grading system. At the same time, school providers (or principals) – in the public as well as in the independent sector – have significant authority when making decisions that concern hiring, wage setting, allocation of resources within the school, and allocation of (a minimum total amount of) instruction time between courses and over the school year. In all essence, the same degrees of freedom offered by law thus applies to both public and independent providers, except for a few provisions regarding independent providers, including the possibility to organize as for-profit and to have a religious profile. However, both independent and public schools can profile themselves according to their offering of educational tracks, optional courses, and voluntary special instruction in sports, arts, or in other academic subjects.19
3. Data
The bottom line data set contains information on all individuals in Sweden that applied to upper secondary schools in 2009-2013, in the following referred to as the “application register”.20 This data set has been merged with a number of different population-wide registers held by Statistics Sweden (SCB), such as the upper secondary school attendance register, and registers containing information on students’
graduation status, grades and test results, parental and student background characteristics, early work life, and post-secondary education. The sections below contain a presentation of sample restrictions, and descriptions of the variables used in the analysis.
3.1 Sample restrictions
Our sample restrictions are primarily motivated by the aim to obtain a more homogenous and comparable sample of independent and public school students. In some cases, however, observations have to be dropped because of difficulty of interpretation or suspected errors.
Starting with the original data set21 , we limit the data set to students who start upper secondary education immediately after completing lower secondary school, at the age of 16, and who are qualified to enter a regular track in upper secondary school (i.e. who do not have to attend a preparatory track). These two restrictions reduce the sample size from 575,276 to 447,388 individuals. We then continue to make restrictions i.–v. below, based on information in the application register, and we thereby shrink the sample size to 296,890 individual observations. That is, we drop students who:
19 For more detailed information about regulatory differences, see Section A1.1 in Appendix A.
20 2009 is the first year for which we observe the schools that students applied to – prior years of data show only listed track choices.
21 The “original data set” refers to the sample size (575,276) after observations with missing observations on the following essential variables have been dropped: school ownership, educational track, and personal ID.
i. Have ranked only one school and track combination in the upper secondary application, because we want to be able to study students who have ranked both an independent and a public school.
ii. Are recorded as being admitted to more than one ranked preference, because for these students we cannot be sure which admission information is correct.
iii. Have applied to several different admission regions, or to independent schools with separate admission processes, because for these students there are several lists of ranked schools.
iv. Have applied to tracks for which they are unqualified, e.g. due to fail grades in certain subjects.
v. Were not admitted to their 1st or 2nd ranked alternative. Most students (approximately 80 percent of the applicants in the raw application data) are admitted to choice 1 or 2.
The fact that we can control for preferences for independent and public schools, as reflected in the application registers, is one of our most important contributions to the existing literature that, like us, relies on conditional-on-observables approaches. Our preferred strategy is to restrict the sample to students who have ranked both types of schools among their top two choices, which leaves us with a sample of 72 745 observations. When restricting preferences, we can be fairly certain that none of the students in the sample have any direct aversions, either rooted in political preferences or in experiences, to either of the school types, thus closing one selection channel.22
In addition to the observational sample above, we generate a separate sample to be used for an RD/DID- based analysis. This sample contains observations that are located around all competitive admission thresholds to independent and public schools. In other words, the sample will include cases where a student was marginally accepted (or not) to an independent instead of a public school, or to a public instead of an independent school. We therefore make an additional set of sample restrictions (please note, when reading what follows, that admission group is defined as combinations of 𝑠𝑐ℎ𝑜𝑜𝑙 × 𝑡𝑟𝑎𝑐𝑘 × 𝑦𝑒𝑎𝑟 based on the student’s top ranked alternative). The restrictions vi.–x., listed below, shrink the sample size to 12,060 individual observations. We keep only:
vi. Competitive admission groups, where the grade sum was actually binding in the sense that not all who applied were accepted.
vii. Admission groups that contain observations close to the admission threshold on both sides of the threshold. “Close” is defined as a ±10 grade sum unit interval; recall that the maximum grade sum is 320. (For regression specifications using a smaller data window of ±5 around the thresholds, we restrict the sample to admission groups with observations on both sides within this interval.)
viii. Individuals whose admission threshold for the lower ranked alternative was lower than that of the higher ranked alternative, and whose grades were higher or equal to than the threshold of the lower ranked alternative, so that the lower ranked school is a realistic fallback option.
ix. Individuals who have listed the same educational track for both the higher and lower ranked preference, so that we can isolate the independent/public-effect from potential track-effects.23 x. Individuals who apply to only non-artistic tracks, since admission to artistic tracks is not solely
based on the grade sum, but also on practical admission tests.
22 However, we will also, as a robustness check, perform estimations on the full observational sample and include dummy variables to control for the ranking of school types. These results are shown in Appendix C.
23 We will also provide results after we relax this restriction.
In Table 2, we present an overview of the different samples and their sizes. We also show the distribution within the samples with respect to the exact rankings of the different types of schools. A more detailed exposé over sample restrictions can be found in Appendix B, Section B1.
TABLE 2.SAMPLES OVERVIEW
Observational samples Observations RD/DID-samples Observations
Full sample 296 890 Full sample 12 060
Main sample (preference restriction) 72 745 Main sample 4 408
Independent (1) / Public (2) 34 320 Independent (1) / Public (2) 2 399 Public (1) / Independent (2) 38 425 Public (1) / Independent (2) 2 009
Supplementary samples 224 145 Supplementary samples 7 652
Independent (1) / Independent (2) 34 911 Independent (1) / Independent (2) 1 401 Public (1) / Public (2) 189 234 Public (1) / Public (2) 6 251
Note: (1) and (2) denotes the ranking of the school type.
3.2 Student background variables and other covariates
The richness of Swedish register data allows us to control for a comprehensive list of covariates on student background characteristics when estimating the effect of attending an independent school. In Table 3 we display the full list, and the averages values, of covariates for students attending independent and public schools respectively (columns 1–2).24 The table also shows p-values (column 3) and normalized differences (Imbens and Rubin, 2015) (column 4). The sample used is the main observational sample that only includes students who have listed a mix of independent and public schools among the top two choices.
Student background characteristics in independent and public schools come across as remarkably similar according to the averages in Table 3. The (normalized) differences are less than 2 percent of the pooled standard deviations for 16 out of 20 variables. The strong balance in covariates between the samples is partly a result of restricting the sample to students who have listed both independent and public schools among the top two choices. As can be seen in Table B.6 in Appendix B, the selection into different types of schools is more pronounced when this restriction is not imposed.
Nevertheless, according to Table 3, independent school students are more likely to live in metropolitan municipalities, whereas students attending a public school are more likely to live in urban municipalities.25 Independent school students are also somewhat more likely to have attended an independent school in grade 9. Since school choices made in lower level education are potentially important for later educational choices, we will control for 9th grade school in the empirical analysis.
24 See also Table B.4 in Appendix B for basic summary statistics for all covariates.
25 An urban municipality is where a majority of the population lives in urban areas, and in a rural municipality the opposite is the case. A metropolitan municipality is where at least 80 percent of the population lives in an urban area, and where the combined regional (metropolitan) population amounts to at least 500,000.
Additional controls for geography are added through the inclusion of upper secondary school municipality dummies.
TABLE 3.STUDENT BACKGROUND CHARACTERISTICS IN INDEPENDENT/PUBLIC SCHOOLS
Independent Public P-value Normalized diff.
(1) (2) (3) (4)
Household disposable income 246 095 243 240 0.177 0.010
One parent business income 0.145 0.142 0.237 0.009
One parent unemployed 0.186 0.179 0.019 0.017
One parent post-sec educ 0.550 0.554 0.276 -0.008
Both parents born in Sweden 0.730 0.729 0.739 0.002
One parent born in Sweden 0.124 0.123 0.545 0.004
No parent born in West 0.082 0.082 0.729 0.003
Born in Sweden 0.946 0.944 0.149 0.011
Born in West 0.024 0.026 0.144 -0.011
Born in non-West 0.030 0.030 0.554 -0.004
Female 0.514 0.519 0.237 -0.009
Independent9 0.186 0.172 0.000 0.037
GPS9 226.7 226.9 0.417 -0.006
High MA Test9 0.117 0.118 0.661 -0.003
High SW Test9 0.089 0.087 0.317 0.007
High EN Test9 0.224 0.216 0.015 0.018
Metropolitan municipality 0.453 0.411 0.000 0.086
Urban municipality 0.435 0.479 0.000 -0.087
Rural municipality 0.111 0.111 0.803 0.002
Regional independent share 0.254 0.248 0.000 0.078
Observations 35,098 37,647 72,745 72,745
Household income is represented per individual and in year 2016 monetary value. P-values refer to the raw differences. Missing values are replaced with imputed pooled averages. The normalized difference between samples 1 and 2 for covariate 𝑋 is calculated as (𝑋̅1− 𝑋̅2)/√(𝑆12+ 𝑆22)/2 (Imbens and Rubin, 2015).
3.3 Outcome variables
The cohorts in our data enter upper secondary education in 2009–2013 and are thus expected to graduate in 2012–2016. As 2016 is the last year recorded in our data, all post-graduation outcomes will be short- term in nature. While we are restricted to short-term outcomes, we have aimed to make use of the detailed register data to capture a broad range of the options available to students after upper secondary school.
Our outcome variables thus include not only university/college educations, but also other post-secondary educations and labor income. The outcome variables are listed and categorized into three different groups in Table 4.
The outcomes in panel A are measured during, or at the end of, upper secondary school. This includes an indicator for switching school type during upper secondary school – from an independent school to a public school, or the reverse; the final 12th grade GPA, measured as the percentile rank by year among all graduating students; and the outcome “graduate on time”, i.e. after three years in upper secondary school.
Students who have failed to graduate on time may either leave school without a degree (they will instead
obtain a transcript of the completed courses), or complete upper secondary education by remaining in the incumbent school (or by switching to an adult education program).26 We also define a dummy variable for remaining in upper secondary school for a 7th term, i.e. after the expected graduation.
TABLE 4. OUTCOME VARIABLES IN INDEPENDENT/PUBLIC SCHOOLS
Independent Public P-value Normalized diff.
(1) (2) (3) (4)
Panel A. Graduation and grades
Switch independent/public 0.088 0.061 0.000 0.103
Pctile GPA12 57.143 53.217 0.000 0.140
Graduate on time 0.823 0.808 0.000 0.039
7th term 0.093 0.103 0.000 -0.034
Panel B. Standardized tests Mathematics
High test grade 0.059 0.050 0.000 0.038
Pass test grade 0.763 0.772 0.003 -0.021
Test grade>Course grade 0.017 0.013 0.000 0.032
Test grade<Course grade 0.301 0.271 0.000 0.066
Swedish
High test grade 0.095 0.069 0.000 0.092
Pass test grade 0.948 0.944 0.000 0.019
Test grade>Course grade 0.098 0.099 0.614 -0.004
Test grade<Course grade 0.305 0.285 0.000 0.045
English
High test grade 0.128 0.103 0.000 0.076
Pass test grade 0.978 0.977 0.812 0.002
Test grade>Course grade 0.108 0.111 0.195 -0.010
Test grade<Course grade 0.183 0.148 0.000 0.093
Panel C. Post-graduationa
Study 0.383 0.368 0.000 0.032
Study no-prepb 0.312 0.295 0.000 0.037
UC≥15 0.152 0.144 0.011 0.022
Work≥50% 0.259 0.279 0.000 -0.045
The normalized difference for covariate X is calculated as (𝑋̅1− 𝑋̅2)/√(𝑆12+ 𝑆22)/2 (Imbens and Rubin, 2015).
a Post-graduation outcomes are measured in the year following the graduation year, i.e. 4 years after entering upper secondary school.
b The pre-registered snapshot version of this table contained an error in this variable. This has been corrected, which is why the variable content for this variable differs from the same table in the snapshot.
In panel B we collect outcomes that are based on standardized tests taken in Mathematics, Swedish, and English throughout upper secondary school.27 Our data lacks information on the exact test scores, but we
26 Students have a general right to complete their upper secondary education, and have the right to retake or retest a failed course.
The right to continue upper secondary education however transpires if the student is absent without valid reason for more than a month in a row.
do have information on the grades awarded on the tests. Based on this, we generate one outcome variable indicating whether the student was awarded a “high” grade or not, and one indicating whether the student was awarded a “pass” grade (all grades above fail) or not. In addition, we observe the final course grade corresponding to each test. The standardized tests are supposed to be a support and guide for the teachers’
assessments of students, but they are not strict determinants of the final course grades. In order to study how the test and course grades correspond to each other, we construct two dummy variables indicating if the test grade is higher or lower, respectively, than the course grade.
The timing and the number of tests taken varies across the educational tracks, and students in some tracks are tested in several sub courses in the same subject, resulting in multiple test observations per student.
We will in our baseline estimations run the regressions on the student level averages for each outcome, such that each student gets the same weight.28
Panel C lists our post-graduation outcomes. We measure post-secondary school studies in the fall and create two indicator variables: the first takes on value 1 for all types of post-secondary studies, including both tertiary education (advanced and vocational training), and “complementary” types of studies such as adult complementary education, active labor market educational programs and Swedish for immigrants (see complete list in section B4.3 in Appendix B). The second dummy variable excludes the
“complementary” types of studies. We also capture university studies separately by creating a dummy variable that takes on value 1 for taking university credits (UC) equivalent to 50 percent or more of a term of fulltime studies (≥15 UC). Finally, we measure labor market earnings in the form of a dummy variable for earning a “substantial amount” of labor income. We follow Forslund et. al. (2017), and define this amount as yearly earnings of at least half of the median annual work income among 45-year-olds.29
We recognize that studying post-graduation outcomes in the same year as graduation is probably premature, since many students choose to take a sabbatical year to work or study abroad, and we will therefore show results when measuring outcomes one year after graduation in our main results tables (4 years after entering upper secondary school). This in effect means that we are excluding the 2013 cohort from the analysis of post-graduation outcomes.30
Table 4 shows outcome variable averages for students attending independent and public schools respectively, as well as the normalized differences and p-values. Similar to Table 3, we use the main observational sample, which is restricted to students who have applied to a combination of independent and public schools as the two top choices. According to the raw differences in Table 4, independent school students are: more likely to switch school type, to have a higher GPA12, and somewhat more likely to graduate on time. In all test subjects, students in independent schools are more likely to receive the highest grade. The test grade is also commonly found to be more loosely connected to the final course
27 In order to account for the fact that the exact timing and number of tests take varies across tracks, and sometimes even across schools within a track, we include fixed effects for the timing in terms of the year, school grade and term and for the course tested. These are also relevant to include due to the fact that the grading system changed during the time studied, see section B4 of Appendix B for details.
28 We have also, as a robustness test, estimated the regressions when using each test as the level of observation. The results are overall very similar.
29 According to Forslund et al (2017), this corresponds roughly to six months’ worth of wages for a full time employed janitor in the municipal (public) sector. A “substantial amount” is redefined as a quarter of the median income among 45-year olds, when studying outcomes in the graduation year, as the students were still in upper secondary education approximately half of that year.
30 We also show estimates when measuring post-graduation outcomes in the same year as graduation, see Appendix C.
grade in independent schools. In particular, the course grade given is more often higher than the grade awarded on the standardized test. When looking at post-graduation outcomes, we see that the largest difference, in favor of students in public schools, is found in the propensity to work at least 50 percent one year after graduation. Furthermore, students in independent schools are somewhat more likely to register in any kind of studies in the year after graduation.
4. Empirical methods and results 4.1 Overview of the empirical estimations
We estimate the effect of attending an independent school using both conditional-on-observables approaches and quasi-experimental approaches. That is, we run added-value regressions (VAMs) that are combined with coarsened exact matching (CEM), as a way of enforcing common support, but we also estimate a regression discontinuity inspired difference-in-difference analysis (RD/DID) around admission thresholds. We argue that these different strategies, by making use of somewhat different sources of identifying variation and by relying on partly different assumptions, together provide a more comprehensive analysis than each analysis would yield on its own. Before going into the details, we write down the basic regression equation for our analysis problem as:
𝑦𝑖 = 𝛼 + 𝛽𝐼𝑁𝐷𝑖+ 𝑢𝑖 (1),
where 𝑦𝑖 denotes some outcome for upper secondary student i; 𝛼 is an intercept; and 𝐼𝑁𝐷𝑖 is a dummy variable for whether or not the students attended an upper secondary independent school instead of a public school as measured at the start of upper secondary education; and 𝑢𝑖 is the error term.
If independent and public students were comparable in all aspects apart from what type of school they attended, the 𝛽-coefficient from equation (1) would capture the average causal effect of attending an independent – instead of a public – upper secondary school. In practice, however, independent and public students may very well differ systematically in ways that are correlated with the outcomes studied, and that may or may not be observable to the researcher.
As was explained in the introduction we deal with the selection problem by restricting the data sample to students who have expressed a relatively strong preference for both independent and public schools, more precisely, those that have listed a combination of the two school types as the top two choices in the upper secondary school applications. We then address potential remaining student selection by either conditioning on observable characteristics (combining CEM and VAM), or by further restricting the sample to students who can (more or less) plausibly be assumed to be comparable in an RD/DID-type analysis. The below sections present these respective approaches and their results in order.
4.2 Combination of CEM and VAM
4.2.1. Overview of CEM/VAM
Since admission to Swedish upper secondary school is not based on lotteries31, we follow Dobbie and Fryer (2019), and Hinnerich and Vlachos (2017), and implement an empirical approach that combines CEM (coarsened exact matching) and VAMs (value-added-models). Abdulkadiroglu et. al. (2011) show that a conditional-on-observables approach yields test score estimates for oversubscribed Boston charter schools that are similar to the estimates obtained when leveraging charter school lotteries.32 In a follow- up,study, Angrist et. al. (2017) conclude that the bias contained in VAMs is, in the charter school context, moderate and statistically significant. Importantly, they suggest that the bias is small enough to render observational estimates useful from a policy perspective. We take this as suggesting that VAMs may yield policy relevant estimates also in the present Swedish case; in particular as we have access to a broad set of student background variables including prior achievement and stated preferences for different types of schools.
We use CEM in order to obtain common support with respect to combinations of a set of background variables that can be considered particularly important for school choice and subsequent outcomes. We force exact matching on the following variables: gender, parents’ country of birth (three dummies), GPS9 quintile, and, depending on the specification, either the county33 where the student attends lower secondary school, or the school the student attended in 9th grade. After carrying out the matching procedure, we keep only the cells that contain both independent- and public school students.34
The two samples generated, one matching on upper secondary school county, and the other on 9th grade school, are then alternately used for estimating VAMs; regression models where student background variables and student’s prior academic achievements are controlled for in a flexible manner (see the table notes to Tables 5.A–C for the exact covariate specification). The empirical model, which builds on equation (1), is displayed below:
𝑦𝑖𝑡𝑚𝑐𝑝= 𝛼𝑡+ 𝛽𝐼𝑁𝐷𝑖+ 𝛿𝑨𝒊𝒕−𝟏+ 𝜑𝑿𝑖+ 𝑢𝑚+ 𝑢𝑐 + 𝑢𝑝+ 𝑢𝑖𝑡𝑚𝑐𝑝 (2)
Similarly to equation (1), 𝑦𝑖𝑡𝑚𝑐𝑝 denotes some outcome variable for student i, at time t (but note that the time when the outcome variable is measured varies), in upper secondary school municipality m, who attended 9th grade school c, and who is enrolled in track p. Furthermore, 𝛼𝑡 denotes time (or cohort) fixed effects; 𝐼𝑁𝐷𝑖 is a dummy variable for attending private school (measured in October of the first year of upper secondary school); 𝑨𝒊𝒕−𝟏 denotes prior academic achievement; 𝑿𝑖 denotes the remaining set of
31 Lotteries may only be used in cases where several students with equal grade sum compete for the last available slot.
32 Studying the effectiveness of charter schools in New York City, Dobbie and Fryer (2013) also show that observational estimates and lottery estimates can be qualitatively similar, although in their case the observational estimates are somewhat smaller in size. Deming (2014) present observational estimates that are similar to lottery estimates, using data on charter school lotteries in Charlotte-Mecklenburg, North Carolina.
33 There are 21 counties in Sweden, sometimes referred to as “regions”.
34 The variables used for exact matching have been chosen to align ourselves with the previous literature, in particular with Hinnerich and Vlachos (2017), but also Dobbie and Fryer (2019). When we match on 9th grade school, we also add cohort dummies, since 9th grade school IDs cannot always be correctly linked over time.
student background characteristics that are potentially correlated with both independent school attendance and the outcome variable. Upper secondary school municipality fixed effects are included in 𝑢𝑚, 9th grade school fixed effects are included in 𝑢𝑐, track fixed effects are included in 𝑢𝑝, and 𝑢𝑖𝑡𝑚𝑐𝑝 is the error term.
Under the assumption that the included covariates and fixed effects successfully capture all systematic background differences between independent and public school students that remain in the restricted and matched samples and that are correlated with the outcome variable, the 𝛽-coefficient in equation (2) corresponds to the average treatment effect (ATE) of attending an independent school, for the sample population.
This conditional independence assumption cannot be tested. We will however construct bounds for the 𝛽- coefficient under different assumptions on the relation between unobserved and observed selection.35 This analysis will tell us: i) what are the bounds of the 𝛽-coefficient if we assume that the selection on unobserved student characteristics is the same, half of, or double, that of the observed selection; and ii) how large would the selection on unobserved variables need to be, as a share of the observed selection, for the true 𝛽-coefficient to be zero.
4.2.2. CEM/VAM results
The VAMs are estimated on three different samples, resulting in three 3-column tables, one table for each outcome group of Table 4. Results for outcomes in group A, “Graduation and grades”, are shown in Table 5.A. Column 1 shows the results for the most restricted sample, where we enforce both the preference restriction (having applied to a combination of independent and public schools for the top two options), and common support with respect to 9th grade school, gender, parents’ country of birth (three dummies) and GPS9 quintile. The specification in Column 2 is our preferred specification; here we also enforce the preference restriction, but enforce common support with respect to upper secondary school county instead of 9th grade school, which has the benefit of retaining substantially more observations. In Column 3 we use the full observational sample without adding preference restrictions or preference controls. The difference between Column 3 and the remaining two columns thus shows the potential importance of utilizing information on student preferences.
Although sample sizes vary greatly as a result of alternating the restrictions discussed above, the results in Table 5.A are fairly stable across specifications. The results in the first row suggest that independent school attendance has an impact on the probability of switching school type: independent school students are more likely to switch to public schools than the other way around. The effect size of 2.3 p.p. in column 2 (our preferred specification) is quantitatively similar to the raw difference presented in Table 4.
One could suspect that the effect is a result of the bankruptcy of the John Bauer (JB) group, described in Section 2.1, forcing students to switch to public schools. In a robustness analysis, which is presented in section C2 in Appendix C, we however find that the estimate for switching school is similar (0.018, compared to 0.023 in Table 5.A) also when we exclude the JB-school cases from the sample.36 The higher
35 This analysis will be carried out using the STATA command psacalc, see Oster (2019).
36 More specifically, this is done in the following manner: As we lack access to school names, we cannot drop students attending JB-schools. Instead, we have dropped all observations belonging to a track×municipality×year combination where a JB-school was present.
propensity to switch from an independent to a public school rather than the reverse could thus be related to higher volatility in the independent school market in general, or to the fact that students in independent schools are more often dissatisfied with their school. The results from the analysis excluding the JB-cases are also very similar to the baseline estimates for the other outcome variables, see section C2 in Appendix C.
TABLE 5.A.GRADUATION AND GRADES – CEM/VAM
Results in Table 5.A. also suggest that independent school attendance has a positive impact on the ranking of the student’s GPA in 12th grade. The effect size of 4.49 percentiles in column 2 is quantitatively similar to the raw difference in Table 4. The positive independent school impact on the likelihood of graduating on time at 2.88 percentage points in column 2 is somewhat larger than the raw difference, while the negative independent school impact of staying behind for a 7th semester at 1.53 p.p. is in line with the raw difference.
Results for standardized test outcomes are shown in Table 5.B. We study four outcomes, a dummy for obtaining a pass test grade, a dummy for obtaining a high test grade, a dummy for obtaining a final course grade that is lower than the test grade, and finally, a dummy for obtaining a final course grade that is higher than the test grade. The result that stands out the most is the positive coefficient on the probability
(1) (2) (3)
Switch independent/public 0.0279*** 0.0231*** 0.0343***
Standard error (0.0057) (0.0049) (0.0042)
P-value [0.0000] [0.0000] [0.0000]
Observations 28837 70623 288762
Pctile GPA12 4.4302*** 4.4906*** 4.5410***
Standard error (0.3474) (0.3080) (0.2955)
P-value [0.0000] [0.0000] [0.0000]
Observations 25578 61898 254937
Graduate on time 0.0229*** 0.0288*** 0.0200***
Standard error (0.0051) (0.0041) (0.0038)
P-value [0.0000] [0.0000] [0.0000]
Observations 29440 72220 294580
7th term -0.0133*** -0.0153*** -0.0099***
Standard error (0.0038) (0.0028) (0.0024)
P-value [0.0004] [0.0000] [0.0000]
Observations 29440 72220 294580
Preference restriction YES YES NO
CEM on 9th grade school YES NO NO
CEM on county NO YES YES
Note: All regressions above include the following covariates: upper secondary school municipality dummies, prior achievement as controlled for by a cubic form of GPS9 and GPS9 quintile dummies, as well as 6 dummies representing pass/high test result in Math/Swe/Eng in 9th grade, 9th grade school dummies, track dummies, log household income, income decile dummies; and dummies indicating the following: gender, born in western country (excl. Sweden), born in non-western country, one parent post-secondary education, both parents born in Sweden, one parent born in Sweden, both parents born in non-western country, negative or zero household income, one parent is self-employed, one parent is unemployed, and cohort. Columns 1 and 2 also include a dummy indicating admission to first ranked school. Standard errors are clustered on upper secondary school.*** p<0.005, ** p<0.01, * p<0.05
of getting a final course grade that is higher than the grade obtained on the standardized test. The coefficient is statistically significant and/or economically interesting across all samples and across all subjects. The increased probability is the highest in Mathematics at 4.62 percentage points in column 2, and the smallest in Swedish at 2.39 percentage points in column 5. The increased probability is 4.01 percentage points in English, see column 8.
A second striking result in Table 5.B is the positive coefficient on the probability of getting a high grade on the standardized tests in Swedish and in English. The probability increases with 2.04 percentage points in Swedish, and 1.55 percentage points in English. The coefficient for Mathematics is only statistically significant in the full sample estimation, but the coefficient is then relatively small at 0.5 percentage points. Remaining results that are statistically significant are also relatively small in economic terms. To summarize the findings reported in Table 5.B; attending an independent school increases the probability that you will become upgraded in your final course grade compared with the grade you received on your standardized test. This is true for all subjects. The probability of actually getting a high test grade only increases in Swedish and in English but not in Mathematics.
Finally
,
Table 5.C shows results for post-graduation outcomes. In column 2 the estimate of 1.99 suggests that attending an independent school has a positive impact on the probability of participating in any type of studies one year after graduation. This is the case even when excluding studies that are of a preparatory type, and the estimate then even increases to 2.44 percentage points. The effect on the probability of obtaining at least 15 university credits is also positive at 1.42 percentage points. However, the effect on the probability of earning labor income corresponding to at least a half time job is negative at -1.70 percentage points. The effects shown in Table 5.C. have the same signs as the raw differences in Table 4, and the effect sizes are quantitatively similar.It can be noted that our results are not hugely sensitive to restricting the sample to only include individuals with preferences for both types of schools (compare full sample results in column 3 with other columns). Our results thus provide some support for the conditional on observables analysis in Hinnerich and Vlachos (2017), which does not make use of information on student preferences.
