Evaluating the Effectiveness of the SMART Contract Signing Strategy in Reducing the Growth of Swedish Adolescents’ Substance Use and Problem Behaviors
Cristian Bortes
Master’s Program in Public Health with Prevention Focus
Örebro University
Submitted in Partial Fulfillment of Master’s Program in Public Health with Prevention Focus
Independent Research Course
Spring 2015
Abstract
Aim: To evaluate the effectiveness of the SMART contract concept strategy in reducing the growth of youth substance use and other problem behaviors amongst Swedish adolescents.
Method: By a longitudinal design, students from five schools within a mid-size Swedish municipality were surveyed in three waves from 7th to 9th grade of primary school.
Data analysis: We used GLM repeated measures ANOVA to test if the outcome measure smoking, use of snus and alcohol, drunkenness, delinquency, and bullying changed
significantly over time differently in groups of long, short and sporadic- or non-attendees in the SMART program. Groups were compared on demographic background variables, and outcome measures assessed at all measurement occasions by a one-way ANOVA. The
magnitude of group differences at the end of the study was estimated according to Cohen’s d.
Results: Number of years with contract have an effect on the levels of self-reported youth problems by 9th grade. We found small to medium-size differences in measured outcomes between students who attended the program for the longest period of time, 5 years, compared to students who attended the program for the shortest time, 0-2 years.
Conclusion: Findings suggests that the SMART program have preventive effects on adolescent substance use.
Evaluating the effectiveness of the SMART contract signing strategy in reducing the growth of Swedish adolescents’ substance use and problem behaviors
Adolescence is an important period in people’s lives during which identity and personality formation occurs (Steinberg, 2014; Hill et al., 2013; Kroger, Martinussen & Marcia, 2010; Meeus, 2011; Tanti, Stukas, Halloran & Foddy, 2011), and lifestyles are established (Lake, Mathers, Rugg-Gunn, & Adamson, 2006; Talema, 2009). Poor health behaviors in adolescence affect adult life (Rose, Winter, Viken & Kaprio, 2014; Wennberg et al., 2013), and as such adolescence can be seen as a foundation for future health (Sawyer et al., 2012).
The initiation of alcohol, tobacco and other illicit drugs usually take place during the adolescent years (HBSC, 2012). The risks associated are well known, as these problematic behaviors are among the important causes of disease and mortality worldwide (Rehm, Taylor & Room, 2006; Ezzati et al., 2002; Ezzati & Lopez, 2003) and put an economic cost-burden on society as well (Rehm et al., 2009). Early-onset of substance initiation in particular has a significant role in later substance-related problems and is also related to psychosocial problems during young adulthood (Griffin, Bang, & Botvin, 2010; Moss, Chen & Yi, 2014; Magid & Moreland, 2014). The risk of becoming nicotine dependent is greater if starting to smoke early in life, compared to start smoking later (Kendler, Myers, Damaj & Chen, 2013), and so the risk of becoming a regular smoker decreases as the onset age increases (Kendal & Logan, 1984; Kandel, Yamaguchi & Chen, 1992). Underage drinking and especially first use of alcohol at ages 11-14 has been linked to a range of later health problems as well (DeWit, Adlaf, Offord & Ogborne, 2000; Pitkänen, Lyyra & Pulkkinen, 2005; Windle et al., 2008; Hingson & Zha, 2009; Dawson, Goldstein, Chou, Ruan, & Grant, 2009). Preventing or delaying youth substance use is therefore of great public health concern.
Youth substance use can be a complex phenomenon to get an absolute and complete understanding of and how to best tackle it. We know that attitudes and norms (Lipperman-Kreda, Grube & Paschall, 2010), and characteristics of the social environment (Jung & Chung, 2012; Edvardsson, Lendahls, Andersson & Ejlertsson, 2012); such as peer
socialization (Trucco, Colder, & Wieczorek, 2011), parental expectations (Nash, McQueen & Bray, 2005) and smoking rules in school (Pinilla, González, Barber & Santana, 2002), as well as sociocultural influences (Morgenstern, Sargent, Engels, Florek, & Hanewinkel, 2013; Sargent & Hanewinkel, 2009), are some of the determining factors, among others (García-Ridríguez, 2014; Joffer et al., 2014), for smoking and alcohol use in adolescence. It is however possible, based on knowledge on risk and protective factors, to reduce youth
problems from developing into increasingly more harmful and long-term disorders (Catalano et al., 2012). School is an important arena for a healthy development and make up a large part of young peoples’ social environment. In Sweden, where schooling is compulsory, this provides the possibility to reach virtually all children. School-based interventions are therefore appropriate for health promoting purposes.
In 2013, around 40% of the schools in Sweden had structured programs to prevent tobacco and alcohol debuts in primary school (Public Health Agency of Sweden, 2014). It has unfortunately been a lack of scientific evidence for most of the prevention methods focusing on primary prevention in schools in Sweden (Sundell & Forster, 2005). Hence, there is a need for evaluations of school-based preventive programs in this national context. Especially the concept of signing a contract for preventing tobacco and alcohol use has a long history in Sweden (SMART, 2015). The Non-Governmental Organization (NGO) SMART carries out one popular program in which positive reinforcement and signing of contracts with school children are core components. The purpose of this present study is to evaluate the
effectiveness of their prevention program in reducing the growth of Swedish adolescents’ substance use and problem behaviors.
Turning to the scientific literature, there are a variety of school-based programs for preventing alcohol, tobacco, and other drugs reported. There is heterogeneity between types of interventions, populations, outcomes, and results between them. The conclusions that can be drawn are consequently limited to the different contexts. Overall, reviews find that interactive and theory driven multicomponent programs are most successful in achieving positive outcomes. These have a combination of social competence enhancement approaches, through which the participants develop personal and social, refusal- and overall life skills, and social influence approaches that address social norms, commitment and intentions not to use (Tobler et al., 2000; Cuijpers, 2002; Botvin & Griffin, 2007; Soole, Mazerolle & Rombouts, 2008; Stigler, Neusel & Perry, 2011; Thomas et al., 2013; Faggiano et al., 2014). It is also acknowledged that an integration of other parts of the community as well as the family is contributing success-factors. However, there are only a couple of programs that include contract signing as a component.
One well-studied school-based prevention program containing components found to be effective, and the element of contract signing, is the ”Smoke-Free Class Competition” (SFC) (Vartiainen, Saukko, Paavola & Vertio, 1996). It has the objective to delay or prevent the onset of smoking in adolescence. The SFC competition is considered to have rather broad evidence for its effectiveness as a school-based prevention program (Isensee & Hanewinkel, 2012; Vartainen et al., 1996; Wiborg & Hanewinkel, 2002; Isensee et al., 2012) and it has been widely implemented throughout Europe, however not in Sweden.
A tobacco prevention program that is well established in Sweden is the “Tobacco-Free Duo” (Nilsson et al., 2006). This program also includes contract signing as a strategy along
with other components and is widely disseminated in Sweden. Despite its proven sustainability within communities however, Tobacco-Free Duo yet lacks to fulfill the standards of evidence and the criteria of effectiveness as postulated by the Society for
Prevention Research (Flay et al., 2005), since only one evaluation study (Nilsson et al., 2006) as part of a dissertation (Nilsson, 2009) has been conducted and published. We argue that there undeniably is a need to broaden the evidence-base for school-based preventive methods in Sweden, particularly the ones with a strategy of contract signing, as these are widely used but lack sufficient evidence for their effectiveness.
In this study, we aim to evaluate whether the contract concept strategy, as carried out by the NGO SMART, is successful in preventing youth substance use and problem behaviors. Has the writing of contracts any effect on the levels of substance use, and is the number of years with contract important for the results?
The Non-Governmental Organization SMART and the contract-concept
SMART was founded in 2001 and is a network for anyone involved in drug
prevention. The aim of SMART is to prevent or delay the onset of alcohol, tobacco and illicit drug initiation among school children through positive reinforcement and signing of contracts. Their method, the contract concept, is based on voluntary participation and encourages young people to consciously opt out unwanted behaviors and to make smart choices. SMART has a whole-community approach, where local actors, with the support of SMART, are designing the method based on local conditions. More than 30 000 youths in Sweden are locally
connected to some form of contract activity (SMART, 2015). Today it exists in approximately 90 Swedish municipalities as well as in 8 municipalities in other countries. Actors behind these contract activities may be county councils, social services, police, schools, sports clubs and NGOs. To have different actors as head of the contract activities also means that there are differences in the implementation and delivery of the program from one place to another. The
emphasis between fun activities as reinforcement for the students and financial incentives such as discounts and lotteries varies between actors. Even though the program in general targets 10-16 year old students in primary school, the decision on what school-grades to target can differ. There are also many different names for the local operations. Membership cards are different, and some operations are without membership cards. SMART represents a general concept, an overall strategy that is adapted to local conditions, and not necessarily a uniform manual-based step by step program. The major exception is Tobacco-Free Duo, the largest variant within the network SMART, which works with a clearly mapped manual. Despite local differences, it is a minimum requirement that the contract must contain an agreement concerning tobacco use. The concept is that students sign a contract in the beginning of the school year, and a parent must give written consent. The contract is the agreement in which the student promise to refrain from smoking, using snus, or other tobacco for the upcoming year. The contract may contain additional items as well, as it has in the program-edition we evaluate in this study; abstain from using drugs (such as tobacco, alcohol, drugs or sniff- and dopants), destroying belongings of someone else, shoplift or steal, and being a good friend and showing respect for other people. When signing a contract the student receives a membership card. This provides benefits such as activities and discounts sponsored by local businesses, reinforcing positive behaviors. The members may choose to prolong their memberships by signing a new contract for one year at a time. At breach of contract the members’ parent / guardian are contacted for calls. The member can be excluded, ranging from one month to the rest of the contract period, but is always welcome back.
Methods Study procedure
In the spring of 2011, a plan regarding the evaluation of the contract-concept was made between SMART, the participating schools and the research team from Örebro University. The parties agreed that SMART would carry out the concept and continuously inform the school staff about the program. The schools’ responsibilities was to have staff implementing the method; providing class lists of addresses to the children's parents that would annually be updated to the research team, and set aside time for the annual surveys in order for students to fill them out during school hours. The school would also report once per academic year, what health promotion and prevention activities were performed during the school years that the contract-concept was carried out. The research team was to survey school students; analyze and report these results to the schools and SMART, and to publish them as international scientific articles as well as Swedish-language articles.
A letter of consent was sent to parents, with information about the study purpose, and that participation was voluntary for their children. Also, parents were given the opportunity of passive consent, to communicate if they wished for their child not to participate in the study. Parents were welcomed to contact the research team if they had any questions. Telephone number and e-mail addresses were provided.
The present study design is non-experimental and the intervention was already running prior to the start of our observations. The research team got involved and started the first data collection when students were in 7th grade. SMART and the schools in this particular
municipality had implemented and worked with the concept since the students who make up this study population were in 4th grade.
The first data collection was conducted in autumn of 2011 (T1) when students were in school year 7. Follow-ups were conducted one year later in autumn of 2012, in school year 8 (T2), and a third in spring of 2014, in school year 9 (T3). The survey includes questions about
family, school and peer relationships, outlook on life, tobacco, alcohol and drugs, health and lifestyle. The students received written and verbal information about the purpose of the study. They were also informed that participation was voluntary, about the confidentiality of the data, and that no identifying information would remain accessible. The students answered the questionnaire in the classroom during school hours, in the presence of a representative from the research team whom was previously unknown to them. The study has been ethically approved by the Ethical Review Board in Uppsala, Dnr. 2011/213.
Participants
The study population consists of adolescents from five different schools within one medium-sized Swedish municipality. At T1, in 7th grade, students are 13-14 years of age and 50.4% are boys. At T2, in 8th grade, students are 14-15 years and 50.6% are boys. At T3, in 9th grade, the students are 15-16 years and 50.1% are boys. Response-rates on the survey at the three time-points of data-collection are shown in Table 1.
Table 1. Total study population and response-rates.
T1 T2 T3
Total-population n = 518 n = 502 n = 476
Response-rate n = 432 n = 458 n = 422
83 % 91 % 89 %
Measures
Gender. Gender was coded 1 for boy and 2 for girl.
One non-Nordic parent. At least one parent born in a non-Nordic country is coded as 0 and if parent born in Sweden or another Scandinavian country as 1.
Monthly allowance. Students were asked how much money they received to spend on their free time and hobbies on a 7-point scale: (1) “0-249”, (4) 750-999, (7) “More than 1500”. This is an indicator of SES.
Books at home. Students were asked “How many books are there in your home?” on a 7-point scale, from (1) “No books”, (3) “11-50 books”, to (7) “More than 500 books”. We did not have information concerning parents’ education and employment, instead we used number of books at home as a socio-cultural indicator (Mullis et al., 2012; Yoshino, 2012).
Type of habitation. The item “How do you live?” with five response options from: (1) “Rental apartment”, (3) “Semi-detached house”, to (5) “other accommodation”. Any other type of living then rented counted as owned and may indicate a higher SES-status. Rented was coded as 1 and owned as 2.
Number of years with contract. Students were asked to report whether they signed a contract for any of the school years ranging from (1) “Yes, in 4th grade”, to (6) “Yes, in 9th grade, and (7) “Never signed”.
Smoking. Students were asked to report whether they smoke on a 7-point scale: (1) “No, never smoked”, (5) Yes, sometimes”, (7) “Yes, every day”. Higher values indicate a more established smoking behavior.
Snus use. Snus is a Swedish sort of moist snuff. Students were asked to report if they used snus on a 7-point scale: (1) “No, never used snus”, (6) “Yes, almost every day”, (7) “Yes, every day”.
Alcohol use. Students were asked to report if they ever drunk alcohol on a 4-point scale, ranging from (1) “Have not consumed alcohol” to (4) “Have drunk several times”. Higher values indicate higher levels of alcohol use.
Drunkenness. Adolescents were asked to report whether they had ever become drunk on a 6-point scale, ranging from (1) “I have never drunk alcohol” to (6) “Yes, every time”. Higher values indicate more frequent binge drinking.
Delinquency. Students were asked to report on two items whether they stolen or intentionally damaged anything the past year on a 5-point scale (two items: r = .47 at T1, .47 at T2, .43 at T3): (1) “Never done it” to (5) “More than 10 times”. Higher scores indicate higher delinquency levels.
Bullying. Students were asked how often they been involved in bullying other students in school this semester on a five-point scale from: (1) “I have not bullied anyone at school this semester”, (2) “Once or a couple of times, to (5) “Several times a week”. Bullying had the following written definition in the survey: A student is being bullied when another student, or group of students, says or does nasty and unpleasant things to him or her. It is also bullying when a student is constantly being teased in a way he or she does not like. It is not bullying when two fairly evenly matched students quarrel or fight. It is also not bullying when a student tease in a kind or friendly manner.
Data analysis
The IBM SPSS software package version 22 was used for statistical analysis. Using the data at T3 on the number of years with contract, students were grouped into three groups in terms of how long they have attended the program: students who signed contract 5 years, from 4th until 7th and 8th or- 9th grade (long-attendees, 22.4%), students who signed contract starting in 4th grade and 2-4 years ahead (short-attendees, 39.9%), and students who signed contract only in some consecutive (0-3 years) or none of the years (sporadic- or non-attendees, 20.6%). This operation turned the analytic sample into a total of 414 participants. Creating a comparison condition using non-participants has previously been used successfully
(Pettersson, Özdemir & Eriksson, 2011) and is accepted as a reasonable strategy for evaluations to increase internal validity of the conclusions (Özdemir, in press).
Our main analysis was GLM repeated measures ANOVA to test if the outcome measure smoking, snus and alcohol use, level of drunkenness, delinquency and bullying changed significantly over time differently in groups. Then, to compare long, short, and sporadic- or non- (SPON) attendees in the program on demographic background variables, and outcome measures assessed at all measurement occasions we used a one-way ANOVA. In order to reach more robust conclusions regarding the magnitude of group differences at the end of the study we estimated effect-sizes according to Cohen’s d (Cohen, 1988).
Furthermore, we used ANCOVA to compare the groups on the outcomes controlling for gender and the SES variable books at home, as preliminary results showed significant
baseline differences between groups specifically on these variables. This would partial out the initial differences between groups so that the differences in outcomes at measurement
occasions may be attributable to number of years with contract. All results were considered significant at p ≤ 0.05.
Missing data
In order to analyze and understand the missing data pattern we first recoded the six outcome variables, 1 for missing cases and 0 for everything else. In this way we could inspect the frequencies of missing cases in the main study variables. This indicated internal
missingness – a consistent level of missingness across all the variables and time-points: 15-17% missing at T1 on all six outcome variables, around 10% missing at T2, and 15-15-17% missing at T3 on all the six outcome variables. Then, a one-way ANOVA was conducted in which all the recoded variables were run by the categorical variable number of years with contract. In this way we could identify differences in missingness at each measurement
occasion between the groups. This revealed a significant difference between the groups on only two outcome variables as shown in Table 1.
Table 1. Differences in missing-data in outcome variables between the groups. Similar superscript letters indicate between which groups there were significant differences (p<0.05, ANOVA, Tukey’s HSD). Long-attendees Short-attendees SPON-attendees M SD M SD M SD F(df) p Smoking T1 .08a .27 .15 .35 .19a .40 2.95(2,412) .043 Smoking T2 .06 .24 .06a .23 .14a .34 3.39(2,412) .035 Results
Number of years with contract has an effect on the levels of youth substance use and problem behaviors at T3. Students who attended the program for the longest time, and signed contract at least 5 years from 4th grade upwards, had significantly lower levels of youth problems, compared to students in the other two groups (Table 3). All outcomes significantly increased over time for the whole sample (Table 4). Further inspection of the data shows a significant difference in gender distribution between the three groups (see Table 3). There are more boys in the group of SPON-attendees as the mean value (M=1.33) loads slightly more towards that direction in the gender-variable. There is also a significant difference between the groups on the SES variable books at home. Students in the group of long-attendees report having more books at home (M=.70) compared to students in the group short-attendees (M=.62), and SPON-attendees (M=.52).
Smoking
We see a significant difference in how the groups increased in smoking over time, F(4,656) = 4.69, p = .001, η2 = 028, suggesting that number of years with contract has a significant effect on smoking. The longer students attended the program and signed a contract,
the less smoking they report at T3 (see Figure 1). At baseline (T1), the groups are alike in self-reported smoking behavior. Differences between the groups starts approaching
significance at T2 (p=.054), and is significant at T3 (p=.001). The effect size between long-attendees and short-long-attendees at T3 is nearly medium (d = .48). Between long-long-attendees and SPON-attendees the effect size at T3 is medium (d = .64), and between short-attendees and SPON-attendees at T3 the effect size in smoking is small (d = .18). Additionally, when
controlling for the SES variable books at home F(1,325) = .39, p = .532, η2 = .001, and gender F(1,327) = .22, p = .642, η2 = .001, using analysis of covariance (ANCOVA), this failed to explain the variance in differences between groups in smoking at T3. The differences in smoking between the groups in 9th grade can thus be attributable to the number of years with contract after controlling for gender and the SES variable books at home.
Snus use
Regarding the use of snus, repeated measures ANOVA reveals a significant difference in how groups increased in snus use over time, F(4,652) = 4.53, p = .001, η2 = 027,
suggesting that number of years with contract has a significant effect on youth snus use at T3. There are significant differences between the groups only at T3 (p=.001). Long-attendees have the lowest reported snus use (M=.16) compared to short- (M=.59) and SPON-attendees (M=.73). However, the ANCOVA results suggests that the variance in the differences
between groups at T3 in snus use is not solely explained by the number of years with contract, but also by gender, F(1,325) = 20.01, p =.001, η2 = .071. Boys tend to use snus more than girls. The effect size between long- and short-attendees at T3 is small to moderate (d = .40), between long- and attendees it is medium (d = .56), and between short- and SPON-attendees it is small (d = .10).
The rate of change from T1 to T3 in alcohol use, as revealed by repeated measures ANOVA, is significantly different between groups, F(4,450) = 2.50, p =.041, η2 =.015. This suggests that number of years with contract has a significant effect on alcohol use at T3. We see a significant difference between the groups in self-reported alcohol use starting at T2 (p=.002). Long-attendees have lowest levels of alcohol use (M=.1.57). Interestingly, short-attendees have the highest levels of alcohol use at T2 (M=1.98), and SPON-short-attendees slightly below them (M=1.91). The variance between the groups in alcohol use at T2 is attributable to attendance in the program and number of years with contract, after controlling for gender F(1,324) = .181, p =.671, η2=.001, and SES variable books at home F(1,322) = 0.00, p =.999, η2 = .000. At T3, long-attendees have increased but still have significantly lower levels of alcohol use (M=2.04), compared to short- (M=2.57) and SPON-attendees (M=2.59).. Also after controlling for the initial difference in gender distribution F(1,324) =3.70, p = .055, η2=.011, and books at home F(1,322) = 2.35, p=.126, η2=.007 which fails to explain the variance between the groups at T3. The effect size between long-attendees and short-attendees in alcohol use at T3 is medium (d = .47), as well as between long-attendees and
SPON-attendees (d = .50). However, between short- and SPON-SPON-attendees this difference at T3 is almost non-existent (d = .01). The longer students attended the program and signed a contract, the lower levels of alcohol use they report at T3.
Drunkenness
Despite an inconsistent pattern over time, we see an overall significant difference in how groups increased in levels of reported drunkenness from T1 to T3, F(4,652) = 6.28, p=.001, η2 = .037. Again, this suggests that number of years with contract matters. Youth drunkenness is the only outcome measure on which we see a significant difference between groups already at T1 (p = .048). Long-attendees, who at this time-point already had been
Table 3. The rate of changes in youth problem behaviors over time. Attendance (in the program) = number of years with contract.
F(df) p η2 Smoking Time effect 30.19(2,656) .001 .084 Time * Attendance 4.69(4,656) .001 .028 Snus Time effect 22.47(2,652) .001 .064 Time * Attendance 4.53(4,652) .001 .027 Alcohol use Time effect 83.00(2,650) .001 .203 Time * Attendance 2.50(4,650) .041 .015 Drunkenness Time effect 53.76(2,652) .001 .142 Time * Attendance 6.28(4,652) .001 .037 Delinquency Time effect 3.99(2,648) .019 .012 Time * Attendance .33(4,648) .868 .002
Figure 1. Changes in levels of self-reported smoking behavior.
Figure 2. Changes in levels of self-reported alcohol use.
0 0,5 1 1,5 2 2,5 3 T1 T2 T3 S m ok in g Long,attendees Short,attendees Sporadic, or8non, attendees 0 0,5 1 1,5 2 2,5 3 T1 T2 T3 A lc oh ol u se Long,attendees Short,attendees Sporadic, or8non, attendees
Figure 3. Changes in self-reported levels of drunkenness.
signing contract for three years every year since 4th grade, report significantly lower levels of drunkenness (M=1.34) compared to short-attendees (M=1.48) and SPON-attendees (M=1.60). The variance in difference between the groups in drunkenness at T1 is however also explained by books at home as a covariate, F(1,323) = 6.99, p = .009, η2 = .021. At T2, this difference in youth drunkenness between the groups is no longer, but regains significance at T3.
Drunkenness levels increased more sharply for the groups of short- and SPON-attendees from T2 towards T3 (see Figure 3). The effect size between long-attendees and short-attendees in youth drunkenness at T3 is medium (d = .53), as well as between long-attendees and SPON-attendees (d = .54). The difference in youth drunkenness at T3 between short-SPON-attendees and SPON-attendees is small (d = .01). In sum, the longer students attended the program and signed a contract, the lower levels of drunkenness they report at T3.
Delinquency
Repeated measures ANOVA revealed a non-significant difference in how groups changed over time from T1 to T3 on indicators of delinquency, F(4,648) = .33, p = 868, η2 = .002. However, significant differences between groups in delinquency are visible at T3.
0 0,5 1 1,5 2 2,5 3 T1 T2 T3 Dr u n k en n es s Long,attendees Short,attendees Sporadic, or8non,attendees
Table 4. Means and standard deviations, and differences between groups on study variables. Groups with similar superscript letters did differ (p > 0.05) on measured outcomes (ANOVA, Tukey’s HSD).
Long-attendees Short-attendees SPON- attendees
(n= 112) (n= 200) (n= 102) M SD M SD M SD F-test p
Baseline demographic and
SES variables Gender 1.59 a .49 1.51b .50 1.33ab .47 7.14(2,357) .001 One non-Nordic parent .82 .38 .88 .33 .77 .42 2.37(2,352) .095 Monthly allowance .19 .23 .24 .26 .18 .24 1.69(2,353) .186 Books at home .70abc .24 .62abc .23 .52abc .26 13.10(2,354) .001 Type of habitation 1.06 .24 1.07 .26 1.08 .28 .194(2,357) .824 Smoking T1 .12 .78 .19 .60 .19 .53 .35(2,354) .704 T2 .16 .79 .36 .94 .47 .99 2.94(2,380) .054 T3 .28a .71 .84b 1.47 1.12ab 1.71 10.64(2,411) .001 Snus use T1 .07 .60 .06 .42 .08 .31 .04(2,357) .961 T2 .07 .62 .13 .58 .26 .79 2,00(2,379) .137 T3 .16 a .55 .59b 1.41 .73ab 1.33 6.75(2,408) .001 Alcohol use T1 1.47 .69 1.64 .77 1.73 .86 2.96(2,353) .053 T2 1.57 a .77 1.98b 1.01 1.91ab 1.00 6.56(2,379) .002 T3 2.04 a 1.06 2.57b 1.21 2.59ab 1.13 8.65(2,410) .001 Drunkenness T1 1.34a .78 1.48 .61 1.60a .82 3.06(2,353) .048 T2 1.44 .88 1.60 .90 1.67 1.06 1.64(2,381) .195 T3 1.72a 1.03 2.43b 1.60 2.41ab 1.53 9.52(2,408) .001 Delinquency T1 2.14 .86 2.25 .83 2.40 1.11 1.87(2,352) .156 T2 2.24 1.00 2.37 1.11 2.47 .86 1.22(2,380) .295 T3 2.23a .72 2.52 1.33 2.76a 1.62 4.59(2,410) .011 Bullying T1 1.1 .58 1.08 .32 .108 .31 .14(2,356) .868 T2 1.05 .23 1.14 .49 1.16 .45 1.80(2,379) .167 T3 1.06 .41 1.11 .46 1.16 .48 1.17(2,409) .313
This variance in differences at T3 is also explained by gender, F(1,323) =11.11, p = .004, η2 = .026. Between long- and short-attendees the difference in reported delinquent acts at T3 is significant (p = .001) but small (d = .27), as well as between short-attendees and SPON-attendees (d = .16). Between long- and SPON SPON-attendees the difference in reported delinquency at T3 is however moderate (d = .43).
Bullying
Results revealed a non-significant difference between the groups in the outcome bullying. All three groups have almost lowest values possible in this measure at all measurement occasions.
Discussion
The present study was conducted as part of an evaluation of the contract concept as implemented and conducted by a NGO, and contributes to the development of the evidence base related to school-based alcohol and tobacco interventions. One goal of the study was to obtain information on whether writing a contract has any inhibiting effect on levels of substance use among school children. The clear findings in this study are that the longer individuals attend the SMART program and sign a contract, the less they smoke, use snus, drink alcohol, get drunk, and commit delinquent acts when they are in 9th grade. Above all, the present study meets the need of scientifically evaluated methods focusing on primary prevention in schools within the Swedish context.
Contrary to most prevention programs, the method as carried out in these particular schools within this municipality that this study encompasses has in a certain sense a wider ambition. Although the primary objective is to prevent and delay the initiation of substance use, a contract containing additional items on good conduct behavior and peer victimization
several precursors of multiple problems is an efficient approach in prevention work (Catalano et al., 2012). Thus, this study further adds that signing of contracts might be used for purposes in reducing various youth problems.
However to merely sign a contract in itself is insufficient and should be accompanied by other components as well. When signing a contract, the individual’s intention to abstain from doing certain things becomes reified and concrete. This action and commitment is shared with classmates and significant others. This in turn creates a sense of community – having a membership and a membership card, which is further positively reinforced by the positive gains and benefits to which members are entitled to access. So even though the signing of a contract is a core component, there is much more to it. In addition to the existing Swedish ANDT-education (alcohol, narcotics, doping, tobacco) in schools, which already is interactive (Skolverket, 2012), the SMART contract-concept is a multicomponent intervention using positive reinforcement, engaging the adolescents’ social environment, parents and community. The parental component of having a parent leave written consent leads to conversations on the items of the contract at home (Eriksson et al., 2010). Furthermore, it addresses norms, as well as commitment and intentions not to use. These are several
components which previous research has identified as effective (Tobler et al., 2000; Cuijpers, 2002; Botvin & Griffin, 2007; Soole, Mazerolle & Rombouts, 2008; Stigler, Neusel & Perry, 2011; Thomas et al., 2013; Faggiano et al., 2014), and the concept has furthermore a strong advantage in its flexibility and possibility to be adapted to local conditions.
Regarding the study findings, some further aspects should be considered. Despite some few baseline differences, we can infer program effect to some extent because the rate of change over time in outcomes amongst students in the short- and sporadic- or non-attendees group was significantly higher than the rate of change in the long-attendees group. Our missing-data analysis (Table 2) demonstrates that the group of short-attendees and especially
sporadic- or non-attendees, on the question of smoking, had the most missing cases. These results may indicate that these groups had yet higher levels as they neglected to report it. Furthermore the time effect, that an overall and significantly observable increase in for instance youth smoking or alcohol use as our measures is constructed (i.e. “have sipped from someone else’s glass”) occurs from 7th to 9th grade, is more or less expected to occur in all groups and is less surprising. Another thing regarding the findings should be emphasized: the groups of individuals who attended the program less number of years and scored significantly higher on the surveys compared to the group of individuals who attended the program for several years do still not report soaring levels of substance use relative to our measurement scale. Indeed, there are statistically significant differences between them. However on the 7-point scale as we for instance measured smoking; none of the groups scored above a mean-average of 1.71. So as a group, on the question “Do you smoke”, the sporadic- or
non-attendees who are the highest-scoring group, approach “No, but I have tried” in 9th grade. For drunkenness, the highest score on the 6-point scale was the mean-average 2.43. Meaning that the group of short-attendees, who is the highest-scoring group on drunkenness, but also the group of sporadic- or non-attendees, had up to the time point in 9th grade been drinking alcohol to the limit of intoxication, once. Not to trivialize youth drunkenness, but when pointing at the practical difference between the groups and the practical significance, it is fairly low. This is of course a good thing and may be explained by the prevailing culture and other established approaches amongst the schools in the particular municipality. Especially as at least two of the schools provided other health promotion and prevention activities in addition to ordinary curriculum and the SMART program. It is most certainly also a result of more comprehensive national and regional strategies that has achieved a population-level impact (Swedish Government, 2010). The latter has been suggested as necessary since school-based alcohol, tobacco and drug prevention programs alone, have in general small effects
(Faggiano et al., 2014; Ström et al., 2014). Nevertheless, on a group level, the present study show a solid medium-sized difference between having been involved in the SMART program for at least 5 years starting in 4th grade, and not, when looking at binge drinking/drunkenness. It has been suggested in previous studies, that primary prevention programs particularly for reducing alcohol (Pasch, Perry, Stigler, & Komro, 2009), and tobacco use among adolescents (Edvardsson, Lendahls & Håkansson, 2009), should be provided before 6th grade, or at least before initiation occurs. That is in order to influence views and attitudes towards substances before adolescents come across them elsewhere, as they usually do in the upper grades of primary school. The present study adds yet further support to these notions. But our findings also suggest the necessity of prolonging participation in the program. Because looking at the differences in alcohol use and levels of drunkenness between the group of short-attendees and the group of sporadic-or non-attendees in 8th and 9th grade we see they are small, and not even significant. This indicates that students, who quit attending the program, as short-attendees, do not opt out to the same extent as those attending the program year after year, and as a result their levels of substance use were rising. The annual reoccurring signing of a new contract for the coming year can thus be seen as booster sessions (Nation et al., 2003).
Limitations and strengths
There are some factors of uncertainties and threats to validity to take into consideration as well. We did not control if and how many individuals ever breached a contract, and how this might have affected the outcomes. We also do not know how teachers and other students acted when a contract-participant possibly did something that could have affected their contract. We lack information if every possible breach of contract was detected and handled, especially as it most likely would have occurred outside school-hours. The contract-concept largely rests on the individual’s own conscience and sense of responsibility. There are possibilities that someone signed contracts and violated it, enjoyed the benefits, and
reported false answers in the survey. However, on the validity of self-reports of socially unacceptable behaviors, such as adolescent smoking and alcohol drinking, a cross-sectional, biochemically verified analysis of a Swedish cohort sub-sample, confirmed that adolescents self-reported tobacco use are reliable (Post et al., 2005). A further note regards the measures we use. They are very limited and far from exhaustive, and should be considered as
approximations. That our findings show non-significant differences between the groups or effect of the program on the level of bullying could be understood as a methodological
technical issue. Even though the participants did report almost lowest possible scores, the one-item measure we used is probably not adequate enough. Because in the bullying and peer-victimization research literature we find more comprehensive assessment tools with more nuanced scales for measuring this more accurately (Hellström, 2015; Hamburger, Basile, & Vivolo, 2011). But even so, the low levels of bullying detected even by our measure, may also be due to the interdisciplinary plan for working against bullying, with student welfare, student participation, and parent involvement, which at least two of the schools explicitly stated that they had in an interview with a representative from the research team. Moreover, this
information on health promotion and prevention activities carried out in the schools, which originally were agreed to be collected and documented each academic year during the study, was done so only one year during the study. Another matter of concern regards the study design being non-experimental; hence we cannot demonstrate a true cause and effect relationship between the program and outcomes. Neither can we claim robust evidence for program effect. On the other hand, if individuals with greater exposure to the program show greater change in the outcomes, it strengthens the argument that the program led to changes. The exposure in our study would be the reoccurring annual signing of the contract; the stated intention not to use and the active decision to opt out unwanted behaviors. Further strengths in the study are the sample-size, providing adequate statistical power that makes it possible to
detect effect sizes (Cohen, 1988). We also chose not to have categorical or dichotomous outcome variables as there is a risk of losing one to two thirds of the information of the variance of the total sample (Cohen, 1983). The high participant response-rate and low missingness on the outcome measures further strengthens our conclusions.
This study leaves however some unanswered questions. Individuals who have chosen to attend the program and sign contract several years might be those who would refrain from smoking and drinking and have lower levels of substance use without the contract anyway. We thus need to know who the individuals that choose to sign a contract every year are and what characterizes them. What predicts that those who report having contract all the years have had it? More importantly, who are the individuals declining participation in the
program? If we find out we might do better in reaching them before developing more costly and adverse problems.
Conclusion
Our findings show that individuals who were less exposed to and engaged in the SMART contract program developed significantly higher levels of substance use throughout the study, compared to those who were more exposed to the program. This suggests that the SMART program have some preventive effects on adolescent substance use. The current findings are not conclusive and future research is needed to reach more robust conclusions about the effectiveness of the SMART contract strategy. Implications of these findings for practice are the support to actors who work with this form of contract-activity to continue, but also to direct efforts on engaging more students to write contracts during several years.
Botvin, G.J., & Griffin, K.W. (2007). School-based programmes to prevent alcohol, tobacco and other drug use. International Review of Psychiatry, 19(6), 607-615. doi:
10.1080/09540260701797753
Catalano, R.F., Fagan, A.F., Gavin, L.E., Greenberg, M.T., Irwin, E.C., Ross, D.A., & Shek, D.T.L. (2012). Worldwide application of prevention science in adolescent health. The Lancet, 379(9826), 1653-1664. doi: 10.1016/S0140-6736(12)60238-4
Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7(3), 249-253. doi: 10.1177/014662168300700301
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale NJ: Lawrence Erlbaum Associates.
Cuijpers, P. (2002). Effective ingredients of school-based drug prevention programs: A systematic review. Addictive Behaviors, 27(6), 1009-1023. doi: 10.1016/S0306-4603(02)00295-2
Dawson, D.A., Goldstein, R.B., Chou, S.P., Ruan, W.J., & Grant, B.F. (2009). Age at first drink and the first incidence of adult-onset DSM-IV alcohol use disorders. Alcohol Clinical Experimental Research, 32(12), 2149-2160. doi:
10.1111/j.1530.0277.2008.00806.x.
DeWit, D.J., Adlaf, E.M., Offord, D.R., & Ogborne, A.C. (2000). Age at first alcohol use: a risk factor for the development of alcohol disorders. American Journal of Psychiatry, 157(5), 745-750.
Edvardsson, I., Lendahls, L., Andersson, T., & Ejlertsson G. (2012). The social environment is most important for not using snus or smoking among adolescents. Health, 4(12), 1247-1255. doi: 10.4236/health.2012.412184
Eriksson, C., Geidne, S., Larsson, M., & Pettersson, C. (2010). Med kraft och vilja. Alkohol- och drogförebyggande arbete inom Socialstyrelsens stöd till frivilligorganisationer 2003 – 2009. Studier i folkhälsovetenskap, Örebro universitet, 2010:1.
Ezzati M, Lopez A.D. Estimates of global mortality attributable to smoking in 2000. Lancet 2003; 362(9387):847-852.
Ezzati, M., Lopez, A., Rodgers, A., Vander Hoorn, S., & Murray, C. (2002). Selected major risk factors and global and regional burden of disease. Lancet, 360(9343), 1347-1360. doi:
Flay, B.R., Biglan, A., Boruch, R.F., Castro, F.G., Gottfredson, D., Kellam, S… & Ji, P, (2005). Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prevention Science, 6(3), 151-175. doi: 10.1007/s1121-005-5553-y
García-Rodríguez, O., Blanco, C., Wall, M.M., Wang, S., Jin, C.J., & Kendler, K.S. (2014). Toward a comprehensive developmental model of smoking initiation and nicotine dependence. Drug and alcohol dependence, 144(), 160-169. doi:
10.1016/j.drugalcdep.2014.09.002
Griffin, K., Bang, H., & Botvin, G. (2010). Age of alcohol and marijuana use onset predicts weekly substance use and related psychosocial problems during young adulthood. Journal of Substance Use, 15(3), 174-183. doi: 10.3109/14659890903013109 Hamburger, M.E., Basile, K.C., & Vivolo, A.M. (2011). Measuring bullying victimization,
perpetration, and bystander experiences: a compendium of assessment tools. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Injury
Prevention and Control. Retrieved from:
Hellström, L. (2015). Measuring peer victimization and school leadership – a study of definitions, measurement methods and associations with psychosomatic health. (Doctoral dissertation). Retrieved from Open Access DiVA. (urn:nbn:se:kau:diva-35192)
Hill, P.L., Allemand, M., Grob, S.Z., Peng, A., Morgenthaler, C., & Käppler, C. (2013). Longitudinal relations between personality traits and aspects of identity formation during adolescence. Journal of adolescence, 36(2), 413-421. doi:
10.1016/j.adolescence.2013.01.003
Hingson, R., & Zha, W. (2009). Age of drinking onset, alcohol use disorders, frequent heavy drinking, and unintentionally injuring oneself and others after drinking. Pediatrics, 123(6), 1477-1484. doi:10.1542/peds.2008-2176
Isensee, B., & Hanewinkel, R. (2012). Meta-analysis on the effects of the smoke-free class competition on smoking prevention in adolescents. European Addiction Research, 18, 110-115. doi: 10.1159/000335085.
Isensee, B., Morgenstern, M., Stoolmiller, M., Maruska, K., Sargent, J.D., & Hanewinkel, R. (2012). Effects of smokefree class competition 1 year after the end of intervention: a cluster randomised controlled trial. J Epidemiol Community Health, 66(4), 334-341. doi: 10.1136/jech.2009.107490
Joffer, J., Burell, G., Bergström, E., Stenlulnd, H., Sjörs, L., & Jerdén, L. (2014). Predictors of smoking among Swedish adolescents. BMC Public Health, 14(1), 1035-1051. doi: 10.1186/1471-2458-14-1296
Johnston, V., Liberato, S., Thomas, D. (2012). Incentives for preventing smoking in children and adolescents. Cochrane Database of Systematic Reviews, 10, CD008645. doi: 10.1002/14651858.CD008645.pub2
Jung, M., & Chung, D. (2012). Evidence of social contextual effects on adolescent smoking in South Korea. Asia Pacific Journal of Public Health, 25(3), 260-270. doi:
10.1177/1010539512461667
Kandel, D.B. & Logan, J.A. (1984). Patterns of drug use from adolescence to young adulthood: I. Periods of risk for initiation, continued use, and discontinuation. American journal of public health, 74(7), 660-666. doi: 10.2105/AJPH.74.7.660
Kandel, D.B., Yamaguchi, K., Chen K. (1992). Stages of progression in drug involvement from adolescence to adulthood: Further evidence for the gateway theory. Journal of Studies on Alcohol, 53(5), 447-457. doi: http://dx.doi.org/10.15288/jsa.1992.53.447 Kendler, K.S., Myers, J., Damaj, M.I., & Chen X. (2013). Early smoking onset and risk for
subsequent nicotine dependence: a monozygotic co-twin control study. The American Journal of Psychiatry, 170(4), 408-413. doi:
http://dx.doi.org/10.1176/appi.ajp.2012.1203021
Kroger, J., Martinussen, M., & Marcia, J.E. (2010). Identity status change during adolescence and young adulthood: A meta-analysis. Journal of adolescence, 33(5), 683-698. doi: 10.1016/j.adolescence.2009.11.002
Lipperman-Kreda, S., Grube, J.W., & Pascall, M.J. (2010). Community norms, enforcement of minimum legal drinking age laws, personal beliefs and underage drinking: an explanatory model. Journal of Community Health, 35(3), 249-257. doi:
Magid, V., & Moreland, A.D. (2014). The role of substance use initiation in adolescent
development of subsequent substance-related problems. Journal of Child & Adolescent Substance Abuse, 23(2), 78-86. doi: 10.1080/1067828X.2012.748595
Meeus, W. (2011). The study of adolescent identity formation 2000-2010: A review of longitudinal research. Journal of research on adolescence, 21(1), 75-94. doi: 10.1111/j.1532-7795.2010.00716.x
Morgenstern, M., Sargent, J.D., Engels, R.C.M.E., Florek, E., & Hanewinkel, R. (2013). Smoking in European adolescents: Relations between media influences, family affluence, and migration background. Addictive Behaviors, 38(10), 2589-2595. doi: 10.1016/j.addbeh.2013.06.008
Moss, H.B., Chen, C.M., Yi, H. (2014). Early adolescent patterns of alcohol, cigarettes, and marijuana polysubstance use and young adult substance use outcomes in a nationally representative sample. Drug & Alcohol Dependence, 136, 51-61
Mullis, I.V.S., Martin, M.O., Foy, P., & Arora, A. (2012). TIMSS 2011 International Results in Mathematics. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College Chestnut Hill, MA, USA and International Association for the Evaluation of Educational Achievement (IEA) IEA Secretariat Amsterdam, the Netherlands.
Nash, S.G., McQueen, A., & Bray, J.H. (2005). Pathways to adolescent alcohol use: family environment, peer influence and parental expectations. Journal of adolescent health, 37(1), 19-28. doi: 10.1016/j.jadohealth.2004.06.004
Nation, M., Crusto, C., Wandersman, A., Kumpfer, K.L., Seybolt, D., Morrisey-Kane, E., & Davino, K. (2003). What works in prevention: principles of effective prevention programs. American Psychologist, 58(6-7), 449-456. doi:
Nilsson, M. (2009). Promoting health in adolescents – preventing the use of tobacco. (Doctoral dissertation). (URN: urn:nbn:se:umu:diva-21239)
Nilsson, M., Stenlund, H., Bergström, E., Weinehall, L., & Janlert, U. (2006). It takes two: Reducing smoking uptake through sustainable adolescent-adult partnership. Journal of Adolescent Health, 39(6), 880-886. doi: 10.1016/j.jadohealth.2006.07.004
Pasch K.E., Perry, C.L., Stigler, M.H., & Komro, K.A. (2009). Sixth grade students who use alcohol: do we need primary prevention programs for “tweens”? Health Education & Behavior, 36(4), 673–695. doi: 10.1177/1090198107308374
Pettersson, C., Özdemir, M., & Eriksson, C. (2011). Effects of a parental program for preventing underage drinking – the NGO program strong and clear. BMC Public Health, 11, 251. doi: 10.1186/1471-2458-11-251
Pinilla, J., González, B., Barber, P., & Santana, Y. (2002). Smoking in young adolescents: an approach with multilevel discrete choice models. Public health policy and practice, 56(3), 227-232. doi: 10.1136/jech.56.3.227
Pitkänen, T., Lyyra, A.L., & Pulkkinen, L. (2005). Age of onset of drinking and the use of alcohol in adulthood: a follow-up study from age 8-42 for females and males. Addiction, 100(5), 652-661. doi: 10.1111/j.1360-0443.2005.01053.x
Post, A., Gilljam, H., Rosendahl, I., Meurling, L., Bremberg, S., & Galanti, M.R. (2005). Validity of self reports in a cohort of Swedish adolescent smokers and smokeless tobacco (snus) users. Tobacco Control, 14(2), 114-117. doi: 10.1136/tc.2004.008789
Public Health Agency of Sweden. (2014). National Report (2013 data) to the EMCDDA. Retrieved from:
Rehm, J., Mathers, C., Popova, S., Thavorncharoensap, M., Teerawattananon, Y., & Patra., J. (2009). Global burden of disease and injury and economic cost attributable to alcohol use and alcohol-use disorders. The Lancet, 373(9682), 2223-2233. doi:
10.1016/S0140-6736(09)60746-7
Rehm, J., Taylor, B., & Room, R. (2006). Global burden of disease from alcohol, illicit drugs and tobacco. Drug and alcohol review, 25(), 503-513. doi:
10.1080/09595230600944453
Sargent, J., & Hanewinkel, R. (2009). Comparing the effects of entertainment media and tobacco marketing on youth smoking in Germany. Addiction, 104(5), 815-823. doi:10.1111/j.1360-0443.2009.02542.x
Skolverket. (2014). Undervisning om alkohol, narkotika, dopning och tobak (ANDT) – en praktiknära litteraturgenomgång. Retrieved from:
www.skolverket.se/publikationer?id=3325 SMART. (2015). Retrieved from www.smart.org.se
Steinberg, L.D. (2014). Adolescence. New York: McGraw-Hill.
Stigler, M.H., Neusel, E., & Perry, C.K. (2011). School-based programs to prevent and reduce alcohol use among youth. Alcohol Research & Health, 34(2), 157-162.
Ström, H.K., Adolfsen, F., Fossum, S., Kaiser, S., & Martinussen, M. (2014). Effectiveness of school-based preventive interventions on adolescent alcohol use: a meta-analysis of randomized controlled trials. Substance Abuse Treatment, Prevention, and Policy, 9(48), doi: 10.11.86/1747-597X-9-48
Swedish Government (2010). A cohesive strategy for alcohol, narcotic drugs, doping and tobacco (ANDT) policy. A summarized version of Government Bill 2010/11:47. Stockholm: Ministry of Social Affairs.
Tanti, C., Stukas, A.A., Halloran, M.J., & Foddy, M. (2011). Social identity change: Shifts in social identity during adolescence. Journal of adolescence, 34(3), 555-567. doi: 10.1016/j.adolescence.2010.05.012
Tobler, N. S., Roona, M. R., Ochshorn, P., Marshall, D. G., Streke, A. V., & Stackpole, K. M. (2000). School-based adolescent drug prevention programs: 1998 meta-analysis. The Journal of Primary Prevention, 20(4), 275–336.
Trucco, E.M., Colder, C.R., & Wieczorek, W.F. (2011). Vulnerability to peer influence: A moderated mediation study of early adolescent alcohol use initiation. Addictive Behaviors, 36(7), 729-736. doi: 10.1016/j.addbeh.2011.02.008
Vartiainen, E., Saukko, A., Paavola, M., & Vertio, H. (1996). ”No smoking class”
competitions in Finland: their value in delaying the onset of smoking in adolescence. Health Promotion International, 11(3), 189-192. doi: 10.1093/heapro/11.3.189
Wennberg, P., Gustafsson, P.E., Dunstan, D.W., Wennberg, M. & Hammarström, A. (2013). Televison viewing and low leisure-time physical activity in adolescence independently predict the metabolic syndrome in mid-adulthood. Diabetes Care, 36(7), 2090-7. doi: 10.2337/dc12-1948.
Wiborg, G., & Hanewinkel, Reiner (2002). Effectiveness of the “Smoke-Free Class
Competition” in Delaying the Onset of Smoking in Adolescence. Preventive Medicine, 35: 241-249. doi: 10.1006/pmed.2002.1071
Windle, M., Spear, L.P., Fuligni, A.J., Angold, A., Brown, J.D., Pine, D.… Dahl, R.E. (2008). Transitions Into Underage and Problem Drinking: Summary of Developmental
Processes and Mechanisms: Ages 10–15. Pediatrics, 121(Suppl 4), S273–S289. doi:10.1542/peds.2007-2243C
Yoshino, A. (2012). The relationship between self-concept and achievement in TIMSS 2007: A comparison between American and Japanese students. International Review of Education 58(2), 199-219. doi: 10.1007/s11159-012-9283-7
Özdemir, M. (in press). How much do we know about the long-term effectiveness of parenting programs? Advances, shortcomings, and future directions. Journal of Children’s Services.
