Nurse Education in Practice 47 (2020) 102841
Available online 16 July 2020
1471-5953/© 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Clinical education
Translation and further validation of a global rating scale for the assessment
of clinical competence in prehospital emergency care
Anders Bremer
a,b,c,*, Magnus Andersson Hagiwara
a,b, Walter Tavares
d,e,f, Heikki Paakkonen
g,
Patrik Nystr¨om
g, Henrik Andersson
a,baPreHospen – Centre for Prehospital Research, University of Borås, Sweden bFaculty of Caring Science, Work Life and Social Welfare, University of Borås, Sweden cFaculty of Health and Life Sciences, Linnaeus University, V¨axj¨o, Sweden
dThe Wilson Centre, Department of Medicine, University of Toronto/University Health Network, Toronto, Canada
ePost-MD Education (Post-Graduate Medical Education/Continued Professional Development), University of Toronto, Toronto, Canada fParamedic and Senior Services, Community and Health Services Department, Regional Municipality of York, Newmarket, ON, Canada gDepartment of Health and Welfare, Arcada University of Applied Sciences, Helsinki, Finland
A R T I C L E I N F O
Keywords:
Ambulance nurses Clinical competence Educational measurement Global rating scale
A B S T R A C T
Global rating scales are useful to assess clinical competence at a general level based on specific word dimensions. The aim of this study was to translate and culturally adapt the Paramedic Global Rating Scale, and to contribute validity evidence and instrument usefulness in training results and clinical competence assessments of students undergoing training to become ambulance nurses and paramedics at Swedish and Finnish universities. The study included translation, expert review and inter-rater reliability (IRR) tests. The scale was translated and culturally adapted to clinical and educational settings in both countries. A content validity index (CVI) was calculated using eight experts. IRR tests were performed with five registered nurses working as university lecturers, and with six clinicians working as ambulance nurses. They individually rated the same simulated ambulance assignment. Based on the ratings IRR was calculated with intra-class correlation (ICC). The scale showed excellent CVI for items and scale. The ICC indicated substantial agreement in the group of lecturers and a high degree of agree-ment in the group of clinicians. This study provides validity evidence for a Swedish version of the scale, sup-porting its use in measuring clinical competence among students undergoing training to become ambulance nurses and paramedics.
1. Introduction
Providing prehospital emergency care involves managing acute, ur-gent and non-emergency needs of patients in uncontrolled and
some-times adverse environments, often with limited resources (Bremer,
2016; Elmqvist et al., 2008). Emergency Medical Services (EMS) personnel encounter care situations with chaotic and dangerous ele-ments of unique complexity, facing unexpected challenges, with sudden
changes in pace and disruption of the care relationship (Bremer et al.,
2012). EMS personnel also encounter situations with a variety of ethical
problems (Becker et al., 2013; Bremer et al., 2015; Sandman and
Nordmark, 2006). Hence, decision-making is complex, difficult and
often creates pressure and unease for personnel (Hagiwara et al., 2013;
Sandman and Nordmark, 2006). All these factors make prehospital care
a high-risk activity from a patient safety perspective (Bigham et al.,
2012; Colld´en Benneck and Bremer, 2019).
Regardless of EMS system and personnel’s educational background, the education and training of EMS personnel is essential to attain acceptable clinical competence to protect patient safety and optimize patient outcome in both high-stress and low-stress care situations (LeBlanc et al., 2012). Kane defined competence as “the degree to which an individual can use the knowledge, skills, and judgment associated with the profession to perform effectively in a domain of possible
en-counters defining the scope of professional practice (Kane, 1992,
p.166).” Clinical competence refers to how competence is reflected in
clinical practice (Eurat, 1998).
* Corresponding author. Faculty of Health and Life Sciences, Linnaeus University, SE-351 95, V¨axj¨o, Sweden.
E-mail addresses: anders.bremer@lnu.se (A. Bremer), magnus.hagiwara@hb.se (M. Andersson Hagiwara), walter.tavares@utoronto.ca (W. Tavares), heikki. paakkonen@arcada.fi (H. Paakkonen), patrik.nystrom@safetyfactors.fi (P. Nystr¨om), henrik.andersson@hb.se (H. Andersson).
Contents lists available at ScienceDirect
Nurse Education in Practice
journal homepage: www.elsevier.com/locate/issn/14715953
https://doi.org/10.1016/j.nepr.2020.102841
The expected level of EMS personnel’s clinical competence is high and has grown due to increased demands on accurate clinical
assess-ments and changed competence requireassess-ments (Williams, 2012). Such
demands include an ability to choose appropriate care pathways and
level of care (Zorab et al., 2015), decision-making on non-conveyance
(Lederman et al., 2019), and decision-making to convey older people
to the emergency department (Oosterwold et al., 2018). Regardless of
the changing requirements, a variety of desirable competencies remain, such as the ability to lead, communicate with patients and their families, carry out assessments and measures of patient care, and use available
technical equipment (Holmberg et al., 2017; Wihlborg et al., 2014).
Ultimately, however, both medical and nursing skills are needed to safely complete patient assessments, plan adequate care measures, and
evaluate the effects of implemented healthcare measures (Holmberg
et al., 2017; Houghton and Gray, 2010). All this means that clinical competencies of EMS personnel include abilities, knowledge and in-sights such as being able to handle a variety of events, and
under-standing the consequences of actions planned or taken (Norfolk and
Niroshan, 2013).
1.1. Assessment of student performance
The procedures for determining competence vary. Practical and/or oral examinations are used in academic programs, often by scoring on
skill sheets when evaluating psychomotor skills (Martin et al., 2012).
Task-specific checklists describing performance expectations and
crit-ical errors are often used for EMS professions (Regener, 2005). However,
it may be challenging to use skill sheets or checklists as the EMS per-sonnel’s clinical competence includes both technical and non-technical
elements as well as variations in care processes (Tavares et al., 2012).
In North America, student performance during the education of EMS students is assessed with checklists when evaluating, for example,
iso-lated procedural skills (Norfolk and Niroshan, 2013; Tavares et al.,
2012). In Sweden, assessments are usually performed with clinical
ed-ucation (AssCE) tools and with objective, structured clinical
examina-tion (OSCE) (L¨ofmark and Mårtensson, 2017; Ruesseler et al., 2010). In
Finland, EMS student performance at eight universities with emergency care BSc programs is usually assessed with locally adapted but not fully validated versions of the Paramedic Global Rating Scale (PGRS). Moreover, in schools with EMT programs, checklists and skill sheets are the preferred method of assessment.
1.2. The Paramedic Global Rating Scale
Assessment of clinical competence should be educational, meaning that for example, students undergoing training to become ambulance nurses and paramedics should both learn from tests and receive
feed-back to develop their competencies (Wass et al., 2001). Simultaneously,
a problem is that the assessor in clinical practice or education has lacked appropriate and contextual assessment tools for assessing EMS personnel and EMS students. Global rating scales (GRS) can be used to assess different levels of clinical competence at a general level, based on specific assessment areas compared to, for example, a checklist that mainly answers if an observable action has been performed correctly or
not (Dankbaar et al., 2014; Ilgren et al., 2015; Swanson and van der
Vleuten, 2013). However, the literature indicates that checklists have
poor validity and reliability for assessing skills (Dankbaar et al., 2014)
and that GRS capture nuanced elements of expertise better (Tavares
et al., 2014). To enable a more desirable tool to evaluate paramedic clinical competence, including both technical and non-technical skills, Tavares and colleagues developed and contributed validity evidence for
a seven-dimension, seven-point adjectival GRS (Fig. 1).
The evidence included item analyses, discrimination between levels of performance, and the application of a cut score strategy that
sup-ported prediction (Tavares et al., 2012). Subsequent research provided
further validity evidence including demonstrating how simulation-based assessments with the GRS were associated with mea-surements of paramedics providing care to real patients. Five of the seven GRS dimensions were found to represent the construct of
perfor-mance in real clinical contexts with real patients (Tavares et al., 2014).
Finally, validity evidence for a high-stake, simulation-based assessment strategy was applied using a validity framework in an OSCE. Results indicated that the evidence for scoring, generalization and implications supports the use of simulation-based assessments as a certifying exam, but also suggested that exam scores did not predict clinical error rates (Tavares et al., 2018). Despite careful development of the PGRS, there may be limitations in simply translating and using the instrument in Swedish and Finnish contexts, given the cultural differences that may exist. For example, given that Sweden and Finland have a significant proportion of RNs in the EMS, there may be differences in the view of nursing sciences and medicine compared to the EMS in Canada, which is dominated by paramedics. Consequently, there may be potential dif-ferences in what should constitute important components of clinical competence. In addition, the English version of the PGRS has so far been used in Sweden and Finland, both in the EMS and in ambulance nursing education. This is unsatisfactory given the subjective and different
interpretations this entails.
The aim of this study was therefore to translate and culturally adapt the PGRS, and to contribute validity evidence and instrument usefulness in training results and clinical competence assessments of students un-dergoing training to become ambulance nurses and paramedics at Swedish and Finnish universities.
2. Methods 2.1. Study design
The study design included translation, expert review and inter-rater reliability (IRR) tests. The translation followed recommended processes
conducted by certified translators (Beaton et al., 2000; Maneesriwongul
and Dixon, 2004; Sousa and Rojjanasrirat, 2011). Regional and national experts reviewed the original PGRS and the translations in relation to the content of the PGRS, including a content validity index (CVI). Although the cultural differences between EMS organizations in Canada, Sweden and Finland are believed to be quite small, it was nevertheless considered valuable to consider any differences that may have been expressed in language (semantic, conceptual, experiential, and idio-matic) and/or content of clinical competence in the PGRS. Finally, IRR
tests as recommended by Hallgren (2012) were performed. The tests
included university lecturers and clinical supervisors in the EMS. This study conforms to the ethical principles for medical research
involving human subjects outlined in the Declaration of Helsinki (World
Medical Association, 2018) and adheres to Swedish and Finnish laws and regulations concerning research, informed consent and confidenti-ality. The participants received oral information about the study before voluntarily giving their informed consent to participate. No one chose to refrain from participation.
2.2. Settings
This study was conducted at one Swedish university and one Finnish University of Applied Sciences (UAS). The Swedish university is located in a region with 1.7 million inhabitants where the educational specialist master’s degree program for ambulance nurses is available for registered nurses (RNs). The Finnish university is located in a region with 1.7
million inhabitants (Statistics Finland, 2018). The university offers a
bachelor’s degree program to prospective paramedics, combined with
RN training. All teaching at the Finnish university is in Swedish (Arcada,
2018).
The Swedish EMS is comprised of mostly ambulances staffed by specialist RNs, RNs and EMTs, always with at least one RN. The pro-portion of RNs who are specialist-trained ambulance nurses varies be-tween EMS organizations and represents 20–80% of the nursing staff. The proportion of EMTs varies between 15 and 35% of the total EMS
staff (Bremer, 2016; Langhelle et al., 2004). Ambulance nurses and other
specialist nurses have completed 4-year university training programs including three years to become an RN and one year for specialization. EMTs usually have 2-year high school training as an assistant nurse, which is supplemented by a 6 months to 1-year EMS specialization (Bremer, 2016).
The Finnish EMS varies in the level of care, due to the independence of each municipality. The basic ambulance level mainly employs fire-fighters as EMTs, educated at the national Emergency Services College. They have 18 months’ education, a third of which is related to the EMS. The advanced level employs nurses and paramedics with 4 years of
university training (Langhelle et al., 2004; Krüger et al., 2010).
2.3. Translation process
The translation and cultural adaptation process involved translation from source language, blind back-translation to the target language, translators’ agreement of translation synthesis, and content expert
reviews (Beaton et al., 2000; Maneesriwongul and Dixon, 2004; Sousa
and Rojjanasrirat, 2011).
Two bilingual and certified translators who were fluent in Swedish and English made the translations of the original PGRS in the source language (SL) English to the target language (TL) Swedish, and vice versa. The native countries of the translators were the United States of America and the United Kingdom respectively. Both translators were experienced in healthcare research and instrument construction, and knowledgeable on cultural nuances between North America and Northern Europe and linguistic nuances in Swedish and English. First, one of the translators independently translated the original English/ Canadian PGRS to Swedish. Second, the other translator independently made a back-translation of this TL version to English (BTL). In the third step, both translators worked together, comparing and analysing all three versions; SL, TL and BTL until consensus was reached in one synthesized and preliminary version in Swedish (P-TL). In this process, the translators were forced to agree on which concept is linguistically the most correct, and also take into account the meaning of the concept in a Swedish healthcare context.
2.4. Expert reviews and cultural adaptation
Four expert reviews with two expert panels were performed; one regional panel and one national panel. The regional panel consisted of four male experts in prehospital emergency care, education and simu-lation. The national panel consisted of eight experts from the Network for Education of Ambulance Nurses. These experts, four men and four women, were RNs with clinical experience from the EMS and educa-tional experience from six of the 11 Swedish specialist master’s degree programs for ambulance nurses.
First, before the translations, the regional experts assessed the face validity of the PGRS, i.e. whether the SL version of the scale seemed to measure its intended areas of clinical competence and if these areas were culturally adaptable, reasonable, relevant and useful in the EMS and educational contexts in Sweden and Finland. This initial form of validity is generally argued to be insufficient, but it was considered important in this context to determine whether the instrument was at all appropriate for use in the nurse-dominated EMS and the context of university edu-cation in nursing for the purposes of summative assessments, and if it would provide a valuable complement to the assessment methods already used. We recognized this initial “test” as important to initial
stakeholder buy-in (Fig. 2).
Second, when the translators had agreed on the P-TL version, the regional experts performed a review including conceptual revisions and considerations of a potential cultural adaptation of the P-TL. In support of this work, the experts had access to the source language, target lan-guage and back-translations versions of the scale. This step resulted in the P-TL2 version.
Third, the eight national experts made their assessment of the P-TL2 version. The experts’ task was to increase instrument clarity, compre-hensiveness and cultural relevance by specifying the degree of agree-ment in a CVI of the seven dimensions (items) of the PGRS. The experts were asked to make ratings of three text sections of the PGRS: 1) the instructions, rating labels and definitions, 2) the seven dimensions, and
3) the overall clinical performance (see Fig. 1). The experts were also
invited to make further recommendations and to comment on the
in-strument. As recommended by Polit and Beck (2006), the CVI of the
target language version was calculated on item level (I-CVI), presenting the range of I-CVI values. A scale with excellent content validity should
meet Lynn’s (1986) criteria (here I-CVI ≥0.78 with 6–10 experts). The
experts’ ratings were made using a 4-point scale after being informed to rate 1 for ‘not relevant,’ 2 for ‘somewhat relevant,’ 3 for ‘quite relevant,’
and 4 for ‘highly relevant’ (Davis, 1992). The result from this scale was
then dichotomized for each item, implying that the number of experts giving a rating of 1 or 2 was transformed to one group of ‘not relevant,’ while 3 or 4 was transformed to ‘relevant,’ and finally divided by the
total number of experts. Content validity for scale (S-CVI) followed Polit and Beck’s (2006) recommendations of using average congruity (S-CVI/Ave) with 0.90 as minimum value. For the sake of clarity, CVI-S is also presented as universal agreement (UA) where 0.80 is considered a reasonable criterion.
Fourth, the regional experts made an overall review before the IRR tests. This implied that a final analysis was performed, based on the translators’ views, the results of the CVI, comments made by the na-tional experts, and the regional experts’ own experience of using the English version of the PGRS in teaching and simulation.
Finally, based on translation, cultural adaptation and expert reviews, the researchers decided on a final target language (F-TL) version of the Swedish PGRS, intended for the IRR tests.
2.5. Inter-rater reliability analysis
The IRR analysis was performed on the final instrument (F-TL) in two different groups. The first IRR test was performed using university lec-turers and the second using EMS clinicians, based on the same scenario in both groups.
The IRR analysis process was started with inter-rater training on two occasions, one for the lecturers and one for the clinicians. The training started with information on the instrument. Afterwards, lecturers indi-vidually watched a simulated ambulance mission on digital video, while
the clinicians watched the same mission together. The scenario was from an experiment where an ambulance team assessed and treated a patient simulator with medical symptoms. The scenario included driving in a simulator to the address, assessment and treatment in an apartment, transport to the ambulance, assessment and treatment on route, and hand-over to the physician at the ED. Thus, this was done in a highly
immersive simulation environment (Engstr¨om et al., 2016). During the
training session the raters discussed the rating according to the in-strument’s different dimensions (items), rating labels, definitions, and the overall clinical performance in the instrument. The training session lasted for 30 min.
After the training, IRR tests were performed in the two separate groups of raters. This time the raters were asked individually to rate another scenario than the one previously used in the training session. This scenario was independently rated without any communication between the raters. The IRR tests were performed during a period of one week.
The first group of raters consisted of five lecturers. Four of them were Swedish ambulance nurses with ≥10 years of experience from the field, working with education in Swedish universities. The fifth was an experienced emergency department RN. They (n = 4) had a mean experience from the EMS of 23.0 years (SD = 11.3), mean experience as university lecturers (n = 5) of 4.6 years (SD = 2.9), and their mean age (n = 5) was 51.2 years old (SD = 11.1). All five raters were men. Two had earlier experience from education and evaluation in ambulance organizations and three had experience from university education, simulation and evaluation. Four of the lecturers/raters had a PhD degree and the fifth had an MSc degree.
The second group consisted of six clinicians. They were all experi-enced ambulance nurses, still clinically active in the EMS. They had a mean experience from the EMS of 10.2 years (SD = 4.2), their mean age was 40.0 years old (SD = 7.6), and two were female and four were male. Two of the raters had earlier experience of using PGRS (SL) in rating EMS clinicians’ clinical performance.
2.6. Data analysis
The results of the study were analysed using the statistical program SPSS© version 20 (IBM, Armonk, NY, USA). Descriptive statistics pre-sented frequencies of the participants’ age, gender, years of practice in the EMS, and profession.
The raters scored one video on a 7-point Likert scale. The IRR among the raters was assessed using a two-way mixed, consistency, average-
measures intra-class correlation (ICC) (Hallgren, 2012). An ICC value
over 0.60 is considered a good IRR (Cicchetti, 1994).
3. Results
3.1. Translation process
The P-TL version provided by the certified translators showed a few differences in word choices and formulations when compared to the TL version. In the SL version, the differences were traced to the words “poor” and “highly” in the rating labels, the word “compromised” in the definitions of rating label 1 (unsafe) and 2 (unsatisfactory), and the word “history gathering” as the name of one of the seven PGRS dimensions.
Table 1 shows some of the differences between SL and BTL, which the translators had to take into consideration when they agreed on the P-TL version.
There were more differences between the English versions of the PGRS (SL and BTL) than between the Swedish versions. However, the content of the versions was consistent despite differences in word choice. Prior to the decision on word choice, discussions were held about linguistic nuances of the Swedish words in comparison with the English words proposed in the different versions of the scale. Integrated in these discussions were also considerations of potential cultural differences Fig. 2. The regional and national expert review processes.
that could be reflected in the choice of words. However, the translators’ consideration of potential cultural differences could not be distinguished from the choice of words, but rather lies in the meaning of the words.
3.2. Expert reviews and cultural adaptation
The regional experts’ initial assessment on face validity indicated consensus regarding the content and usefulness of the instrument in the Swedish and Finnish contexts. After the national experts’ valuation of the instrument and comments given by them, the initial assessment of face validity was confirmed.
The regional experts’ review on the P-TL version resulted in some conceptual and language revisions, and cultural adaptation of the P-TL2 version regarding terminology in a Swedish/Finnish EMS context, for example “nursing needs,” “nursing” and “family members.” The first rating label “Unsafe” (SL) was changed to “Security risk” in the P-TL and finally to “Unsure/security risk” in the P-TL2 made by the experts. The second dimension “History gathering” became “Information gathering” while the seventh dimension “Procedural skill” was changed to “Per-formance”. In the texts that explain the seven dimensions, the term “objective” was chosen instead of “requirements” (including inflections of the word), while “competence raising and new examination/exercise” and “with extra support” were chosen instead of “remediation.” The word “progression” was replaced by “next stage of training.” The dimension “Information gathering” resulted from a cultural adaptation due to the need to obtain information about the patient’s need of nursing interventions. Finally, input from the original scale developer regarding the cultural adaptation was obtained and considered.
Comments by the national experts were submitted in connection with the CVI (based on the P-TL2). One expert was concerned by the lack of content and clarity when it came to identification, assessment, action, and evaluation of the patient’s mental, emotional and existential status and needs. Three experts considered that seven assessment levels were too many and one expert argued that the rating label “Marginal” (“Borderline” in SL) was unnecessary.
Additionally, the experts suggested that clear instructions for using
the scale were needed. Therefore, the researchers (AB, MAH and HA) formulated written instructions with ten steps for the use of the PGRS: 1) inform the participants about the conditions for the exam/exercise; 2) start from a documented and well-considered case description; 3) determine the target assessment goals (e.g. based on a course objective); 4) specify what the participant will achieve in each of the seven assessment areas, based on the case description and related to the course or learning objective; 5) encourage the participant to verbally refer to and motivate actions during the exam/exercise; 6) conduct the exam/ exercise; 7) assess the participant’s ability to reach the assessment goal in each of the seven assessment areas; 8) determine a score with one or more values of 1, 2 or 3 as insufficient and require a new exam/exercise; 9) the overall (and approved) clinical competence related to the assessment goal is based on an average of the rating values 4–7; and 10) report results to the participant where strengths and weaknesses in the exam/exercise are discussed using a supportive and reflective approach. The instructions were determined after the researchers had discussed and agreed on formulations and content, specifically adapted to the Swedish and Finnish education systems, based on program curriculum and course or learning objectives. In order to assess the overall clinical performance, based on a single course objective or exercise goal, raters were instructed to calculate the overall performance based on an average of the ratings made for all seven assessment levels.
3.3. Content validity index
The results from the CVI showed excellent content validity for items (0.88–1.00) and scale (Ave, 0.95). For scale validity measured as UA, the
Swedish PGRS was just below (0.75) the strict criterion of 0.80 (Table 2).
3.4. Inter-rater reliability
The resulting ICC for the first IRR group (lecturers) showing sub-stantial agreement, ICC = 0.66, indicates that the instrument is suffi-cient for use by multiple raters. The ICC result for the second IRR group (clinicians) was in the excellent range, ICC = 0.83, indicating that the
raters had a high degree of agreement (Table 3).
4. Discussion
Patients use the EMS for minor to very severe health problems (Bremer, 2016; Tavares et al., 2012). Hence, the task for EMS personnel is to provide care for patients with both urgent and non-urgent
condi-tions (Bremer, 2016). Therefore, a core component is to construct and
develop clinical competencies in both education and clinical practice to ensure quality and safety in the EMS, regardless of patients’ conditions
and needs (Sj¨olin et al., 2015). Hence, testing, making judgments and
giving feedback have become vital in education to make summative and formative assessments of the EMS students’ knowledge, skills and atti-tudes. However, judging whether EMS nursing students are ready for entry-to-practice, or if EMS personnel are maintaining their skills and
knowledge is demanding (Holmberg et al., 2017; Wihlborg et al., 2014).
The dimensions and characteristics of the PGRS enable the use of both clinicians and educators in the assessment of EMS personnel’s and EMS nursing students’ clinical competence. The use of the scale during and after training for an ambulance nurse should be regarded as a clear advantage as it promotes continuity, clarifies expectations regarding clinical competence and bridges students’ transition from university to EMS.
In Sweden and Finland there has been a significant challenge to assess technical and non-technical competencies and to make decisions regarding EMS nursing students’ capability for entry into supervised training and approval of them as fit for work in the EMS. This was one essential reason to validate a Swedish PGRS for measuring clinical competence among EMS nursing students. Another reason was the po-tential risk in using the original PGRS without translation, validation or Table 1
Comparison between the source language (SL) version and the back-translation (BTL) version of the PGRS.
Text sections SL BTL
Dimensions (items) History Gathering Patient history collection Decision Making Decisiveness
Procedural Skill Methodological skills
Rating Label Unsafe Security risk
Poor/Weak Substandard/weak
Marginal Borderline
Definition As required According to requirements
Serious Extensive
Remediation Improvements (support)a
Practice Internship (experience) Progression Next stage of training Cause for concern Troubling
Needed Necessary
Not meet the standard Sub-standard
Standards Requirements
Safe Sure
Independent Supervised (independent)
Concerns Weaknesses
Consistently Consequently Exceeds Surpasses (exceeds) Demonstrates Performs
Enhancing Improved
Be used Serve
Highly Especially
aWords in parentheses = more than one word has been used in the BTL version.
cultural adaptation to Swedish. Hence, our goal was to minimize these threats to validity.
4.1. The Swedish Paramedic Global Rating Scale (PGRS-SW)
Tavares et al. (2012, 2014; 2018) have done extensive and rigorous work on the design and development of the seven-dimensional and seven-point adjectival PGRS with sound psychometric properties. Based on this work, we have focused on three parts as ways of further vali-dating its use in a new context: a) translation of the scale including cultural adaptation to Sweden and Finland; b) validation of the scale content to Swedish/Finnish contexts; and c) assessment of the inter-rater agreement of the PGRS-SW.
The translation and cultural adaptation processes were performed in a rigorous manner. This included a centering process, i.e. recognizing the equal importance of the source language and the target language to enhance a more accurate adaptation and cross-cultural validation of the
translated instrument (Sousa and Rojjanasrirat, 2011). The choice of
translators and expert review members was carefully considered to strengthen instrument validity and to enhance high quality of the translations as well as the adaptation to contextual and cultural aspects,
which in the methodological literature is considered important (Beaton
et al., 2000; Maneesriwongul and Dixon, 2004; Sousa and Rojjanasrirat,
2011). Further, by choosing one American-English translator and one
British-English translator, the effort was to bridge linguistic and cultural
differences between North America (Canada/US) and Northern Europe (Sweden/Finland), and thereby strengthen scale validity. This was achieved in part, but the P-TL version agreed upon by the translators lacked a clear and correct contextual adaptation to the EMS. The experts and researchers in the last two steps before the IRR tests addressed this shortcoming.
Problems with scale ratings that relate to deficiencies in scoring
in-structions for raters have been reported (Aas et al., 2018) and addressed
(Salvi et al., 2008). In our study it was not primarily deficiencies in the scoring instructions regarding how to interpret the definitions of the seven scale dimensions, but rather how to use the PGRS-SW by clearly defined steps aimed to strengthen its use. Since PGRS is a global scale, it would probably be a difficult and possibly limited task to attempt to define precisely how each dimension should be interpreted. Instead, it seems reasonable that each rater defines this based on the specific and predetermined learning and training goals. Consequently, rater vari-ability is unavoidable and should perhaps even be considered desirable in some instances. However, future experiences from raters, students and clinicians from universities, and EMS organizations may reveal whether the recommended supporting instructions should be adjusted. Recognizing that experts’ face validity was insufficient, additional expert reviews and computing a CVI were conducted to provide evi-dence of content validity. The results from the CVI were satisfactory. I- CVI and S-CVI/Ave reached excellent content validity while the S-CVI/ UA indicated good content validity. By achieving good to excellent Table 2
Content Validity Index (CVI) results for the PGRS based on expert assessment by national experts (n = 8).
E1 E2 E3 E4 E5 E6 E7 E8 Number in agreement I-CVIa S-CVI/Aveb S-CVI/UAc
Dimensions (items) Situation Awareness 4 4 4 3 4 4 2 4 7 0.88 History Gathering 4 4 4 3 3 4 3 4 8 1.00 Patient Assessment 4 4 4 4 3 4 3 4 8 1.00 Decision Making 4 4 3 4 4 4 3 4 8 1.00 Resource Utilization 4 4 3 4 4 4 2 4 7 0.88 Communication 4 4 4 4 4 4 3 4 8 1.00 Procedural Skill 4 4 3 4 4 4 2 4 7 0.88
Scale content validity 0.95 0.75
Text sections
PGRS/Candidate/Rater/Case 4 4 3 4 4 4 4 4 8
Rating Label/Definition 3 4 4 3 3 4 2 4 7
Overall Clinical Performance 4 4 4 2 4 4 3 4 7
aExcellent content validity = I-CVI >0.78. b Excellent content validity = S-CVI/Ave >0.90. cExcellent content validity = S-CVI/UA >0.80.
Table 3
Intra-class correlation (ICC) results based on two inter-rater reliability (IRR) groups: lecturers (n = 5) and clinicians (n = 6).
L1 L2 L3 L4 L5 C1 C2 C3 C4 C5 C6 Mdn Q1-Q3a Dimensions (items) Situation Awareness 4 3 3 4 4 6 5 5 5 5 6 5 4–5 History Gathering 4 3 4 5 6 4 5 4 4 5 5 4 4–5 Patient Assessment 5 2 4 5 5 5 6 5 5 5 5 5 5–5 Decision Making 3 2 4 5 5 5 6 5 6 5 6 5 4–6 Resource Utilization 5 4 4 5 5 4 6 5 5 5 5 5 4–5 Communication 4 4 5 5 6 5 5 4 5 4 5 5 4–5 Procedural Skill 5 4 4 5 5 6 5 5 4 4 5 5 4–5 ICC valueb (95% CIc) Lecturers 0.66 (.661) Clinicians 0.74 (.744) aInterquartile range, Q 1–Q3 =quartiles. b Good intra-class correlation = ICC >0.60. cCI as average measures.
values for single items and the scale as a whole, the content in the
Swedish PGRS is considered to be valid. The CVI results meet Polit and
Beck’s (2006) criterion for a scale having excellent content validity when measuring I-CVI and S-CVI/Ave with 6–10 experts. If the CVI/UA method alone had been used for scale validity, the excellent level would not have been reached. However, this method is considered overly
stringent when there are many experts making assessments (Polit and
Beck, 2006). There were no significant differences when looking at the seven single dimensions (items) of the scale. When looking at raters, one of the eight raters differed from the rest by lower ratings compared to the other raters. In regard to CVI/UA, this is described as likely to cause
one expert to disproportionately influence the experts’ agreement (Polit
and Beck, 2006). However, despite this risk, a satisfactory CVI/UA result was achieved.
The intra-class correlation was good (>0.60) in both IRR groups, showing consistency between the raters in each group. However, a dif-ference was found between the groups showing a higher ICC value for clinicians. The explanation for this may be that the two groups have different goals when using a GRS instrument. The clinician’s goal is often to determine if a new colleague is ready for duty and the goal for the university lecturers is often to set a grade on a course. Maybe the latter goal is more complex and to a higher degree at risk for dissenting opinions.
4.2. Limitations
This study has some limitations. One is that no structural pilot testing of the Swedish PGRS was performed among EMS nursing students and/ or personnel. However, during the cultural adaptation process, the re-searchers (AB, MAH and HA) have continuously received feedback from participants in the IRR tests and from clinicians who conducted the tests. Some unstructured evaluation of the PGRS instructions, items and response format has thereby been available and useful to further enhance instrument quality. Another limitation is the absence of a full- scale psychometric testing of the Swedish PGRS. A third limitation may be the way in which we handled the middle option (Marginal) of the 7 Likert-type scale, i.e. we did not address the problem of a “neutral” midpoint despite the fact that the assessment should result in a dichot-omous answer as to whether the student has been approved or failed. Finally, the result might be imprecise due to the small number of par-ticipants included in the various study phases. The strength of this study is the access to regional and national experts in prehospital emergency care and education of EMS students and EMS personnel, which has promoted high quality of the contextual and cultural adaptation of the scale.
5. Conclusions
This study reports on translation, cultural adaptation and further validation of the Canadian PGRS to a Swedish/Finnish context. A structured and carefully planned process for translation, back- translation, expert review, CVI and IRR tests, was conducted and ulti-mately contributed further validity evidence for the Swedish version – PGRS-SW. The scale is considered useful for raters to assess clinical competence in both EMS students and EMS personnel. Despite this conclusion, further item analyses and analysis of the psychometric properties of the PGRS-SW are recommended in order to further enhance scale validity.
Funding source
Two researchers (AB and HA) received educational project funding from the University of Borås, Sweden (2018) for the conduct of the research. The funding body was not involved in any decisions or steps in the research process.
CRediT authorship contribution statement
Anders Bremer: Conceptualization, Data curation, Formal analysis,
Funding acquisition, Investigation, Methodology, Project administra-tion, Validaadministra-tion, Visualizaadministra-tion, Writing - original draft. Magnus
Andersson Hagiwara: Conceptualization, Data curation, Formal
anal-ysis, Investigation, Methodology, Validation, Visualization, Writing - review & editing. Walter Tavares: Conceptualization, Data curation, Validation, Visualization, Writing - review & editing. Heikki
Paakko-nen: Data curation, Investigation, Validation, Visualization, Writing -
review & editing. Patrik Nystr¨om: Data curation, Investigation, Vali-dation, Visualization, Writing - review & editing. Henrik Andersson: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Validation, Visualization, Writing - review & editing.
Declaration of competing interest
We declare that there are no conflicts of interest with regard to the authorship and/or publication of this manuscript. The copyright holder PhD Walter Tavares has given permission to use and translate the PGRS.
Acknowledgments
The authors would like to thank all participating experts and raters in the study and Walter Tavares for the permission to use and translate the Paramedic Global Rating Scale.
References
Aas, I.H.M., Sonesson, O., Torp, S., 2018. A qualitative study of clinicians experience with rating of the Global Assessment of Functioning (GAF) scale. Community Ment.
Health J. 54 (1), 107–116.
Arcada, 2018. Bachelor’s Degree Emergency Care. https://www.arcada.fi/en/bach
elor/emergency-care. (Accessed 6 December 2018).
Beaton, D.E., Bombardier, C., Guillemin, F., Ferraz, M.B., 2000. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine 25 (24),
3186–3191.
Becker, T.K., Gausche-Hill, M., Aswegan, A.L., Baker, E.F., Bookman, K.J., Bradley, R.N., De Lorenzo, R.A., Schoenwetter, D.J., American College of Emergency Physicians’ EMS Committee, 2013. Ethical challenges in emergency medical services:
controversies and recommendations. Prehospital Disaster Med. 28 (5), 488–497.
Bigham, B.L., Buick, J.E., Brooks, S.C., Morrison, M., Shojania, K.G., Morrison, L.J., 2012. Patient safety in emergency medical services: a systematic review of the literature.
Prehosp. Emerg. Care 16 (1), 20–35.
Bremer, A., 2016. Emergency medical services of today. In: Suserud, B.O., Lundberg, L. (Eds.), Prehospital Emergency Care, second ed. Liber, Stockholm, Sweden,
pp. 48–64. In Swedish.
Bremer, A., Dahlberg, K., Sandman, L., 2012. Balancing between closeness and distance: emergency medical services personnel’s experiences of caring for families at out-of-
hospital cardiac arrest and sudden death. Prehospital Disaster Med. 27 (1), 42–52.
Bremer, A., Jim´enez Herrera, M., Axelsson, C., Burjales Marti, D., Sandman, L., Casali, G. L., 2015. Ethical values in the emergency medical services: a pilot study. Nurs. Ethics
22 (8), 928–942.
Cicchetti, D.V., 1994. Guidelines, criteria, and rules of thumb for evaluating normed and
standardized assessment instruments in psychology. Psychol. Assess. 6 (4), 284–290.
Colld´en Benneck, J., Bremer, A., 2019. Registered nurses’ experiences of near misses in ambulance care – a critical incident technique study. Int. Emerg. Nurs. 47, 1–6.
https://doi.org/10.1016/j.ienj.2019.05.002.
Dankbaar, M., Stegers-Jager, K., Baarveld, F., van Merrienboer, J., Norman, G., Rutten, F., van Saase, J., Schult, S., 2014. Assessing the assessment in emergency
care training. PLoS One 9 (12), e114663.
Davis, L.L., 1992. Instrument review: getting the most from your panel of experts. Appl.
Nurs. Res. 5 (4), 194–197.
Elmqvist, C., Fridlund, B., Ekeberg, M., 2008. More than medical treatment: the patient’s
first encounter with prehospital emergency care. Int. Emerg. Nurs. 16 (3), 185–192.
Engstr¨om, H., Andersson Hagiwara, M., Backlund, P., Lebram, M., Lundberg, L., Johannesson, M., Sterner, A., Maurin S¨oderholm, H., 2016. The impact of contextualization on immersion in healthcare simulation. Adv. Simul. 1, 8. https://
doi.org/10.1186/s41077-016-0009-y.
Eurat, M., 1998. Concepts of competence. J. Interprof. Care 12 (2), 127–139.
Hagiwara, M., Suserud, B.O., Jonsson, A., Henricson, M., 2013. Exclusion of context knowledge in the development of prehospital guidelines: results produced by realistic evaluation. Scand. J. Trauma Resuscitation Emerg. Med. 21, 46. https://doi.
org/10.1186/1757-7241-21-46.
Hallgren, K.A., 2012. Computing inter-rater reliability for observational data: an
Holmberg, M., Fagerberg, I., Wahlberg, A.C., 2017. The knowledge desired by emergency medical service managers of their ambulance clinicians – a modified Delphi study. Int. Emerg. Nurs. 34, 23–28. https://doi.org/10.1016/j.ienj.2017.03.007.
Houghton, A.R., Gray, D., 2010. Symptoms and Signs in Clinical Medicine. An
Introduction to Medical Diagnosis, thirteenth ed. Hodder Arnold, London, UK.
Ilgren, J.S., Ma, I.W., Hatala, R., Cook, D.A., 2015. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment.
Med. Educ. 49 (2), 161–173.
Kane, M.T., 1992. The assessment of professional competence. Eval. Health Prof. 15 (2),
163–182.
Krüger, A.J., Skogvoll, E., Castr´en, M., Kurola, J., Lossius, H.M., ScanDoc Phase 1a Study Group, 2010. Scandinavian pre-hospital physician-manned emergency medical
services – same concept across borders? Resuscitaton 81 (4), 427–433.
Langhelle, A., Lossius, H.M., Silfvast, T., Bj¨ornsson, H.M., Lippert, F.K., Ersson, A., Søreide, E., 2004. International EMS systems – the Nordic countries. Resuscitation 61
(1), 9–21.
LeBlanc, V.R., Regehr, C., Tavares, W., Scott, A.K., MacDonald, R., King, K., 2012. The impact of stress on paramedic performance during simulated critical events.
Prehospital Disaster Med. 27 (4), 369–374.
Lederman, J., L¨ofvenmark, C., Dj¨arv, T., Lindstr¨om, V., Elmqvist, C., 2019. Assessing non-conveyed patients in the ambulance service: a phenomenological interview study with Swedish ambulance clinicians. BMJ Open 9, e030203. https://doi.org/
10.1136/bmjopen-2019-030203.
L¨ofmark, A., Mårtensson, G., 2017. Validation of the tool assessment of clinical education (AssCE): a study using Delphi method and clinical experts. Nurse Educ. Today 50,
82–86.
Lynn, M.R., 1986. Determination and quantification of content validity. Nurs. Res. 35
(6), 382–385.
Maneesriwongul, W., Dixon, J.K., 2004. Instrument translation process: a methods
review. J. Adv. Nurs. 48 (2), 175–186.
Martin, M., Hubble, M.W., Hollis, M., Richards, M.E., 2012. Interevaluator reliability of a
mock paramedic practical examination. Prehosp. Emerg. Care 16 (2), 277–283.
Norfolk, T., Niroshan, S., 2013. A comprehensive model for diagnosing the causes of individual medical performance problems: skills, knowledge, internal, past and
external factors (SKIPE). Qual. Prim. Care 21 (5), 315–323.
Oosterwold, J., Sagel, D., Berben, S., Roodbol, P., Broekhuis, M., 2018. Factors influencing the decision to convey or not to convey elderly people to the emergency department after emergency ambulance attendance: a systematic mixed studies review. BMJ Open 8, e021732. https://doi.org/10.1136/bmjopen-2018-021732.
Polit, D.F., Beck, C.T., 2006. The content validity index: are you sure you know what’s
being reported? Critique and recommendations. Res. Nurs. Health 29 (5), 489–497.
Regener, H., 2005. A proposal for student assessment in paramedic education. Med.
Teach. 27 (3), 234–241.
Ruesseler, M., Weinlich, M., Byhahn, C., Müller, M.P., Jünger, J., Marzi, I., Walcher, F., 2010. Increased authenticity in practical assessment using emergency case OSCE
stations. Adv. Health Sci. Educ. Theor. Pract. 15 (1), 81–95.
Salvi, F., Miller, M.D., Grilli, A., Giorgi, R., Towers, A.L., Morichi, V., Spazzafumo, L., Mancinelli, L., Espinosa, E., Rappelli, A., Dessì-Fulgheri, P., 2008. A manual of guidelines to score the modified Cumulative Illness Rating Scale and its validation in
acute hospitalized elderly patients. J. Am. Geriatr. Soc. 56 (10), 1926–1931.
Sandman, L., Nordmark, A., 2006. Ethical conflicts in prehospital emergency care. Nurs.
Ethics 13 (6), 592–607.
Sj¨olin, H., Lindstr¨om, V., Hult, H., Kurland, L., 2015. What an ambulance nurse needs to know: a content analysis of curricula in the specialist nursing programme in
prehospital emergency care. Int. Emerg. Nurs. 23 (2), 127–132.
Sousa, V.D., Rojjanasrirat, W., 2011. Translation, adaptation and validation of instruments or scales for use in cross-cultural healthcare research: a clear and user-
friendly guideline. J. Eval. Clin. Pract. 17 (2), 268–274.
Statistics Finland, 2018. Population. https://www.stat.fi/til/vrm_en.html. (Accessed 6 December 2018).
Swanson, D.B., van der Vleuten, C.P., 2013. Assessment of clinical skills with standardized patients: state of the art revisited. Teach. Learn. Med. 25 (Suppl. 1),
S17–S25.
Tavares, W., Boet, S., Theriault, R., Mallette, T., Eva, K.W., 2012. Global rating scale for the assessment of paramedic clinical competence. Prehop. Emerg. Care. 17 (1),
57–67.
Tavares, W., Brydges, R., Myre, P., Prpic, J., Turner, L., Yelle, R., Huiskamp, M., 2018. Applying Kane’s validity framework to a simulation based assessment of clinical
competence. Adv. Health Sci. Educ. Theor. Pract. 23 (2), 323–338.
Tavares, W., LeBlanc, V.R., Mausz, J., Sun, V., Eva, K.W., 2014. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehosp. Emerg.
Care 18 (1), 116–122.
Wass, V., Van der Vleuten, C., Shatzer, J., Jones, R., 2001. Assessment of clinical
competence. Lancet 357, 945–949.
Wihlborg, J., Edgren, G., Johansson, A., Sivberg, B., 2014. The desired competence of the Swedish ambulance nurse according to the professionals – a Delphi study. Int.
Emerg. Nurs. 22 (3), 127–133.
Williams, R., 2012. Nurses who work in the ambulance service. Emerg. Nurse 20 (2),
14–17.
World Medical Association, 2018. WMA Declaration of Helsinki Ethical Principles for Medical Research Involving Human Subjects. https://www.wma.net/policies-post/ wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-huma
n-subjects/.
Zorab, O., Robinson, M., Endacott, R., 2015. Are prehospital treatment or conveyance decisions affected by an ambulance crew’s ability to access a patient’s health information? BMC Emerg. Med. 15 (26) https://doi.org/10.1186/s12873-015-0054- 1.
