• No results found

Development of a scoring rubric for A to F grading and assessment of Master’s Theses that are using quantitative methodology

N/A
N/A
Protected

Academic year: 2022

Share "Development of a scoring rubric for A to F grading and assessment of Master’s Theses that are using quantitative methodology"

Copied!
22
0
0

Loading.... (view fulltext now)

Full text

(1)

Development of a scoring rubric for A to F grading and assessment of Master’s Theses that are using quantitative methodology

Helena Jernström

Department of Oncology, Clinical Sciences, Lund University, Sweden Helena.Jernstrom@med.lu.se

Pedagogiskt projektarbete i högskolepedagogik, five weeks,

2010-12-13

Supervisor: Associate Professor Gudrun Edgren, MedCUL

(2)

Summary

Malmö University uses ECTS grades (A to Fail) in parallel with Swedish grades for some courses. During 2008 and 2009 I was course director for the course Masters Thesis at the International Masters Programme for Public Health at Malmö University, Sweden. After our first examination it was evident that lecturers from different backgrounds had widely

disparate grading criteria and the same quantitative thesis was graded from A to Fail. It is key for teachers to adopt a common standard in grading.

The aim of this project was to develop a detailed scoring rubric for grading of master’s theses based on quantitative methodology according to a seven-grade scale (A to Fail) that reflect how well the learning objectives for the course are met. The rubric could be used for both formative as well as summative assessment.

A detailed rubric was created. Students, examiners and opponents received the rubric. As only one thesis was completed and graded during the period of this project, it was not possible to formally evaluate the validity and reliability of the rubric. However, all parties found the rubric useful and easy to use and gave me constructive feedback, which has been incorporated in the updated version of the rubric that is included in this paper.

In conclusion, the rubric should be aligned with the learning outcomes. The student needs access to the rubric from the start of the thesis work and the rubric should also be provided to the supervisor, opponent, and examiner. The rubric may be a useful tool for formative assessment during thesis writing and may also improve the objectivity of the final grading. As several higher education institutions in Sweden have introduced a criterion based A to F scale, the rubric may also be useful for summative grading. Since only one essay was graded based on the criteria listed in the rubric, the validity and reliability of the rubric warrants evaluation in an independent setting.

Introduction

The Bologna process aimed to make the European higher education systems more transparent.

As a first step, the implementation of the ECTS scale required the collection of statistical data in the institutions who were willing to participate (p41) (Office for Official Publications of the European Communities, 2009). The European Credit Transfer and Accumulation System (ECTS) scale was developed to facilitate comparisons between grades from different countries. The national grading systems were so different that comparison was more or less impossible (Karran, 2005). In Denmark, Norway, and Sweden the Bologna process led to reforms of higher education which included a change of grading scales (Dahl et al, 2009).

Denmark had a 13-grade norm-references assessment system, a relative scale. This scale was abolished and developed into a new 7-step scale comparable to ECTS and the new scale was used as of September 2007 (Dahl et al, 2009). As of 2003 Norway had either a 6-step A to F scale or pass/fail. The A to F scale was supposed to be criterion based, but at the same time the ministry clearly stated in May 2004 that if the scale was used as intended, there would be no need to translate Norwegian grades into ECTS (Dahl et al, 2009). However, the ECTS scale was developed as a relative scale that was intended to show if the original national grade was awarded for a smaller group of students or a larger group and not as a criterion based scale. The intention was that 10% of the students would be awarded an A, 25% a B, 30% a C, 25% a D, and 10% an E, plus Fx and F according to the ECTS system (Karran, 2005; Office for Official Publications of the European Communities, 2009; Åkesson, 2007).

(3)

In Sweden, the choice of grading scales in decentralized and each university and sometimes each faculty have the right to choose grading scales. Normally the grades ”Väl godkänd (VG)” -- ”Pass with distinction”, ”Godkänd (G)” --”Pass” and “Underkänd (U)” –“ Fail” are used, unless the university decides to use a different scale (Högskoleförordningen (1993:100) 6 kap 19 §). The decentralization means that usually between two and four levels are used.

The introduction of a multi level A to F scale is now being considered. The grades awarded, irrespective of which scale is used, should be absolute and reflect how well the intended learning outcomes are met. The International Masters Programme for Public Health at Malmö University uses the Swedish grades ”Väl godkänd (VG)” -- ”Pass with distinction”, ”Godkänd (G)” --”Pass” and “Underkänd (U)” –“ Fail”. These grades are absolute. In addition to the Swedish grades, the students may also request ECTS grades at the IMER programme (Åkesson, 2007) and at the International Masters Programme for Public Health.

The fact that the ECTS grades are intended for grading on a scale in contrast to the Swedish absolute grades complicates the conversion of the grades between the two systems, especially when there are few students in a class. At Malmö University, an A or B has roughly been equal to a high pass (VG) while the C-E grades have been translated into a pass (G). For obvious reasons it has been harder to translate these Swedish grades into ECTS than vice versa.

During 2009 it became evident on an international level, and not just at Malmö University, that it was too difficult to implement the second step of the ECTS, which consisted of translating national grades based on few grades to the ECTS. It was next to impossible to make the division according to the intention of A-10%, B-25%, C-30%, D- 25%, E-10%, plus Fx and F, especially for courses with few students. “In the light of the experience made with the ECTS 5-point grading scale in the past years, it can be said that the second step described above proved to be far too ambitious and difficult to implement, especially in those national grading systems with only five or fewer passing grades, which could hardly fit into the predetermined percentage structure provided by the ECTS scale. In fact, the use of the ECTS scale by European institutions has been rather limited.” (Office for Official Publications of the European Communities, 2009). It was decided that the A to F system should be abandoned and instead “…institutions only need to provide in a standard table form the statistical

distribution of their own grades. Therefore, the ECTS grading scale based on a predetermined percentage structure is to be replaced by a simple statistical table completed for each degree programme or group of homogeneous programmes.” (p42) (Office for Official Publications of the European Communities, 2009). However, as Swedish universities have the option of using a criterion based A to F scale, there may still be a need for objective multi-level criterion based grading scales.

Jarsmaa et al. (Jaarsma et al, 2007) reported that there is an explicit lack of criteria for evaluation of students’ research reports, and a large spread of results during re-grading of already graded reports, which is in line with my own experience. Since students have the right to an explanation for why a certain grade has been awarded, it is essential that the grading criteria are as objective as possible (p 59) (Högskoleverket, 2008). If holistic scoring is used the teacher gives an overall score compared to analytical scoring, which assesses each of the dimensions. According to Biggs “The worst thing that can be said about grading on the curve is that it precludes Criterion Referenced Assessment and aligned teaching. It is a procedure that can not be justified on educational grounds.” (Biggs, 2003). A rubric is based on analytical scoring and explicitly states what is required for a certain grade to be awarded. It

(4)

is thus compatible with the Swedish requirement that the grade should be absolute and not based on the performance in relation to fellow students performances.

During 2008 and 2009 I was course director for the course Masters Thesis at the International Masters Programme for Public Health at Malmö University, Sweden. Public Health is a multi- disciplinary field, comprising medicine, social sciences, behavioral sciences etc. After our first examination it was evident that lecturers from different backgrounds had widely disparate grading criteria and the same quantitative essay was rated from A to Fail. A clear difference was seen depending on whether the lecturer had a qualitative or quantitative research profile. The teachers had wide differences in their standards and judgments. In order to be fair to the students and also to avoid an academic drift it is key for teachers to adopt a common standard in grading. There are also legal aspects to grading. The student is entitled to find out from the examiner on what criteria a certain degree has been awarded

(Högskoleverket, 2008).

The learning outcomes for the master’s thesis in Public Health at Malmö University read as follows:

After the course the student should be able to:

• formulate a scientific problem,

• choose research methods relevant for a scientific problem, and be able to apply them in order to solve this problem,

• apply models and theories relevant for a chosen research topic,

• compile, critically analyse and present research results, independently, with the support of a tutor,

• conduct an empirical or a theoretical study in public health, and

• consider relevant ethical issues at all stages of the study, including conducting an ethical application for the chosen project.

The aim of this project was to develop a scoring rubric for grading according to a seven-grade scale (A to F) of how well a quantitative master’s thesis met the learning objectives for the course. The rubric is intended for both formative and summative assessment. Qualitative and quantitative thesis work put different emphasis on different parts of the work. For example, statistical models, statistical adjustments, and false positive findings due to small number of subjects, are more relevant for a quantitative thesis than for a qualitative thesis. The rubric developed during the course of this project is thus only intended for grading quantitative thesis work.

Material and methods

There are various taxonomies for describing the level of learning outcomes, such as Bigg’s structure of observed learning outcomes (SOLO) taxonomy (Biggs, 2003) and the Bloom taxonomy (Bloom & Krathwohl, 1956). The SOLO taxonomy consists of five levels of understanding pre-structural, uni-structural, multi-structural, relational, and extended abstract.

A student who has reached the highest level is able to conceptualize the previous integrated knowledge as a whole at a higher level of abstraction and generalize it to a new topic or area.

The SOLO taxonomy enables assessment of the quality of the work of the student and not just whether certain pieces have been completed. Bloom identified three domains of educational activities: cognitive, affective, and psychomotor activities. The cognitive domain includes knowledge and development of intellectual skills, while the affective domain reflects attitude and the psychomotor domain stands for manual or physical skills (Bloom & Krathwohl, 1956). Malmö University uses a model of Bloom’s taxonomy regarding cognitive skills that

(5)

has been adapted by Khalid El Gaidi after Grönlund N.E. (Grönlund, 2000). The grade should then reflect how well the learning outcome has been met. At Malmö University it was decided that the following adjectives should be used for the ECTS grades. A-excellent, B-very good, C-Good, D-Satisfactory, and E-Sufficient. I have therefore used these adjectives in the rubric.

Since the rubric is intended for the level of a master’s thesis, the quality of the work should reflect the higher levels represented by the two taxonomies.

In June 2008 Peter Gill, professor in pedagogic at Högskolan in Gävle, gave a seminar entitled “Allmänna strategier för autentisk betygsättning: Bedömningsmatriser och

medbedömareskattning”. “Common strategies for authentic grading: Scoring rubrics and peer- assessment” (Gill, 2008)

Peter Gill showed us the scoring rubric he had developed and currently used. This rubric was also given to his students and was used for grading of fellow student’s essays as well as for awarding the final grades. We were given a copy of the power-point presentation, which included the rubric. I have used parts of this rubric as a template for my own rubric.

I was also given a draft of the suggested grading criteria for term papers and theses in Communication in English III, English Studies and Language and Cultural Studies at IMER programme at Malmö University. This document also served as an inspiration for the construction of the rubric.

I then developed my rubric and presented it at a seminar with colleagues using both qualitative and quantitative methodology. I received valuable feedback from several colleagues and a revised version was created and later handed out to the students.

Results

In the fall of 2009 one student submitted a thesis that was returned to the student without a formal examination. The student was given constructive critique. The student was asked to re-write the thesis according to the guidelines prior to re-submission. This student and one other student were also given a copy of the rubric. After a few months the first student re- submitted a new version of the thesis in the spring of 2010. The opponent and the examiner also used the same rubric for scoring this thesis.

I attended the defense and talked to the student, supervisor, opponent and examiner. The student found the rubric very useful and expressed that it would have been very useful if the rubric had been available from the very start. Both the opponent and the examiner used the rubric to grade the essay and were in agreement in all but two sections where one of them gave a C and the other a D and vice versa. They both agreed on the same final grade.

I asked if they felt that any sections were missing, could be omitted or improved upon. Their comments were the following. They felt that the rubric lacked a section on readability and central thread. This section should receive a high weight, five units, and has now been added.

My colleagues argued that method is not the same as study design. However, I feel that the way the study is designed is an intrinsic part of the methodology. I have therefore now

explicitly included study design in the methods section. My colleagues also thought that some of the criteria for obtaining an A were possibly more suited for the doctoral thesis level than the level of a master’s thesis. However, I have not lowered the criteria for any of the sections as I feel that more than one thesis will need to be graded prior to making this change.

(6)

In the original version of the rubric, there were no guidelines as to how many figures or tables could be presented and a section on figures and tables worth three units has now been added.

We agreed that a total of maximum six figures and tables should be included in the results section. In addition, a flowchart that shows the inclusion and exclusion criteria and numbers in each step for selection of the study population may be included.

The time aspect came up as another important issue. My colleagues suggested that in order to obtain a high pass (equivalent to an A or a B), the student should complete the thesis work within the stipulated time frame, i.e. within one semester if it is a master’s thesis. It was suggested that the grades A and B (VG) should not be awarded to students who presented their thesis work after the deadline, no matter how good the work was. Special reasons for the delay such as parental leave or serious illness would be valid reasons for extending the deadline date. There is an optional section on completion of the thesis on time added to the rubric.

Discussion

The main result was that the student, teachers, supervisor, opponent and examiner found the rubric useful and easy to use. It did, however, need some adjustments, which have been added to the rubric enclosed with this project paper. Jönsson and Svingby have reviewed the

reliability, validity and educational consequences of scoring rubrics (Jönsson & Svingby, 2007). The validity of the rubric examines what it is intended to measure, i.e. the quality of the thesis and that it fulfills the learning objectives. It is also important that the rubric is not user dependent, i.e. that it has a high inter-rater reliability. Since only one thesis was graded, it was not possible to formally examine validity and reliability of the rubric.

The rubric has been developed for both formative and summative assessment. According to Heidi Goodrich Andrade, a rubrics expert, rubrics “…are powerful tools for both teaching and assessment. Rubrics can improve student performance, as well as monitor it, by making teachers’ expectations clear and by showing students how to meet these expectations. The result is often marked improvements in the quality of student work and in learning. Thus, the most common argument for using rubrics is they help define “quality.”” (Goodrich Andrade, 1996). Conversely, Biggs argue that multiple steps do not promote learning and that

fulfillment of the learning objectives is the most important factor (Biggs, 2003). The detailed scoring rubric facilitates the evaluation as to how well a thesis fulfills the learning objectives and may benefit from multiple steps at least in the formative setting. The rubric should be aligned with the learning objectives. For example, the learning objectives state ”apply models and theories relevant for a chosen research topic”. I have given examples in the rubric with respect to quantitative methodology. Statistical models, statistical adjustments, and false positive findings due to small number of subjects are relevant for a quantitative thesis and there are therefore detailed sections on these. However, these sections may be less relevant for a rubric for a thesis using qualitative methodology.

As of now there is no strict deadline for handing in a thesis. It may therefore be beneficial for students to work longer than one semester in order to increase their chances for a higher grade (VG). This creates an unfair advantage in relation to their fellow students who may actually deliver the thesis on time. However, students have the right to a re-examination if they did not pass the first exam. They will then have more time to study than those who passed at the first attempt. Unless the time-aspect is included and considered a crucial part of the learning objectives it is probably not possible to legally include a section that states that an essay can only be awarded a top grade if it is handed in on time. Given that the university suffers

(7)

economical consequences when a thesis is delayed as it does not receive full economical compensation for a student until the thesis is passed, putting weight on the time-aspect may be worth testing, if legal. Otherwise it may be sufficient with more structured group sessions in order to improve the percentage of theses completed on time.

During the spring of 2009 there were bi-weekly group sessions to promote progress of the thesis work. These sessions focused on quantitative and qualitative essays every second time.

Not all students attended these group sessions. Since some students carried out their thesis work abroad it was not possible to make the sessions obligatory. Andersson and Persson have carried out a project where the effect of group mentoring and coaching was evaluated

(Andersson & Persson, 2002). The students were encouraged to start the essays already during the prior course on methodology. They were divided into groups of six to seven students who met once a month. Prior to each meeting it was decided how far the essay should have progressed. This created a group pressure to keep on track. Prior to the project, less than 10%

of the essays were completed within the stipulated timeframe. After the project, over 50% of the essays were completed on time without lowering of the scientific quality. The authors noted that the essays were more even in quality and that both the worst and the best essays were rare. Andersson and Persson also discussed the possibility of letting the grade be affected by the time-aspect (Andersson & Persson, 2002).

A scoring rubric will facilitate the objective grading and the explanation of why a certain grade has been awarded. If the rubric has also been used during the thesis’ writing as a formative tool, the final grade should not come as a surprise to the student. The rubric will help the student figure out how the thesis will be evaluated and may help the students judge and revise their own work before handing it in. Multiple steps in the rubric may thus be useful in the formative setting, whether or not the final grade consists of multiple steps or simply a pass/no pass. One example of local rules regarding examinations and grading is Lund University guidelines for the relation between institutions and students at Lund University

”Efter varje examination ska examinationsgenomgång hållas då studenten ska kunna få förklaring till varför ett visst betyg givits. Examinator bör vara närvarande…...” “After each examination shall a examination session be held where the student should receive an

explanation for the given grade. The examiner should be present…..” (Högskoleverket, 2008).

Swedish universities have the option to use a criterion based A to F scale. Therefore the scoring rubric may be useful even if the way grades are awarded according to the ECTS system has changed since the beginning of this project.

The rubric was developed for essays within the field of public health. However, it could, after slight adjustment, be used for formative assessment and summative grading of any master’s thesis using quantitative methodology within other fields of medicine. If only a pass/no pass is awarded as the final grade, it may still be useful to keep the many assessment steps during the formative process in order to improve the quality of the work. Currently there is ongoing work at the medical faculty at Lund University where the quality of thesis essays is discussed among course-directors for thesis courses. I previously talked to Associate Professors

Elisabeth Holst and Martin Garwicz who are responsible for the ISEX courses at the fifth and eleventh semesters of the medical students program. They do not use a rubric for scoring the essays. As of 2010-03-29 there is an updated list of criteria for passing the fifth semester course and how the essay should be written

(http://www.med.lu.se/laekarutbildning/termin_5/lund_malmoe_individuell_studieperiod/indi viduell_studieperiod_examensarbete_isex). The syllabus with criteria for the eleventh

(8)

semester thesis is less detailed and is dated 2007-09-19

(http://www.med.lu.se/plain/laekarutbildning/termin_11/isex).

In conclusion, the rubric should be aligned with the learning outcomes. The student needs access to the rubric from the start of the thesis work and rubric should also be provided to the supervisor, opponent, and examiner. The rubric may be a useful tool for formative assessment during thesis writing and may also improve objectivity of the final grading. As several higher education institutions in Sweden have introduced a criterion based A to F scale, the rubric may also be useful for summative grading. Since only one essay was graded based on the criteria listed in the rubric, the validity and reliability of the rubric warrants evaluation in an independent setting.

References

Andersson G, Persson A (2002) Coaching och handledning av grupper: inom universitets- och högskoleutbildning: Lunds Studentlitteratur

Biggs J (2003) Teaching for quality learning at University, 2nd ed edn. Berkshire: The society for Research into Higher Education & Open University Press

Bloom BS, Krathwohl DR (1956) Taxonomy of Educational Objectives: The Classification of Educational Goals, by a committee of college and university examiners. Handbook I:

Cognitive Domain. New York: Longmans, Green

Dahl B, Lien E, Lindberg-Sand Å (2009) Conformity or confusion? Changing higher

education grading scales as a part of the Bologna Process: the cases of Denmark, Norway and Sweden. Learning and teaching 2: 39-79

Gill P (2008) Allmänna strategier för autentisk betygsättning: Bedömningsmatriser och medbedömareskattning (Seminar at Malmö University June 6 2008)

Goodrich Andrade H (1996) Understanding Rubrics. Educational Leadership 54: 14-18 http://learnweb.harvard.edu/alps/thinking/docs/rubricar.htm

Grönlund NE (2000) How to Write and Use Instructional Objectives, 6th ed edn. Upper Saddle River, NJ: Prentice Hall

Högskoleförordningen (1993:100) 6 kap 19 §

Högskoleverket (2008) Rättssäker examination Andra omarbetade upplagan. In Högskoleverkets rapport serie 2008;36R, Högskoleverket (ed). Stockholm

Jaarsma DA, Kranenbarg WJ, Dolmans DH, Muijtjens AM, Scherpbier AJ, Van Beukelen P (2007) Assessing students' research reports: development of a rating scale. Med Teach 29:

160-5

Jönsson A, Svingby G (2007) The use of scoring rubrics: Reliability, validity and educational consequences. Education Research Review 2: 130-144

(9)

Karran T (2005) Pan-European grading scales: lessons from national systems and the ECTS.

Higher Education in Europe 30: 5-22

Office for Official Publications of the European Communities (2009) ECTS Users' Guide.

Luxembourg http://ec.europa.eu/education/lifelong-learning-policy/doc/ects/guide_en.pdf

Åkesson E (2007) ECTS-skalan eller något annat? Några aspekter på valet av betygssystem.

Bidrag till högsoleverkets kvalitetskonferens i Umeå 10-11 oktober 2007

Acknowledgement

First I would like to express my sincere gratitude to my supervisor Gudrun Edgren at

MedCUL for her continuous encouragement and support during the whole process of creating the rubric to the finalization of this paper. Your input and knowledge of the field of pedagogic at the university level has been invaluable. I would also like to express my sincere gratitude to my former colleagues at Malmö University, Agneta Bjuväng, Oscar Andersson, Staffan Berglund, Ellis Janzon, Azzam Khalaf, Per-Anders Tengland, and Slobodan Zdravkovic for all your valuable input and for being great critical friends! Thank you!

(10)
(11)

Helena Jernström 11

F (0) Fx (0) E (1)

Sufficient

D (2) Satisfactory

C (3) Good

B (4) Very good

A (5) Excellent

Topic (2) Not relevant

to public health

Not relevant to public health

Barely relevant to public health

Somewhat relevant to public health

Relevant to public health

Clearly relevant to public health

Highly relevant to public health Abstract (2) Missing Too long or

too short or key words missing does not contain background, aim, results and

conclusion

200-300 words with 4-6 key words. Contains background, aim, results and conclusion

200-300 words with 4-6 key words, with relevant background, aim, results and conclusion

200-300 words with 4-6 key words. Well written with

relevant

background, aim, results and conclusion

200-300 words with 4-6 key words. Very well written with relevant background, aim, results and conclusion

200-300 words with 4-6 keywords. Excellently written, clear and concise abstract with highly relevant

background, aim, results and conclusion

Title (1) Unclear Unclear

poorly stated

Sufficiently stated title includes some key words in aim

Satisfactory title, includes some key words in aim

Complete title, includes some key word from aim

Complete title, includes key words found in aim and mentioned introduction

Complete title, includes clear key words also found in the aim and properly introduced in introduction

Introduction/background (3)

Missing or not relevant to public health

Little relevance to public health

Gives sufficient perspective on the public health problem.

Gives some explanation to the aim and hypothesis.

Sufficient use of references to the literature

Gives satisfactory perspective on the public health problem.

Gives

explanation to the aim and hypothesis Uses adequate references to

Gives a good perspective on the public health problem on a national or

international level.

Gives good

explanation for why the aim and

hypothesis are important to study

Gives a very good

perspective on the public health problem on a national and

international level. States why the aim and hypothesis are

Gives a clear perspective on the public health problem the thesis deals with on a national and

international level. What is known and unknown.

Clearly states why the aim and hypothesis are important to study in public health. Uses

(12)

Helena Jernström 12

the literature Uses relevant references to the literature

important to study Uses relevant

references to the literature.

highly relevant references to the literature.

Aim (3) Missing Not relevant

to public health or poorly formulated

Sufficiently formulated aim, somewhat relevant to public health

Satisfactorily formulated aim relevant to public health

Well stated aim.

Relevant to public health. Has been studied previously in similar

population.

Clearly stated aim. Relevant to public health.

Has been studied previously in similar population as opposed to in a new population.

Adequate for a one semester thesis.

Clearly stated. Highly relevant to public health. Investigates something new or puts a new angle on previously known problem or investigates the aim in a new population. Well suited for a one semester thesis

Hypothesis (3) Missing Not related to the aim or

poorly formulated, cannot be tested

Sufficiently formulated hypothesis, not entirely in line with aim, can only be partly tested with material and methods chosen, or too many

Satisfactorily formulated but not entirely in line with the aim.

Well formulated, but not entirely in line with the aim, Testable with current material and methods.

Clearly formulated, interesting and relevant to aim and public health. Well suited for the material and methods chosen.

Clearly formulated, highly interesting or creative and highly relevant to the stated aim and public health.

Very well suited for the material and methods chosen.

(13)

Helena Jernström 13

hypotheses.

Material (2) Not suited for testing hypothesis

Poorly suited for testing hypothesis, poorly described

Sufficiently suited for testing

hypothesis, and sufficiently described

Adequately suited for testing

hypothesis and adequately described

Well suited testing hypothesis. Well described.

Interesting material, well suited for testing hypothesis and clearly described.

Highly interesting material, clearly

described and very well suited for testing the hypothesis. Eg case- control study for rare problems.

If a data base is used, data has been quality controlled and how this is done is clearly descried.

Justifies the sample size or explains why a smaller sample size was used.

Methods (3)

including study design

Not suited for testing hypothesis

Poorly suited for testing hypothesis, poorly described.

Sufficient method for testing hypothesis.

Sufficiently suited for chosen material.

Sufficiently described and referenced.

Adequately described method with references.

Method is satisfactory for testing the chosen hypothesis in the chosen material.

Well described method with correct references.

Appropriate method for testing the chosen hypothesis in the chosen material.

Clearly described method with appropriate references.

Correct method for testing the chosen hypothesis in the chosen material.

Questionnaire

Clearly described method with highly appropriate references.

Preferred method for testing the chosen hypothesis in the chosen material. Eg. case- control study for rare problems. The reader should be able to replicate the study following the methods

(14)

Helena Jernström 14

may not be fully validated or it is not stated whether validated questions were available, but some form of quality control of data is made.

section. If a

questionnaire was used, it was to a large extent based on validated questions with

appropriate references to where it has been previously used.

Ethics (1) Ethics

application is missing

Ethics application is submitted but not

approved/or ethics approved for other parts of project but not for current research

Ethics application is submitted and approved but not mentioned in thesis, only in oral

presentation

Ethics approved, number and written consent obtained from most subjects

Ethics approved, number and written consent obtained from all subjects.

Ethical implications of current study mentioned in background or in discussion.

Ethics approved, number and written consent obtained from all subjects.

Ethical

implications of current study mentioned in background and in discussion.

Ethics approved, number and written consent obtained from all subjects. Current and future ethics

implications of the public health issue investigated and the conduct and results of current study are mentioned in background and

discussed in discussion.

Statistics (3) No statistics section

Section only mentions which program was used, but not which analyses

Statistical program may be stated.

Statistical methods used are sufficiently described and

Statistical program stated.

Statistical methods used are mentioned Some tests correctly used,

Statistical program stated. Statistical methods used are well described.

Some tests correctly used, Adjustments may be missing

Statistical program and version stated.

Correct statistical method used and well

Statistical program and version stated and referenced. Correct statistical method used and clearly described.

Appropriate adjustments done and described. P-

(15)

Helena Jernström 15

used. all are not

optimal.

described.

Appropriate adjustments mostly done and clearly

described. P- value taken as significant stated.

value taken as

significant clearly stated and whether one or two- tailed tests were used.

The section should be detailed enough to re- run the analyses.

Results (3) Results not related to aim and hypothesis or mostly missing.

Results not related to aim and

hypothesis.

Too many or too few results presented, not clear which are the main ones.

Either unadjusted or adjusted results missing.

Too many or too few results presented, not clear which are the main ones.

Either unadjusted or adjusted results missing.

Adequately structured.

Results based on the hypothesis are presented. Results from unadjusted and some adjusted analyses presented.

Appropriate adjustments only partly presented.

Well structured.

Main results based on the hypothesis are clearly

presented.

Results from unadjusted and adjusted analyses presented in structured order.

Appropriate adjustments mostly done in statistical model. Clearly structured.

New and interesting results. Main results based on a highly relevant public health related hypothesis are clearly presented.

Appropriate adjustments done in statistical models. Results from unadjusted and adjusted analyses presented in clear and structured order.

Figures and Tables (3) Tables and/or figures are missing.

Tables and/or figures without relevant

There is between less than four or more than eight

There is between four and eight figures and/or

There is between four and eight figures and/or tables for the results

There is a maximum of six figures and/or tables for the

There is a maximum of six figures and/or tables for the results section.

These should contain

(16)

Helena Jernström 16

characteristics of study population and main findings.

figures and/or tables for the results section.

These contain some of the relevant characteristics of the study population on which the results are based. They also contain secondary findings. No flow-chart with the inclusion and exclusion criteria leading to the final study

population is presented.

tables for the results section.

These contain most of the relevant characteristics of the study population on which the results are based.

Secondary findings should not be

emphasized No flow-chart with the inclusion and exclusion criteria leading to the final study

population is presented.

section. These contain the relevant characteristics of the study population on which the results are based. The main results based on the correct numbers of the population for each analysis should be highlighted, for example risk factors with crude RRs with confidence intervals. Secondary findings should not be emphasized No flow-chart with the inclusion and exclusion criteria leading to the final study population is presented.

results section.

These should contain the relevant characteristics of the study population on which the results are based. The main results based on the correct numbers of the population for each analysis should be highlighted with and without relevant

adjustments, for example risk factors with crude and adjusted RRs with confidence intervals.

Secondary findings should not be

emphasized In addition, a flow-chart with

the relevant

characteristics of the study population on which the results are based. The main results based on the correct numbers of the population for each analysis should be highlighted with and without relevant adjustments, for example risk factors with crude and adjusted RRs with confidence intervals. Missing data should be clearly indicated. Secondary findings should not be emphasized

In addition, a flow-chart with the inclusion and exclusion criteria leading to the final study population may be presented.

(17)

Helena Jernström 17

the inclusion and exclusion criteria leading to the final study

population may be presented.

Discussion (4) Validity, Reliability Generalizability Bias

Discussion not relevant to study.

Poor use of references.

Sections are plagiarized.

Discussion not relevant to study.

Discussion mostly missing on validity, reliability, generalizabilit y and bias.

Poor structure and issues mentioned in the

introduction/b ackground are not followed- up and discussed.

Poor use of references.

Sufficient discussion of results. One or two of validity, generalizability and reliability of method may not be

addressed or properly referenced.

Sufficient discussion of the study’s strengths and/or weaknesses and/or potential bias. Issues mentioned in the

introduction/bac kground are not fully followed- up and

Satisfactory discussion of results in relation to aim and hypothesis.

Some

mentioning of validity, generalizability and/or

reliability of method.

Satisfactory discussion of the study’s strengths and/or weaknesses and/or potential bias. Issues mentioned in the

introduction/bac kground are satisfactorily

Well written discussion of main results in relation to aim and hypothesis.

Adequate discussion of validity,

reliability and generalizability of the study. Most of the study’s strengths and/or weaknesses and/or potential bias are discussed and properly referenced.

Issues mentioned in the

introduction/backgro und are followed-up and discussed.

Implications of the study in a public health perspective are addressed and appropriately

Very well written discussion of main results in relation to aim and hypothesis.

Good discussion of validity, reliability and generalizability of the study’s material, methods and results, with relevant references. The study’s

strengths and weaknesses and potential bias are discussed in detail. Issues mentioned in the

Creative and interesting discussion of main results in relation to aim and hypothesis. The validity, reliability and generalizability of the study’s material, methods and results are scrutinized and

references used are excellent. The study’s strengths and

weaknesses and potential bias are discussed and put into a new perspective. Issues mentioned in the

introduction/background are clearly followed-up and discussed in the light of the new findings of the current study. The implications of the findings of the study in

(18)

Helena Jernström 18

discussed.

Implications of the study in a public health perspective are sufficiently addressed and referenced.

followed-up and discussed.

Some

implications of the study in a public health perspective are addressed and referenced.

Adequately structured.

referenced. Well structured

introduction/bac kground are properlay followed-up and discussed in the light of the new findings of the current study.

Major

implications of the study in a public health perspective are addressed and referenced.

Well structured.

a short and long-term public health

perspective are very well addressed with excellent references.

Conclusion (1)

Missing Not relevant to aim

hypothesis and results

States some finding but not main finding.

States some finding/s and some

implications of study

Satisfactorily formulated

States main finding/s and some implication of study. Well formulated.

States main finding/s and the immediate public health implications of study. Very well formulated.

States main finding/s and its implications in a short and long-term public health perspective. Future implications in a national and

international perspective are mentioned and very well formulated.

References (2) Poorly organized, not in order, typos,

Poorly organized, some typos Some

Sufficient use of references in correct order in relation to text,

Satisfactory use of references in correct order in relation to the

Good use of relevant references used. Properly referred to in the

Clearly relevant references used.

Properly referred to in

Selected highly relevant references used.

Properly referred to in the text. All references

(19)

Helena Jernström 19

wrong, or missing

irrelevant references.

Some typos.

Not always optimal references.

text, but not always in same format.

text. All references in correct order, no typos.

the text. All references in correct order.

References in the same format

in correct order. All references in the same format (using ref works, end note or similar program)

Language (2) Poor,

difficult to understand, plenty of grammatical errors. Poor organization of text.

Contains plagiarized material.

Poor, difficult to understand, plenty of grammatical errors. Poor organization of text.

Sufficient use of English to enable reader to understand the text. Some incomplete sentences or sentences that use colloquial English.

Sufficient organization of paragraphs.

Many grammatical errors.

Satisfactory use of the English language.

Satisfactory organization of paragraphs.

Quite a few grammatical errors.

Well written.

Complete sentences.

For example: if something is said to be of poor quality, it is not always explained why.

Paragraphs well organized. Some grammatical errors.

Very well written, fluent.

Easy to read.

Complete sentences. For example: if something is said to be of poor quality, it is explained why.

Paragraphs well organized and easy to follow.

Few

grammatical errors.

Fluent, clear concise.

Easy and interesting to read. Complete and well written sentences. For example: if something is said to be of poor quality, it is clearly explained why.

Paragraphs in excellent order. Issues mentioned in background, results and discussion are organized in the similar order. Few, if any, grammatical errors.

Readability (5) There is no central thread The paper appears to be a patchwork of sections

There is a tendency to a central thread in some sections. The paper appears to be a patchwork of other sections.

There is a central thread in some sections, but it is missing in two sections.

There is a central thread in the paper but it is missing in one section.

There is a clear central thread in most of the paper, but it may be less evident in one section.

There is a clear central thread throughout most of the paper, it may be less clear in one section.

There is a clear central thread throughout the paper.

(20)

Helena Jernström 20

Oral presentation (2) Poor Poor structure, poor slides,

poor

presentation.

The slides do not follow the content of the thesis, or the font is too small, or too much

information is crammed on some slides.

Too many or too few slides, not 15-25.

Sufficient to enable a person to follow why the study was done, how it was done, what was found, and some of the implications of the findings in relation to public health.

The majority of the 15-25 slides follows the thesis, the font is readable.

May contain some typos.

Satisfactory slides, and presentation to enable a the audience to follow why the study was done, how it was done, what was found, and most of the

implications of the findings in relation to public health.

The 15-25 slides are satisfactorily organized according the content of the thesis and all have readable font, may contain some typos.

Good presentation and organization of slides. It is easy for the audience to follow why the study was done, how it was done, what was found, and the importance of the implications of the findings in relation to public health. The 15-25 slides are well organized, the font and figures are easy to read for the audience, may contain some typos.

Very good presentation and slides.

Interesting for the audience and very easy to follow why the study was done, how it was done, what was found, and the importance of the implications of the current findings in relation to public health on a national or international levels. The 15- 25 slides are very well organized, the font and figures are easy to read for the

audience.

Excellent presentation and slides. Highly interesting for the audience and very easy to follow why the study was done, how it was done, what was found, and the importance of the implications of the current findings in relation to public health and health economic factory on national and international levels. The 15-25 slides are

excellent and very well organized, the font and figures are easy to read for the audience. No typos.

Defence (2) Poor, Poor, not

engaged, appears as if

Sufficient, can defend and explain why

Satisfactory, can defend different parts

Defends thesis well, can explain why things were done,

Defends thesis very well. Can discuss thesis in

Excellent defense on a high scientific level.

Excellent knowledge of

(21)

Helena Jernström 21

the student has plagiarized parts of the study and cannot defend or explain the different parts of the thesis.

how and what was done in the study.

and discuss different issues of thesis with opponent/exami nator. Familiar with the references used in thesis

why material and method were chosen and pros and cons with alternative methods and how results could have been different with alternative methods.

Good knowledge of references used.

a wider perspective.

Demonstrates that the choices of aim,

hypothesis, material and methods were well

substantiated.

Can very well discuss alternative methods and the implications of the findings if these alternative methods were used or if a larger sample size were used.

Good

knowledge of both literature used and literature the student chose not to include.

topic and how the thesis work was performed in detail. Demonstrates that the choices of aim, hypothesis, material and methods were highly substantiated. Excellent discussion of alternative methods and the

implications of the findings in an

immediate and future public health

perspective and how they can be

implemented or what more it takes before they can be

implemented. Clear understanding of alternative methods and what would have happed if they were used or if a larger sample size were used.

In dept knowledge of both literature used and literature that was not used.

(22)

Helena Jernström 22

For each section the grade A=5, B=4, C=3, D=2, E=1, Fx=0 and F=0 is multiplied by the points for the section.

There is s total of 47 points for the section. Max score is 5*47=235 =A range 231-235 An average of 188 equals a B range (184-230)

An average of 141 equals a C range (137-183) An average of 94 equals a D range (90-136) An average of 47 equals an E range (47-89)

Unless it has been agreed that a certain section should not be part of the examination it is not allowed to have any Fx or F. In the event that a section is not part of the examination and grading, the scores for each grade will be adjusted downwards with the same number of points.

Optional

Time-frame The thesis is not

handed in within the stipulated time- frame. The maximum final grade is C.

The thesis is handed in before deadline, i.e. if it is a one-semester thesis, it is completed and examined within this time-frame. The

maximum final grade is A.

References

Related documents

Note that in the original WRA, WAsP was used for the simulations and the long term reference data was created extending the M4 dataset by correlating it with the

Fordi amming er normen i Norge og de aller fleste kvinner starter å amme, kan det oppleves desto mer traumatisk når de ikke får det til eller må slutte før de hadde ønsket.. I

BRIM HEALTH’S main objective was to ensure the proper development of the public health function in the countries through support to national strategies for public health

In section 3.2.1 the usage of barrier and trolley was selected for further investigation. In order to continue the design of the new test method, different concepts in terms of

A health content analysis is a useful tool to know how media deliver health messages and the way in which population receive this health information –amount of space related to

“Det är dålig uppfostran” är ett examensarbete skrivet av Jenny Spik och Alexander Villafuerte. Studien undersöker utifrån ett föräldraperspektiv hur föräldrarnas

Kjønnsperspektivet når det gjelder støtte eller behandlingstiltak for kvinner, er et virksomt redskap når det gjelder å forstå, ikke bare kvinners, men også menns behov for

“ recent expression of the connection be- tween human rights, including the rights of in- digenous peoples, and environmental law was made by the independent expert on the issue of