http://www.diva-portal.org
This is the published version of a paper presented at Proceedings of the Pupils Attitudes Towards Technology (PATT-34) 2017.
Citation for the original published paper:
Doyle, A., Seery, N., Canty, D., McGuiness, J. (2017)
Professional Continuity: Investigating the Alignment of Technology Teachers’
InternalCapability Constructs.
In: In Proceedings of the Pupils Attitudes Towards Technology (PATT-34) Conference:
Technology & Engineering Education - Fostering the Creativity of Youth Around the Globe.
10-14th July, Philadelphia, Pennsylvania, USA.
N.B. When citing this work, cite the original published paper.
Permanent link to this version:
http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-217098
1 | P a g e Professional Continuity: Investigating the Alignment of Technology Teachers’ Internal
Capability Constructs
Mr. John McGuiness
University of Limerick, Limerick, Ireland
Abstract
In contemporary education, teachers’ epistemological beliefs governing what and how to teach are important due to their influence on practice, pedagogy, assessment, and the learner.
Teachers’ beliefs are perhaps of more significance in technology education as defining clear subject boundaries regarding ‘what to teach’ has traditionally proven difficult. Despite this, there are recognisable practices, processes and outputs that are considered of value to the learner. This research sets out to explore the level of professional continuity among educators regarding such outputs in technology education.
Initial data collection involved the generation of authentic evidence in response to an open design task. Participants were sought from five schools across Ireland. The cohort consisted of first- and second-year technology education pupils (n=64) in post-primary education.
Following this, technology teachers (n=27) were engaged in the holistic assessment of pupil Mr. Andrew Doyle
University of Limerick, Limerick, Ireland andrew.doyle@ul.ie
Dr. Niall Seery
KTH Royal Institute of Technology, Stockholm, Sweden Athlone Institute of Technology, Westmeath, Ireland nseery@ait.ie
Dr. Donal Canty
University of Limerick, Limerick, Ireland donal.canty@ul.ie
2 | P a g e work utilising the Adaptive Comparative Judgment (ACJ) method. The ACJ method relies on a series of binary judgments between two pieces of evidence, effectively producing a rank order of evidence.
In alignment with previous studies utilising the ACJ method, teachers generated very high levels of reliability when tasked with the adjudication of pupil work, despite the lack of assessment criteria. This suggests an implicit understanding of capability among teachers, irrespective of variables such as culture, context or curricula. Interestingly however, not all teachers engaged in judgements agreed consensually. Taking cognisance of this, an analysis of constructs of capability highlighted five criteria that governed teachers’ adjudication on portfolios. The significance of these criteria and the continuity between teachers’ constructs of capability is discussed.
Key Words: Technology Education, Constructs of Capability, Professional Continuity
Introduction
In the Irish educational context, the concept of technological capability has come to the fore as the ultimate aim of technology education (NCCA, 2004, 2007). Although the term has traditionally proven difficult to define (Gagel, 2004), several conceptions have contributed to a collective understanding of what it means to be technologically capable. Black and Harrison (1985) base a definition on one’s capacity to combine designing and making skills, cognisant of the process, and content required. The NCCA (2004) developed a framework for technological capability based upon a foundational knowledge and skills base which focuses on the variety of tasks involved, including designing, production, and evaluation. Similarly, the Assessment and Performance Unit (APU) presented a dialectic model emphasising the importance of the process, suggesting that technological capability cannot be developed solely from an underpinning knowledge base (Kelly, Kimbell, Paterson, Saxton, & Stables, 1987). Instead resulting from interactions between the mind and hand, where ideals are bounced back and forth until suitable solutions are found (Kimbell, Stables, & Green, 1996).
Apparent from all conceptualisations is the task-centred nature of technological activity, Kimbell (2011) develops this further by suggesting that the development of technological capability requires interactions between skills, knowledge, and values. A consolidation of conceptualisations was offered by Gibson (2008), defining technological capability as
“meaningful practical solutions to real problems framed within an appropriate set of values and underpinned by appropriate knowledge” (p.11). To be considered technologically capable, it is necessary to apply both knowledge and skills in solving practical problems while acknowledging and engaging with value-laden decisions, also ensuring that the traditional task centred nature of technology education is not lost.
As a free agent, teachers have the opportunity to embrace, reject, or modify new knowledge, skills, and practices. Regulated by teachers’ belief and value system, constructs of capability in technology will influence teaching and learning in the discipline. This is perhaps of more significance in technology than most subjects due to the nature of the domain. For example, in defining technology education from a content perspective, McGarr and Lynch (2015) highlight the nature of the domain as having blurred boundaries, residing in the weakly classified and weakly framed quadrant of Bernstein’s (1975) framework of curriculum codes, as the domain tends to draw on the subject knowledge of a range of areas. Despite the weakly classified nature of the domain, there are recognisable practices, processes, and outputs that are indicative of technological capability. As well as this, the advantage of not having prescribed content is that ownership lies with the teacher, who can draw upon their own as well as their students’ interests, and recent developments to engage learners with relevant concepts when required (Spendlove, 2012). As highlighted by Jones and Compton (1998),
3 | P a g e issues do arise when teachers’ understanding of technological capability is limited as there is a tendency for the teacher to focus on the production of a product rather than the thinking skills, creativity, processes, issues, and key learning involved.
Research Focus
This study set out to explore the level of professional continuity between technology educators in terms of the level of consensus between teachers’ understanding of evidence of capability in the discipline, and was guided by the following questions; when presented with evidence of capability, without mandated criteria, do teachers respond collectively in the same way, identifying standards and qualities? If so, what are these standards and qualities (criteria) relevant to the task? Adaptive Comparative Judgment (ACJ) provides an idyllic platform for this research as it has the potential to illuminate both level of consensus and teachers’
constructs of capability.
Method
To capture the level of continuity between technology educators’ constructs of capability, a mixed methods approach was devised. Creswell (2009) highlighted that a combination of both qualitative and quantitative methodologies provide greater insight into a research problem.
Similarly, Reams and Twale (2008) argue that mixed methods are necessary to uncover new information and perspectives that will result in less bias and more accurate conclusions.
Although the focus of this research was on capturing the level of professional continuity between technology educators, it was first necessary to generate authentic evidence that is typical of an Irish technology classroom. Thus, in phase one of this research, five qualified and practicing technology teachers were approached to engage with the research. All teachers were teaching technology education at lower secondary level (12-15 years) at the time the study was conducted. Teachers were afforded approximately five weeks to facilitate engagement with a conceptual design task in a way that was flexible and that suited their original targets for that period.
Secondary school pupils (n=64) were tasked with developing “a product to separate and store coins” within individual student portfolios. However, before implementation in schools, a scheme of work was designed to capture evidence of pupil learning through a multiple of communicative tools in solving the design task. Within the scheme, the focus of the design task was place on the authenticity of the learning experience as recommended by Kimbell and Stables (2008). The task was designed so that pupils could engage in creative endeavours facilitated by a disposition of enquiry. Pupils were required to respond to the design task on A3 sheets of paper. Furthermore, students were asked to audio record their thoughts, problems, and/or findings in real time, as previous studies have highlighted that requiring participants to articulate their ideas through audio recordings may refine cognitive processes (Seery, Lane, & Canty, 2011). A pilot study was conducted to highlight any shortfalls with the scheme prior to wider implementation. No major amendments were required within the facilitation of learning experiences. Due to the open nature of the task, the portfolios generated ranged from two to six pages in length. Once the portfolios were collected, they were scanned and compiled into individual PowerPoint files. Audio files were added at this stage in preparation for the ACJ session.
Table 1 Participating student demographics
School No. of Pupils Subject Year Group
1 19 Technical Graphics 2nd
2 12 Materials Technology (Wood) 2nd
3 14 Technical Graphics 1st
4 8 Technology 1st
5 11 Metalwork 2nd
4 | P a g e Phase two of data collection involved the ACJ software. Qualified technology teachers unfamiliar with the project were contacted via email to engage with the ACJ session. A total of 31 teachers were contacted for this phase of the research; 27 engaged with the study. The ACJ method relies on a series of binary judgments between two pieces of evidence, effectively producing a rank order of evidence. Practicing teachers were tasked with the adjudication of pupils’ portfolios independent of set criteria. However, in an effort to unpack teachers’
constructs of capability, participants were required to highlight the specific criteria that influenced judgment on each portfolio. Once this commentary had been provided, teachers also had the opportunity to leave adjudicating comments, detailing precisely what influenced their binary decision.
The subsequent data was analysed through a process of inductive development of codes and categories supported by the qualitative data analysis package NVivo 11. All judge commentary was inductively coded and allocated to categories using the thematic coding process advocated by Braun and Clarke (2006). Following this, a deductive analysis of adjudicating commentary was undertaken for each of the 505 judgment made.
Findings
To analyse the data it was first necessary to elicit the performance rank created during the ACJ session. Each portfolio attained a specific parameter value based on the judgments it was involved in. The rank order stabilised statistically after nine rounds with judging continuing until 15 rounds were completed. The rank (Fig. 1) illustrates a very high level of interrater reliability of 0.968.
Fig. 1 Portfolio parameter values and standard error bars indicating ACJ rank position
THEMATIC ANALYSIS
The thematic review of judging commentary revealed that the criteria that influenced judgments could be grouped into five broad themes; ‘conceptual’ and ‘unique’ ideas,
‘sequence’ and ‘order’, alignment with design brief, ‘thinking’ and ‘telling’, and judges implicit values.
-10 -8 -6 -4 -2 0 2 4 6 8 10
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61
Parameter Values
Rank Position
5 | P a g e
‘Conceptual’ and ‘unique’ ideas
Judge responses indicated that presenting evidence of the generation of initial ideas was of value. Cited most often when pupils were generating ideas, participants valued the presentation of “conceptual” and particularly “unique” ideas. Much of the commentary surrounding this theme centred on how an “interesting idea” distinguished a portfolio from the norm, or more “basic” solutions. In addition, it appeared important to detail how an idea was generated as presenting a particularly unique idea without specifying the origin of the idea generated suspicion regarding authenticity.
‘Sequence’ and ‘order’
Inextricably linked to the model of the design process presented by the NCCA (1991), judges cited the different “stages” or “sequence” of the design process as important criteria. A common response was that a portfolio followed the “correct order”. In addition, participants highlighted where pupils’ portfolios “missed” a stage of the design process. To some participants, evidence of pupils detailing a “journey” through their project was important, due to its perceived alignment with pupil learning. Clarity in presenting the evolution of the portfolio was cited as important, and an “improvement” or “progression” of initial ideas distinguished this from theme one. As well as the development of the design solution, judges consistently referenced evidence of “logical” and “coherent” progression through the design process as important to the project.
Alignment with design brief
Given the open nature of the design brief, it was interesting to see that judges were quite particular about pupils meeting the requirements. Some participants viewed the “sorting” or
“separation” element of the brief as paramount, to the point that it overrode pupils’ creative skills evidenced in portfolios. Where cited, this criterion often specified that a solution was merely a “container” for storing loose coins, leaving the user to sort coins themselves, highlighting the importance of functionality.
‘Thinking’ and ‘telling’
In keeping with the visual nature of the discipline, participants valued the use of “sketches”,
“working drawings”, “mind-maps”, and “graphic organisers” in the effective communication of design ideas. It was notable that this theme permeated almost the entire commentary.
However, it appeared to be most influential where there was a clear discrepancy between two portfolios in terms of presentation. The way in which the quality of presentation so readily influenced decisions suggested its position as a tacit skill expected of pupils, and that effective communication skills were not necessarily rewarded, rather poor communicative skills were penalised.
Judges’ implicit values
Furthering this, some judges’ implicit criteria regarding what was relevant to the task informed their decision, with criteria such as the “use of electronics”, “consideration for function” or, level of design “sophistication”. It was unclear if this criterion governed each judgment by participants or if this arose as a “delta” criterion (Seery, Canty, & Phelan, 2012, p. 224) that distinguished a pupil as having gone above and beyond the status quo.
ANALYSIS OF ADJUDICATING COMMENTRY
A deductive analysis (Cohen, Manion, & Morrison, 2007) of adjudicating comments was carried out once the inductive thematic analysis of initial comments was conducted. Of the 505 judgments made, adjudicating comments accompanied 338 judgments. Due to a lack of specificity by participants, it was only possible to code 314 of these comments. The findings of which are presented in Fig. 2.
6 | P a g e Fig. 2 Correlation of themes with adjudicating commentary
MISFIT PORTFOLIOS AND JUDGES
ACJ has the capacity to measure the consistency of judgments, which in turn has the potential to highlight misfit portfolios and/or misfit judges. The adaptive nature of the software will identify and re-send portfolios for further adjudication to confirm its position on the rank (Pollitt, 2012). Two portfolios (portfolio 5 and 25) remained outside the misfit criterion (WmnSq = Mean
± 2 x SD) at the end of the 15 rounds of judgments, identifying the position of these portfolios on the rank as contentious. Similarly, two judges (judge 13 and 25) were identified as misfits.
Upon purposive sampling (Cohen et al., 2007), it emerged that judge 25 did not provide individual portfolio commentary or adjudicating commentary on judgments. The findings from judge 13’s sampling are presented in Fig. 3.
Fig. 3 Judge 13 purposive sampling 0
20 40 60 80 100 120 140 160
'Conceptual' and 'unique' ideas
'Sequence' and 'order'
'Thinking' and 'telling'
Alignment with design brief
Judges implicit values
Quanitity of references
Themes
0 2 4 6 8 10 12
'Conceptual' and 'unique' ideas
'Sequence' and 'order'
'Thinking' and 'telling' Alignment with design brief
Judges implicit values
Quantity of references
Themes
7 | P a g e Discussion and Conclusion
In alignment with previous studies utilising the ACJ method (Seery et al., 2011), teachers generated very high levels of reliability when tasked with the adjudication of pupil work.
Achieving a high level of reliability despite the variables of culture, context, curricula, and the lack of assessment criteria suggests continuity in understanding of capability amongst 25 of the 27 teachers engaged with this research. This continuity, although implicit by nature, was made explicit through engagement with the ACJ software.
The five criteria outlined through the thematic analysis; ‘conceptual’ and ‘unique’ ideas,
‘sequence’ and ‘order’, alignment with design brief, ‘thinking’ and ‘telling’, and judges’ implicit values, are useful as they depict the communal understanding of capability between judges.
Particularly interesting is the commonality between the findings presented herein at the results presented by Kimbell et al., (2004). Specifically, the alignment between having (‘conceptual’
and ‘unique’ ideas), growing (‘sequence’ and ‘order’) and, proving ideas (alignment with design brief), confirming the nature of technological activity through aligning evidence with that presented from the UK context.
The inclusion of both ‘thinking’ and ‘telling’ and judges implicit values indicate levels of implicit values about the nature of technological activity, both generalizable to the task, and more specific to individual teachers. The generalizable nature of the evidence supporting the
‘thinking’ and ‘telling’ theme suggested the effective communication of ideas as an expected standard of student work. Used to discriminate against evidence that was sub-standard the magnitude of this finding is corroborated by its prevalence in 151 (Fig. 2) of the 314 coded adjudicating commentaries. The inclusion of judges’ implicit values is interesting due to its variability between teachers. Described as a delta criterion (Seery et al., 2012), the ability to recognise and reward evidence of capability that moves beyond the status quo is the unique province of ACJ. It’s absence from judge 13’s (Fig. 3) commentary suggest its importance in depicting competency in adjudications, as exposure to a broad range of pupil responses required judges to develop appraisal skills beyond their initial interpretation of the task, engaging their construct of capability. For the most part, teachers illustrated their ability to recognise evidence of capability that has traditionally been difficult to represent on criterion- based assessment (e.g. critical thinking). However, the lack of a tacit understanding of capability, exposed where two very similar portfolios were presented together, highlighted misfit judges from the rest of the cohort. Further research, determining the reasons for such misfits may provide useful insight on the consistency of judgments and understandings of capability espoused by judges. By proxy, highlighting specific attainment targets and judges for continual professional development has the potential to increase the degree of professional continuity between teachers in the discipline.
References
Bernstein, B. (1975). Class, codes and control: Towards a theory of educational transmission (Vol. III) (3rd ed.). London: Routledge and Kegan Paul Ltd.
Black, P., & Harrison, M. (1985). In place of confusion: Technology and science in the school curriculum: a discussion paper. Nuffield-Chelsea Curriculum Trust and the National Centre for School Technology, Trent Polytechnic.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology ISSN:, 3(2), 77–101.
Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education. (6th ed.).
London: Routledge.
Creswell, J. W. (2014). Research design: Qualitative, quantitative and mixed methods approaches (4th ed.). London: Sage Publications, Inc.
8 | P a g e Gagel, C. W. (2004). Technology profile : An assessment strategy for technological literacy.
Journal of Technology Studies, 30(4), 38–45.
Gibson, K. (2008). Technology and technological knowledge: A challenge for school curricula. Teachers and Teaching: Theory and Practice, 14(1), 3–15.
Jones, A., & Compton, V. (1998). Towards a model for teacher development in technology education: From research to practice. International Journal of Technology and Design Education, 8(1), 51–65.
Kelly, V., Kimbell, R., Paterson, V., Saxton, J., & Stables, K. (1987). Design and technology:
A framework for assessment. London.
Kimbell, R. (2011). Wrong ... but right enough. Design and Technology Education: An International Journal, 16(2), 6–7.
Kimbell, R., Bain, J., Miller, S., Stables, K., Wheeler, T., & Wright, R. (2004). Assessing design innovtion. London.
Kimbell, R., & Stables, K. (2008). Researching design learning: Issues and findings from two decades of research and development. Springer Netherlands.
Kimbell, R., Stables, K., & Green, R. (1996). Understanding practice in design and technology. Buckingham: Open University Press.
McGarr, O., & Lynch, R. (2015). Monopolising the STEM agenda in second-level schools:
exploring power relations and subject subcultures. International Journal of Technology and Design Education. Retrieved from https://link.springer.com/article/10.1007/s10798- 015-9333-0.
NCCA. (1991). Junior certificate technical graphics syllabus. Dublin, Department of Education and Science.
NCCA. (2004). Review of technology education in the junior cycle. Dublin: Department of Education and Science.
NCCA. (2007). Leaving certificate technology syllabus. Dublin: Department of Education and Science.
Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education:
Principles, Policy & Practice, 19(3), 281–300.
Reams, P., & Twale, D. (2008). The promise of mixed methods: Discovering conflicting realities in the data. International Journal of Research & Method in Education, 31(2), 133–142.
Seery, N., Canty, D., & Phelan, P. (2012). The validity and value of peer assessment using adaptive comparative judgement in design driven practical education. International Journal of Technology and Design Education, 22(2), 205–226.
Seery, N., Lane, D., & Canty, D. (2011). Exploring the value of democratic assessment in design based activities of graphical education. In 118th Annual American Society of Engineering Education Conference. Vancouver, British Columbia: American Society of Engineering Education.
Spendlove, D. (2012). Teaching technology. In P. J. Williams (Ed.), Technology Education for Teachers (pp. 35–54). Rotterdam, The Netherlands: Sense Publishers.
