Quality Assessment
1. Introduction
As Thurstone (1959) asserted ‘If it exists it can be measured and if it can’t be measured it doesn’t exist.’ Thus in the domain of professional development training, if learning exists we must be able to measure it. The question is then - what do we measure and how do we measure it?
Particular problem areas for trainers include the design and creation of assessment that is valid, reliable and flexible. In Australia, these are the key principles outlined in 1992 by the National Framework for Recognition of Training. However, what do they really mean when applied to a workplace assessment plan?
This essay will present and justify an assessment plan designed to support professional development training for management trainers. It will justify the worth of the plan in the three areas mentioned above, in addition to analysing the level of cohesion demonstrated by the plan. Furthermore, it will show the appropriateness of the plan for the participants, their employer and the wider community.
Intellitrain Pty Ltd is a corporate training consultancy specialising in the finance industry. The management team have agreed to work one day per quarter with people for whom the usual cost of training would be prohibitive. This quarter (Q2, 2011), they have begun to prepare for a workshop entitled “Job Skills for the Corporate Sector”, aimed at a group of seventeen unemployed youth.
2. Assessment Plan - Description
2.1. Context
Intellitrain is headquartered in Wharf Street, Brisbane City. They have appropriate training facilities on site and can easily accommodate both the training and the assessment. On Friday, 3rd June 2011, the morning will be blocked off and devoted to the assessment process. There will be four trainers being assessed at this time, the Chief Executive, Mr Paul Eldridge, the Chief Operating Officer, Mr Andrew Hetherington, the General Manager, Mr Byron Gray and Senior Trainer, Mr Eric Reid. The assessment will be conducted as a group in the training room in the interests of time, in addition to the need for an “audience” for the role play component.
Whilst the team are highly experienced and competent corporte trainers, their experience with unemployed youth is extremely limited. The management team have asked their Learning and Development Manager to lead the development of pedagogical sessions to ensure the training is effective and enjoyable for the youth. Given the very broad nature of the issues facing the youth, the plan has been designed to foster those skills that will ensure the trainers can effectively impart knowledge and skills without causing any undue dissonance within the group.
As corporate trainers, a very high level of technical subject matter competency is assumed, however for these trainers, the challenge of training unemployed youth is diametrically opposed to their usual audience of high level managers and professionals. This assessment plan has been developed to ensure the trainers are as prepared as possible for challenges arising from this unique group.
2.2. Overview
The assessment plan has one objective. That is, to assess the ability of a group of experienced and highly skilled management trainers in delivering a workshop on job skills for a group of unemployed youth. This learning intervention is far from the deficit or clinical approach, as these are hardly ‘struggling students’ (Taylor, 2007). The approach taken is a developmental one created to scaffold and build on existing knowledge and skills.
These trainers know how to train, and are completely at home with the subject matter. The purpose of both the training and assessment plan is to provide only that which will ensure the trainers can more effectively deal with the unique needs of a particular group. Thus, a generalisation has been made regarding the trainers and therefore the content, and this has informed the way in which the scaffolding may occur (Vygotsky, 1986). That generalisation is simply that the trainers need no additional subject matter or generic skills training, but rather supplemental knowledge and pedagogical skills.
2.3. Assessment Matrix
2.4. Outcomes
The outcome for the program is to equip the management team of Intellitrain Pty Ltd with the ability to facilitate the workshop Job Skills for the Corporate Sector to a group of unemployed youth. There are four modules included in the program. They are Literacy Skills, Computer Skills, Socio-Economic Factors and Counselling Skills.
For the purposes of this essay, the focus is on the first module in the program is entitled Literacy. This module requires the trainers to be assessed across four behavioural objectives. The objective is to ensure trainers demonstrate the necessary knowledge, skills and behaviours required to deliver a program to an audience for whom literacy may be an issue.
2.5. Targets
The plan could be effectively summarised as ensuring the trainers understand the issues facing, can create an atmosphere of acceptance and resolve issues for, and appropriately direct the unemployed youth. The plan demands trainers utilise conceptual, procedural and factual knowledge, whilst using the cognitive processes of remembering, understanding and applying knowledge to the various assessment methods.
2.6. Assessment Methods
The assessment involved is varied across objective, subjective and performance measures, with the trainers being asked to respond to multiple choice, true/false, fill in the blank and matching methods. In addition, they will be challenged to provide answers to short answer tests, case studies and role play. In the case of the module on Literacy Skills, they will be asked to complete an exam involving multiple choice questions to test their factual recall, in addition to performing a role play to demonstrate methods of problem solving should issues affecting literacy arise during the course of the training.
3. Assessment Plan - Appraisal
3.1. Validity
It would be very possible to fill the remaining pages wholly with an appraisal of the inferences made in determining the appropriateness and meaningfulness of the assessment results. However as Kane asked in 2004 with regard to validity – are the assessment results of interest and are they helpful in making good decisions?
This plan was formulated throughout with the aim of ensuring the principles of validity remained upheld as much as possible. The items to be measured were clearly defined in that the plan was devised to measure the very knowledge, skills and attitudes required to meet the behavioural objectives. The exam specifications were carefully prepared to represent that which was taught and a representative sample of items were chosen from the module (Gronlund, 2009).
The inferences about validity made of this plan on this basis are that a passing grade of 100% (as required) represents complete knowledge of the subject matter tested. Whether this grade then will represent that the trainer in question has a superior ability to deliver a course to the youth is too wide an assumption and required greater evidence. As we are observing only those factor in the assessment, it serves simply as a pointer or guide from which to infer learning (Griffin, 1997).
To highlight the more pragmatic elements of the plan, the tasks are set in not only realistic but the actual workplace, the evidence related directly to the outcomes being assessed (particularly in the case of behavioural objective 1.3) and both the exam and performance assessments are appropriate and typical of the training industry (Booth, R et al, 2002). A possible limitation in drawing valid inferences with regard to this plan may exist in the more subjective short answer and role play elements. Given the predictable propensity for assessors to interact and interpret scoring rubrics subjectively, some inadequacies may exist, however explicit the instructions may be (Gronlund, 2009).
There also exists significant potential to explore further that which could be inferred with regard to the construct and criterion related evidence, in addition to noting the consequences of its usage. The content related evidence apparent in this assessment provides confidence in a balanced and high level of content validity. The sampling of material taken from the literary module is highly representative of the overall content presented to trainers. In addition, the assessment was carefully and properly prepared with a similar level of diligence being applied to its administration and scoring (Gronlund, 2009).
A high level of performance in this assessment may indeed predict performance on a future assessment, or gauge present performance in relation to another concurrent measure. Moreover, performance in this assessment could be used to infer, for example a high level of ability to cope with change, a superior emotional empathic response or simply a good memory. To establish this however, would require a comprehensive study and greater additional evidence than at this stage is available (Gronlund, 2009). Given the above, in this instance the assessment plan has been created with and will be executed in a manner promoting as high a degree of validity as the limited scope and content allowed.
3.2. Reliability
The question of reliability or consistency of the results obtained by this plan highlights several considerations. Firstly, the objectives for which the assessment method of examination are used. Multiple choice, oral assessment and short answer essays are all open to questions of limiting factors such as too few items, limited range of scores, inadequate testing conditions and subjective scoring (Gronlund, 2009).
In this case however, the assessment plan and its objectives were devised with careful consideration as to the applicability and authenticity of the tasks involved, as asserted by Gulikers in 2004. As stated earlier, the focus needed to remain on the unemployed youth and what was needed to enhance the trainer knowledge and skill to present effectively in front of them.
To ensure this outcome, the content matter was therefore limited to the issues faced by the youth and how the trainers could best support and guide them during the session. This meant a limited amount of new, objectively assessable content matter and more emphasis on ensuring the trainers skills were up to the task. The factual and knowledge based elements of the assessment plan were thus limited in number.
The testing conditions have been controlled with the minimum possible disruptions allowed and scoring for these methods is as objective as possible given the highly prescriptive questions involved in objectives 1.1, 1.2 and 1.4. As a result, in this instance the test scores and range of scores are more uniform in this case as there is very little objective content from which to draw.
The reliability of the results obtained from objective 1.3 (role play) could be challenged due to the subjective nature of the task, along with the fact that there will be only one assessor present during assessment. Balancing these factors are the careful design of detailed instructions provided to both the trainers and the assessor prior to the assessment and the prescriptive nature of the scoring rubric included with the kit.
Given the short duration of the training itself and associated limited content matter, all that might have been done to ensure consistency of scores, judging bias, scoring and procedures has been addressed in a way that promotes an appropriate degree of reliability.
3.3. Flexibility
In 1992, the National Framework for the Recognition of Training outlined key principles of assessment, stating that it must be ‘valid, reliable, flexible and fair’. Hager, Athanasou and Gonczi maintained in 1994 that flexibility principles in assessment should cover both the on the job and off the job components of training, provide for the recognition of competencies no matter how, where or when they have been acquired and should be made accessible to learners so that they can proceed readily from one competency standard to another.
The attached assessment plan is flexible in accordance with these principles in that its objectives cover off aspects of the training that are principally set within a live training room situation, in addition to the knowledge based content which could be considered ‘off the job’. Moreover, should a trainer have acquired the competencies in another context, several years ago in another state, the assessment will remain highly relevant. Finally, the assessment plan can readily be made accessible to the trainers and adapted to a range of delivery modes, sites and needs. For example, all off the objectives (1.1 through 1.4) could be adapted to be assessed by distance, or even live via such technologies as Skype or video conferencing. There is also the potential to design and develop an eLearning version using such software tools as Lectora and Articulate.
There are no methods utilised in this assessment which would preclude the adaptation for variations in context or the needs and personal situations of the trainers (Rumsey, 1994). Indeed, in this corporate training context the need to incorporate a performance assessment to demonstrate objective 1.3 could easily be adapted as the live training abilities of the trainers is not in question. This couples well with the other more conceptual, procedural and factual targets in demonstrating a very high level of flexibility, and resulting integrity in this assessment plan (Misko 1994).
3.4. Cohesion
The question as to whether the assessment plan is integrated in such a way as to ensure it performs as intended and meets the requirements of all stakeholders begins with looking at its initial intent. The program outcome was to equip experienced trainers with the ability to run a workshop on job skills for unemployed youth. In module one, the focus was on literacy issues. The reason this module is vitally important is simply that, for the trainers, literacy issues have long been relegated to that knowledge and skills which are unconsciously competent. That is in terms of literacy, the trainers are unable to effectively teach something they have long forgotten they know.
To remedy this, the literacy model had four objectives. 1.1 was designed to ensure that the possible literacy issues faced by the youth were made clear. Objective 1.2 highlights the procedures that may be useful in creating a climate in which these issues are handled appropriately in a training room. Objective 1.3 allows trainers to demonstrate the concepts applicable should these issues arise and the last objective (1.4) simply provides the trainers with the knowledge with which to assist the youth in locating additional assistance outside the training room.
The assessment methods are clearly consistent with the desired program and module outcomes in addition to the learner needs, as they provide the most direct method of trainer performance specified by the outcomes (Gronlund, 2009).
3.5. Social Context
The assessment cycle is governed by both formal teaching arrangements and components of the social contexts (Rowntree, 1987). These contexts relate for our purposes to the learner, employer and wider community needs.
The trainers’ needs, as previously examined are met through the understanding and alignment of the program and module outcome, along with the assessment methodology. As to the wider social implications, the generic benefits to the employer include the improvements made in both training and assessing through various iterations of plans, the effect on future training plans and resulting curriculum and business planning. With respect to the appended assessment plan, the inferred results of the plan may influence future training and assessment in meaningful ways.
Further, this assessment plan and its results may provide an important case study and data which could be used by the community to duplicate, enhance existing or encourage the deeper pursuit of other such programs which also incorporate the bringing together of corporate trainers and unemployed youth groups.
4. Conclusion
It is clear therefore, that the measurement of professional development training by inferring results from an assessment plan can be a complicated task. Not only must the assessment plan be designed and created with a focus on the factors of validity, reliability, flexibility, cohesion and more but also with an awareness of the varying needs of stakeholders such as learners, employers and the wider community. Quality assessment must ensure that as many needs and considerations with regard to the above factors are taken into account before cementing the assessment objectives, targets and methods.
The assessment plan presented in this essay demonstrates an understanding of these factors and demonstrates this in recognising the existing skills and knowledge of the trainers, and creating an appropriate pedagogical scaffold to deliver the required results. It is brought together in a concise, valid, reliable and cohesive manner through the use of appropriate evidence, administration, instructions and scoring. In addition, the plan presents a high degree of authenticity and is suitable for adaptation to cater to the diverse needs of all stakeholders, should the need arise.
The design and creation of assessment requires a deep commitment in time and effort to ensure the kind of outcomes that the term ‘quality’ demands. In truth, no form of assessment could be considered perfect. However, further studies to identify opportunities for improvements, as well as a deepening of our understanding of how we learn and grow as human beings, should result in more robust and meaningful assessments in the future.
5. References
Booth, Berwyn and Clayton, Robin and Roy, Sue (2001) Maximising confidence in assessment decision-making: a springboard to quality in assessment. In: AVETRA 2001, 4th Annual Conference: "Research to reality: putting VET Research to work", 28-30 March 2001, Hilton Adelaide, Victoria Square, Adelaide, South Australia.
Griffin, P. (1997) Assessment in schools and workplace. Inaugural professorial lecture, University of Melbourne, September.
Gronlund, N. E. & Waugh, C. K. (2009). Assessment of student achievement (9th ed .). New Jersey: Merrill-Pearson.
Gulikers, J., Bastiaens, T., & Kirschner, P. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52 (3), 67-85.
Hager, P., Athanasou J. & Gonczi, A. (1994) Assessment Technical Manual. DEET, Canberra: Australian Government Publishing Service
Kane, M. T. (2004). Certification Testing as an Illustration of Argument-Based Validation. Measurement, 2, 135-170.
Misko, Josie (1994) Learning styles, National Centre for Vocational Education Research, Leabrook, ISBN 0863971431
Rowntree, D. (1987) Assessing Students: How Shall We Know Them? London: Kogan Page.
Tayler, C. (2007). Challenges for early learning and schooling. Education, Science & the Future of Australia: A public seminar series on policy. University of Melbourne, Woodward Centre, 23 July.
Thurstone; L. L. (1959). The Measurement of Values. Chicago: University of Chicago Press
Vygotsky, L. S. (1986) Thought and Language. Boston: MIT Press
6. Appendices
1. Assessment Plan – Complete
2. Exam – Multiple Choice
3. Exam Instructions – Candidate
4. Exam Instructions – Assessor
5. Performance Test – Role Play
6. Test Instructions – Candidate
7. Test Instructions - Assessor
6.1. Appendix 1 – Assessment Plan – Complete
6.2. Appendix 2 - Exam – Multiple Choice
6.3. Appendix 3 -
Exam Instructions – Candidate
10.4. Appendix 4 -
Exam Instructions – Assessor
10.5. Appendix 5 - Performance Test – Role Play
10.6. Appendix 6 - Test Instructions – Candidate
10.7. Appendix 7 - Test Instructions – Assessor
