However, many RTOs are moving their assessments online, not just as a repository of documents but truly online, and what may be a compliant paper-based assessment tool may not cut the mustard when it comes to online assessment. This article provides five key areas to consider when moving assessments online.
This article is brought to you by Kerri Buttery, Director of VETNexus and Partner in the VET Gurus. Kerri is a well known and respected Trainer and Assessor, Consultant and Guru when it comes to Assessment Compliance and VETtech.
1. Knowledge assessment question format
I am finding that many RTOs make the decision to go online with their assessment due to the promise of being able to auto-mark all their student submissions. While this is great for certain types of questions, not all questions should be recreated in a format that can be auto-marked.
Many Learning Management Systems (LMSs) provide the ability to set multiple choice, true/false, categorisation, matching questions, etc. There are a range of formats that can be auto-marked. Sometimes the questions in the existing assessments are already in a format that can be directly transferred across and this is easy.
In other circumstances the question needs to be reworded, rearranged, and thought of differently to be able to put it in a format that can be auto-marked. Most of the time this is possible – it just requires some creative thinking and sometimes to cover the content it means that it might be split into multiple questions.
However, we need to think about what is happening to our assessments when we do this. Are we losing rigour by giving students a choice of answers (just in different formats, not always multiple choice)? Are we developing the foundation skills of reading and writing sufficiently? Are we still meeting the requirements of the verb from the unit?
For example, it is important to consider that not everything should be auto-marked when it comes to knowledge assessments, and that good old typing in a response or having some verbal questions is still the best way to go for some situations.
Key tip: Analyse the situation with each individual question and whether changing it to an auto-marked question will still meet the principles of assessment and rules of evidence.
2. Practical assessments
Two common problems that I see with practical assessments online are:
a) not actually doing a practical assessment, and
b) not providing evidence around the decision making.
- Sometimes the assessment is asking students what they would do in a particular situation, for example, how would you conduct CPR? (An extreme example, but using it to make a point.)Instead of a practical demonstration on what to do, the student is asked to describe the order of steps, what they would do, how long for, what would they look for, etc., and describe it in detail. However, they aren’t asked to actually do it and be observed. This seems to be more prevalent in online situations where students do not have any face-to-face component of training and assessment.It is critical to note that while a student may tell you how they would do CPR, would you actually feel comfortable in saying they can do it if you had never seen them physically do it? No. Online doesn’t mean we can skip that practical observation.
- Internet access, cloud computing and small devices such as smart phones and tablets have made it possible to capture video evidence of observations. This is becoming common place for face-to-face and online assessment. Let’s use the technology to do this!Having a video doesn’t show how you made your decision as an assessor though. If you weren’t taking a video and you were in the workplace conducting an observation you would have an observation checklist. The checklist is the justification of how you decided that student has demonstrated performance to a satisfactory level. The same applies when you have a video. The justification of the decision still needs to be there, the video is evidence, the checklist is the decision making tool.
Key tips: Always include the practical assessment as you would in face-to-face assessment, the evidence may be captured differently though via video or teleconference. There must be a decision making tool included.
3. Assessor guides including checklists & decision making rules
As already mentioned, we need to have checklists and decision making rules for assessments. Just because it is online doesn’t mean this can be skipped. This does not only apply to observations but all assessments.
If a student completes a project, how have you evaluated that project? What were you looking for? There must be a checklist to show what has been considered and what has been judged as satisfactory. This links back to the Principles of Assessment, in particular reliability.
Assessor guides also relate to reliability and these are essential. They are not just for theory tasks, they are for everything. Too much to explain in this article, but it needs to be considered how this will be accessed in alignment with the online assessments.
Does the LMS allow for assessor guidelines to be included for the assessor, but be invisible to the students? Or is this feature not included, in which case what is the workaround to ensure the assessor has version controlled access?
Key tip: To ensure reliability of judgements there always needs to be checklists, decision making rules and detailed assessor guides available, regardless of the assessment being online or paper-based.
4. Authenticity
This is huge area and often what causes concern. How can you make sure that it is the student doing the assessment and not someone else doing it on their behalf? It really comes down to some of the same principles you would use in face-to-face assessment plus a few extra things.
Firstly, agreements around use of logins and passwords. I know this is just an agreement, and you don’t really know if they will follow up, but it still needs to be there upfront to make sure it is in the student’s mind when they sign up that they are not to give that password to anyone else – not even their workplace supervisor.
Secondly, online authenticity statements. If the LMS can’t do this, then you may need the student to sign a document to upload, or use an online signing system that you can integrate with your LMS. Again, they may sign saying it is their own work and you don’t know for sure that it is, but it makes them think about it.
The type of evidence that is collected can provide a third way of ensuring authenticity. Just because assessment is online doesn’t mean it can’t be ‘live’. Use of webinar or meeting software means you can interview the student and see that it is them on camera – if used for some verbal questioning or observation of skills then this provides direct evidence where you know what they can do.
Also, inclusion of unedited videos can show to an extent the skills or knowledge of your student. Some LMSs allow for recording directly into the system, rather than students creating a video, doing some creative editing, and then uploading.
Finally, there are some more tech savvy options such as proctoring systems, checking for plagiarism, taking random photos of students (be careful with that one) and other more advanced applications.
Key tips: To assist with authenticity, have students sign declarations and ensure a range of direct evidence is included in the assessment process.
5. Mapping tool
The process that brings it all together! A mapping tool is what will show you if you have covered the requirements of the unit. This can be challenging in online assessment.
There are some LMSs that will allow you to map directly inside the system (usually those that are built for RTOs), while others have no way of doing this (usually those built for other purposes and have been adopted by RTOs).
If your system can do this, then that is awesome – just make sure you are following good mapping practices to avoid over or under mapping (a common non-compliance). If your system doesn’t do this then you need to revert to your manual process, possibly using a spreadsheet or table document.
It needs to be really clear where parts of the unit are covered though, and the challenge can be labelling. Again, the document will need to be version controlled and stored appropriately given it won’t sit on the system with the assessments.
Finally, be prepared to re-map your assessment. Don’t be so worried about the mapping process that you don’t redesign your assessments appropriately for the online environment just to avoid mapping them again.
Key tips: Mapping must be just as detailed as it would be if working with paper based materials. Be prepared to re-map assessment once redesigned for the online environment.
There are many, many more areas that need to be considered when moving your compliant assessment online, but hopefully these five key areas can provide some discussion points for consideration.
