Transforming Assessment Webinar 1 April 2020
Presenter: Mathias Magdowski (Otto von Guericke University Magdeburg, Germany)
Ideally we should be able to provide a personalised task for every student in an engineering exercise or seminar. This would reduce the risk of copying and plagiarism, but is the effort too high for the lecturer? This session explores how digital tools can help to solve these problems and how students can effectively assess each other’s work. Each student receives their own assignment by email, is able to work through it it without time pressure and then submit it via our learning management system Moodle for assessment. In order to reduce marking load of teachers, the students then assess each other using a sample solution, which is also personalised. The assessment process is automated and therefore easily scalable. In contrast to simple multiple choice or numerical response questions, the calculation method and approach can also be evaluated. Examples are provided from STEM disciplines.
This session was hosted by Dr Mathew Hillier, Macquarie University, Australia starting 07:00AM UST/GMT. Duration 58 minutes.
Further resources for this session (slides and chat log):
http://transformingassessment.com/events_1_april_2020.php
See more about e-assessment and webinars at Transforming Assessment: http://transformingassessment.com
Okay welcome to the transforming assessment webinars this is uh the session for first of april 2020. we’ll be looking at uh personalized tasks and anonymous peer grading by matthias from germany sorry i can’t pronounce the name of your university but thank you very much mathias for coming along
Yeah good morning to everyone uh so actually the the university is pronounced autofocus it was a major of our city and he was also a physicist who found about uh about vacuum and did some very famous experiments at least in germany with vacuum and to be honest my my first name is
Also pronounced matias we don’t have this th in germany but but anyhow this is um i’m used to it okay so the topic that i’m trying to explain and by the way i will switch off my video now because everyone has seen me and this will hopefully save some bandwidth for us all
Um yeah it’s as already said personalized tasks and anonymous peer grading so the plan is that i explain a little bit what is the motivation behind this why i have developed this feature or this method and then i will show some typical tasks they are from our course about fundamentals of electric engineering
So if you are familiar with electrical engineering you will you will maybe understand these tasks otherwise it’s just to give you a brief idea how these tasks look like then i will explain how the procedure works and i will show some results and we can maybe have a little discussion about this
So why all this um what is the motivation behind and before i start to explain this i would have to like a short survey and ask you what is your background so um are you let’s say from general education from school or high school or are you from an university or college
And then social sciences economics natural sciences technology or do you already come from electric engineering um just post maybe the number we have to tried earlier the survey tool in this blackboard is is not so nice that’s why we just said it would be easier to do this just via the chat
Okay so i have some threes and fours and uh i think lots of universities but not necessarily engineering sciences and also yeah like i think some some learning centers and teaching units um that support other structures in your university okay thanks thanks for this um so so not too many electrical
Engineers i see okay um so i will i would try to not to go into too much detail about electrical engineering stuff thank you okay so um the motivation is that i said we have a course about fundamentals of electric engineering it goes over two semesters
And of course at the end uh over this two semesters or over one year we have some exam and the classical exam looks like this that um of course we force the students to solve tasks alone without any support of books or formularies um so they cannot use their um
They cannot use their smartphones they cannot use their computers they can just use their calculator and they can also not discuss with each other because of course we are interested in what they can do on their own um still i think the problem is that i’m i’m a little bit confused about the
Chat okay um the problem is that no engineering problem like this um or no no no real world engineering problem is solved like this i mean no and no big companies the boss says here go to some room um be there alone for three hours just with
Your calculator and a sheet of paper and a pen and now develop the new charging system for the electric vehicle this won’t work out so um yeah of course universities are somehow traditional and uh stick to this classical performance assessment but yeah it’s no real replication of the real world
And then of course it does a lot of work for for the lecturers to grade all these papers so i said the demands in the real working world is that um with something like the internet students can use computers there are lots of simulation tools um for all engineering sciences also for
Electrical engineering and persons can also discuss with each other and can solve problems and collaboration and i think this is nothing that we really assess now at least in germany in our classical enzymes maybe maybe it’s a bit different in australia or somewhere around the world
But at least in germany where we are very very traditional about this and so then we also have something like e-learning tasks um so where where we can have something like formative assessment assessment during the course um and yeah so i’ve i’ve inserted here some classical or some typical tasks um
From our moodle from our e-learning course um i’ve not translated it into english but there’s a schematic and there are some values for this elements and then uh the students should calculate a current or certain voltage in the schematic and the real task for the student at the end
Is just to fill in um i mean of course this is interesting but at the end what the student has to do is fill in a number here um the right number was the right unit click some button and check if it’s right or wrong and what typically happens um at least what
We have found out is that um only these students do these tasks that would not really have to be because they are already good and the um students that yeah have the the um the greatest potential for improvement let’s say like this they at least um on a voluntary basis
They do not do these tasks and then sometimes if they do then it’s done in a way that they they send some good student and say solve this task give me the number and at the end they just fill in this number and say hey i’ve i’ve somehow solved this task
At least if you give them points for this so um it’s somehow nice of course because it scales you can give such tasks to a lot of students and the learning management system like moodle will automatically grade them um yeah but in my opinion it’s it’s still not really um
Good tasks a good task to check if people have really understood about what they are doing okay and there is a lot of mental effort let’s say um in really developing a freehand written solution now start with a white sheet of paper and write down formulas for a problem think about what
Could be what would be a good approach uh what formula do i need how to insert all the values that are known into these formulas how do i rearrange these formulas um to develop some solution for the engineering problem for the physical problem take care about the units uh
Take care about the calculus and so on and so on and what what can also be done um in such free hand written solutions of course is that students can draw diagrams they can draw plots they can draw schematics they can use schematics they are already there and Draw errors in schematics for there’s a voltage um in this direction there’s a current flowing through this in this element uh or there are elements in some flowchart connected to each other and so on and so on um and this is some some solutions like this are very
Difficult to develop in the classical e-learning systems like moodle i think i mean you can have multiple choice questions and you can have like some drop-down questions um but it’s really difficult to um to really test these yeah kind of freehand written solutions so our idea was that students get tasks that
They solve by hand on a piece of paper because this is a little bit how our exam works and where you really have to think about the problem if you want to start with a blank sheet of paper and write down everything by yourself from scratch and then students should be able um
To to great each other or to review each other even if they were not able to find a good solution for their own problems so they must be really good solutions for these tasks so that students can rate each other even if they did not really understood about their own problem and
The tasks like this or the solutions like this they should be personalized so that the students can can talk to each other about their problems and can discuss their problems but they cannot just copy from each other or they cannot plug your eyes this was the idea
And um yeah i will show later on the procedure show some some first um some how some typical tasks look like um and i have developed since now i think 11 or 12 of these tasks and um because this is a typical question it it
Takes me about like a weekend or so to develop and to program such a task so how do these tasks look like and here i have translated it into english again if you’re not from electric engineering you might not be able um let’s say to find a solution for this task but
Typically the students get all the very same tasks like there’s a display curve of a current and from this current the charge um should be calculated or the charge is searched for and um there are some time sections and some initial charges given um and so all students have exactly the
Same task but every student gets a different diagram so and these diagrams are generated by the matriculation number every student in germany has a six digit number at our university and with the six digit number um i feed some random number generator so every other matriculation number gives a different diagram
So another number gives a different diagram that for example looks like this but if i insert the same number in this random number generator i get the same diagram again so i can regenerate these tasks in case of something fails or if in case of something is wrong and i just
Have to redo it and so again a different number gives a different diagram and now you can already see a little bit how it works in this first section here it’s always a linear function that goes through the origin and then in this second and third and fourth section is always a concept
Ah so um the effort in calculating this for every student is the same but at least they all have a different diagram and they all have different numbers so everyone has to think about his problem himself they can discuss each other and they can they can try to find the solution
Together but at the end everyone has to find his own solution even if they are quite similar they are still different in the numbers okay and as said the good thing is because the students have to review each other and they have to grade each other and they
Should be able to do it even if they were not able to find the proper or the correct or the right solution for their own task i can generate a very detailed simple solutions for all these tasks so again this is based on this on the matriculation number and for this number
Um i get the solution yeah so um What really changes theologies in this case really just these numbers this is quite simple example are just the numbers inside um these equations here so for this diagram there’s a one there and then we integrate over this one and at the end we get 0.5 here and so on and so
On so but everything is step by step written down very detailedly and um so this is just for the first um of these four time sections and uh and so then for the next student the solution would look like this so there’s in this case really just a different
Number here and here and everywhere um and so for every student i said i can generate without problems computer algorithm the program does it automatically i can generate all these sample solutions and if you ask how it’s done it’s it’s based on late age and what’s done in
Late or what can be done in late age is as these diagrams before i can also draw the diagrams automatically how would the final solution for the charge now we start with the current as the function of the time and what was searched for is charge
The charge it’s as a function of the time and so for every student again based on this matriculation number we get a different diagram how um the charge develops over time and now you can see that even um even if the task is the same for everyone and just the diagram that
We start with is different of course at the end we also end up with a very different diagram for every student also this should give you some basic idea on how how such a typical task looks like and this brings me to a second survey and the question is can you think of
Some Kind of the same way personalized or randomized tasks that could be also used in your branch in your field of science in your department and again i will ask you just to paste the number into the chat box and i see some threes and some tools and some ones
And that’s good because it means yes it’s possible uh yeah and i know i know a colleague from from our university in machterborg who has developed similar tasks for fluid dynamics um so for another branch of engineering sciences okay thank you um so then i i want to show a second task
It’s number seven here because it was the seventh task that i’ve developed and i have a larger set of slides where the tasks are numbered like this and i would not confuse myself that’s why it’s maybe confusing for you but for me it’s task number seven and um yeah again
Um the task is the same for all here the idea was that the students should draw such a phasor diagram and if you don’t know what the phasor diagram is don’t mind but the idea is that they draw such kind of a diagram and from this diagram at the end it’s
Really difficult to see if it’s right or wrong if you see just the complete diagram and if it’s wrong it’s very difficult to say until which stage it is right and when it gets wrong so here um the student should not just write down their solution and
Upload a photograph or a scan of it here the idea was they should create a video of how they draw this diagram um uh so i said because it’s they should not show the final result of this question of this task but they should show the process of how they created their solution
Um and i gave them some hint and say if you have a good tablet you can make a screencast of how you draw this with your pen on the tablet or you can create a simple tripod for your smartphone camera and just film your hand
While you draw this on a piece of paper and give them some hints on what they should look at and so on um and said also there should be no sound in the video because it should be anonymous everyone has a certain voice we do not
Want to hear this voice um do not take too high quality videos because otherwise it would be too large uh it should almost also not be longer than 10 minutes and 500 megabytes 10 minutes because someone has to watch these videos and review it it should not take too long
And 500 megabytes was somehow the upload limit of our e-learning movie system and then just yeah when drawing they should use different colors because it’s easier to then see what what line belongs to which phasor in this diagram and they should look at the scale and so on and this is the um
The idea that i gave them on how to build such improvised tripod for filming the handwriting very easy to build from household materials and to give them some brief idea i also created some example video and uploaded this to our e-learning system and to our twitter account um
Where i would solve a task like this and i think it took me something like five minutes and then i said okay if i can do it in five minutes ten minutes should be enough for the students um and so again then everyone got this very same task but each student again based
On the matriculation number based on the random number generator um got a different schematic so one schematic looks like this and another schematic looks like this and another schematic looks like this and now you already get the idea that it’s always five elements um so for example you two resistors and some
Adductor and some capacitor and another resistor and um so every circuit has five elements and there are always um two changes of uh the topology so we have um um a serious circuit here then we have some parallel circuit here and then again we have some serious connection of these whole branches
And so these five elements and three changes of this topology we have in every one of these circuits so the effort in drawing these phasor diagrams at the end is the same for a student even if these schematics all look very different and again students can discuss
These problems with each other they can also look in the internet but they will not find exactly the same circuit so they have to think about a solution for themselves and um so the the the sample solutions and here this final phaser diagram then looks for example like this and like this
Or like this or like this and you can see it’s always nine phases that are inside this phasor diagram but each phasor diagram looks very very unique because every circuit is unique and every circuit has also different values for the elements so that’s the idea behind this task and
I said so i have by now i have 11 or 12 different tasks like this okay so then um i want to briefly explain how the procedure works so what happens technically in the background and this was also the schematic that was used to advertise this webinar
And now i’ve translated it also into english so The students here on top these are always the students the students register in our e-learning system and in our case the e-learning system is a moodle and i download a list of students from this moodle system and feed it into matlab i have programmed everything in matlab weimar
Club okay i’m an engineer my network electric engineer electric engineers know how to program in matlab i know that um Yeah it’s it’s not so it’s not so well known in in other um fields of science and my colleague in magdeborg olivier who did this for fluid dynamics who who did he did all this stuff in python for example but at the end it’s just a programming tool
It would be good at the end if one could do all the programming again back in in moodle but it was much easier and much faster for me to program everything in matlab than a moodle that’s why i did it in matlab okay and so from matlab
I use the matriculation number and with the matriculation number i feed some latest source and the latex source then generates all these tasks and for every task i get back some pdf file and this pdf file is sent via email to the students and the students then have some time to think
About their solution and at the same time i also generate these simple solutions and just store them intermediately into some folder and we need this folder later on to to assess these simple solutions okay so then the next step is that the students they found they have usually they have one
Week to find the solution the deadline for submission is always on monday at 10 pm in the evening and they upload their solution with a task form into the moodle system and so then i download all the submissions for from all the students at once using a zip file
And again put this into my matlab algorithm and what the matlab algorithm does is it now mixes up these tasks and sends every student um two emails and the emails always say okay here is one solution of another student and the corresponding sample solution please create this and upload it back into the
Moodle system and the second email says okay here’s the second solution from another student the second corresponding sample solution again please create this and upload it back into our mule system so then the students um think about how to review these papers and they say okay this is right or this is wrong
Um based on comparing the solutions with the simple solutions so again they usually have one week to do this and again the deadline then is monday evening 10 pm and what they upload back then in the moodle system is the the reviewed or the graded tasks of their peers and they insert
The the point number the score that the students have their fellow students have reached into a text field again back into our moodle system um and yeah these points here they are unfortunately they are necessary otherwise no one will really solve these tasks and we don’t give these points so
That they are the the points are necessary um At least from our point of view as some extrinsic motivation unfortunately it is like this um and if something is wrong with the points and the students write long emails or my fellow student has forgotten that this is right but he said it’s wrong and i think it’s right and
It’s really right and can i get this point and so and then we always say oh it’s not really important it’s not about the points it’s about that you have thought about this problem and that you have developed some method um on how to solve this task and um
That you learn something and it’s not really about these points but the students are really really really um upset if they don’t get their proper points okay so then i have all these rated um solutions in moodle again i download them all as a zip file put them into my matlab algorithm
And the matlab algorithm then automatically sends back each student these two reviews and the corresponding sample solutions so that then the students can have a look at um yeah was i graded correctly um does this correspond to the simple solution was everything right was everything wrong and so on and so on and
These points they are stored in some score list and the score list is not really used for the exam but the students have to reach a certain score to be allowed to take the exam so it’s like a pre-exam or something i i don’t know how you would like to call
This but um the points are not really used for the final exam just to be able to register for it okay so to discuss the results what um what has happened um and so a typical cycle looks like this that i sent these tasks to about 200 students this is roughly the number
Of students in our course and yeah usually something like 150 students would uh take part and would upload a solution um and so the other students are not interested or they forgot about it or maybe sometimes i think they also already have enough points and then they
Say okay i’m fine i can do the exam i do not need further exercise anymore and um then i usually lose some some five to ten percent um of students for the review so yeah typically let’s say 10 less upload reviews um then of course some reviews are missing but
Still i mean for every student i off for for every task i get two reviews so if one of them is missing this is not really a big deal and the students are told that they only get points for their own solution if they also upload the reviews so if they
Forget uh or if they are not interested in reviewing each other then they also do not get points for their own solution and as i said they are very very interested in these points so they um the percentage here is usually quite high so i would say this is um
Yeah compared to other tasks that we give out to students and to these e-learning tasks and so on it’s some excellent activation of the students we also reached the students that would usually typically on a voluntary basis would not do these tasks and would not develop a solution
And i think it’s also good preparation for the exam without this teaching to the test that you give the students exactly the same tasks for exercising then the same task that would be asked for in the exam and typical distribution of the points we always for each task we give 16 points
So that these tasks are comparable with each other and a typical distribution of the points looked like this that really most of the students would get full points or almost the full points um and yeah i think this is also okay because they have one week
Um i said they can ask each other they can discuss with each other they can ask the internet they can um i mean at the end they have to write down a handwritten solution but they can also do a simulation they can also do some calculation in matlab
They have a lot of time to check the result so i think that’s why it’s quite common that most of the students get full points and the difference between the reviews between two um scores is usually also quite small just zero or one point um and yeah i mean still there are
Quite a high number where let’s say one reviewer just gave seven points and the other one gave 13 out of 16 and then my my algorithm gives me a warning and says oh look there’s a huge difference between review one and two maybe someone should have a look that really has knowledge about
This topic and then i or some colleague looks about these solutions um and then yeah these are sometimes really difficult solutions so the idea of the student was was good but then still how it was done was really bad and then it’s always a question if do you count points or do you
Do you count mistakes and how you do it you will end up with a different number um then what’s also quite interesting because my moodle system tells me at which time the students upload these tasks of course i do not really know at which time of the day they do their tasks but
I assume that if you have done the task you will not wait for three hours and upload then but if you have done the task you will submit it so um here out of something like 2 400 submits i have made a histogram of at which time of the
Day the students are active and upload their tasks and see i mean this is over the whole week our deadline is monday at 22 p.m so here’s the deadline that’s why of course a lot of this happens before the deadline and much less after this but students are
Quite active during the night and uh you know early in the morning between five and six or seven you can shut down all servers and install updates on your moodle system no one will care about this so um one interesting question for me and a lot of things happen in the chat already
To be honest i i cannot speak in english and at the same time look and read the questions in the chat i will hopefully come back to this later so the the survey now would be what what questions would come to your mind uh regarding this evaluation um yeah so i have lots
Of data from the students um let’s say at which time they submitted with which point score and so on and so on um and i said i’m an engineer i’m not a deduct uh from a pedagogical point of view what would be interesting questions to do with this evaluation oh thanks
Okay there’s uh i think this was just a comment about blind peer review um yeah so for example what would what would be an interesting idea for me maybe i will do this in this starting summer semester and this starting summer term that i when the students upload their results
That i also asked them how difficult was the task for you how would you grade the difficulty of the tasks because i have not shown this curve but it’s what is also quite interesting is if you look at the curve how the students maybe i can draw this here uh so if if
This is the time the one week that they have and if the deadline is here then of course what happens the students have to full week nothing happens nothing happens no one’s uploads it no one uploads and something happens something happens and most of the uploads are really really shortly before the deadline
And what i’ve seen interestingly is that um if if the tasks for me if i would say it’s an easier task then the curve is flatter and if it’s a more difficult task then the curve is really really steep at the end so from my point of view this would be
Something interesting to discuss interestingly what happens if the students review each other and they have one week to upload the reviews it’s more or less always a linear curve so if you develop a solution you ask really for the latest moment to to to do it and
To upload it and then if you are asked to review each other the students would i think at the moment when they get the email and read this email they would directly do it and upload it um and that’s why it’s more linear over a week and
Well i have no real idea about this but some students told me that is um if you review each other of course you also get some idea if your own solution is right or wrong and this is what you want to know at the end okay some fun facts um i could not
Translate this because these are screenshots here a student has uploaded his solution under the file name let’s say full score of fullpoints.pdf i think that’s very interesting another student has uploaded his solution under the file name my solution or my own solution uh which is also kind of funny i think because
If if there’s a file which is called my own solution is there also a file that’s called not my own solution i don’t know and here someone has called his his file name for submittance submit me dot pdf um yeah and then what we also see is that students have problems in operating
Their computers so here student uploaded the file link and was not aware that this of course will not work and was also that the filing of course is just let’s say 500 bytes or something like this and then i wrote him an email and said look you have just uploaded a link to
The file the link would not work on my computer and he said no but it works on my computer yes it is of course because you have the file on your computer but i don’t have the file it’s just a link so um i think we we also observe
I mean these are first semester students um but i think we observe a lot of problems with that students do not are not really able to operate their computer operate their smartphones their laptops then another problem is poor image quality of submissions or corrections so that they make a photograph of their solution
But there’s some shadow on it or there is some glare and you cannot really see what happens there and that then people um yeah also do not really care about this and they are very very good smartphone apps i think where you can make a photograph in it will automatically collect correct the
Contrast and will also correct this this misalignment here so that you can always get the perfect picture and yeah we also suggest the students to use these apps but yeah sometimes still difficult another question is of course the credibility of the corrections again i could not translate this because
It’s screenshots but here yeah they’re spelling errors um and so if you think okay students are not able to write their sentence correctly is also sometimes strange um and then there is some potential of improvement in our e-learning system especially when the students insert these numbers for these points
And so the the last question is which e-learning system do you use at your college or university okay and looks like that most people would use moodle so um okay thanks a lot and the last two slides are just questions that i would like to discuss with students on a roundtable discussion
For further development but to be honest i’ve not found time to do this and at the moment of course it’s also quite difficult to do round table discussions with students thanks for your attention and i will be happy to answer some questions matthew have you have you collected
Questions in between from the chat uh they did go flying past um i did have a question that i had um put in before was um have you used your uh random sort of question technique for uh giving students end of semester term exams or are these just sort of
Like formative learning activities they are just formative learning activities so the the exams at the end are really in this i mean maybe we will change it in the future but at the moment they’re really in this way that all the students get exactly the same tasks um so that it’s um
That it’s really really comparable I mean i think the problem is that i’m also not an expert in this but that there are some legal issues in the background that um exams should be comparable at least for one batch of students and that you could run into legal problems if you give different tasks to different students
For the final exam sure yeah yeah um i think there’s probably ways of statistically working out if if the questions have equivalence in terms of their difficulty i’m not an expert in that area but i believe there is probably some method to do so okay i am Um so there’s a question from peter who asks do you wise do you use poise or similar i i i think i’ve heard about peerwise um i’ve no idea at the moment what poise is i think it’s a tool to do um peer reviews um no i’ve i’ve not used
It i don’t know it in detail i can tell you if you like um wise is like a crowd sourcing tool so that students can create a whole bunch of multiple choice questions for you and then they the students then do and rate the questions so the act the idea is
You know the idea if you teach if you’re able to teach somebody else or explain a concept to somebody else then you yourself are learning a lot so from a point of view of a student having to create a question themselves then they are learning and then they have to you
Know uh explain it and include feedback for the wrong like if it’s a multiple choice question they have to include uh valid feedback for the um for the wrong answers it was the correct one um yeah it’s it’s quite increasingly common i think quite a few universities in
Australia use it um it’s used all around the world but definitely peer wise you can look it up if you have moodle there’s also a plugin i think it’s called student quiz that’s a similar idea of crowdsourcing you’re just using the standard question types in moodle and you get the students
To create questions and then they can rank each other’s questions etc i’ve just pasted the link to another tool from the technical university of dressen in germany that is also for peer assessment and i think there’s also um a tool for peer assessment in uh in the moodle itself
But the problem there is that you can just give one task for all students and you can also just give let’s say one solution or one grading one joint grading form for all the students that is used for the for the peer review and you cannot have this kind of personalized or randomized
Tasks that i do okay so then there is a question from james what is the best citation that describes this approach um at the moment there are just german papers about this my colleague olivier from fluid dynamics and me we have written in english paper that is submitted and reviewed that
Should be published soon but um i don’t have a proper citation for this at the moment but there are some yeah if you if you write me an email james i can at least send you some german citations and i can also send you a copy of this english paper if you like
Um Oh very good okay so ah there are more questions unfortunately i cannot uh scroll the chat at the same time while people paste something in the chat um are there are there more questions yeah folks the chat is not brilliant so folks can if you could just type your uh
Questions into the text chat again if you happen to do them a lot earlier um and we can get to them okay there’s a there’s a good question what i’m interested in is the feedback the students actually self-generate if you ask them not the teacher to make
Comparisons of their own work with other submissions or their own work by comparing to others with each other yeah so the students i said students grade each other based on these also automatically generated simple solutions i think it would be possible to do it just comparing their own solution to
Other solutions but it’s it’s engineering tasks i mean there is not so much to discuss um you can have different approaches for for certain problems um but it’s it’s not let’s say like writing an essay where it’s um not just right or wrong but there’s a lot of gray
In between i think it would be difficult if they just um make comparisons or make a comparison of their own work with other submissions but maybe what also answers this question is that the quality of the reviews is usually very very good so we have made very good experience also in other
Subjects with this peer review that the students do it really really fair and also really really thoroughly so i think they also have a very good feeling how much effort they put in their own solution and if they look at the different solution i think they not not by looking if it’s technically
Right or wrong but just in a way how it’s written and in a way how it’s presented i think in most cases they also have some some quite good idea or impression if um the other student a fellow student has invested some effort in creating this solution and
Has thought about it or if it’s just drafted very very quickly in five minutes and um yeah put very very quickly on the piece of paper then there’s another question what is the likelihood of using pure grading and saying assessing essay tasks we do this um also
I’ve not not presented this year but we also do it we have some seminar about scientific writing on scientific working where students get a lecture about how to write paper how to give a presentation how to draw a proper diagram how to cite correctly what is plagiarism and so
On how how it can be avoided and then they should write a very short paper about a non-technical topic so why is so much tomato juice drunk drunken on airplanes why is snow white this uh coca-cola invented santa claus some non-technical topics that’s why it’s called non-technical project and
Their students also um grade their papers each other and this also works very very well but you have to explain it to them so we usually use one full lecture 90 minutes to explain on a sample paper how this review works and what they should look at and how to
Fill out this review form and if you explain it properly then they also do it very very well yeah but it needs a strong rubric exactly yeah you cannot say okay here’s the paper do some review then everyone will do something else you need to explain it very well and then it
Works usually very very good and the students do it very fair ais and for i said for this essay we do it and practice it for this um electrical engineering we have too many students to do it really in the seminar but we do it then in the lecture where we
Have all the students and we shortly show okay um this is how some solution looks like this is how the corresponding sample solution looks like um and then you go through it step by step and check and please also be careful check not only the numbers also check the units
And what would be proper signs to mark if it’s right or wrong okay and there’s a question are there tips in supporting students to internalize the success criteria and to use the rubrics effectively for marking and feedback um to be honest i do not really get the question um
I think personally it’s about having students practice with it um i’ve seen it done really well in other stem courses as well where you have students evaluate each other against the criteria in classes and then they discuss about the criteria so then there becomes like a group understanding about what the
Criteria and what the standards each level mean so i i’ve saw a tutor at university of queensland she um she did a very good job of this led the tutor training program along these lines where they had where they had their people that first they gave them the work they asked them
To say oh how much would you give this a credit a distinction or whatever they would put up their hands they had a voting and then they handed out the the criteria sheet that the rubric and they did the exercises again and then after a discussion they did the exercise for a
Third time and what they found is by the time they got to the third time the group had converged on uh the standards around where how and the standards are and and and the the criteria or measuring the standards of the criteria reasonably well so i think
It’s just like you do with marking moderation between you know a class of tutors or practical demonstrators you get them together and you do a sample work then you mark it together and you discuss it so i think it’s the same we we do if we’re asking students to do
Peer reviews i think we have to teach them how to do peer reviews we have to teach them about the the criteria and the standards and we discuss it with them so that’s my first stance on it yes calibration exactly yes jackie you’re right yeah i think we we do this um
At least for our class or for our lecture we do it in the seminars so we we do not have just only the lecture with all students but we have um seven or eight or nine Um smaller classes every week where we just have a group of 15 or 20 students and yeah they also have some kind of peer teaching model would be too difficult to explain now but i think we also do it not with this with this personalized peer review tasks but in our usual seminars
Students write solutions on the blackboard and then the the solutions are discussed and the solutions are corrected and so on so we i i think we do it but in a different way but yeah these uh self-generated own criterias are um in comparing them with other criteria it definitely helps to calibrate
These internal criterias yes thanks david for this comment yeah i also have to say that this is a an important skill for professional practice i mean they have to be able to judge their own you know success in in the workplace so it’s a good idea to do it now when when
You know the lives and money are not at risk as it were okay are there further questions uh matteos i think we are just about at the end um it is nearly uh nearly 7 p.m here in sydney so if there’s any last minute questions folks and now’s the time
If you would like to ask a question for your microphone in the last minute please put up your hand and you’re welcome to do that as well so okay thank you very much folks yeah yeah uh let me just um we do have a new session coming up um
That’s just only been stuck on our website if i can manage to find out uh what i did with that um helps to be prepared for these things doesn’t it okay so the session will be coming up on the 29th of april it’s about uh detecting and addressing contract cheating in online assessment
Uh i think it’s probably going to be a timely kind of a a topic because at the moment everybody is changing their assessment modalities they’re changing their assessment mixes uh the russia online with covert 19 etc is forcing people to rethink about their assessment mix and often that has been
Turning away from exams towards things like take-home assignments and that sort of thing um my feeling is that there will be consumed concern in the academic community about the potential for the uptick in the contract cheating um so from that point of view i’d like to
Present the session uh it’s going to be presented by phil dawson who’s a well-known person in australia with work on uh academic integrity and contract cheating and things like that um so that will be 29th of april uh look out for the email shortly you can sign up now at the
Transforming assessment website so i’m going to stop it here thanks matthias uh excellent session i’m sure many people particularly in stem disciplines will find it very useful so thank you much thank you