Foto: Deepmind, Domhnall Malone från Unsplash.com.
What are AI writing tools?
You may have heard about the use of Artificial Intelligence (AI) being used in new writing tools which are becoming widely available. These tools take a prompt from the user and produce some text in response. For instance, we created an account on ChatGPT, and entered the prompt:
“Why should a university use digital tools in education?”
Within a few seconds, we received a list of seven suggestions, which included:
- Improved student learning outcomes: Digital tools can enhance the teaching and learning experience by providing interactive, engaging and personalized content for students. Digital tools, such as virtual reality and simulations, can also provide new ways to visualise and understand important concepts…
- Increased Access to Educational Resources: …
And so on. You can see the full set of answers at the end of the article if you are interested. The list is a pretty good start, and would have given me some things to think about at the start of a writing project.
When I start to read in detail, I am likely to be more critical: I’m not aware of any actual evidence for the use of digital tools improving student learning outcomes, but the rest of that first paragraph is reasonable. Would you be able to tell that it was generated by a computer rather than a student? Even if you did spot some signs now, I could refine the prompt to improve the choice of language, and the system will improve over time.
These tools will soon be available to most teachers and students internationally, as they are likely to be integrated into commonly used software such as Microsoft Word and Google Docs.
The tools are software programs which can respond to natural-sounding language prompts, search for relevant information in a database, and then provide a natural-sounding answer. The databases contain vast quantities of information, but early versions of the tools cannot search on the internet for data – in the case of the November 2022 version of ChatGPT, known as ChatGPT3, it was trained using data up till 2021. The programs have been ‘trained’ to improve their responses by comparing their answers against those previously written by humans and also by being ‘corrected’ by people reviewing the answers. Whilst the term ‘artificial intelligence’ is widely used, the tools are “neither artificial nor intelligent” (Dahlskog 2023) – they are products of human interpretation and judgements.
What are the limitations?
These tools create sentences and paragraphs based on previous times the same words and concepts have been seen together in their database. This means that they may present with you sentences which are not true, and it is likely that the output will reinforce existing over-representation of particular ideas, as it may not have had access to a wide range of contexts. In addition, we know that this process of putting together plausible-sounding combinations means that it also creates citations and webpage links which are fictional.
What is the potential impact of these tools on our assessments in higher education?
The first message is: don’t panic. We have coped with the introduction of many ‘disruptive’ technologies in higher education before. The lecturer’s role in early universities was to read aloud from scarce books. The invention of the printing press, then the compilation of encyclopedias, the ability to draw on a blackboard, and then to create copies of notes for students all changed what happened in classrooms. Previous generations of teachers have used their knowledge of teaching and student learning to adapt their approaches to accommodate new tools. As a school student in the 1970s, I was not allowed to use a calculator in an examination, in case it interfered with my ability to use a log book and a slide rule, two items which have been completely superseded by the much more useful calculator.
AI tools will have an impact on assessment and you need to think about what is best for your course. In thinking about this, you should return to your knowledge of good assessment practice. According to Biggs (2003), teaching and assessment should be aligned, using SMART learning outcomes and clearly defined assignment tasks. Biggs and Collis (1982) created the SOLO taxonomy to classify learning outcomes in a structured way. It is worth revisiting these concepts to help you decide what to do in your course.
Three main options for considering AI writing tools in assessment
We will provide more detail on different approaches to designing assignments on other pages, but as an overview, the main options for considering these AI writing tools in assessment are:
1. To ban their use (like the calculator example)
The only way to do that reliably is to have the assessment in constrained conditions – a supervised examination room with no use of external technology, or using a computer which is locked down so only certain software can be used. This may be the right approach if you need students to reproduce factual knowledge under constrained conditions.
You might do this if these are the kinds of learning outcomes you have: The student will be able to: define, identify, reproduce, describe, sequence, combine, classify…
Advantages
- You can test factual knowledge
- You know the answers are the students’ own work, or at least produced from their memory
Disadvantages
- You may have wanted the students to spend more time and thought on the task than is possible in an examination situation
- In future work, students may need to use these kinds of technologies in more realistic conditions and they will not have experienced them.
2. To assume that they may be used, and set tasks which incorporate them
This means thinking about what skills students need to prompt the tools and how they will assess the results.
You might do this if these are the kinds of learning outcomes you have: The student will be able to: compare, explain, analyse, create…
Advantages
- You allow students to explore the power of these tools in relation to your subject area
- You can work with students to explore academic honesty and the purpose of academic assessments.
Disadvantages
- You will have to rewrite the assignment tasks to focus on critique and analysis – this may require changing the course learning outcomes.
- Not all students may currently have access so you might have to set a choice of questions – one using these tools, one not using them, and ask students to say which they have picked.
3. To require them to be used – fully incorporate the tools in the assessment
This is not possible at the time of writing (March 2023), we have not yet studied the GDPR implications of the service, and as capacity on the free service is limited, and you couldn’t guarantee that students would be able to create an account. But in the future these tools will be more generally available so you could be preparing for how to do this in the future.
You might do this if these are the kinds of learning outcomes you have: The student will be able to: evaluate, theorise, generalise, predict, hypothesise, imagine, reflect…
Advantages
- You ensure that students can explore the power of these tools in relation to your subject area
- You can work with students to explore academic honesty and the purpose of academic assessments.
Disadvantages
- You will have to ensure everyone has access
- You will have to rewrite the assignment tasks to focus on critique and analysis – this may require changing the course learning outcomes.
References
ChatGPT website – chat.openai.com
Biggs, J. (2003, 10/11/2006 16:20:43). Aligning teaching for constructing learning.
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome). Academic Press.
Link to the book Evaluating the quality of learning – sciencedirect.com
Dahlskog, S (2023), private communication
Example of output from ChatGPT3, January 2023
