In a recent seminar in our current series on teaching programming (with or without AI), we were delighted to be joined by Dr Stephen MacNeil, Andrew Tran, and Irene Hou from Temple University. Their talk showcased several research projects involving generative AI in undergraduate education, and explored how undergraduate research projects can create agency for students in navigating the implications of generative AI in their professional lives.

Differing perceptions of generative AI

Stephen began by discussing the media coverage around generative AI. He highlighted the binary distinction between media representations of generative AI as signalling the end of higher education — including programming in CS courses — and other representations that highlight the issues that using generative AI will solve for educators, such as improving access to high-quality help (specifically, virtual assistance) or personalised learning experiences.

As part of a recent ITiCSE working group, Stephen and colleagues conducted a survey of undergraduate CS students and educators and found conflicting views about the perceived benefits and drawbacks of generative AI in computing education. Despite this divide, most CS educators reported that they were planning to incorporate generative AI tools into their courses. Conflicting views were also noted between students and educators on what is allowed in terms of generative AI tools and whether their universities had clear policies around their use.

The role of generative AI tools in students’ help-seeking

There is growing interest in how undergraduate CS students are using generative AI tools. Irene presented a study in which her team explored the effect of generative AI on undergraduate CS students’ help-seeking preferences. Help-seeking can be understood as any actions or strategies undertaken by students to receive assistance when encountering problems. Help-seeking is an important part of the learning process, as it requires metacognitive awareness to understand that a problem exists that requires external help. Previous research has indicated that instructors, teaching assistants, student peers, and online resources (such as YouTube and Stack Overflow) can assist CS students. However, as generative AI tools are now widely available to assist in some tasks (such as debugging code), Irene and her team wanted to understand which resources students valued most, and which factors influenced their preferences. Their study consisted of a survey of 47 students, and follow-up interviews with 8 additional students. 

Responding to the survey, students stated that they used online searches or support from friends/peers more frequently than two generative AI tools, ChatGPT and GitHub Copilot; however, Irene indicated that as data collection took place at the beginning of summer 2023, it is possible that students were not familiar with these tools or had not used them yet. In terms of students’ experiences in seeking help, students found online searches and ChatGPT were faster and more convenient, though they felt these resources led to less trustworthy or lower-quality support than seeking help from instructors or teaching assistants.

Some students felt more comfortable seeking help from ChatGPT than peers as there were fewer social pressures. Comparing generative AI tools and online searches, one student highlighted that unlike Stack Overflow, solutions generated using ChatGPT and GitHub Copilot could not be verified by experts or other users. Students who received the most value from using ChatGPT in seeking help either (i) prompted the model effectively when requesting help or (ii) viewed ChatGPT as a search engine or comprehensive resource that could point them in the right direction. Irene cautioned that some students struggled to use generative AI tools effectively as they had limited understanding of how to write effective prompts.

Using generative AI tools to produce code explanations

Andrew presented a study where the usefulness of different types of code explanations generated by a large language model was evaluated by students in a web software development course. Based on Likert scale data, they found that line-by-line explanations were less useful for students than high-level summary or concept explanations, but that line-by-line explanations were most popular. They also found that explanations were less useful when students already knew what the code did. Andrew and his team then qualitatively analysed code explanations that had been given a low rating and found they were overly detailed (i.e. focusing on superfluous elements of the code), the explanation given was the wrong type, or the explanation mixed code with explanatory text. Despite the flaws of some explanations, they concluded that students found explanations relevant and useful to their learning.

Using generative AI tools to create multiple choice questions

In a separate study, Andrew and his team investigated the use of ChatGPT to generate novel multiple choice questions for computing courses. The researchers prompted two models, GPT-3 and GPT-4, with example question stems to generate correct answers and distractors (incorrect but plausible choices). Across two data sets of example questions, GPT-4 significantly outperformed GPT-3 in generating the correct answer (75.3% and 90% vs 30.8% and 36.7% of all cases). GPT-3 performed less well at providing the correct answer when faced with negatively worded questions. Both models generated correct answers as distractors across both sets of example questions (GPT-3: 11.1% and 10% of cases; GPT-4: 9.9% and 17.8%). They concluded that educators would still need to verify whether answers were correct and distractors were appropriate.

Undergraduate students shaping the direction of generative AI research

With student concerns about generative AI and its implications for the world of work, the seminar ended with a hopeful message highlighting undergraduate students being proactive in conducting their own research and shaping the direction of generative AI research in computer science education. Stephen concluded the seminar by celebrating the undergraduate students who are undertaking these research projects.

You can watch the seminar here:

If you are interested to learn more about Stephen’s work on generative AI, you can read about how undergraduate students used generative AI tools to create analogies for recursion. If you would like to experiment with using generative AI tools to assist with debugging, you could try using Gemini, ChatGPT, or Copilot.

Join our next seminar

Our current seminar series is on teaching programming with or without AI. 

In our next seminar, on 16 July at 17:00 to 18:30 BST, we welcome Laurie Gale (Raspberry Pi Computing Education Research Centre, University of Cambridge), who will discuss how to teach debugging to secondary school students. To take part in the seminar, click the button below to sign up, and we will send you information about how to join. We hope to see you there.

The schedule of our upcoming seminars is available online. You can catch up on past seminars on our blog and on the previous seminars and recordings page.

The post Empowering undergraduate computer science students to shape generative AI research appeared first on Raspberry Pi Foundation.

The basis of these tools is a ‘natural language to code’ approach, also called natural language programming. This allows users to generate code using a simple text-based prompt, such as “Write a simple Python script for a number guessing game”. Programming-specific AI models are trained on vast quantities of text data, including GitHub repositories, to enable users to quickly solve coding problems using natural language. 

As a computing educator, you might ask what the potential is for using these tools in your classroom. In our latest research seminar, Majeed Kazemitabaar (University of Toronto) shared his work in developing AI-assisted coding tools to support students during Python programming tasks.

Majeed argued that natural language programming can enable students to focus on the problem-solving aspects of computing, and support them in fixing and debugging their code. However, he cautioned that students might become overdependent on the use of ‘AI assistants’ and that they might not understand what code is being outputted. Nonetheless, Majeed and colleagues were interested in exploring the impact of these code generators on students who are starting to learn programming.

Using AI code generators to support novice programmers

In one study, the team Majeed works in investigated whether students’ task and learning performance was affected by an AI code generator. They split 69 students (aged 10–17) into two groups: one group used a code generator in an environment, Coding Steps, that enabled log data to be captured, and the other group did not use the code generator.

Learners who used the code generator completed significantly more authoring tasks — where students manually write all of the code — and spent less time completing them, as well as generating significantly more correct solutions. In multiple choice questions and modifying tasks — where students were asked to modify a working program — students performed similarly whether they had access to the code generator or not. 

A test was administered a week later to check the groups’ performance, and both groups did similarly well. However, the ‘code generator’ group made significantly more errors in authoring tasks where no starter code was given. 

Majeed’s team concluded that using the code generator significantly increased the completion rate of tasks and student performance (i.e. correctness) when authoring code, and that using code generators did not lead to decreased performance when manually modifying code. 

Finally, students in the code generator group reported feeling less stressed and more eager to continue programming at the end of the study.

Understanding how novices use AI code generators

In a related study, Majeed and his colleagues investigated how novice programmers used the code generator and whether this usage impacted their learning. Working with data from 33 learners (aged 11–17), they analysed 45 tasks completed by students to understand:

1. The context in which the code generator was used
2. What learners asked for
3. How prompts were written
4. The nature of the outputted code
5. How learners used the outputted code 

Their analysis found that students used the code generator for the majority of task attempts (74% of cases) with far fewer tasks attempted without the code generator (26%). Of the task attempts made using the code generator, 61% involved a single prompt while only 8% involved decomposition of the task into multiple prompts for the code generator to solve subgoals; 25% used a hybrid approach — that is, some subgoal solutions being AI-generated and others manually written.

In a comparison of students against their post-test evaluation scores, there were positive though not statistically significant trends for students who used a hybrid approach (see the image below). Conversely, negative though not statistically significant trends were found for students who used a single prompt approach.

Though not statistically significant, these results suggest that the students who actively engaged with tasks — i.e. generating some subgoal solutions, manually writing others, and debugging their own written code — performed better in coding tasks.

Majeed concluded that while the data showed evidence of self-regulation, such as students writing code manually or adding to AI-generated code, students frequently used the output from single prompts in their solutions, indicating an over-reliance on the output of AI code generators.

He suggested that teachers should support novice programmers to write better quality prompts to produce better code.  

If you want to learn more, you can watch Majeed’s seminar:

You can read more about Majeed’s work on his personal website. You can also download and use the code generator Coding Steps yourself.

Join our next seminar

The focus of our ongoing seminar series is on teaching programming with or without AI. 

For our next seminar on Tuesday 16 April at 17:00–18:30 GMT, we’re joined by Brett Becker (University College Dublin), who will discuss how generative AI may be effectively utilised in secondary school programming education and how it can be leveraged so that students can be best prepared for whatever lies ahead. To take part in the seminar, click the button below to sign up, and we will send you information about joining. We hope to see you there.

The schedule of our upcoming seminars is online. You can catch up on past seminars on our previous seminars and recordings page.

The post Using an AI code generator with school-age beginner programmers appeared first on Raspberry Pi Foundation.

We are committed to make computing and creating with digital technologies accessible to all young people, including through our work with educators and researchers. In our current online research seminar series, we focus on computing education for primary-aged children (K–5, ages 5 to 11). In the series’ second seminar, we were delighted to welcome Dr Jean Salac, researcher in the Code & Cognition Lab at the University of Washington.

Jean’s work sits across computing education and human-computer interaction, with an emphasis on justice-focused computing for youth. She talked to the seminar attendees about her work on developing strategies to support primary school students learning to program in Scratch. Specifically, Jean described an approach called TIPP&SEE and how teachers can use it to guide their learners through programming activities.

What is TIPP&SEE?

TIPP&SEE is a metacognitive approach for programming in Scratch. The purpose of metacognitive strategies is to help students become more aware of their own learning processes.

TIPP&SEE scaffolds students as they learn from example Scratch projects: TIPP (Title, Instructions, Purpose, Play) is a scaffold to read and run a Scratch project, while SEE (Sprites, Events, Explore) is a scaffold to examine projects more deeply and begin to adapt them. 

Using, modifying and creating

TIPP&SEE is inspired by the work of Irene Lee and colleagues who proposed a progressive three-stage approach called Use-Modify-Create. Following that approach, learners move from reading pre-existing programs (“not mine”) to adapting and creating their own programs (“mine”) and gradually increase ownership of their learning.

Proponents of scaffolded approaches like Use-Modify-Create argue that engaging learners in cycles of using existing programs (e.g. worked examples) before they move to adapting and creating new programs encourages ownership and agency in learning. TIPP&SEE builds on this model by providing additional scaffolding measures to support learners.

Impact of TIPP&SEE

Jean presented some promising results from her research on the use of TIPP&SEE in classrooms. In one study, fourth-grade learners (age 9 to 10) were randomly assigned to one of two groups: (i) Use-Modify-Create only (the control group) or (ii) Use-Modify-Create with TIPP&SEE. Jean found that, compared to learners in the control group, learners in the TIPP&SEE group:

• Were more thorough, and completed more tasks
• Wrote longer scripts during open-ended tasks
• Used more learned blocks during open-ended tasks

In another study, Jean compared how learners in the TIPP&SEE and control groups performed on several cognitive tests. She found that, in the TIPP&SEE group, students with learning difficulties performed as well as students without learning difficulties. In other words, in the TIPP&SEE group the performance gap was much narrower than in the control group. In our seminar, Jean argued that this indicates the TIPP&SEE scaffolding provides much-needed support to diverse groups of students.

Using TIPP&SEE in the classroom

TIPP&SEE is a multi-step strategy where learners start by looking at the surface elements of a program, and then move on to examining the underlying code. In the TIPP phase, learners first read the title and instructions of a Scratch project, identify its purpose, and then play the project to see what it does.

In the second phase, SEE, learners look inside the Scratch project to click on sprites and predict what each script is doing. They then make changes to the Scratch code and see how the project’s output changes. By changing parameters, learners can observe which part of the output changes as a result and then reason how each block functions. This practice is called deliberate tinkering because it encourages learners to observe changes while executing programs multiple times with different parameters.

You can read more of Jean’s research on TIPP&SEE on her website. There’s also a video on how TIPP&SEE can be used, and free lesson resources based on TIPP&SEE are available in Elementary Computing for ALL and Scratch Encore.

Learning about learning in computing education

Jean’s talk highlighted the need for computing to be inclusive and to give equitable access to all learners. The field of computing education is still in its infancy, though our understanding of how young people learn about computing is growing. We ourselves work to deepen our understanding of how young people learn through computing and digital making experiences.

In our own research, we have been investigating similar teaching approaches for programming, including the use of the PRIMM approach in the UK, so we were very interested to learn about different approaches and country contexts. We are grateful to Dr Jean Salac for sharing her work with researchers and teachers alike. Watch the recording of Jean’s seminar to hear more:

Free support for teaching programming and more to primary school learners

If you are looking for more free resources to help you structure your computing lessons:

• Download our comprehensive classroom resources in The Computing Curriculum, which includes units of programming lessons for age 5 and upwards.
• Take one of our online courses about teaching computing. The course Teaching programming to 5- to 11-year-olds is especially relevant to this seminar.
• Read The Big Book of Computing Pedagogy, a special edition of our Hello World magazine.

Join our next seminar

In the next seminar of our online series on primary computing, I will be presenting my research on integrated computing and literacy activities. Sign up now to join us for this session on Tues 7 March:

As always, the seminars will take place online on the first Tuesday of the month at 17:00–18:30 UK time. Hope to see you there!

The post Supporting beginner programmers in primary school using TIPP-SEE appeared first on Raspberry Pi Foundation.

Conrad has been an influential figure in the areas of AI, data science, and computation for over 30 years. The company he co-founded, Wolfram Research, develops computational technologies including the Wolfram programming language, which is used by the Mathematica and WolframAlpha programs. In the seminar, Conrad spoke about his work on developing a mathematics curriculum “for the AI age”.

Computation is everywhere

In his talk, Conrad began by talking about the ubiquity of computation. He explained how computation (i.e. an operation that follows conditions to give a defined output) has transformed our everyday lives and led to the development of entire new sub-disciplines, such as computational medicine, computational marketing, and even computational agriculture. He then used the WolframAlpha tool to give several practical examples of applying high-level computation to problem-solving in different areas.

The power of computation for mathematics

Conrad then turned his attention to the main question of his talk: if computation has also changed real-world mathematics, how should school-based mathematics teaching respond? He suggested that, as computation has impacted all aspects of our daily lives, school subjects should be reformed to better prepare students for the careers of the future.

His biggest criticism was the use of hand calculation methods in mathematics teaching. He proposed that a mathematics curriculum that “assumes computers exist” and uses computers (rather than humans) to compute answers would better support students to develop a deep understanding of mathematical concepts and principles. In other words, if students spent less time doing hand-calculation methods, they could devote more time to more complex problems.

What does computational problem-solving look like?

One interesting aspect of Conrad’s talk was how he modelled the process of solving problems using computation. In all of the example problems, he outlined that computational problem-solving follows the same four-step process:

1. Define the question: Students think about the scope and details of the problem and define answerable questions to tackle.
2. Abstract to computable form: Using the information provided, students translate the question into a precise abstract form, such as a diagram or algorithm, so that it can be solved by a computer-based agent.
3. Computer answers: Using the power of computation, students solve the abstract question and resolve any issues during the computation process.
4. Interpret results: Students reinterpret and recontextualise the abstract answer to derive useful results. If problems emerge, students refine or fix their work.

Depending on the problem, the process can be repeated multiple times until the desired solution is reached. Rather than being proposed as a static list of outcomes, the process was presented by Conrad as an iterative cycle than resembles an “ascending helix”:

A curriculum for a world with AI

In the later stages of his talk, Conrad talked about the development of a new computational curriculum to better define what a modern mathematics curriculum might look like. The platform that hosts the curriculum, named Computer-Based Math (or CBM), outlines the need to integrate computational thinking into mathematics in schools. For instance, one of the modules, How Fast Could I Cycle Stage 7 Of The An Post Rás?, asks students to develop a computational solution to a real-world problem. Following the four-step problem-solving process, students apply mathematical models, computational tools, and real-world data to generate a valid solution:

Some future challenges he remarked on included how a computer-based mathematics curriculum could be integrated with existing curricula or qualifications, at what ages computational mathematics should be taught, and what assessment, training, and hardware would be needed to support teachers to deliver such a curriculum. 

Conrad concluded the talk by arguing that the current need for computational literacy is similar to the need for mass literacy and pondering whether the UK could lead the push towards a new computational curriculum suitable for learners who grow up with AI technologies. This point provided food for thought during our discussion section, especially for teachers interested in embedding computation into their lessons, and for researchers thinking about the impact of AI in different fields. We’re grateful to Conrad for speaking about his work and mission — long may it continue!

You can catch up on Conrad’s talk with his slides and the talk’s recording:

More to explore

Conrad’s book, The Math(s) Fix: An Education Blueprint for the AI Age, gives more details on how he thinks data science, AI, and computation could be embedded into the modern maths curriculum.

You can also explore Wolfram Research’s Computer-Based Maths curriculum, which offers learning materials to help teachers embed computation in their maths lessons. 

Finally, try out Wolfram’s tools to solve everyday problems using computation. For example, you might ask WolframAlpha data-rich questions, which the tool converts from text input into a computable problem using natural language processing. (Two of my favourite example questions are: “How old was Leonardo when the Mona Lisa was painted?” and “What was the weather like when I was born?”)

Join our next seminar

In the final seminar of our series on cross-curricular computing, we welcome Dr Tracy Gardner and Rebecca Franks (Raspberry Pi Foundation) to present their ongoing work on computing education in non-formal settings. Sign up now to join us for this session on Tues 8 November:

We will shortly be announcing the theme of a brand-new series of research seminars starting in January 2023. The seminars will take place online on the first Tuesday of the month at 17:00–18:30 UK time.

The post Building a maths curriculum for a world shaped by computing appeared first on Raspberry Pi Foundation.