MIT CS Professor Tests AI’s Impact on Educating Programmers

Long-time Slashdot reader theodp writes: “The Impact of AI on Computer Science Education” recounts an experiment Eric Klopfer conducted in his undergrad CS class at MIT. He divided the class into three groups and gave them a programming task to solve in the Fortran language, which none of them knew. Reminiscent of how The Three Little Pigs used straw, sticks, and bricks to build their houses with very different results, Klopfer allowed one group to use ChatGPT to solve the problem, while the second group was told to use Meta’s Code Llama LLM, and the third group could only use Google. The group that used ChatGPT, predictably, solved the problem quickest, while it took the second group longer to solve it. It took the group using Google even longer, because they had to break the task down into components.

Then, the students were tested on how they solved the problem from memory, and the tables turned. The ChatGPT group “remembered nothing, and they all failed,” recalled Klopfer. Meanwhile, half of the Code Llama group passed the test. The group that used Google? Every student passed.

“This is an important educational lesson,” said Klopfer. “Working hard and struggling is actually an important way of learning. When you’re given an answer, you’re not struggling and you’re not learning. And when you get more of a complex problem, it’s tedious to go back to the beginning of a large language model and troubleshoot it and integrate it.” In contrast, breaking the problem into components allows you to use an LLM to work on small aspects, as opposed to trying to use the model for an entire project, he says. “These skills, of how to break down the problem, are critical to learn.”