Doing my (past) Homework with GitHub Copilot
GitHub Copilot is an AI-based programming tool created by OpenAI. I’m going to use it to do my homework.
Copilot
Copilot is currently in a closed preview state, and is accessed via an extension for VSCode. Put simply, Copilot can generate code in nearly all major languages, based on simple descriptions, or even just method signatures. Here is a quick sample of some Python code generated by Copilot.
We simply add a method signature and comment describing what our code should do:
def is_palindrome(string):
"""
returns true if string is a palindrome, false otherwise
"""
Wait a couple seconds for Copilot to come up with a suggestion, and voila:
def is_palindrome(string):
"""
returns true if string is a palindrome, false otherwise
"""
return string == string[::-1]
Copilot comes up with a correct solution: check if the string is the same after reversing it. Note that we can actually explore multiple solutions, here are a few more that copilot came up with for this example.
# A: more verbose version of the first solution
if string == string[::-1]:
return True
else:
return False
# B: A recursive implementation, check the ends, slice, and recurse
if len(string) <= 1:
return True
if string[0] == string[-1]:
return is_palindrome(string[1:-1])
return False
While the first solution is probably the cleanest and most idiomatic, it’s interesting to see the different variations that copilot comes up with. While I’m sure I could use Copilot for a plethora of useful tasks, let’s talk about what we’re really here for: using cutting edge AI to avoid doing my homework.
The Problem
CSE 143 is the second half of the University of Washington’s introductory CS series. It focuses on Object-Oriented patterns, and introductory data-structures. It includes core topics like dynamic lists, stacks, queues, sets, maps, recursive algorithms, etc. I originally took this class during my first quarter at the UW, Autumn 2017. Today we are going to try to tackle the first of 8 homework projects using GitHub Copilot to see if a student taking the course for the first time could do the same.
LetterInventory
While the full specification can be found online, here is the short version:
We are to design a class called LetterInventory, a data-structure that is used to keep track of an inventory of letters in the alphabet. Given a String, the structure tracks how many of each letter there are. We then define a few basic operations to query and mutate our structure. The specification dictates what methods we must implement, their signatures, and provides a basic description for each one. The assignment also specifies we are to implement this using an array of 26 counters, one for each letter in the alphabet. Let’s use the information we have so far to scaffold out the class:
public class LetterInventory {
private static final int NUM_LETTERS = 26;
private int[] letterCounts;
public LetterInventory(String data) {
// TODO
}
public int get(char letter) {
// TODO
}
public void set(char letter, int value) {
// TODO
}
// More methods below...
}
Implementing
Starting with the constructor, copilot gives a few suggestions before we even add any comments:
public LetterInventory(String data) {
letterCounts = new int[NUM_LETTERS];
for (int i = 0; i < data.length(); i++) {
letterCounts[data.charAt(i) - 'a']++;
}
}
Without any additional context, this is actually a pretty solid suggestion. Initialize the array using our constant, loop through our data, and increment the corresponding index (computed by subtracting ‘a’ from each character, so a=0, b=1, c=2, etc.). This solution does have a flaw however, what if our data contains characters outside of the range a-z? We will end up with an IndexOutOfBoundsException!
Lets try including more details from the specification in a comment and see if Copilot can do better. Namely, we should ignore the case of our letters, and omit non-alphabetic characters.
// Constructs an inventory (a count) of the alphabetic letters in the given string,
// ignoring the case of letters and ignoring any non-alphabetic characters.
public LetterInventory(String data) {
letterCounts = new int[NUM_LETTERS];
for (int i = 0; i < data.length(); i++) {
char c = data.charAt(i);
if (Character.isLetter(c)) {
letterCounts[Character.toLowerCase(c) - 'a']++;
}
}
}
Fantastic! Copilot uses the Character.isLetter method to ensure c is part of the alphabet, and Character.toLowerCase to handle different casings. Note that the comment given is the exact, word-for-word, description given in the specification. Copilot allowed us the write this without having to understand a single thing about the assignment so far.
Let’s move on to the next method, get:
public int get(char letter) {
letter = Character.toLowerCase(letter);
if (letter >= 'a' && letter <= 'z') {
return letterCounts[letter - 'a'];
} else {
return 0;
}
}
This time, Copilot comes up with an even better solution right off the bat, normalizing case and checking the bounds without having to be explicitly told via a comment. Interestingly copilot opts for the check letter >= 'a' && letter <= 'z' instead of Character.isLetter like it did in the constructor. The solution is not perfect yet though, the spec says we should throw an IllegalArgumentException if our argument is non-alphabetical. Lets add the exact description from the spec again and see how our solution changes.
// Returns a count of how many of this letter are in the inventory. Letter might
// be lowercase or uppercase (your method shouldn’t care). If a nonalphabetic
// character is passed, your method should throw an IllegalArgumentException.
public int get(char letter) {
if (!Character.isLetter(letter)) {
throw new IllegalArgumentException("Not a letter: " + letter);
}
return letterCounts[Character.toLowerCase(letter) - 'a'];
}
Much better, we check that letter is in fact a letter, this time using the same method from the constructor, throwing our exception otherwise.
So far, Copilot has done great with explicit descriptions. Let’s try a more vague method and see how it fairs. LetterInventory’s size method returns the number of letters in the inventory, this can be accomplished trivially by looping over letterCounts and returning the sum. The specification however, states that “This operation should be ‘fast’ in that it should store the size rather than having to compute it each time this method is called.” This hint is meant to allude to the fact that we should keep another field for the structure’s size and progressively update it as we modify the inventory so we can return it here instead of computing it each time it is needed. Without adding our field, lets see what Copilot does.
// Returns the sum of all of the counts in this inventory. This operation should
// be “fast” in that it should store the size rather than having to compute it each
// time this method is called.
public int size() {
int sum = 0;
for (int i = 0; i < letterCounts.length; i++) {
sum += letterCounts[i];
}
return sum;
}
Unsurprisingly, Copilot comes up with the (correct) trivial solution. In general, Copilot is great at writing code locally using the global context, but its current implementation doesn’t edit our code beyond that. Let’s try adding the field ourselves and revisiting this problem. Note that this will also require us to update our constructor to initialize the new size field correctly. This was a bit finicky, at first Copilot tried to introduce a non-existent getIndex method (which might actually be a good abstraction), but after cycling through a few suggestions, we end up with this implementation. It’s exactly the same as our initial one, but now we properly initialize and increment the size field.
public LetterInventory(String data) {
letterCounts = new int[NUM_LETTERS];
size = 0;
for (int i = 0; i < data.length(); i++) {
char c = data.charAt(i);
if (Character.isLetter(c)) {
letterCounts[Character.toLowerCase(c) - 'a']++;
size++;
}
}
}
Now the size method looks like this:
public int size() {
return size;
}
Beautiful.
Skipping ahead a bit, Copilot did fine with the most of the rest of LetterInventory’s simple methods. I had to coax it a bit to get it to update size relative to count in the set method, it initially wanted to just set size equal to count which would of course produce the wrong results. Additionally, Copilot was able to get the correct (or at least a very close) answer without comments as we progressed, suggesting that it was using the other methods we wrote already as context.
toString was a bit of a struggle as well, Copilot insisted on using the numerical counts of each letter in the string, while the spec suggests that letters should be repeated based on their counts. For example, an inventory for the string "I would not fly on a plane programmed by Copilot quite yet", the spec wants us to produce: [aaabcddeeeefgiiillllmmnnnoooooopppqrrttttuuwyyy] while Copilots solution produces [a3b1c1d2e4f1g1i3l4m2n3o6p3q1r2t4u2w1y3], not an unthinkable solution, but certainly not what we’re after. If we start writing an inner loop in our method, Copilot does catch on however:
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append('[');
for (int i = 0; i < NUM_LETTERS; i++) {
// I just had to type 'for' here and copilot did the rest
for (int j = 0; j < letterCounts[i]; j++) {
sb.append((char) ('a' + i));
}
}
sb.append(']');
return sb.toString();
}
The last method I want to look at is subtract. This method takes another LetterInventory as a parameter, and returns a new LetterInventory that is the result of subtracting one from the other, returning null if any counts in the result would be < 0. In my opinion this is one of the more complicated methods in this assignment as it requires us to handle multiple inventories at once, as well as compute the size of the new inventory accordingly. This is what Copilot produced with no comments.
public LetterInventory subtract(LetterInventory other) {
LetterInventory result = new LetterInventory("");
for (int i = 0; i < NUM_LETTERS; i++) {
result.letterCounts[i] = this.letterCounts[i] - other.letterCounts[i];
}
result.size = this.size - other.size;
return result;
}
In fact, not only did I not need comments, I didn’t even need to write the method header, Copilot suggested this to me immediately after I used it to write the corresponding add method above it. Lets add the detail about returning null and see how it changes.
// Constructs and returns a new LetterInventory object that represents the result
// of subtracting the other inventory from this inventory (i.e., subtracting the
// counts in the other inventory from this object’s counts). If any resulting
// count would be negative, your method should return null. The two
// LetterInventory objects being subtracted (this and other) should not be
// changed by this method
public LetterInventory subtract(LetterInventory other) {
LetterInventory result = new LetterInventory("");
for (int i = 0; i < NUM_LETTERS; i++) {
result.letterCounts[i] = this.letterCounts[i] - other.letterCounts[i];
if (result.letterCounts[i] < 0) {
return null;
}
}
result.size = this.size - other.size;
return result;
}
No sweat. Copilot reads the comment, and includes the appropriate check.
Analysis
In total, I had to write less than 5 lines of code myself to complete this assignment. The prerequisite understanding needed to do this with Copilot was pretty low. It might not be powerful enough for someone who has never written code, but for a student in this class, Copilot trivializes this homework.
Obviously this is not the intended use of Copilot. As the name suggests, Copilot is meant to be an assistant to the developer. It isn’t meant to write entire programs on its own. Despite that, this exercise does raise some interesting questions about the future of learning to code. I’m sure people made the same argument when intelliSense first hit the market, but how will Copilot help (or hurt) the education of future developers? How will classes manage software like this? There is no restriction against using VSCode and autocomplete while working on assignments for this class, but I imagine Copilot would be frowned upon. Where do we draw the line between a helpful tool and academic misconduct? As a note to any current 143 students reading this, Copilot certainly makes use of methods and concepts not covered in class, so I wouldn’t recommend using code from it directly, but it does remove a lot of the thinking required to do this entirely on your own. For me, Copilot currently sits in a gray area between helpful and harmful. It might be interesting to revisit this over the next few years to see how Copilot evolves. LetterInventory was by no means the hardest assignment in CSE 143, let alone my academic career, but this exercise was a thought provoking look into the usage of AI-based tools and the future of CS education.
There is another perspective from which this could be seen as a great thing however; in the field of computer science we stand on the shoulders of giants. Every piece of modern technology is built with layer upon layer of abstraction. Just writing “Hello, World!” in Java requires the compiler and JRE, both of which were at one point written or bootstrapped from C or C++, which were at some point bootstrapped from early assembly languages. Copilot is trained on millions upon millions of open source examples. Just as higher level languages allow us to focus on business-logic instead of the underlying bits and bytes, perhaps this is just an early stage of the next abstraction that one day professional engineers will rely on to productively build software.