- Third semester of virtual teaching assistants underway at Georgia Tech
- Virtual TAs are built on IBM's artificially intelligent Watson platform
- AI assistants save time for human TAs and are improving student participation
Many of us know Jack and Jill went up a hill to fetch a pail of water. But did you know that Jill then went to the Georgia Institute of Technology?
That's right — Jill went on to college and is now a teaching assistant in a course on artificial intelligence (AI) in Georgia Tech's computer science program. Jill assists Ashok Goel, professor in the School of Interactive Computing.
Jill, implemented on IBM's Watson platform, was first used during the spring 2016 semester to successfully answer frequently asked student questions without the help of humans. The students weren't told her identity until the final day of the class.
Building on Jill's success, Goel then introduced two 'Jill Watsons' this past fall to work alongside 13 human TAs.
"I told the students at the beginning of the semester that some of their TAs may or may not be computers," says Goel. "Then I watched the chat rooms for months as they tried to differentiate between human and artificial intelligence."
With Jill no longer a secret, Goel gave 14 of 15 TAs pseudonyms (only the head assistant kept his real identity). Jill Watson became Stacy Sisko and Ian Braun. Stacy interacted with the 400 enrolled students during class introductions and posted weekly updates while Ian answered common questions.
Stacy dove into the discussion forum first. All members of the class were encouraged to introduce themselves. She responded to about half of them without human assistance, chiming in with short paragraphs and relevant details.
"If a student mentioned that they lived in, say, Chicago and worked at a specific company, Stacy might comment on the city or the workplace," Goel says. "If a student mentioned they were taking another Georgia Tech course, she would sometimes make a comment about the instructor."
Goel said there were a few mistakes, but nothing alarming. Stacy wrote her own weekly previews of the content, then summarized them on Fridays. Sometimes her wrap-ups referenced conversations among students. For instance, if she noticed a helpful online discussion from a few days prior, she would highlight it during her summary and encourage students to check it out for added insight.
Virtual teaching assistants like Jill were recently recognized as one of the most transformative technologies to impact college within the past 50 years by the Chronicle of Higher Education.
The other non-human TA, Ian, wasn't much different from the original Jill Watson. He answered routine questions typically asked each semester, such as the allowed length and format of written assignments.
"Ian wasn't as efficient in fall as Jill was in spring. He didn't answer as many questions as we had expected," Goel admits. Ian only posted responses if he was 97 percent confident. "We're still sorting through the data, but it looks like some students may have deliberately tried to outsmart the computer by asking questions in new ways."
And because Ian could only pull answers from his episodic memory of previous offerings of the class, Goel thinks the variety of the student questions may have been a bit overwhelming. So his research team has developed a new version of Jill based on semantic analysis that he will introduce to the incoming class this semester.
At the end of the term, the students were polled about who was human and what was AI. Slightly more than 50 percent of the students correctly guessed that Stacy was a computer. Sixteen percent figured out that Ian wasn't human. On the other hand, more than 10 percent mistakenly thought two of the human TAs weren't real.
"We're seeing more engagement in the course. For instance, in fall of 2015 before Jill Watson, each student averaged 32 comments during the semester. This fall it was close to 38 comments per student, on average," Goel says. "I attribute this increased involvement partly to our AI TAs. They're able to respond to inquiries more quickly than us."
This isn't something Goel expected when he began the Jill Watson project. He just wanted to free up more time for his staff so they could concentrate on tasks computers can't do.
Also in the fall, approximately 40 students built chatbots (their own avatars of Jill Watson) that could converse about the course. This allowed the students to operationalize some of the techniques they were learning in the class.
"When we started, I had no idea that this would blossom into a project with so many dimensions. It's been a bonanza of low-hanging fruit we're just starting to pluck."