Peter Norvig Interview

Question: In your perspective and experience what was the quality difference between education in person and in distance?

Peter: I guess its a couple of things.
1) There is a wider background for these distance courses because of the way they are organized, so that makes it a little bit harder for the teacher and students because there isn’t that much shared experience. You can expect Stanford students to have the same kind of background, because they have actually taken the same prerequisite courses or similar classes elsewhere. But for the online class, its such a wide variety that it is hard for everybody to get on the same page.
A weakness in the past in online classes has been a lack of a community of feeling like you are doing it yourself and I think that seems to be getting much better now with these various online forms that have much cheaper cost.
I remember we had a teacher from Teach for America came to Google and talked about teaching this math class which was two levels behind. By the end of the year, she had almost all of them caught up. So I said “Wow, that’s amazing. How did you do that?” And one thing that struck me is that Google has this mission statement of  organizing all of the world’s information and I said “Wait a minute, she didn’t do better in that class because she had better information. She had the same textbook as the year before and the year before that. And they did lousy before and they did better once she came. So, what was different?” It wasn’t information. It was motivation, determination or something like that. So I realized that you got to have a minimum floor of information and once you have that, its the determination that counts. So I wanted to think about how you can do classes that do that better. And one thing I realized was a lot of the online classes say let’s take advantages of being online, and one of these advantages is that you can do it any time you want. You see these ads on television showing a student that saying “I took my classes in my pajamas. Oh great. I can do my classes wherever and whenever I want.” But part of the problem is that if you can do it whenever you want you can do it tomorrow. Then you end up not doing it. We wanted to make sure that although week to week its asynchronous (you can do it any time you want during the week), there has to be a homework assignment. Because if there is no homework assignment, you are going to push it off and not do it, at least I know I would. So I thought that part was important. And when you ask “What works with distance learning?”, I would say it really depends on how you organize distance learning and solve these different aspects that I think are more important than whether you are in the same room or not.

 

Question: Other than this compartmentalization of giving people discrete quantities of time, is there anything else that you found about distance learning?

Peter: It is the time for learning. In the lecture hall, there is this problem that everybody has to go at the same pace. And there is an advantage in that, that can be customized. So, Sebastian [Thrun] can stop and ask a question and then wait for the answers to come back. Sometimes the answer is going to come immediately, sometimes it will take a minute, and that’s ok, because he can adjust to that. But  he can adjust to that for the people that speak up, not for everybody else. And when online, if you do it right, you can adjust infinitely. You don’t get the advantage of kind of hearing the answer come back and modifying the lecture based on it. But if you give a quiz,you see whether its right or wrong, you do get to pause and let the student figure something out on their own. So, in that sense its better because there is more involvement but its worse that you can’t modify based on it.
I guess at some point, if somebody really wants to put in a lot of effort (we are already putting a lot of effort to the [ai-class.com] class), you can have a full branching structure. For example, if the student says one thing, you do one thing. If the student says something else, you do another. You can never anticipate everything that the student is going to say but you cover the popular cases. So far, we don’t have a branching structure. We have a straight line structure with loops of “did you get this question right or wrong?” That’s all we can do.
And then one of the things Salman Kahn says is videos are less intimidating than real life. I think most people agree that the best is having a one-on-one tutor and not having a lecture in a large group. And then where does online fit? Its not like it fits in between or its sort of all over the map depending on what part of it you are talking about. So Kahn says even when you have a tutor, if the student is really stuck and he says “I don’t get it” and the tutor explains it again. That’s great that you can have an alternative explanation. Its hard to do that online. But after 3-4 times, the social pressure on the student to say “Oh yeah yeah I get it, I’m not gonna make you try again”. But in the online realm, students rewind the videos twenty times.

 

Question: It would be interesting to get statistics on how often they are rewinding so they can see which parts they have trouble with.

Peter: It would be better for them instead of rewinding, they could see an alternate explanation and if it could somehow be customized to them. But just the fact that they dont have to say “I understand it now. I’m going to feel stupid so let’s move on.” They’re not embarrassed interacting with a computer rather than a person, so they can take as much time as they need”.

 

Question: Since the classes began, did you notice any disadvantages or drawbacks that you hadn’t anticipated?

Peter: I guess one thing is we developed this style of being even more interactive and we hadn’t even thought about it when we started. We liked the Kahn style videos more than the hour-long videos. We thought the hour-long videos would be completely tedious on the small screen and it makes us think “Oh we hope its not that tedious in the lecture hall”. But Kahn has 5 or 10 minute videos and we have even shorter ones. A lot of ours are even 1 or 2 minutes before we ask a quiz question. And we thought that was good. We hadn’t really sat down beforehand and thought about what the mean length of a video going to be. And then also that we can be a little bit more interactive. Its a pain to be recording these videos and sometimes I wish that if it were a written problem it would be so much easier. Because there are all these questions that come up because of ambiguities. And if you want to edit a word-processing document and change one word, takes you a second and you are done. But if you want to edit a video, usually the easiest way to edit a video is to rerecord the whole thing. Because its hard to go in there and splice out one word and put in another word, so you are looking at a 5-minute commitment rather than a 5-second commitment. So that makes it harder.
But there are some things that are nicer in a way. Like when you are doing A-star search, and ask what nodes are next, it really is nicer to be clicking on the nodes rather than describing the problem in paper here and then having the answer be node a, node b, node c etc. Using that interactivity and graphical layout, there were some nice ways to do that. And I think we can do more. I would also like to have a computing environment to do these things in. We like the idea of drawing on paper and making it seem low-teach because we wanted to have the feeling of personal, one-on-one, you and I are sitting on a table. Its not this big Hollywood production. But if we could have an environment that we could share. We could say “Here is an algorithm, here is a Pacman running around.” But if we both had that, we could say “Here is an algorithm, here is a Pacman, change the algorithm and the Pacman does something else. Now you play with it”. If we had a dozen different environments set up like that, that would be a powerful way to explore.
It seems that there are a lot of contradictions here of things we are trying to resolve. So one is, we said we want to bring a lot more education to a lot more people at a cheaper cost. But then we said the way we are disrupting it is making our online class be more like a traditional class than others in that we have these weekly homework assignments and you have to learn at our pace rather than your own schedule. So we thought that we are bringing back the good parts of an on-campus experience while keeping it online.
And then, the other part was most people agree that having a tutor one-on-one is a very good way to learn. And then another part is a constructivist approach where you are figuring things out on your own and you are doing projects rather than a road-learning or fill-in the blank. But we are kind of limited there in that the only way we have to interact with people is fill in this little box. So we have to be creative and try to have a way with the fill-in-the box such that a student doesn’t just memorize something or does a simple calculation but actually thinks about the question. We have done an okay job. We are hitting against the boundaries because if I want you to think about it, that means the question is a little open ended. That means its ambiguous. If its ambiguous, you might get the wrong answer, and you are going to be really mad because you got the wrong answer because of the ambiguity.
Question: And there are probably not as many Q/A sessions.

Peter: Yeah.

 

Question: What do you think the future is going to look like via education?

Peter: It looks like this is starting to do really well. I don’t know if you saw, but there is an announcement of more classes coming up in January. So I think it seems to have really resonated with people. Looks really powerful. I guess another surprise for us is that our average age is about 30, so it looks like its more professionals than high school or college students.

 

Question: So you are not eating away at the applicant pool for colleges.

Peter: Yeah. We are not doing that as much. But we are providing education to people who are in a stage in their lives where they are more ready to accept it and see a need for it more. So I think you will see more of this and more flexibility. People won’t have to come together into campus as much. But its not like we are against it (on-campus education), but we are more complementing it. And I think there is certainly room for innovation and change in what the materials look like and how its presented. I think more important is the community around it. How do you build up these groups, both for an individual and for a class and something that persists longer than a class? Not only am I in a class and a discussion group but I’m in a course of study and I can build up longer commitments with my friends that I am studying with. And what this feeling of “I didn’t do this homework. Its not only bad for me but I am also letting my friends down.” So we are trying to build a group commitment rather than an individual commitment.

 

Question: We also saw that this is a part of a start-up. Is this going to be a part of the startup’s future goals?

Peter: I’m doing this to further education. I guess I am providing a service to the startup by doing the course but I am not really part of it. Daphne Koller has her own start-up, but I think there will be other competitors in the form of startups. When you have startups you start thinking about them as being competitive and my hope is that it will be more collegial. I am teaching at Stanford and I have friends at Berkeley that teach the same topics and it is not like we are against each other. I hope that companies that step into this space can keep that collegial academic friendship than be competitive.

 

Question: Since they are free, there will be no monetary competition?

Peter: Eventually there will be. What models will we get to eventually pay for this? I think that the right model is that courses like this should remain free, that the material should be out there. But if you want accreditation in some way, then it makes sense to pay for that. Because there are initial services required by that. For example, for this class, we didn’t really worry about cheating too much, mainly because we say “We give you a letter which is not an official Stanford letter. Its just a virtue of me and Sebastian”. We hope that that would be valuable enough that students would want to do the work and not so valuable that they would want to cheat. I don’t know if we hit it right or not. But if you are giving credit, you really have to verify who is this person and that means that you need to do ID checks and maybe have  a person come into a testing center somewhere. That has additional costs and it makes sense that you pay for that. And also a model to pay for it would be having information about the students, especially in a field like computer science where hiring is competitive. Companies now will pay a recruiter to come and find somebody. And its pretty big, its 20% of a first year salary, so there is a lot of money there per hire. And similarly companies pay Stanford just to have access to their students, so they have these job fairs and companies pay quite a bit to set up their booth. And so you could offer the class for free and tell the companies “Look, we have a lot of students. Do you want to hire some?” You’re better off coming to me than coming to Stanford because you get a transcript with one letter per semester per course, but we can give you a history with 1000 clicks, so if they ask us for somebody with specific skills, we can tell them who the best students are for those skills. Maybe companies would pay for the whole thing and students could get the ride for free.
I don’t know what the financial models are going to be but those are some of the possibilities.
And then I would hope that there would be more prestige for the Professors who offer the course. Right now, in some schools, teaching matters to some degree, but not so much for others because they are more focused on research. But, Professors put a lot of time into writing books, and some of them make a little bit of money from that but most of them don’t. So really most of them do it to further their reputation. And you would think that they might be willing to do this kind of thing at the same level where mostly they make a course so they have a reputation as one of the top teachers of this course, and maybe they will make a little bit of money from it. But they don’t necessarily have to make a lot if that is counted towards their career in the form of reputation.
And I guess its all different in the K-12 level rather than the university level. And beyond that there is education for non-academic topics as well. If you want to learn how to be a plumber. You can think long course such as becoming a plumber or an electrician or short course such as a specific thing like baking a cake. Now that’s served pretty well by doing a search and finding a video, but it could be better.

 

Question: Interesting model: getting companies to pay for histories of students. That brings up an ethical question too. When you sign up for a class, you might not want that to happen…

Peter: Well, you have to agree. There will be a privacy statement about what we’re going to do with the data.

 

Question: I certainly wouldn’t want people looking at how fast I am reading a novel or something for a class, or how quickly I am doing the problem sets. But it could work.

Peter: Yeah. There are lots of ways to structure. Probably what you want to do is have something where there’s anonymous data and aggregated data and you can get permission from both sides to reveal who’s who. Like a dating service.