RoboProf is an online program submission and correction system. It is WWW based and can easily be integrated with other WWW media (e.g. images, sound and video). It follows the format of an electronic book in that a student must show understanding of previous material before advancing to more advanced topics.
An introductory computer programming course must cover two main areas, the computer model (syntax and semantics of a programming language) and program design. In this paper I show how RoboProf can be effectively used to help teach the syntax and semantics of a programming language.
Learning programming is not easy. Students must simultaneously learn a programming language, program design and problem solving. It is hard to teach one part in isolation from the others. Teaching design without the language is boring, as the student gets no practice in the techniques they are learning. Learning the language without the design tends to encourage bad programming practice as well as being boring for the students. Learning both together is a juggling act that students find quite difficult.
RoboProf teaches the language first using artificial motivation to encourage the students. Students log onto RoboProf and read the html notes about the topic they are learning. When they are ready they are asked questions or given exercises on the topic. Their submissions are marked immediately and if they don’t get full marks they are shown the correct answer. There are no penalties for an incorrect submission, a student must simply complete a similar problem. In this manner a student would gradually complete a course and the exercises.
At DCU we have a 12-week introductory programming course. There are two lectures and two laboratory sessions per week. This year RoboProf was used for the first 6 weeks to help teach a subset of the syntax, semantics and some idioms of C++. The module was delivered to 50 maths students.
The RoboProf exercises were part of the continuous assessment (25% of the module marks) and the only deadline was the end of the semester. Despite this, all the students passed the RoboProf section of the course two months before the deadline.
Almost all the exercises were small programming problems. For example, one of the exercises showed a for loop that printed the numbers 1 to 10. The student was then asked to adapt the program so that it printed the numbers 5 to 20. The student would submit the modified program and be marked based on output. The marking engine is separate and can be replaced with a more sophisticated test, but the current problems are marked almost purely on output. For the above problem, RoboProf did ensure that a for loop was used to control the printing. Most exercises followed this format. I.e. the student had to modify a short program (maybe 10 lines) and adapt it (maybe change 1 to 4 lines). The problems covered if statements, loops, nested loops, conditional expressions, arrays and strings.
If the student's program didn't get full marks, he/she was shown what the program did produce along with what it should have produced. The student could then correct the program and resubmit. Although there was no penalty for resubmission the students were told that if the programs were being continuously modified in a hope that one version might accidentally work then they would be penalised. All program submissions were filed away and could be later checked by the lecturer.
Students could log into RoboProf using a standard WWW browser and do the exercises at any time and anywhere that they had Internet access. However RoboProf was most heavily used during the timetabled (and supervised) lab sessions.
In an introductory programming course students must practice in order to develop a programming skill. It is however a lot of work to set programming tasks and to provide feedback on all student submissions. Since many of the lecturers are themselves programmers it is not surprising that many systems have been developed, e.g. [1] [2] and [3].
For the student the immediate feedback encourages the student to try the problem and to feel satisfaction when he/she solves it. Each problem tends to build on the ones that went before and so the students gradually increase their confidence in programming. It would obviously be very difficult to have so many exercises in a manual system and even if you did, the students would not get the benefits of immediate feedback.
Frequently in our labs students are quite shy about asking tutors for help. However because they had short clear goals, the students were focused on solving the problem and forgot their shyness. This increased their demand for tutors which we could only satisfy when we had a tutor for every 10 students. RoboProf helped with the administration of the exercises but it didn't eliminate the human help required to explain how one might go about solving a particular problem. This did place a large burden on the tutors and it didn't stop when the labs had finished as the students had learned the power of email. Undergraduate tutors are a useful resource that we used to fill the need.
The feedback provided by RoboProf is minimal but it is fast. More detailed feedback can be provided later as tutors get a report (via the Web) of students' progress. A tutor can send a message to a student if they notice anything funny about the code. This could include suggestions for alternative ways of solving the problem etc.
RoboProf comprises a student database, a course database and a Java servlet (a servlet is the Java equivalent of a CGI program) which presents the module notes and problems. The servlet runs on a Solaris server. Java An applet handles the program submissions. When a program is submitted, the applet compiles the program and - assuming there were no errors - proceeds to test the program. The program is run against a set of test data and the student is shown the output of the program and expected output for each set of test data. The marking system can be specified separately for each problem. It is possible to plug in modules that analyse the submitted program and give a mark based on this. There is a number of default marking schemes, which mark a program depending on its output. This may compare the numbers in the output with the expected output, or expect an exact match or some similar mechanism. When a new programming problem is added to the system you can specify that you want to use a default marking scheme or you may wish to supply your own marking program. The test data can be fixed for each submission or generated randomly, with the appropriate output generated by the engine to check the student’s output. This could be useful if a student adapted a program to check what the input data was and simply print the appropriate output. In fact students frequently submitted a program that they knew wasn’t working just so they could see the sample input and output.
One big advantage of Java is that it allows dynamic loading of program modules. This means that the engine can load specific problem generation and testing modules without having to adapt the engine itself. Each problem can be generated at run-time and adapted to the ability of the student. The marking software can be made arbitrarily sophisticated for each problem. Of course this would require writing programs to add this behaviour. The default behaviour is to present a constant specification, to mark based on output and to read a file containing the required test data. Time considerations meant that almost all the problems were designed using the default.
Fifty students used RoboProf over a six-week period. All the students passed (got over 40%), and two-thirds got 80% or more. The following graph shows RoboProf submissions during the six-week period.
Most submissions were on a Wednesday (the lab session). On the first main lab session 230 correct programs were submitted. This averages at over four programs per student. And this is without the pressure of deadlines. Although students were encouraged to finish within six weeks (to keep up with the lectures) the deadline wasn’t till the end of the semester (two months later). The number of submissions to RoboProf gradually decreases over time. Some of the more enthusiastic students completed all the problems early and stopped using the system.
Here we see the usage by student, i.e. the number of students who submitted a program on a given day. Over 41 students submitted a program on day 13. As students became more familiar with RoboProf they became less reliant on the labs and the supervisors and used the system on non-lab days.
The next graph shows the number of submissions per problem. There were 39 problems and the average number of submissions was 2.6. (I.e. on average a student attempted each problem two or three times.) One student managed to attempt problem 14 twenty times. If the number of attempts is high then it could indicate that the problem is difficult or that the specification is unclear.
RoboProf provided an environment where students could improve their programming skills by completing gradually more difficult tasks. Students completed the tasks well ahead of the deadlines. The challenge and the immediate feedback kept the students interested. They also appeared to be more focused in the lab.
The fact that a student had to compile and run the programs ensured that they were well versed in the low-level details of compiling a problem. RoboProf did not attempt to teach problem solving techniques. Rather the aim was that a student would have a good grasp of the syntax and semantics of a programming language and also become acquainted with a large number of programming idioms. This would place the student in a good position to learn program design and problem solving – the second part of the course.
For those who think that the student doesn’t have enough control in this environment I would point out that the RoboProf progression rules are almost exactly analogous to those of games such as Quake. In these games a player has to complete a level in order to move to higher levels. These rules help make the game challenging and keep the player interested. In fact one problem was that some students found RoboProf quite addictive and spent too much time at it. It might be necessary to limit the time students spend on the system.
Finally I suspect that the activity of surfing (whether surfing the Web or channel surfing with television) allows, in fact almost encourages, students to develop bad habits of concentration. It is too easy to click away from any difficult concepts. RoboProf endeavours to focus students in a way rarely achieved with Web based media.
[1] Rosbottom, John Computer managed, open question, open book assessment proceedings ITiCSE Uppsala Universitet 1997.
[2] Thorburn, G and Rowe, PASS - an automated program assessment system Proceedings CTC Dublin City University, 1996.
[3] Benford S. D. et Al A system to teach programming in a quality controlled environment The software quality journal, V2, 1993.
When a student logs on he gets an overview page that shows the course layout and the marks achieved for each topic. A topic can have any number of exercises; the array topic has about programming problems. Selecting a topic brings the student to the course notes for that topic and the option to improve the marks for any particular problem.
This is the problem specification page. Pressing the 'Submit' button brings up a file dialog box allowing the user to select a program and test it.
Here RoboProf marks the program. The numbers match, but the output wasn't an exact match so the program gets 40%. Correcting the output message ensures that the program will get full marks.