David Mudrak's blog

commit eb123debc6279c0e7fe92d7cb8979f4db651a46c
Author: skodak <skodak>
Date:   Thu Oct 8 22:34:34 2009 +0000

MDL-19474 One year and a half ago we started designing
and implementing our new database abstraction layer
which included brand new moodle native drivers.
Today we are finally removing old legacy drivers
that were still using adodb internally. From now on
the adodb will be used only in some authentication
and enrolment plugins, we are probably going to
drop adodb there too later in 2.x.
Thanks everybody who helped test and improve our
new drivers, even bigger thanks to all the great
developers who created ADOdb!

One of the core Moodle developers put an almost innocent note into our Jabber chat recently and I can’t stop thinking about it. And no, it’s not about hating Google. The Quiz module is one of the most advanced activity module we have in Moodle. It seems to be popular and often requested by the newcomers. It even may be considered as a feature that helped Moodle to become first class LMS. I personally use it sometimes as a teacher, too. It’s just so quick and easy to prepare a question bank and let students to prepare themselves for the evaluation quiz. Yes, I trivialize the issue here. I know it’s not so easy to prepare valid and relevant questions, analyse the responses, modify the questions with low discrimination capability etc. But honestly – how many teachers do this?

Conditional activities subsystem has been committed into the repository and is one of the Moodle 2.0 highlights. It enables course designers to restrict the availability of any activity according to certain conditions such as dates, grade obtained, or activity completion. The world has been waiting for this feature since ages. And Sam really did a good job – kudos for him!

Now, the little worm in the back of my mind starts to whisper: Shh! Wait! This all starts to smell like a SCORM too much! Is not it a little bit strange that the automatically evaluated Quiz is so matured, while the Forum, the essential tool for constructivistic online courses, still lacks such essential features like to subscribe to individual threads, to mark (or tag) the thread as closed, to collapse the parts of the displayed threads etc.? When did we last improve our good old Forum in a really significant way? Why do we not support real cooperation of the students? We are still not able to group students into a team and grade the team work instead of individual students, for example.

I am all for these Moodle 2.0 changes, portfolios, repositories, database layers, webservices, output rendering frameworks, core architecture clean-ups and refactoring. Moodle becomes really nice cathedral in many areas. I only hope I can still start my Moodle presentations saying that the Moodle development has been always driven by a solid pedagogical theory and by its users’ needs. But also, Moodle has always helped teachers to get constructivism into the class simply because using Moodle forced teachers to think in terms of students activity. Not only the Moodle development is driven by teachers’ needs. Moodle learns teachers how to teach in a different (hopefully better) way. I consider this the most important face of Moodle and I wish it would not be put into the shadow in the light of all the bright and glittering cherries of the future release.

As Galadriel said – the world has changed. Just day after Microsoft submitted driver source code for inclusion in the Linux kernel under GNU/GPL license, they have released Microsoft Live Services Plug-in for Moodle – again under GNU/GPL. It seems to me that the Moodle worldwide popularity together with the strict conditions ruled by GNU/GPL forced the software gigant to the next step on their relationships with the open-source movement. NetworkWorld.com comes with a nice overview of Microsoft/open-source milestones (including their patch to ADOdb library for PHP).

As a teacher I know from my own experience that the best method to understand something is to try to explain it to somebody else. It really works. Often, I realize some new important details or concepts or relationships while giving a lesson to the class.

So, when I wanted to sort out all my ideas about the Workshop random allocation method, I decided to document it so the teachers (and students, too!) know how the assignment is done and what results they can expect. I have described the method of random assigning submissions for peer-review and published it in the Workshop forum at moodle.org. If you have a minute, please check the PDF attached to the post and leave a comment in the forum. Thanks in advance!

I am stuck on random allocation subsystem for the new Workshop module. What looks like a trivial algorithm at the beginning, turns into a complex computation once you take various aspects into account. I have spent literally hours and hours by sketching squares, circles and arrows between them trying to discover all possible situations the workshop activity can get into.

In the most trivial case, everything is clear and easy. Say you have ten students in your course and you want each of them to review and assess three submissions. So you just randomly select three submissions for each student and you are done. If you want, you may order the allocator to ensure the self assessment. So it simply adds yet another allocation for every student, setting reviewees to the students themselves. Easy, ha?

Now, groups and groupings would like to play the game. Still no big issue, though. From my point of view, “Separate groups” mode is for those situations where you want or need students being enrolled in the single course but you do not want the groups to interact in any way. Ideally, students from different separate groups should not even know about each other. On the other hand, teachers use “Visible groups” mode, when they want some sort of group work support.

So, if the workshop is in the “Separate groups” mode, the allocator just have to be careful to select a reviewer of a given submission from the same group as the author is from. What if the author is in several separated groups? If I follow the precedent of the Forum behaviour, I can require the author’s submission must belong to a single separate group (like a post in a forum). As every user is allowed to have one submission per workshop at most, this results in that (s)he can’t submit into other separate groups (s)he is member of. Can this be an issue? Probably not – typical students are not expected to be members of several separate groups. But the software design is not about typical cases only. It must work in very rare situations, too.

If the workshop is in the “Visible groups” mode, the allocator shall select reviewers from all available groups. It shall try its best to have the reviews allocation balanced. Ideally, if your paper is going to be reviewed by three peers and there are three visible groups in the workshop, the allocator should pick one reviewer from each group. Advanced teachers could use this feature to get more objective assessment.

So far, I have assumed that reviewers are the same people as authors – it est “students” (peers) in your course. This has always been a must for the Workshop module. In Workshop 2.0, this is to change. There are two capabilities to control which role or roles can submit and/or assess submissions. By default, the legacy role Student is allowed both to submit and to assess others’ submissions. But you can localy override the permissions and – voilà – workshop behaves as an universal peer-reviewing system to be used as a conference management system, for example. So, some of your course participants (let us say Students) can be allowed to submit their work and others (for example Non-editing teachers or Assistants) are responsible for the assessment. You can have a course with 1000+ students divided into small groups (randomly or according their preference) and a cohort of reviewers, each of them having a group of students in charge. In this scenario, the random allocator is responsible for allocating submissions for reviewers in as balanced way as possible.

Other settings like the activity “Default grouping” or “Available for group members only” make the situation even more complicated. So, what looked like a simple game with a die, now has a thick book full of complex rules, tables and strange diagrams…

I have a working prototype of new feature in Workshop 2.0 – manual allocator. As I have described in my recent blog post, manual allocator is a Workshop subplugin that allows teacher to allocate submissions to the selected peers for review. The user interface may be a bit unusual at the beginning. However, I think it presents all relevant information in an understandable way once you realize how it works.

To demonstrate the new UI, I installed wink (thanks for the tip, Helen!) and prepared a screencast of the user interface (my first screencast ever!). Now I am awaiting for some feedback from the community. If they like it, the similar layout could be used consistently in the whole module.

The screencast of the manual allocation user interface

Modularity is one of the key features of Moodle. We have pluggable modules almost everywhere around a Moodle site: course activities, course formats, enrolment plugins, question types, authentication plugins, content filters, side blocks etc. Plugins may define their own database tables, permissions and language strings. Having independent modules implementing the given interface allows developers to relatively easily extend the standard distribution and customize it for their needs.

I have decided to follow the principle of modularity in the new Workshop module. The core of the module defines the required behaviour for the main features. Then, there are plugins that implement the functionality of the given component. Workshop consists of three main process components: submission, allocation and assessment.

The process of a student’s work submission is controlled by a submitter. Technically, submitter is an instance of a class implementing the submission interface. Its main purpose is to render a submission form and save submitted data. There will be a default submitter shipped with the workshop, which offers a submission of an online text with optional attachments.

The process of allocating submissions for review is controlled by an allocator. Allocator is a tool that assigns submissions to peers to be reviewed. So far, I plan to have two allocators available: Random allocator and Manual allocator. As the name suggests, random allocator implements the behaviour of the pre-2.0 Workshop. The manual allocator is a new feature that allows teachers to manually assign submissions. See the screenshot of the user interface (click to enlarge).

The screenshot of the manual allocation user interface

Other allocators can be simply added by writing a custom class that implements the allocation interface. So, I can imagine a plugin called Sorting Hat that checks the students’ grades received so far and tend to assign a work to a reviewer at the same grade level, for example. Similarly, the method described in a paper by Joan Codina and Josep Fontana can be implemented as an allocator. The important point is that allocation can be combined. For example, teacher can allocate several submissions manually and then let the rest of submissions allocate randomly.

Finally, the process of assessment is controlled by so called grading strategy. Grading strategy encapsulates the assessment form definition and the computation of grades. Four default plugins that implement this component will be available and they are described in the Workshop 2.0 specification.

So far, I have working drafts of the core code and some component plugins implemented. I have refactored them several times and I suppose I will probably do yet in some details. During the development, I am still learning the object oriented style of thinking. Luckily, there are three new friends that help me with this task: git branch and rebase, unit testing and design patterns

Thanks to a post by Steven O’Donnell at moodle.org, I have discoverd very interesting tool called Marginalia. Marginalia is an open source Javascript web annotation system written by Geof Glass. It allows users of web applications to highlight text and write margin notes as they can do in a desktop word processor. There is a demo site available where you can check how this stuff works.

As Steven noticed in his post, this would be a great feature for teachers and/or students to comment forum posts, online submission texts or (what a surprise ) a peer work in the new workshop module. Eventualy this might become a standard part of the Moodle core extending the current Comments 2.0 proposal. I am definitely going to consider this in the future.

The Workshop module specification has been marked as “Waiting for the approval”. I have incorporated valuable comments, ideas and suggestions that came during the community feedback period. Many thanks to Penny Leach, Mary Parke, Rick Barnes, Dan McGuire, Tim Hunt, Daniel Brouse, Mark Pearson, Matt Gibson and Vernellia Randall for their replies in the request for comments thread.

According to the Gnome Time Tracker, I have spent 51 hours and 32 minutes working on the specification (includes the time I was testing the current behaviour and was studying the source code). I think I’ve got a clear vision how 95% of the new module should work, how 85% of the UI should look, and how 75% of the internal API should be. So, I think it’s time to hack some code and chew bubblegum. And I’m all out of bubblegum.

Working on the Workshop 2.0 specification, I studied the current behaviour of the module. I needed to figure out how the calculation of grade for assessment (also known as grading grade) works. This part of the module has always been a mystery for me – and I was not alone. Before I learned how the calculation actually works, I proposed to rewrite it from scratch. Interesting. Is it just me tending to push own solutions instead of trying to understand someone else?

After some time studying the code I realized the ideas behind are pretty clever. Now I think there is no need to reinvent the wheel. If the calculation is documented well and the Workshop module has advanced reporting features that help to understand and explain why a student received the grade, the current algorithm should be kept.

Some non-trivial issues and questions emerged, however. The calculation is based on some basic statistical estimations. To be able to measure the quality of assessments, it is assumed that for a given student’s submission there is the only one theoretical objective assessment. Something like “if Zeus assessed the submission, he would give it 67/100″. Is this philosophically right? Is there the only truth about the work out there? And also – the grading in Workshop is determined by the grading form designed by the teacher. So even if Zeus is absolutely objective, his assessment is paltry if he has to use a crappy grading form.

It is clear the Workshop has a potential to be very mighty evaluation tool. In hands of a reckless teacher it becomes a dangerous weapon, however.