Moodle core itself has a solid versioning scheme (see the documentation page describing it). However, authors of contributed plugins are pretty free to use their own system of versions numbering. As the author of some contributed plugins, I was looking for a good versioning scheme for my contributions. From my point of view, such a “good” solution should (1) be stable in the long-term period and (2) intuitively describe the required Moodle core version the version of the plugin is for.

After considering several alternatives, I ended with a system similar to the one used by the Moodle core itself. So I use versions consisting of three numbers separated by dots, for example “2.1.3″. The first two of these numbers declare the Moodle branch the plugin is intended for. So the plugin with the version 2.1.3 would be for Moodle 2.1.x. The last number is the sequential update number. And of course, the plugin with the version 2.1.3 would be the fourth update of the plugin as all real geeks who were breast-fed by C like languages start counting with zero (2.1.0 was the initial release of the plugin).

This scheme fits well the convention of using standard Moodle names for development branches in Git. The repositories of my contrib plugins have branches MOODLE_21_STABLE etc like the Moodle core has. So the plugin version 2.1.3 would be born on MOODLE_21_STABLE branch in its repository. Naturally, I use Git tags for the version releases points. Luckily there was no need to re-invent anything here as together with the switch from CVS to Git, Moodle started using standard tags like v2.1.3 instead of legacy MOODLE_213 style.

Just today, I realized this approach has yet another aspect I did not fully realize initially. It forces the plugin maintainer to release new plugin version together with the Moodle major release. Even if there is no real update of the plugin. As an example, the Course contents block is pretty feature complete and there are no known bugs in it (ok, that’s a bit of blaspheming). The most recent version of it was 2.1.2. Now when Moodle 2.2 is released, I had to go and release 2.2.0 of the block to follow my guidelines. Even when the code of the block has not changed.

Is it a disadvantage of this scheme? I believe it’s not. It’s actually a great opportunity to confirm that the plugin still works on the most recent Moodle version. And also it’s the information for the users that the plugin is still actively maintained. Even by releasing the new version for purely formal reasons, I as the maintainer declare publicly that I tested the current version of the block with Moodle 2.2 and consider it stable and working. And that is worth going through the whole release procedure – branching, tagging and uploading the new version to the plugins directory.

Since very early versions of Moodle, the Workshop activity module has been the only plugin offering advanced grading based on assessment forms (like rubrics, marking criteria etc). No surprise there was a demand for such feature in other modules too – most notably in the Assignment. The feature request MDL-795 was reported for it in September 2003. Today, that issue can be marked as resolved – advanced grading methods (including Rubric as the first implemented plugin) survived the integration review and were merged into the main development tree. In other words, they will be part of Moodle 2.2. For the user documentation of this new feature see the Advanced grading methods and Rubrics wiki pages (or, if you prefer a screencast, see a nice preview prepared by Mary Cooch linked from there).

Me and a colleague of mine – Marina Glancy from Moodle HQ – formed the development team working on this. The project started by initial discussions and research in late July and August that led to the functional specification document. The spec was published and the community was asked for early feedback on the overall design. The actual coding started in the second half of September and after two months of heavy development, we managed to produce what we consider as a production ready feature. And – which is important in the context of the current development model – we did it in time (well, almost ). It’s a nice opportunity for a short reflection of the process.

For me personally, there were several factors that helped us to finish the project successfully. Let me mention some of them in random order.

Manageable goals and reasonable milestones

The project scope is pretty huge. At the end, advanced grading methods should become natural part of grading interfaces in all activity modules that support grading (Assignments, Glossaries, Databases, …) as well as other places dealing with grading (like essay questions in the Quiz, manually created grade items in the Gradebook). It would take much longer to make such a big change in one step. As a proof of concept, we focused on (a) supporting advanced grading in the Assignment module only and  (b) implementing the only one grading method plugin – rubrics. However, the framework is supposed to be easily extended to other activities and to support other grading methods via separate plugins. On the other hand it also means that the new API fits well the single use case. We are aware of that and we expect that the API will probably have to be improved in the future.

Minimal impacts on the current Moodle code

The advanced grading framework has been designed to extend current Moodle features instead of replacing them. The very last thing in the world I want to do is to rewrite Gradebook. Really, the Gradebook in Moodle is Pandora’s box for every developer. Better not to touch it unless you have written permission (and order) to do so. If you change something, half of the community report it as a bug. If you make it optional, the second half of the community groan the UI is too difficult.

So, instead of replacing current grading interfaces, the framework allows us to “attach” complex grading forms to any part of Moodle that can be described via a combination of context, component, gradable area name and item identifier. For example, let us have a single instance of the Assignment module. It has its context (CONTEXT_MODULE instance), component (“mod_assignment”), gradable area name (“submissions”) and item id (the submission id). This pattern in nothing new in Moodleverse. We use it with success in all the new APIs – files, comments or ratings.

This design allowed us to implement advanced grading into the Assignment module without touching the module’s database tables. Grading forms are seen as tools that calculate the final grade and provide it back to the module. All the data are kept in separate database space. In fact, the activity module does not need to know whether the grade was given directly from the grades selector (the standard way in Moodle) or calculated from a complex rubric.

As a result of this approach, even though the patch touches more than 60 files and adds more than 6000 lines of new code, it removes/modifies just 50 lines of the current code. I believe it definitely made the final integration review easier.

Keep the form definitions close to the place they are used

As you know, things like scales or questions in Moodle are typically defined at higher context levels (in the course, in the course category or for the whole site). That introduces a lot of issues when it comes to modifying the scale or question; or during backup and restore. For example, when you make a backup of an activity using a scale, you need to include the scale definition. But then, when you restore it, you need to carefully decide (read: guess ) whether or not a new scale should be created from the backup. For the same reason, scales in Moodle can’t be altered once they are used by some activity.

In this project, we tried an alternative approach. The grading form definitions are associated with the gradable area they are actually used at. In other words, every single assignment has its own copy of the grading form. It is possible to easily re-use these forms other activities by cloning the form. But as long as a new copy is always created, it was pretty easy to write backup and restore support (well, as easy as writing any backup and restore code in Moodle 2.x is ) for the forms and control modifications of them.

Splitting the development workload

I must admit I would not be able to finish the project on my own. Working on it in the team was a must – and actually a new experience for me personally. All my previous big DEV projects (the new Workshop module, AMOS or restoring 1.9 backups into 2.x) were sort of one-man show. This was the first time I had to cooperate closely with another developer. Those who were involved know that this was pretty challenging for me and the coding was salted with many emotional grrr’s and wtf’s. Today, I must say I am happy I had Marina working on this project with me. It would not happen without her.

The nature of the design allowed us to split the work so I could work on the backend framework that provides general support for all grading methods (managing the gradable areas, cloning the forms, sharing them as templates etc). Marina did a great job developing the Rubric plugin itself and most notably integrated it into the Assignment module. It was not an easy task given the current state of the Assignment module’s internals. Marina has my respect and I sent her Kudo for this.

Continuous peer-review and feedback

We were working on opposite sides of the planet on this project and the time zones difference worked against us. In such situation, I found Github very helpful. It allowed us to work so that Marina sent me pull requests regularly and I merged them into the pre-integration feature branch. We usually started our day by reviewing the changes introduced by the other one.  Github allows to comment the code easily in-line and send feedback to each other. While implementing the rubric plugin, Marina suggested significant improvements of the API this way.

Also, Martin Dougiamas was involved, feeding us regularly with ideas on how the user interface and the general user experience should look and feel like. So, if you like it, praise Martin. If you don’t, blame me and Marina

Current stable version 2.0.1+

Total of 64 patches were accepted by the integration and testing teams during this week for Moodle 2.0 stable branch (which still lives on Git ‘master’). This is a new record in terms of number of pull requests. Also, with only 5 pull requests rejected, this is the best accepted/rejected ratio achieved so far. It also means I am no longer able to provide an overview of all accepted changes so I will focus only on those I helped to test or review.

Moodle can now be installed on PostgreSQL server with ‘standard_conforming_strings’ option enabled. This option is on by default since PostgreSQL 9.1 and Bruce Momjian’s blog post explains why (MDL-26351). ♦ It has been known for a while that MSSQL and Oracle databases do not like SELECT DISTINCT if the list of returned columns contains some TEXT ones. This is tricky especially in a case like SELECT DISTINCT t.*, if the table t contains some TEXT fields. As most Moodle developers use PostgreSQL or MySQL as a part of their environment, some affected queries still survive in Moodle code. Eloy Lafuente found and fixed couple of them, using a nice inner join with subselect solution (MDL-26371). ♦ Mark Nielsen spotted a bug in the implementation of print_collapsible_region() function. Glenn Ansley took this as an opportunity for his first contribution to Moodle, created a branch at his moodle.git clone at github.com and prepared a patch there (MDL-26131). ♦ Andrew Davis fixed a really nasty bug in blogging subsystem that was causing accident removal of all recent records in the table ‘post’. This table holds not only blog posts but also user notes too, for example (MDL-26010). A regression of a recent change in the upgrade code of SCORM module was fixed by Eloy Lafuente (note for myself: next time I test SCORM upgrade code, I shall have some SCORM module instance actually created at the site… MDL-26361) ♦ And finally, images embedded into forum posts that are sent by email to the forum subscribers are now correctly displayed in the email clients. Given that the client has access to the forum either via the current user’s session (in case of web email clients) or the course grants access for anonymous hosts (MDL-25944).

Previous stable version 1.9.10+

All 9 submitted patches were accepted and they landed on MOODLE_19_STABLE branch. ♦ Petr Škoda fixed a problem with redirecting to a login page on https protocol and a bug mediaplugin filter. ♦ Eloy Lafuente found some queries using SELECT DISTINCT from TEXT fields in 1.9 version, too. ♦ Dan Marsden fixed a bug in a pop-up form in SCORM module. ♦ Aparup Banerjee fixed the number of glossary entries being displayed in the recent activity block. ♦ Tim Hunt fixed a bug in quiz that used to display overall feedback based on the rounded grade instead of the real value.

Quotes of the week

“We will ALWAYS have blockers and critical issues, they never end. If we tried to solve them all, we’d never release anything.”
– Martin Dougiamas has a pragmatic view on release policy

“Any code you write yourself should follow all the coding guidelines perfectly. However, when looking at old code, you should be tolerant of what you find.”
– Tim Hunt in a discussion on a policy of naming database tables in Moodle

“Never trust user input.”
– Jonny Barnes at an excellent page summarizing what should a developer know before building a public web site

Is it there yet?

If you are watching some issues in the tracker, you probably know those emails informing you about the change of the issue status. If you receive an email that your favourite issue has been resolved or closed, it is good to understand what exactly it means.

As you probably know, the current Moodle development workflow is based on so called fork and pull model. All developers submit their patches into separate branches in their own public forks (clones) of the official Moodle git repository. Then they ask the integration team for pulling the changes from these forks into the official repository. Although most Moodle contributors seem to use github.com as a place to publish their repositories, we do not use pull requests feature offered by Github. Instead, a PULL issue in Moodle tracker must be created for each submitted branch (patch). In the created PULL issue, the contributor describes the submitted patch and provides information for the integration team necessary for including the patch.

The relevant MDL issue being fixed by the submitted patch is marked as resolved immediately after the PULL request is created. The contributor is supposed to link the PULL issue with the MDL issue and resolve the MDL issue. So resolved means there is a patch available for this issue, but the patch is not part of Moodle package yet.

Every week on Monday, the integration team goes through the list of submitted PULLs and reviews the patches. If the patch is accepted, it is merged into integration.git repository and the status of the PULL issue is changed to “ready for testing”. When all submitted patches are reviewed (that is they are either accepted or rejected), the testers come and test all changes. If the test passes, the tester closes the MDL issue (alternatively, the release manager closes all MDL issues with their tests passed at the end of testing). So closed MDL means the patch has been accepted by both integration team and testing team. But the patch is still not part of the official Moodle package.

You must wait until all integrated patches are tested and the patches are pushed from integration.git repository into the official moodle.git repository. At the same time, ZIP and TGZ packages are regenerated and can be downloaded. This typically happens on Wednesday afternoon (European time).

As an example, let us say I work on some bug. On Thursday morning, I commit a fix into my repository at github.com. Then I go to the tracker, create a new PULL issue and resolve the linked MDL issue. On Monday, a member of the integration team reviews my patch and if I am lucky enough, they accept it. On Tuesday, a member of testing team follows the steps I provided in the PULL requests and hopefully confirms that the patch really fixes the reported issue. The linked MDL issue is closed then. Therefore the next weekly build generated on Wednesday will contain my fix.

Current stable version 2.0.1+

Total of 27 submitted patches for Moodle 2.0 were accepted this week. Eight pull requests were rejected. To highlight some of the accepted patches, Andrew Davis fixed a blocker issue reported by Aaron Cowell causing errors in RSS feeds generated by Moodle (MDL-24870). ♦ Dongsheng Cai fixed a bug in Database activity module’s templates editor (MDL-25671). ♦ Eloy Lafuente prepared an emergency fix of a regression blocking the upgrade at Oracle servers.

Previous stable version 1.9.10+

There is a single accepted patch for Moodle 1.9 from the last week. Eloy Lafuente fixed a regression in Database activity module reported by Paul Nijbakker. Some module functions redirected user to the first record in the database instead of to the correct one (MDL-26052). The second submitted patch for Moodle 1.9 improving inefficient computation of tag correlations had to be rejected because the included SQL statement was not cross-platform and worked under MySQL only (MDL-24355).

Quotes of the week

“I think we need to become serious and start rejecting any commit with [trailing] whitespace.”
– Eloy Lafuente strongly recommends all developers and contributors to set up their IDE properly

“Open source sometimes actually works, even without doing the programming yourself.”
– Sam Marshall knows that reporting a bug is sometimes enough to make the thing fixed

Testing Moodle on various databases

Moodle 2.0 officially supports four major database systems: PostgreSQL, MySQL, MSSQL and Oracle. I have PostgreSQL and MySQL installed on my Linux workstation and I use PostgreSQL 8.3 at the moment as the major platform for the development – simply because I believe that if a code works at PostgreSQL, it is generally more cross-platform than a code that was tested on MySQL. Though there are exceptions, of course.
To help with testing patches related to MSSQL, I decided to set up MSSQL server for me. I have successfully installed and set up MSSQL Server 2008 R2 Express edition. The server runs in a virtual machine on my notebook. I use VirtualBox from Oracle (Yes, Oracle made VirtualBox and we’ve always been at war with Eastasia) networked via a bridge between the Linux host and Windows guest. That allows me to have Moodle installed in my Gentoo being connected to the MSSQL database in Windows via freetds driver. To test the new sqlsrv driver, I downloaded Moodle Windows package to the guest and was able to install it there, too.

There were 53 patches submitted for integration this week; 10 of them were rejected, 43 survived both review and testing and got into the official repository. Regular weekly builds were released on Wednesday.
Today on Friday morning, Tomaz Lasic spotted a critical regression at School Demo Moodle site. Due to recent changes in modinfo library, Workshop module’s navigation stopped working completely, throwing coding exception because the new modinfo API was not backward compatible by mistake. This issue was promptly fixed by Sam Hemelryk and emergency weekly build of Moodle 2.0 was re-released.

Current stable version 2.0.1+

Total of 40 accepted pull requests affect Moodle 2.0 branch. ♦ Andrew Davis added new index to the table user_info_data consisting of userid and fieldid fields to increase performance when searching over custom user fields (MDL-17201). ♦ Dan Marsden fixed several SCORM module bugs and cleaned up its code. ♦ Dongsheng Cai fixed a bug in file picker. Internet Explorer was trying to download repository_ajax.php instead of executing it via AJAX request when users had .php files associated with an external editor. Dongsheng also fixed a bug in Wiki module’s upgrade code, dealing with wikis with the same title (some issues with Wiki upgrade seem to be still open – see comments in PULL-196 for details). ♦ Eloy Lafuente submitted a set of patches, mainly dealing with database access – wrong usages of get_text() and get_recordset_xxx() and using words reserved in Oracle as bind placeholders. ♦ John Stabinger fixed various issues in several Moodle 2.0 themes. ♦ Petr Škoda submitted a bulk of patches fixing many areas of the code. ♦ Rossiani Wijaya improved Multimedia plugins filter so that it now supports YouTube playlists (MDL-25573). ♦ Sam Hemelryk improved performance of the new navigation system (MDL-25291). The reporter Petr Škoda noticed that the new administration block is loaded for navigation purposes at almost every page even though most users (students) can not see anything in it. Sam implemented a new session flag to indicate whether there is actually something to load for the administration block. Not trying to load an empty block makes quite a noticeable performance difference in number of included files, RAM consumption and get_string() calls. ♦ Sam Marshall submitted another patch leading to the proposed improvements of get_fast_modinfo() function. ♦ Tim Hunt submitted three patchsets, all dealing with issues in Quiz module and Question bank.

Previous stable version 1.9.10+

Three accepted pull requests affect Moodle 1.9 branch. ♦ Andrew backported MDL-17201 discussed above. ♦ Dan Marsden backported a SCORM issue MDL-25320. ♦ Petr Škoda fixed a bug in an enrolment plugin – sites using IMS Enterprise file for enrolments should upgrade.

Quotes of the week

“Ooh duh – somebody actually documented it. I wasn’t expecting that ”
– Sam Marshall was lucky enough to find up-to-date information at a documentation page for developers

“git blame or blame git – that’s the question”
– Eloy Lafuente comments on Sam Hemelryk’s experience in MDL-25791

Performance matters, women say

Although Moodle developers focus on performance these days, not many performance related patches landed this week yet. And I can understand why. From my own experience, working on performance is pretty time consuming job requiring a lot of comparison tests on production data to get valid results. Luckily, Moodle 2.0 has inbuilt support for xhprof – a profiling tool written by Facebook developers. Large sites admins and Moodle developers can use this tool to compare how various tiny improvements save milliseconds of execution time and to analyse the complex function call graphs, identifying performance bottle necks.
From what we could see so far, the performance of the new navigation system (used at almost every page) and the recently improved filtering system (which makes major part of frequent format_text() function) should be our primary targets now. I started to study the current implementation of text filters and the mechanism of how they are chained, looking for places to improve. While reading the code, some important questions emerge that will need design decision soon.
One of them is a well known “nolink” feature. Moodle users know this as a button in HTML editor that is supposed to “prevent automatic linking” within the selected block of text. This feature was introduced back in 2003 as a part of glossary terms auto-linking feature. Later the feature was used by other parts of Moodle – activity names linking, database records linking etc. As all these automatic linking features are now implemented as standard filters, the concept became less clear. Is the text wrapped by “nolink” marks supposed to be processed by other filters that do not actually create links? Shouldn’t it be replaced with some “do not apply any filter on this block of text” feature?
Beside this conceptual question, there is an issue with the actual implementation of the feature, too. Originally, the part of the text where no automatic linking was supposed to happen had to be wrapped by <nolink> and </nolink> tags. Later as a part of transition to full XHTML support, these tags were replaced with valid <span class=”nolink”>…</span> pair. It works pretty well for short strings – typically glossary terms. But this syntax fails badly when the inner text contains another span tags as it becomes difficult (if not impossible) to identify the correct substring by regular expressions (we can not afford full DOM parsing for obvious performance reasons). Also, using spans makes it impossible to start the protected block in the middle of one paragraph and end it in the middle of the second one because HTML tags must nest correctly.
The solution I am about to suggest is to introduce a new syntax using a couple of empty span tags. The start of the protected block would be defined using something like <span class=”nofilterstart”></span>. The block would end at the place where <span class=”nofilterend”></span> is found. XHTML specification recommends not to use the minimized tag syntax for empty span elements, so even if shorter <span class=”xxx” /> would work (and Moodle will support it probably), our HTML editor would produce the syntax with closing tag. This way, both issues with proper nesting and inner spans would be solved.
You see – I got into a completely different area than the performance of format_text() I started to work on But it is quite related in fact. If the filter manager is able to detect some common parts of the text that no filters are expected to modify, it can save time spent on search & replace job that every filter does. Any feedback, idea or contra-proposal is welcome, of course.

Current stable version 2.0+

There are 23 requests for accepting a patch into the 2.0 branch of the new Moodle git repository from the last week. During the today’s review, 15 of them have been accepted by the integration team and graduated to quality assurance testing; 8 of them have been rejected.

Previous stable version 1.9.10+

There are 9 requests for accepting a patch into the 1.9 branch from the last week; 6 of them have been accepted and graduated to quality assurance; 3 of them have been rejected during the integration review.

Quotes of the week

“This is my last commit to CVS. Goodbye old friend – you’ve served us well.”
– Martin Dougiamas says goodbye to CVS that held the history of changes for 9 years of Moodle development.

“The point is to fix more than you break.”
– Petr Škoda’s simple recipe for an acceptable commit (originally said in a context of upcoming major version release but still applies)

Wind of change

Last week was the first one running under new development procedures. I helped to set up git repositories moodle.git and integration.git that are key part of the new “pull” model of code contribution. Although I am not member of the integration team, I was trying to assist guys in their work, observing the process and discussing the patch workflow.
It is too early for any strict and definitive rules. However, the whole process is getting the final shape. Let us illustrate the key points of it on an example of MDL-25647. The patch fixing the described issue is pretty trivial. In the previous model, I would modify my local checkout and commit the change into CVS central repository immediately. So we can call that “push” model.
In the current “pull” model, I do not have write access to the official Moodle repository any more. Actually almost nobody has – even Moodle HQ developers lost this permission. Instead, I have a clone of moodle.git repository at my local computer. I created a local branch forked off the remote master branch (master branch is where the current Moodle 2.0 development happens) and named it MDL-25647_mnet_peer_setting_not_saved:

$ git checkout -b MDL-25647_mnet_peer_setting_not_saved origin/master

I committed the fix into the branch. By running

$ git cherry -v origin/master MDL-25647_mnet_peer_setting_not_saved

I can make sure that the branch contains just the commit I want to be included in Moodle core (the line starts with the plus sign). I use `git diff`, `git log -p` or `git show` to see the change introduced by the commit.
Once I am happy with the change, I have to publish the branch in some public place so that integrators can pull the change from there and incorporate it into Moodle. I use my repository at github.com for these purposes but I could use any other git hosting site (like legendary repo.or.cz or gitorious.com) or my own git server if I had one. It really does not matter once the repository grants public read access (I could actually export the commit into a file and email it to Moodle or send via tracker – but the public repo method is preferred by Moodle integrators). As I already have github.com repository aliased to “github” remote at my notebook, I simply run

$ git push github MDL-25647_mnet_peer_setting_not_saved

Now I need to ask Moodle integration team to pull from that branch. This time, I had to create an issue in Moodle tracker PULL project which led to PULL-18. But there is an ongoing discussion that we could use single project workflow so that the information about the pull request would be attached to MDL issue directly.
And that’s all! I just needed to wait until Monday when the integration window opens. But that is not a work stopper as I would create other topic branches for all other things and work on them. So the delay is not a blocker and it gives me some extra time to change my mind, for example.
On Monday, integrators go thru the list of submitted branches and review them. They use the information I provided in PULL request and run the following commands to fetch from my public repository into their local clone of integration.git and review the change I propose:

$ git fetch git://github.com/mudrd8mz/moodle.git MDL-25647_mnet_peer_setting_not_saved
$ git log -p ..FETCH_HEAD

If they accept the change, they would just merge it

$ git merge FETCH_HEAD

If they ralize they overlooked something and change their mind immediately after merge, they can undo it by

$ git reset --hard ORIG_HEAD

The same procedure is used for all other pull requests so at the end of the day, the master branch at integration.git contains all accepted patches (of course, git preserves authorship information, commit messages, commit timestamps etc). The repository is then deployed at quality assurance test site and QA team make sure that the issue is fixed from the user’s point of view. After QA tests pass, the integration.git is pushed to moodle.git and a snapshot of it is exported into CVS mirror (that should happen on Wednesdays). So the result is backward compatible with legacy weekly builds.
When moodle.git is updated, I fetch new changes from it and use the following command again:

$ git cherry -v origin/master MDL-25647_mnet_peer_setting_not_saved

This time, the commit line starts with the minus sign indicating that the commit has been included upstream (note that this check would work even if the patch got into upstream via cherry-picking or via email and therefore its commit hash would be different). So it is time to delete the topic branch:

$ git push github :MDL-25647_mnet_peer_setting_not_saved
$ git branch -d MDL-25647_mnet_peer_setting_not_saved

More coming soon as a part of a git course for Moodle contributors I am currently working on. Stay tuned!

Post scriptum

I found gitster’s journal excellent place for learning about git

Latest stable version 1.9.9+

There are 3 commits into the stable branch from the last development week (Tue 28 to Mon 4). ♦ Andrew Zoltay spotted a bug in login form on sites with “Use HTTPS for logins” setting on and provided a patch, included by Petr Skoda (MDL-24225). ♦ Two other commits are just trivial code cleanups. Rossiani Wijaya removed whitespace in the code she committed week before. ♦ Andrew Davis fixed a table comment in one of XMLDB files, spotted by Eloy Lafuente.

Moodle 2.0 RC1

There are 106 commits into the future release branch from the last week. The main community site http://moodle.org has been upgraded to 2.0 engine during the weekend and helped the core developers to discover some forgotten bugs and incompatible customizations.

Quote of the week

“Raarrrrrrrr fixed way old bug from dml conversion”
– Sam Hemelryk really enjoys bug fixing rampage

Parsing uploaded string files in AMOS

I was working on a new AMOS feature that allows users (language pack maintainers and translation contributors) to upload their translation to AMOS and include it in the main repository or offer it for inclusion. The first supported format of such file is standard Moodle string file format, which is valid PHP code defining associative array $string. The problem was that for obvious security reasons, I can not just let anonymous users to upload whatever PHP code and execute it. Imagine what would happen if hacker came and uploaded a file like

<?php
global $CFG;
$string['something'] = $CFG->dbpass;

If I just executed such file via include(), the user would get access to sensitive server configuration data. So it was clear I have to write my own parser that extracts the array from the file without actually executing the file.

I initially tried to solve it by searching for some patterns in the file contents using regular expressions. That worked pretty well in simple cases but unit tests (of course I started with them) failed quickly for complex samples that included commented lines, block comments or strings that contained the interesting patterns themselves.

<?php
// $string['lalala'] = 'Knock knock';
/*
$string['grrrr'] = 'Who\'s there?';
*/
$string['nasty'] = '$string[\'nasty\'] = \'Funny heh?\';';

So I realized regexps are not suitable for this kind of task. The other approach how to deal with comment blocks would be stream processing of the file contents, with a lot of flags like “inside a comment”, “after line comment mark” or “waiting for variable name”. But that was evident reinventing of the wheel which would have a square shape at the end of the day, anyway. PHP itself has to do this boring job when analysing the source so I just had to learn how to use its results.

Wise may already know I ended with tokenizer extension. The tokenizer functions provide an interface to the PHP tokenizer embedded in the Zend Engine. Using these functions you may write your own PHP source analyzing or modification tools without having to deal with the language specification at the lexical level. The parser method I have finally implemented calls token_get_all() to get array of tokens found in the uploaded file and picks patterns that are considered as valid string definition.

For the purpose of uploading the string file into AMOS, valid definition is something like ‘T_VARIABLE $string followed by [ followed by T_CONSTANT_ENCAPSED_STRING followed by ] followed by assignment = followed by T_CONSTANT_ENCAPSED_STRING followed by semicolon’. All other tokens like T_WHITESPACE, T_INLINE_HTML, T_COMMENT or T_DOC_COMMENT are just ignored.

As you can see this means that code like

$string['greeting'] = 'Hello ' . 'world';

is considered as syntax error for AMOS import even though it is valid PHP
code. But I am sure it OK as there is no real reason to support all thinkable
ways of string definiton (like heredoc etc).

Latest stable version 1.9.9+

There are five commits into the stable branch from the last development week (from Tue Sep 21 to Mon Sep 27). ♦ Rossiani Wijaya fixed a bug in Forum module that allowed students to delete their posts even after $CFG->maxeditingtime (MDL-24258). Rossie also applied a patch suggested by Ken Gibson that deals with a bug in Internet Explorer 8. This browser does not support links to empty anchors like <a id="gohere"></a> and caused that section links from Course sections block (and Course contents extension, too) did not work in it. Credit goes to Matthew Davidson for discovering this IE8 behaviour (MDL-18232). ♦ Andrea Bicciolo committed a fix prepared by Daniele Cordella that fixes a regression in Database module, caused by recent work on MDL-24033. ♦ Iñaki Arenaza continues with fixes of CAS authentication module that crashes if there is no LDAP server set up (MDL-16168). Iñaki also applied a correction by Jose Garcia that fixes CAS synchronization with Microsoft Active Directory.

Moodle 2.0 RC1

There are 110 commits into the main branch from the last week. Development is now focused on fixing bugs and polishing the code style. More than 50% of total 176 QA tests already passed the second QA testing cycle, more testers are welcome to confirm the rest.
Eloy Lafuente’s recent work on Question bank backup landed in CVS and he is currently in a heavy combat with their restore. Backup and restore of Questions is not trivial job. By the design, Moodle backup and restore (now stored in *.mbz files) works best with whole course backup. But questions generally live in their own contexts outside the course, they can be used from multiple activities across courses, there is a complex machinery of access rights etc. These factors make the procedure of their backup and restore pretty complex.

Quotes of the week

“I sort of think at this point the best solution is to install an orbital death laser which will be automatically targeted at any server that ever runs MySQL, however I will try to come out with a plan B just in case that one doesn’t work out.”
– Sam Marshall encountered a feature of MySQL: you cannot update a table and select from it in a subquery.

CVS is fine if you aren’t using git.
– Martin Dougiamas originally said this in a different context but I like the sentence as it is :-p

Who to blame for Moodle 2.0

Moodle project is approaching 2.0 release. Let us look back on who contributed to the HEAD branch during this development cycle.
The work on Moodle 2.0 officially started on 10th October 2007 when the branch MOODLE_19_STABLE was forked and the HEAD branch was renamed to ’2.0 dev’. Today, there are 8097 files in the source code tree, containing 1.9 millions lines of code (if they all were considered as text files – some of them are binary files like pictures).
For the purpose of this report, I used moodle.git mirror and a couple of shell lines combining common UNIX data processing tools (like sed, awk, cut, sort and uniq) to go through all these files and executing ‘git blame’. This command displays the content of the file and (among other information) the name of the developer who most recently modified every single line in it. We can roughly estimate a developer’s contribution by counting all the lines reported for them by git-blame. Note that this method does not say anything about the amount of the real work – the most recent modification could by a fix of a stupid typo or even just a removal of trailing whitespace. Also, if a developer just commits some other’s work, git-blame will consider them as the author. Which is not that bad actually as the person is really the one who is responsible for that code being part of Moodle. While having in mind what this really represent, let us look at the aggregated figures.
The following table contains the list of top 20 contributors into Moodle 2.0 according the number of most recently modified lines of code. The sum of numbers of lines modified by these people represents around 97% of all lines in the current Moodle 2.0 tree.

At first, I was surprised by the amount of ‘my’ lines. Then I realized that while working on new format of Moodle strings in 2.0, I re-committed all the language files from legacy /lang/en_utf8/ folder into new lang/en/ folders. Therefore I am the one reported as the most recent editor of most of 15000 strings in Moodle and git blames me instead of the real string authors. Let us take it as an example that similar statistics should be always interpreted very carefully.

Post scriptum

Yesterday, I had my last John Smith’s Extra Smooth beer can and so the last relic of my holiday trip to Wales has gone.

Latest stable version 1.9.9+

There are 15 commits into the latest stable branch from the last development week (from Tue Sep 14 to Mon 20 Sep) and the current weekly build seems to contain several important fixes. ♦ Andrew Davis fixed two critical bug in the Gradebook. The first one caused that the calculated total course grades in the user report were incorrect when all users were selected. The first student’s grade was displayed instead of the valid values (MDL-22841). The second bug, spotted by Chris Bandy, was that Moodle actually never recorded the id of the user making a change when archiving it in the grades history (MDL-23928). ♦ Iñaki Arenaza fixed a bug in CAS authentication plugin. Now there is no need to have an LDAP server to fetch user data from and you can fetch your CAS users into Moodle from other sources (like a text file), too. Credit goes to Matthew Turney for the patch (MDL-16168). ♦ Martin Dougiamas fixed a problem reported by Stefan Schramm. Forum module put invalid In-Reply-To and References headers in mails for the initial thread posts. That led to broken sorting of emails in mail clients with mail threads support (MDL-22551). ♦ Sam Hemelryk fixed the Assignment module so that it still allows students to view their submission even after they do not have the capability to submit it any more (MDL-23848). ♦ Andrea Bicciolo fixed a small regression of his recent patch in the Database module (MDL-24033). ♦ Pierre Pichet committed Oleg Sychev’s patch that replaces one !empty() call with the correct isset() in the form for editing a question with extra_question_fields so the zero value is accepted now (MDL-24241). ♦ Eloy Lafuente made new Dzongkha language pack available for Moodle installation. ♦ I have applied a patch provided by Hubert Chathi that fixes addGroupRule() in form elements containing square brackets and fixed client side validation for the Location field in the Resource so that the field is required now.

Moodle 2.0 RC1

There are 403 commits into the main development branch from the last week. Most of them are results of general code clean up made by Petr Škoda. Petr used code analysis feature of his currently favourite IDE PhpStorm and discovered many malicious typos, forgotten global variable declarations and other mistakes in Moodle 2.0 code.
Even without an official announcement, many community members noticed that 2.0 development cycle reached the Release Candidate 1 milestone. The development team considers the branch feature-complete. See more details in the Release notes.
Hot gossip, Martin Dougiamas confirmed that Moodle development will definitely switch from CVS to git in a very near future. Even though some doubt about it, I can’t imagine any development work without this tool any more. Yes, I admit that mastering git takes some time but I can just recommend it to all developers and contributors – and especially to those making their own customizations of the code.

Quotes of the week

“In any case, the important point to learn is to never use $PAGE for deciding contexts as far as it causes all the ‘remote’ operations to fail.”
– Eloy Lafuente reminds us that global $PAGE is supposed to be used only for output rendering purposes and that abusing it at lower levels leads to problems.

“Software development is meant to be fun. If it isn’t, the process is wrong.”
– Pete McBreen in his Software Craftsmanship: The New Imperative

Mahara 1.3 RC1 has been released and the list of new features is promising. Does it mean that the book Mahara 1.2 ePortfolios: Beginner’s Guide is now outdated? I am pretty sure it is not. If you are just about to start with Mahara, either current stable version or the new one, you will find the book useful.

The book was published by Packt Publishing in February 2010 and I got a chance to read it recently. I was looking forward to it because as a teacher who had some personal experience with implementing Mahara at secondary and high school level, I was wondering what way of introducing e-portfolios to the students the authors recommend.

The guide was written by Derrin Michael Kent, Glenys Gillian Bradbury, Margaret Anne Kent, Richard William Hand. This team represents a balanced group of people with solid background in educational technology, open-source development, end-user training and learning design, too.

While reading the book, I enjoyed the focus on instant action. Since very first pages, reader can start experimenting with Mahara features being introduced. The concepts of the software are illustrated in many use cases from various environments like primary school, international corporate university or vocational training provider. These examples and scenarios helped me to extend my existing attitude to Mahara as I tended to see it as a software for schools only.

After the initial introduction into the environment, creating an account and setting up the user profile, the book guides the reader through key elements of Mahara system – artefacts, blocks and views. I appreciate that the authors managed to satisfy both kinds of readers. Those who need detailed step by step instructions on how to work with Mahara user interface will use the book as very well written manual. And those who prefer to explore the environment on their own by just playing with it will find some inspirative examples to think about.

In seven chapters, the book covers basically all available features of Mahara 1.2 including the users interaction, groups and institutions management and site administration. Touching all these areas, it offers a clear overview of what Mahara is and help you to decide if it would suite your needs.

When I was explaining Mahara concepts to my students, we were able to prepare their first views during the first lesson. Of course, by just telling them click here, click there, it is not difficult at all in fact. Mahara has pretty, modern user interface, intuitive for people who are familiar with Internet social networks. But after reading this book, I was able to provide them more examples and think of other projects and situations where we could use Mahara successfully.

You will be able to become Maharan without this book, definitely. But if you are going to explain what Mahara is and how to work with it effectively to the others, this book is written for you.