David Mudrak's blog

Moodle core itself has a solid versioning scheme (see the documentation page describing it). However, authors of contributed plugins are pretty free to use their own system of versions numbering. As the author of some contributed plugins, I was looking for a good versioning scheme for my contributions. From my point of view, such a “good” solution should (1) be stable in the long-term period and (2) intuitively describe the required Moodle core version the version of the plugin is for.

After considering several alternatives, I ended with a system similar to the one used by the Moodle core itself. So I use versions consisting of three numbers separated by dots, for example “2.1.3″. The first two of these numbers declare the Moodle branch the plugin is intended for. So the plugin with the version 2.1.3 would be for Moodle 2.1.x. The last number is the sequential update number. And of course, the plugin with the version 2.1.3 would be the fourth update of the plugin as all real geeks who were breast-fed by C like languages start counting with zero (2.1.0 was the initial release of the plugin).

This scheme fits well the convention of using standard Moodle names for development branches in Git. The repositories of my contrib plugins have branches MOODLE_21_STABLE etc like the Moodle core has. So the plugin version 2.1.3 would be born on MOODLE_21_STABLE branch in its repository. Naturally, I use Git tags for the version releases points. Luckily there was no need to re-invent anything here as together with the switch from CVS to Git, Moodle started using standard tags like v2.1.3 instead of legacy MOODLE_213 style.

Just today, I realized this approach has yet another aspect I did not fully realize initially. It forces the plugin maintainer to release new plugin version together with the Moodle major release. Even if there is no real update of the plugin. As an example, the Course contents block is pretty feature complete and there are no known bugs in it (ok, that’s a bit of blaspheming). The most recent version of it was 2.1.2. Now when Moodle 2.2 is released, I had to go and release 2.2.0 of the block to follow my guidelines. Even when the code of the block has not changed.

Is it a disadvantage of this scheme? I believe it’s not. It’s actually a great opportunity to confirm that the plugin still works on the most recent Moodle version. And also it’s the information for the users that the plugin is still actively maintained. Even by releasing the new version for purely formal reasons, I as the maintainer declare publicly that I tested the current version of the block with Moodle 2.2 and consider it stable and working. And that is worth going through the whole release procedure – branching, tagging and uploading the new version to the plugins directory.

Since very early versions of Moodle, the Workshop activity module has been the only plugin offering advanced grading based on assessment forms (like rubrics, marking criteria etc). No surprise there was a demand for such feature in other modules too – most notably in the Assignment. The feature request MDL-795 was reported for it in September 2003. Today, that issue can be marked as resolved – advanced grading methods (including Rubric as the first implemented plugin) survived the integration review and were merged into the main development tree. In other words, they will be part of Moodle 2.2. For the user documentation of this new feature see the Advanced grading methods and Rubrics wiki pages (or, if you prefer a screencast, see a nice preview prepared by Mary Cooch linked from there).

Me and a colleague of mine – Marina Glancy from Moodle HQ – formed the development team working on this. The project started by initial discussions and research in late July and August that led to the functional specification document. The spec was published and the community was asked for early feedback on the overall design. The actual coding started in the second half of September and after two months of heavy development, we managed to produce what we consider as a production ready feature. And – which is important in the context of the current development model – we did it in time (well, almost ). It’s a nice opportunity for a short reflection of the process.

For me personally, there were several factors that helped us to finish the project successfully. Let me mention some of them in random order.

Manageable goals and reasonable milestones

The project scope is pretty huge. At the end, advanced grading methods should become natural part of grading interfaces in all activity modules that support grading (Assignments, Glossaries, Databases, …) as well as other places dealing with grading (like essay questions in the Quiz, manually created grade items in the Gradebook). It would take much longer to make such a big change in one step. As a proof of concept, we focused on (a) supporting advanced grading in the Assignment module only and  (b) implementing the only one grading method plugin – rubrics. However, the framework is supposed to be easily extended to other activities and to support other grading methods via separate plugins. On the other hand it also means that the new API fits well the single use case. We are aware of that and we expect that the API will probably have to be improved in the future.

Minimal impacts on the current Moodle code

The advanced grading framework has been designed to extend current Moodle features instead of replacing them. The very last thing in the world I want to do is to rewrite Gradebook. Really, the Gradebook in Moodle is Pandora’s box for every developer. Better not to touch it unless you have written permission (and order) to do so. If you change something, half of the community report it as a bug. If you make it optional, the second half of the community groan the UI is too difficult.

So, instead of replacing current grading interfaces, the framework allows us to “attach” complex grading forms to any part of Moodle that can be described via a combination of context, component, gradable area name and item identifier. For example, let us have a single instance of the Assignment module. It has its context (CONTEXT_MODULE instance), component (“mod_assignment”), gradable area name (“submissions”) and item id (the submission id). This pattern in nothing new in Moodleverse. We use it with success in all the new APIs – files, comments or ratings.

This design allowed us to implement advanced grading into the Assignment module without touching the module’s database tables. Grading forms are seen as tools that calculate the final grade and provide it back to the module. All the data are kept in separate database space. In fact, the activity module does not need to know whether the grade was given directly from the grades selector (the standard way in Moodle) or calculated from a complex rubric.

As a result of this approach, even though the patch touches more than 60 files and adds more than 6000 lines of new code, it removes/modifies just 50 lines of the current code. I believe it definitely made the final integration review easier.

Keep the form definitions close to the place they are used

As you know, things like scales or questions in Moodle are typically defined at higher context levels (in the course, in the course category or for the whole site). That introduces a lot of issues when it comes to modifying the scale or question; or during backup and restore. For example, when you make a backup of an activity using a scale, you need to include the scale definition. But then, when you restore it, you need to carefully decide (read: guess ) whether or not a new scale should be created from the backup. For the same reason, scales in Moodle can’t be altered once they are used by some activity.

In this project, we tried an alternative approach. The grading form definitions are associated with the gradable area they are actually used at. In other words, every single assignment has its own copy of the grading form. It is possible to easily re-use these forms other activities by cloning the form. But as long as a new copy is always created, it was pretty easy to write backup and restore support (well, as easy as writing any backup and restore code in Moodle 2.x is ) for the forms and control modifications of them.

Splitting the development workload

I must admit I would not be able to finish the project on my own. Working on it in the team was a must – and actually a new experience for me personally. All my previous big DEV projects (the new Workshop module, AMOS or restoring 1.9 backups into 2.x) were sort of one-man show. This was the first time I had to cooperate closely with another developer. Those who were involved know that this was pretty challenging for me and the coding was salted with many emotional grrr’s and wtf’s. Today, I must say I am happy I had Marina working on this project with me. It would not happen without her.

The nature of the design allowed us to split the work so I could work on the backend framework that provides general support for all grading methods (managing the gradable areas, cloning the forms, sharing them as templates etc). Marina did a great job developing the Rubric plugin itself and most notably integrated it into the Assignment module. It was not an easy task given the current state of the Assignment module’s internals. Marina has my respect and I sent her Kudo for this.

Continuous peer-review and feedback

We were working on opposite sides of the planet on this project and the time zones difference worked against us. In such situation, I found Github very helpful. It allowed us to work so that Marina sent me pull requests regularly and I merged them into the pre-integration feature branch. We usually started our day by reviewing the changes introduced by the other one.  Github allows to comment the code easily in-line and send feedback to each other. While implementing the rubric plugin, Marina suggested significant improvements of the API this way.

Also, Martin Dougiamas was involved, feeding us regularly with ideas on how the user interface and the general user experience should look and feel like. So, if you like it, praise Martin. If you don’t, blame me and Marina

Current stable version 2.0.1+

Total of 64 patches were accepted by the integration and testing teams during this week for Moodle 2.0 stable branch (which still lives on Git ‘master’). This is a new record in terms of number of pull requests. Also, with only 5 pull requests rejected, this is the best accepted/rejected ratio achieved so far. It also means I am no longer able to provide an overview of all accepted changes so I will focus only on those I helped to test or review.

Moodle can now be installed on PostgreSQL server with ‘standard_conforming_strings’ option enabled. This option is on by default since PostgreSQL 9.1 and Bruce Momjian’s blog post explains why (MDL-26351). ♦ It has been known for a while that MSSQL and Oracle databases do not like SELECT DISTINCT if the list of returned columns contains some TEXT ones. This is tricky especially in a case like SELECT DISTINCT t.*, if the table t contains some TEXT fields. As most Moodle developers use PostgreSQL or MySQL as a part of their environment, some affected queries still survive in Moodle code. Eloy Lafuente found and fixed couple of them, using a nice inner join with subselect solution (MDL-26371). ♦ Mark Nielsen spotted a bug in the implementation of print_collapsible_region() function. Glenn Ansley took this as an opportunity for his first contribution to Moodle, created a branch at his moodle.git clone at github.com and prepared a patch there (MDL-26131). ♦ Andrew Davis fixed a really nasty bug in blogging subsystem that was causing accident removal of all recent records in the table ‘post’. This table holds not only blog posts but also user notes too, for example (MDL-26010). A regression of a recent change in the upgrade code of SCORM module was fixed by Eloy Lafuente (note for myself: next time I test SCORM upgrade code, I shall have some SCORM module instance actually created at the site… MDL-26361) ♦ And finally, images embedded into forum posts that are sent by email to the forum subscribers are now correctly displayed in the email clients. Given that the client has access to the forum either via the current user’s session (in case of web email clients) or the course grants access for anonymous hosts (MDL-25944).

Previous stable version 1.9.10+

All 9 submitted patches were accepted and they landed on MOODLE_19_STABLE branch. ♦ Petr Škoda fixed a problem with redirecting to a login page on https protocol and a bug mediaplugin filter. ♦ Eloy Lafuente found some queries using SELECT DISTINCT from TEXT fields in 1.9 version, too. ♦ Dan Marsden fixed a bug in a pop-up form in SCORM module. ♦ Aparup Banerjee fixed the number of glossary entries being displayed in the recent activity block. ♦ Tim Hunt fixed a bug in quiz that used to display overall feedback based on the rounded grade instead of the real value.

Quotes of the week

“We will ALWAYS have blockers and critical issues, they never end. If we tried to solve them all, we’d never release anything.”
– Martin Dougiamas has a pragmatic view on release policy

“Any code you write yourself should follow all the coding guidelines perfectly. However, when looking at old code, you should be tolerant of what you find.”
– Tim Hunt in a discussion on a policy of naming database tables in Moodle

“Never trust user input.”
– Jonny Barnes at an excellent page summarizing what should a developer know before building a public web site

Is it there yet?

If you are watching some issues in the tracker, you probably know those emails informing you about the change of the issue status. If you receive an email that your favourite issue has been resolved or closed, it is good to understand what exactly it means.

As you probably know, the current Moodle development workflow is based on so called fork and pull model. All developers submit their patches into separate branches in their own public forks (clones) of the official Moodle git repository. Then they ask the integration team for pulling the changes from these forks into the official repository. Although most Moodle contributors seem to use github.com as a place to publish their repositories, we do not use pull requests feature offered by Github. Instead, a PULL issue in Moodle tracker must be created for each submitted branch (patch). In the created PULL issue, the contributor describes the submitted patch and provides information for the integration team necessary for including the patch.

The relevant MDL issue being fixed by the submitted patch is marked as resolved immediately after the PULL request is created. The contributor is supposed to link the PULL issue with the MDL issue and resolve the MDL issue. So resolved means there is a patch available for this issue, but the patch is not part of Moodle package yet.

Every week on Monday, the integration team goes through the list of submitted PULLs and reviews the patches. If the patch is accepted, it is merged into integration.git repository and the status of the PULL issue is changed to “ready for testing”. When all submitted patches are reviewed (that is they are either accepted or rejected), the testers come and test all changes. If the test passes, the tester closes the MDL issue (alternatively, the release manager closes all MDL issues with their tests passed at the end of testing). So closed MDL means the patch has been accepted by both integration team and testing team. But the patch is still not part of the official Moodle package.

You must wait until all integrated patches are tested and the patches are pushed from integration.git repository into the official moodle.git repository. At the same time, ZIP and TGZ packages are regenerated and can be downloaded. This typically happens on Wednesday afternoon (European time).

As an example, let us say I work on some bug. On Thursday morning, I commit a fix into my repository at github.com. Then I go to the tracker, create a new PULL issue and resolve the linked MDL issue. On Monday, a member of the integration team reviews my patch and if I am lucky enough, they accept it. On Tuesday, a member of testing team follows the steps I provided in the PULL requests and hopefully confirms that the patch really fixes the reported issue. The linked MDL issue is closed then. Therefore the next weekly build generated on Wednesday will contain my fix.

Current stable version 2.0+

There are 23 requests for accepting a patch into the 2.0 branch of the new Moodle git repository from the last week. During the today’s review, 15 of them have been accepted by the integration team and graduated to quality assurance testing; 8 of them have been rejected.

Previous stable version 1.9.10+

There are 9 requests for accepting a patch into the 1.9 branch from the last week; 6 of them have been accepted and graduated to quality assurance; 3 of them have been rejected during the integration review.

Quotes of the week

“This is my last commit to CVS. Goodbye old friend – you’ve served us well.”
– Martin Dougiamas says goodbye to CVS that held the history of changes for 9 years of Moodle development.

“The point is to fix more than you break.”
– Petr Å koda’s simple recipe for an acceptable commit (originally said in a context of upcoming major version release but still applies)

Wind of change

Last week was the first one running under new development procedures. I helped to set up git repositories moodle.git and integration.git that are key part of the new “pull” model of code contribution. Although I am not member of the integration team, I was trying to assist guys in their work, observing the process and discussing the patch workflow.
It is too early for any strict and definitive rules. However, the whole process is getting the final shape. Let us illustrate the key points of it on an example of MDL-25647. The patch fixing the described issue is pretty trivial. In the previous model, I would modify my local checkout and commit the change into CVS central repository immediately. So we can call that “push” model.
In the current “pull” model, I do not have write access to the official Moodle repository any more. Actually almost nobody has – even Moodle HQ developers lost this permission. Instead, I have a clone of moodle.git repository at my local computer. I created a local branch forked off the remote master branch (master branch is where the current Moodle 2.0 development happens) and named it MDL-25647_mnet_peer_setting_not_saved:

$ git checkout -b MDL-25647_mnet_peer_setting_not_saved origin/master

I committed the fix into the branch. By running

$ git cherry -v origin/master MDL-25647_mnet_peer_setting_not_saved

I can make sure that the branch contains just the commit I want to be included in Moodle core (the line starts with the plus sign). I use `git diff`, `git log -p` or `git show` to see the change introduced by the commit.
Once I am happy with the change, I have to publish the branch in some public place so that integrators can pull the change from there and incorporate it into Moodle. I use my repository at github.com for these purposes but I could use any other git hosting site (like legendary repo.or.cz or gitorious.com) or my own git server if I had one. It really does not matter once the repository grants public read access (I could actually export the commit into a file and email it to Moodle or send via tracker – but the public repo method is preferred by Moodle integrators). As I already have github.com repository aliased to “github” remote at my notebook, I simply run

$ git push github MDL-25647_mnet_peer_setting_not_saved

Now I need to ask Moodle integration team to pull from that branch. This time, I had to create an issue in Moodle tracker PULL project which led to PULL-18. But there is an ongoing discussion that we could use single project workflow so that the information about the pull request would be attached to MDL issue directly.
And that’s all! I just needed to wait until Monday when the integration window opens. But that is not a work stopper as I would create other topic branches for all other things and work on them. So the delay is not a blocker and it gives me some extra time to change my mind, for example.
On Monday, integrators go thru the list of submitted branches and review them. They use the information I provided in PULL request and run the following commands to fetch from my public repository into their local clone of integration.git and review the change I propose:

$ git fetch git://github.com/mudrd8mz/moodle.git MDL-25647_mnet_peer_setting_not_saved
$ git log -p ..FETCH_HEAD

If they accept the change, they would just merge it

$ git merge FETCH_HEAD

If they ralize they overlooked something and change their mind immediately after merge, they can undo it by

$ git reset --hard ORIG_HEAD

The same procedure is used for all other pull requests so at the end of the day, the master branch at integration.git contains all accepted patches (of course, git preserves authorship information, commit messages, commit timestamps etc). The repository is then deployed at quality assurance test site and QA team make sure that the issue is fixed from the user’s point of view. After QA tests pass, the integration.git is pushed to moodle.git and a snapshot of it is exported into CVS mirror (that should happen on Wednesdays). So the result is backward compatible with legacy weekly builds.
When moodle.git is updated, I fetch new changes from it and use the following command again:

$ git cherry -v origin/master MDL-25647_mnet_peer_setting_not_saved

This time, the commit line starts with the minus sign indicating that the commit has been included upstream (note that this check would work even if the patch got into upstream via cherry-picking or via email and therefore its commit hash would be different). So it is time to delete the topic branch:

$ git push github :MDL-25647_mnet_peer_setting_not_saved
$ git branch -d MDL-25647_mnet_peer_setting_not_saved

More coming soon as a part of a git course for Moodle contributors I am currently working on. Stay tuned!

Post scriptum

I found gitster’s journal excellent place for learning about git

Latest stable version 1.9.9+

There are 15 commits into the latest stable branch from the last development week (from Tue Sep 14 to Mon 20 Sep) and the current weekly build seems to contain several important fixes. ♦ Andrew Davis fixed two critical bug in the Gradebook. The first one caused that the calculated total course grades in the user report were incorrect when all users were selected. The first student’s grade was displayed instead of the valid values (MDL-22841). The second bug, spotted by Chris Bandy, was that Moodle actually never recorded the id of the user making a change when archiving it in the grades history (MDL-23928). ♦ Iñaki Arenaza fixed a bug in CAS authentication plugin. Now there is no need to have an LDAP server to fetch user data from and you can fetch your CAS users into Moodle from other sources (like a text file), too. Credit goes to Matthew Turney for the patch (MDL-16168). ♦ Martin Dougiamas fixed a problem reported by Stefan Schramm. Forum module put invalid In-Reply-To and References headers in mails for the initial thread posts. That led to broken sorting of emails in mail clients with mail threads support (MDL-22551). ♦ Sam Hemelryk fixed the Assignment module so that it still allows students to view their submission even after they do not have the capability to submit it any more (MDL-23848). ♦ Andrea Bicciolo fixed a small regression of his recent patch in the Database module (MDL-24033). ♦ Pierre Pichet committed Oleg Sychev’s patch that replaces one !empty() call with the correct isset() in the form for editing a question with extra_question_fields so the zero value is accepted now (MDL-24241). ♦ Eloy Lafuente made new Dzongkha language pack available for Moodle installation. ♦ I have applied a patch provided by Hubert Chathi that fixes addGroupRule() in form elements containing square brackets and fixed client side validation for the Location field in the Resource so that the field is required now.

Moodle 2.0 RC1

There are 403 commits into the main development branch from the last week. Most of them are results of general code clean up made by Petr Å koda. Petr used code analysis feature of his currently favourite IDE PhpStorm and discovered many malicious typos, forgotten global variable declarations and other mistakes in Moodle 2.0 code.
Even without an official announcement, many community members noticed that 2.0 development cycle reached the Release Candidate 1 milestone. The development team considers the branch feature-complete. See more details in the Release notes.
Hot gossip, Martin Dougiamas confirmed that Moodle development will definitely switch from CVS to git in a very near future. Even though some doubt about it, I can’t imagine any development work without this tool any more. Yes, I admit that mastering git takes some time but I can just recommend it to all developers and contributors – and especially to those making their own customizations of the code.

Quotes of the week

“In any case, the important point to learn is to never use $PAGE for deciding contexts as far as it causes all the ‘remote’ operations to fail.”
– Eloy Lafuente reminds us that global $PAGE is supposed to be used only for output rendering purposes and that abusing it at lower levels leads to problems.

“Software development is meant to be fun. If it isn’t, the process is wrong.”
– Pete McBreen in his Software Craftsmanship: The New Imperative

Latest stable version 1.9.9+

There are 4 commits into the stable branch from the last development week (from Tue Jul 20 to Mon Jul 26) ♦ Dan Marsden removed a possibility to open SCORM player in a new browser window manually because it led to problems with communication between the SCO and the server and no grades were reported back (MDL-21333) ♦ Sam Hemelryk fixed HTML rendering issue in quiz module (MDL-20724) ♦ Petr Skoda improved isguestuser() core function which detects if the given user has logged in via “Login as guest” button. Such user has hard-coded username ‘guest’. But in MNet environments, just user name is not enough to uniquely identify user as there may be accounts with the same user name on different MNet peers. Therefore we must check the user’s mnethostid attribute and make sure it is really our local account to prevent reported problems with multiple guest accounts (MDL-22871) ♦ David Mudrak spotted two typos in the assignment module’s recent activity report. These typos accidentally eliminated each other so that the reporting of the recent activity for students in case of separate groups mode could not work but no error or false positive activity were displayed (e5bc356).

Future version Moodle 2.0 Preview 4

There are 284 commits into the main development branch from the last week. The list of Moodle 2.0 release blockers contains 19 unresolved issues. Thanks to our community QA testers, dozens of other issues have been discovered and fixed.

Quotes of the week

“That’s the second time in as many days I have brought up something and Petr has gone and removed it- I’d better watch out what I highlight next! ”
– Mary Cooch discovers dark side of QA testing

May git be with you

While my recent work on MNet remote enrolment  services for Moodle 2.0, I could not praise git more. Whenever dealing with some MNet development, one usually has to work on two separate instances of Moodle – server side and client side (aka service provider and service subscriber). This is roughly how I used git during the development. Let us say I have moodle.git clone in my ~/public_html/moodle20 directory. It is the checkout I use for everyday development tasks. I started with creating a new branch at the top of the latest upstream Moodle version. At the end of the day, this branch will contain all the commits to be pushed into Moodle CVS repository.

$ cd ~/public_html/moodle20/
$ git fetch
$ git branch mnet-integration origin/cvshead
$ git checkout mnet-integration

Then I prepared checkouts for the two clear installations of Moodle. Due to the lack of invention, let us call them A and B.

$ cd ~/public_html/
$ git clone moodle20 moodle20a
$ git clone moodle20 moodle20b

Thanks to the new CLI installer in Moodle 2.0, it was not actually needed to leave the pleasant dark shell window.

$ cd ~/public_html/moodle20a/
$ php admin/cli/install.php

And the work could start. After I set up both installations, I connected them into MNet network. Moodle A served as a course with user accounts and Moodle B was the one with courses. Simply said, remote enrolment services allows the administrator at Moodle A to enrol their users into remote courses at Moodle B. To make this work within the new enrolment framework in Moodle 2.0, changes had to be done on both of the sides – service provider (B in our scenario as it is the one with the courses) and service subscriber (A as the one with the users).

After a while, I had several commits in both checkouts. When I had a commit in A that I needed in B too, I just run

$ cd ~/public_html/moodle20b/
$ git fetch ../moodle20a
$ git cherry-pick 63891a8

When I finished, it was time to prepare commits to be sent into CVS. I went back to the repository with the integration branch and run

$ cd ~/public_html/moodle20/
$ git fetch ../moodle20a
$ gitk HEAD..FETCH_HEAD
$ git merge FETCH_HEAD
$ git fetch ../moodle20b
$ git merge FETCH_HEAD

git is clever enough to know that some commits from moodle20a were already included in moodle20b due to the cherry picking. Now I could go back to A and B and repeat the merging as many time as needed. So I ended with commits log like this. Sorry for different names of branches at the picture against the ones in the text. You can see how branches can be merged here, there and back again.

When I finished and was ready to publish my work, I rebased the integration branch against the up-to-date upstream head. Once the integration branch is rebased, the merging from the cloned branches can not work any more and would lead to conflicts, problems and duplicated commits. I rebased the branch accidentally so I know that git manual is really right about this. Rebasing is actually re-applying a sequence of commits. During the rebasing, commits order can be modified, commits can be meld into single commits etc. This allowed me to prepare a set of patches to be send into CVS using git cvsexportcommit command.

Post scriptum

When in Rome, do as the Romans. When in Moodle, use forums.

Latest stable version 1.9.8+

There are 7 commits into the stable branch from the last week. Tim Hunt committed David Binney’s patch fixing a question bank SQL query at Oracle (MDL-21828). Iñaki Arenaza fixed an old issue when he committed his patches that significantly increase the quality of images produced by TeX filter. If the binaries of latex, dvips and convert are available at the system, Moodle can generate nice PNG formulas. Otherwise, “ugly” GIFs are created by mimetex (MDL-10197). Another Iñaki’s committed patch allows to use HTML editor when editing login page instructions (MDL-19053) and yet another one fixes small typo in CAS, DB and LDAP authentication plugins (MDL-18689).

Unstable development version 2.0dev

There are 160 commits at the main development branch from the last week, many of them done by Petr Å koda during the weekend strings clean-up. Patrick Malley added a new test theme called Boxxie based on his Greenie theme for Moodle 1.9. On Wed Apr 7 00:31:20 2010 UTC, the last ever run of Eloy’s installer_builder generated and committed strings for Moodle installer and was turned off. In the near feature, it will be replaced with a new script taking the data from our new strings repository called AMOS. So long installer_builder, and thanks for all the strings!

Quotes of the week

“I like git because it might actually force us to fix our sloppy/missing review processes before things hit CVS”
– Petr Å koda guards the code against eager commits – get alarmed!

“NOBUG: Bombing BOM: BOM bombed”
– Eloy Lafuente and his bombastic commit message

Milliseconds count

At the moment, I have 3,740,067  records (yes, 3.74 millions of records) in the new database repository of Moodle strings. The repository (basically one huge SQL table) contains one record per every modification of every single string that happened since Moodle 1.6 in any language. It takes approximately two hours to re-populate the table from our git mirrors of Moodle core and Moodle langs CVS repositories. There are some critical queries that are used quite frequently – for example to take a snapshot of the most recent string values for the given component, language and branch. That is, for example, “what is currently saved in lang/en/moodle.php at the main development branch?” (or what was there at the given timestamp). Writing this query was a big lesson for me. Not only it was actually for the first time I ever used an SQL subselect. Having so many records, the query optimising became very important as I refused to wait dozens of seconds to actually see some results (and I can imagine translators  would not be happy with such delays, too).

Basically, all Moodle strings and their history is now stored in one database table containing fields branch (integer code for Moodle branch, like 1900 representing MOODLE_19_STABLE), lang (language code like ‘es’), component (the string domain – or the file where the string is defined, like ‘forum’), stringid (the key is $string array), text (the string itself) and finally timestamp (since when the change described by this record  is valid). There are other fields defined in the table – to hold the author of the change, commit message, deletion flag and others. There is composited index created for fields  (component, language, branch).

This is the query I was using during the optimisation and performance comparison. It returns the most recent version from several branches, several languages and several components – which is exactly what I need for the new translation interface.

SELECT r.id, r.stringid, r.text, r.timemodified, r.deleted
  FROM  mdl_amos_repository r
  JOIN (SELECT branch,lang,component,stringid,
               MAX(timemodified) AS timemodified
          FROM mdl_amos_repository
         WHERE branch IN (2000,1900)
               AND lang IN ('en','cs')
               AND component IN ('moodle','workshop')
             GROUP BY  branch,lang,component,stringid) j
    ON (r.branch = j.branch
        AND r.lang =  j.lang
        AND r.component = j.component
        AND r.stringid = j.stringid
        AND  r.timemodified = j.timemodified)
  WHERE r.branch IN (2000,1900)
        AND r.lang IN ('en','cs')
        AND r.component IN ('moodle','workshop')
  ORDER BY  r.component, r.stringid, r.lang, r.branch;

The basic idea of the query is to get know (in subselect) which one is the most recent version of every string. Then (via the table self join) get some additional information about such string. I was experimenting with various variants of the query -like putting those WHERE conditions into the inner subselect or the outer main select. None of them was good enough until Petr Å koda gave me a world-saving hint: put the same WHERE conditions into both inner subselect and outer main select! The inner conditions help to filter the rows that are going to be aggregated by GROUP BY. The outer conditions help to join results quickly.

Using this, I managed to get to times like 1500ms at my laptop and 200ms at the testing server, both using PostgreSQL. First attempts to run the same query on MySQL led to significantly worse figures. I am going to do more tests to be sure that both engines have similar resources/conditions available. That massive usage of IN() is supposed to be a challenge for MySQL…

Post scriptum

I fell in love running Moodle 2.0 upgrading script via command line interface – see admin/cli/*.php scripts

(1) make coffee – unfortunately this does not work as Linux command yet on my machine. I should find some time to implement http://tldp.org/HOWTO/Coffee.html

(2) Start-up Vimperator and triage my email inbox, using a variant of GTD method.

(3) Connect to Moodle developers and Moodle commits Jabber rooms

(4) $ cd public_html/moodle-head

(5) $ git fetch

Download the recent commits from our git mirror of the main CVS repository.

(6) $ gitk master..origin

This gives me an overview on what happened in the code since the last time I merged.

(7) $ git checkout master && git pull

Fast-forward my local master branch so it contains the recent commits.

(8) $ git checkout fix/somebug-MDL-xxxxx

I love branching. I do every development (even a simple bug fix) on a separate branch.

(9) $git cvsexportcommit -c -p -u -v <hash>

Once the fix is committed on my branch, I export it to CVS. Thanks to my set-up everything happens really quickly. If the fix is done on a stable branch I have to move the merged-status tag manually yet in the CVS checkout, which is a bit annoying. Once we move to git (note not “If we moved to git” :-p) thing will become even easier.

It was one of that discussions when people prefer leaving the chat room before they say something they would regret later. The Pandora’s box was opened by an innocent question on the recommended procedure to customize a contributed Moodle plugin. The discussion turned into a hot topic on what the current source code management system (CVS) gives to us and what we and our contributors could gain (and lose) from switching the Moodle CONTRIB repository to git.

Imagine you have clients who pay you for implementing a solution based on a contributed Moodle plugin. The quality of the Moodle contributed code varies and some sort of the code review and clean-up is strongly recommended before using it at production server. The contributed code may contain critical security holes and such plugin may easily become an open gate to your server with a big bright sign “Welcome Attackers!” So you have reviewed the code, fixed several obvious bugs and improved the features a bit as requested by your client. And – now what?

Of course you want to deploy your changes to the client’s server – as soon as it is tested  (or at least it does not produce any PHP warnings ) And you, as a follower of the open source spirit, want to contribute your changes back to upstream. So you send your patches to the original component’s maintainer to be included in CVS. Sounds good. In a perfect world it would work with no problems. But the humans are not dead yet and things tend to get complicated. Also, some objections against the real benefits of this procedure were raised. Let me list some issues that may influence this contributing back to upstream.

1. The maintainer does not respond. Such things happen. You can ask the CVS administrator to give you write access and become a maintainer. Do not do this if you know/feel you would not have time to do it responsibly. Remember the lesson #5 from the Eric Raymond’s famous essay: “When you lose interest in a program, your last duty to it is to hand it off to a competent successor.” It’s better if it’s known there is no maintainer than if people think they can rely on you. On the other hand, do it if a silent voice inside your mind whispers that it’s the job you always wanted to do. It’s both nice and good thing to spend your life on.
2. The maintainer does not agree with your improvements. Then it’s time to discuss things. Summarize the issue and open the topic in Moodle forums. Listen to the maintainer’s arguments. Maybe he/she has some further plans with the module? Who else can profit from your changes? Try to not to fork the product unless there is a chance to reach a consensus.
3. The rubbish can get into the code after you reviewed it. Yes, it can. You must remember the state of the code when it was reviewed. Source code management software calls such signs as “tags”. You shall tag a certain point of the source code history and review exactly that point. Then you know what version is sort of safe to be deployed. Also, you can focus on the changes that came after this point later when you review the code again.
4. The maintainer does not understand your fixes. Some say that simply sending the patches to the maintainers is useless as they do not spend any effort on learning from it. I personally do not agree with this. However, I can see the importance of educating the contributors. If the contributed code contains a security bug which is exploited, who is blamed? The administrators? No,  they have always been against this community thingie. The original author? No, nobody knows him/her. Moodle itself? Sure – it’s all Moodle’s fault! And that’s not fair.

All these have nothing to do with the SCM software itself. It is about responsibility, communication, constructive peer-reviews and developer etiquette. However, I think that the features and the characteristics of the SCM may significantly help with sorting out the issues discussed above or with not running into them at all.

Switching Moodle CONTRIB area into a cluster of separate git repositories emerged as an idea during the discussion. The nature of git – a distributed versioning system – seems to perfectly fit into the nature of our contrib, the hive of really interesting features and ideas. Contributors may come with their own trees of source code and the comparison of such two trees is easy. If the maintainer stops responding to the community questions and proposals, his/her tree becomes quickly obsolete and the community may start to prefer some other tree (if such is available). The pull-based model of the the maintainer-contributor interaction requires the maintainer pays attention to the patches to be included and hopefully learns from them. On the other hand, the contributors have to keep their trees up-to-date with the recent versions of the upstream code if they wish it is included in the maintainer’s tree. The point is, with git, everybody can easily become a maintainer of his/her own clone of the product. They have to gain the respect of the community members by keeping the high quality of their version. And remember – respect is everything.

I do not think we should switch to git-only at the moment. Let us give git a chance and see what  Moodle contributors think. To help them with their first steps, I summarized my personal experiences with using git in Moodle development into a sort of tutorial at http://docs.moodle.org. You are welcome to review it and/or put comments with your own solutions.

Recently I spent many hours reading git user’s manual and experiencing git-merge and git-rebase. Thanks to Nigel’s and Penny’s comments, I hope I can see differences between these. Hopefuly I’ll be able to maintain my adminlang repo effectively.

Still, I would like to find what procedure to take when porting a fix from one branch to another. Is cherry-pick intented for this?

Also, I found an interesting note about using git-format-patch | git-am to massive port from a branch to another.

Git is cool Can’t wait Moodle switch ti it as well…