David Mudrak's blog

Current stable version 2.0.1+

Total of 64 patches were accepted by the integration and testing teams during this week for Moodle 2.0 stable branch (which still lives on Git ‘master’). This is a new record in terms of number of pull requests. Also, with only 5 pull requests rejected, this is the best accepted/rejected ratio achieved so far. It also means I am no longer able to provide an overview of all accepted changes so I will focus only on those I helped to test or review.

Moodle can now be installed on PostgreSQL server with ‘standard_conforming_strings’ option enabled. This option is on by default since PostgreSQL 9.1 and Bruce Momjian’s blog post explains why (MDL-26351). ♦ It has been known for a while that MSSQL and Oracle databases do not like SELECT DISTINCT if the list of returned columns contains some TEXT ones. This is tricky especially in a case like SELECT DISTINCT t.*, if the table t contains some TEXT fields. As most Moodle developers use PostgreSQL or MySQL as a part of their environment, some affected queries still survive in Moodle code. Eloy Lafuente found and fixed couple of them, using a nice inner join with subselect solution (MDL-26371). ♦ Mark Nielsen spotted a bug in the implementation of print_collapsible_region() function. Glenn Ansley took this as an opportunity for his first contribution to Moodle, created a branch at his moodle.git clone at github.com and prepared a patch there (MDL-26131). ♦ Andrew Davis fixed a really nasty bug in blogging subsystem that was causing accident removal of all recent records in the table ‘post’. This table holds not only blog posts but also user notes too, for example (MDL-26010). A regression of a recent change in the upgrade code of SCORM module was fixed by Eloy Lafuente (note for myself: next time I test SCORM upgrade code, I shall have some SCORM module instance actually created at the site… MDL-26361) ♦ And finally, images embedded into forum posts that are sent by email to the forum subscribers are now correctly displayed in the email clients. Given that the client has access to the forum either via the current user’s session (in case of web email clients) or the course grants access for anonymous hosts (MDL-25944).

Previous stable version 1.9.10+

All 9 submitted patches were accepted and they landed on MOODLE_19_STABLE branch. ♦ Petr Škoda fixed a problem with redirecting to a login page on https protocol and a bug mediaplugin filter. ♦ Eloy Lafuente found some queries using SELECT DISTINCT from TEXT fields in 1.9 version, too. ♦ Dan Marsden fixed a bug in a pop-up form in SCORM module. ♦ Aparup Banerjee fixed the number of glossary entries being displayed in the recent activity block. ♦ Tim Hunt fixed a bug in quiz that used to display overall feedback based on the rounded grade instead of the real value.

Quotes of the week

“We will ALWAYS have blockers and critical issues, they never end. If we tried to solve them all, we’d never release anything.”
– Martin Dougiamas has a pragmatic view on release policy

“Any code you write yourself should follow all the coding guidelines perfectly. However, when looking at old code, you should be tolerant of what you find.”
– Tim Hunt in a discussion on a policy of naming database tables in Moodle

“Never trust user input.”
– Jonny Barnes at an excellent page summarizing what should a developer know before building a public web site

Is it there yet?

If you are watching some issues in the tracker, you probably know those emails informing you about the change of the issue status. If you receive an email that your favourite issue has been resolved or closed, it is good to understand what exactly it means.

As you probably know, the current Moodle development workflow is based on so called fork and pull model. All developers submit their patches into separate branches in their own public forks (clones) of the official Moodle git repository. Then they ask the integration team for pulling the changes from these forks into the official repository. Although most Moodle contributors seem to use github.com as a place to publish their repositories, we do not use pull requests feature offered by Github. Instead, a PULL issue in Moodle tracker must be created for each submitted branch (patch). In the created PULL issue, the contributor describes the submitted patch and provides information for the integration team necessary for including the patch.

The relevant MDL issue being fixed by the submitted patch is marked as resolved immediately after the PULL request is created. The contributor is supposed to link the PULL issue with the MDL issue and resolve the MDL issue. So resolved means there is a patch available for this issue, but the patch is not part of Moodle package yet.

Every week on Monday, the integration team goes through the list of submitted PULLs and reviews the patches. If the patch is accepted, it is merged into integration.git repository and the status of the PULL issue is changed to “ready for testing”. When all submitted patches are reviewed (that is they are either accepted or rejected), the testers come and test all changes. If the test passes, the tester closes the MDL issue (alternatively, the release manager closes all MDL issues with their tests passed at the end of testing). So closed MDL means the patch has been accepted by both integration team and testing team. But the patch is still not part of the official Moodle package.

You must wait until all integrated patches are tested and the patches are pushed from integration.git repository into the official moodle.git repository. At the same time, ZIP and TGZ packages are regenerated and can be downloaded. This typically happens on Wednesday afternoon (European time).

As an example, let us say I work on some bug. On Thursday morning, I commit a fix into my repository at github.com. Then I go to the tracker, create a new PULL issue and resolve the linked MDL issue. On Monday, a member of the integration team reviews my patch and if I am lucky enough, they accept it. On Tuesday, a member of testing team follows the steps I provided in the PULL requests and hopefully confirms that the patch really fixes the reported issue. The linked MDL issue is closed then. Therefore the next weekly build generated on Wednesday will contain my fix.

Current stable version 2.0.1+

Total of 27 submitted patches for Moodle 2.0 were accepted this week. Eight pull requests were rejected. To highlight some of the accepted patches, Andrew Davis fixed a blocker issue reported by Aaron Cowell causing errors in RSS feeds generated by Moodle (MDL-24870). ♦ Dongsheng Cai fixed a bug in Database activity module’s templates editor (MDL-25671). ♦ Eloy Lafuente prepared an emergency fix of a regression blocking the upgrade at Oracle servers.

Previous stable version 1.9.10+

There is a single accepted patch for Moodle 1.9 from the last week. Eloy Lafuente fixed a regression in Database activity module reported by Paul Nijbakker. Some module functions redirected user to the first record in the database instead of to the correct one (MDL-26052). The second submitted patch for Moodle 1.9 improving inefficient computation of tag correlations had to be rejected because the included SQL statement was not cross-platform and worked under MySQL only (MDL-24355).

Quotes of the week

“I think we need to become serious and start rejecting any commit with [trailing] whitespace.”
– Eloy Lafuente strongly recommends all developers and contributors to set up their IDE properly

“Open source sometimes actually works, even without doing the programming yourself.”
– Sam Marshall knows that reporting a bug is sometimes enough to make the thing fixed

Testing Moodle on various databases

Moodle 2.0 officially supports four major database systems: PostgreSQL, MySQL, MSSQL and Oracle. I have PostgreSQL and MySQL installed on my Linux workstation and I use PostgreSQL 8.3 at the moment as the major platform for the development – simply because I believe that if a code works at PostgreSQL, it is generally more cross-platform than a code that was tested on MySQL. Though there are exceptions, of course.
To help with testing patches related to MSSQL, I decided to set up MSSQL server for me. I have successfully installed and set up MSSQL Server 2008 R2 Express edition. The server runs in a virtual machine on my notebook. I use VirtualBox from Oracle (Yes, Oracle made VirtualBox and we’ve always been at war with Eastasia) networked via a bridge between the Linux host and Windows guest. That allows me to have Moodle installed in my Gentoo being connected to the MSSQL database in Windows via freetds driver. To test the new sqlsrv driver, I downloaded Moodle Windows package to the guest and was able to install it there, too.

There were 53 patches submitted for integration this week; 10 of them were rejected, 43 survived both review and testing and got into the official repository. Regular weekly builds were released on Wednesday.
Today on Friday morning, Tomaz Lasic spotted a critical regression at School Demo Moodle site. Due to recent changes in modinfo library, Workshop module’s navigation stopped working completely, throwing coding exception because the new modinfo API was not backward compatible by mistake. This issue was promptly fixed by Sam Hemelryk and emergency weekly build of Moodle 2.0 was re-released.

Current stable version 2.0.1+

Total of 40 accepted pull requests affect Moodle 2.0 branch. ♦ Andrew Davis added new index to the table user_info_data consisting of userid and fieldid fields to increase performance when searching over custom user fields (MDL-17201). ♦ Dan Marsden fixed several SCORM module bugs and cleaned up its code. ♦ Dongsheng Cai fixed a bug in file picker. Internet Explorer was trying to download repository_ajax.php instead of executing it via AJAX request when users had .php files associated with an external editor. Dongsheng also fixed a bug in Wiki module’s upgrade code, dealing with wikis with the same title (some issues with Wiki upgrade seem to be still open – see comments in PULL-196 for details). ♦ Eloy Lafuente submitted a set of patches, mainly dealing with database access – wrong usages of get_text() and get_recordset_xxx() and using words reserved in Oracle as bind placeholders. ♦ John Stabinger fixed various issues in several Moodle 2.0 themes. ♦ Petr Å koda submitted a bulk of patches fixing many areas of the code. ♦ Rossiani Wijaya improved Multimedia plugins filter so that it now supports YouTube playlists (MDL-25573). ♦ Sam Hemelryk improved performance of the new navigation system (MDL-25291). The reporter Petr Å koda noticed that the new administration block is loaded for navigation purposes at almost every page even though most users (students) can not see anything in it. Sam implemented a new session flag to indicate whether there is actually something to load for the administration block. Not trying to load an empty block makes quite a noticeable performance difference in number of included files, RAM consumption and get_string() calls. ♦ Sam Marshall submitted another patch leading to the proposed improvements of get_fast_modinfo() function. ♦ Tim Hunt submitted three patchsets, all dealing with issues in Quiz module and Question bank.

Previous stable version 1.9.10+

Three accepted pull requests affect Moodle 1.9 branch. ♦ Andrew backported MDL-17201 discussed above. ♦ Dan Marsden backported a SCORM issue MDL-25320. ♦ Petr Å koda fixed a bug in an enrolment plugin – sites using IMS Enterprise file for enrolments should upgrade.

Quotes of the week

“Ooh duh – somebody actually documented it. I wasn’t expecting that ”
– Sam Marshall was lucky enough to find up-to-date information at a documentation page for developers

“git blame or blame git – that’s the question”
– Eloy Lafuente comments on Sam Hemelryk’s experience in MDL-25791

Performance matters, women say

Although Moodle developers focus on performance these days, not many performance related patches landed this week yet. And I can understand why. From my own experience, working on performance is pretty time consuming job requiring a lot of comparison tests on production data to get valid results. Luckily, Moodle 2.0 has inbuilt support for xhprof – a profiling tool written by Facebook developers. Large sites admins and Moodle developers can use this tool to compare how various tiny improvements save milliseconds of execution time and to analyse the complex function call graphs, identifying performance bottle necks.
From what we could see so far, the performance of the new navigation system (used at almost every page) and the recently improved filtering system (which makes major part of frequent format_text() function) should be our primary targets now. I started to study the current implementation of text filters and the mechanism of how they are chained, looking for places to improve. While reading the code, some important questions emerge that will need design decision soon.
One of them is a well known “nolink” feature. Moodle users know this as a button in HTML editor that is supposed to “prevent automatic linking” within the selected block of text. This feature was introduced back in 2003 as a part of glossary terms auto-linking feature. Later the feature was used by other parts of Moodle – activity names linking, database records linking etc. As all these automatic linking features are now implemented as standard filters, the concept became less clear. Is the text wrapped by “nolink” marks supposed to be processed by other filters that do not actually create links? Shouldn’t it be replaced with some “do not apply any filter on this block of text” feature?
Beside this conceptual question, there is an issue with the actual implementation of the feature, too. Originally, the part of the text where no automatic linking was supposed to happen had to be wrapped by <nolink> and </nolink> tags. Later as a part of transition to full XHTML support, these tags were replaced with valid <span class=”nolink”>…</span> pair. It works pretty well for short strings – typically glossary terms. But this syntax fails badly when the inner text contains another span tags as it becomes difficult (if not impossible) to identify the correct substring by regular expressions (we can not afford full DOM parsing for obvious performance reasons). Also, using spans makes it impossible to start the protected block in the middle of one paragraph and end it in the middle of the second one because HTML tags must nest correctly.
The solution I am about to suggest is to introduce a new syntax using a couple of empty span tags. The start of the protected block would be defined using something like <span class=”nofilterstart”></span>. The block would end at the place where <span class=”nofilterend”></span> is found. XHTML specification recommends not to use the minimized tag syntax for empty span elements, so even if shorter <span class=”xxx” /> would work (and Moodle will support it probably), our HTML editor would produce the syntax with closing tag. This way, both issues with proper nesting and inner spans would be solved.
You see – I got into a completely different area than the performance of format_text() I started to work on But it is quite related in fact. If the filter manager is able to detect some common parts of the text that no filters are expected to modify, it can save time spent on search & replace job that every filter does. Any feedback, idea or contra-proposal is welcome, of course.

Current stable version 2.0+

There are 23 requests for accepting a patch into the 2.0 branch of the new Moodle git repository from the last week. During the today’s review, 15 of them have been accepted by the integration team and graduated to quality assurance testing; 8 of them have been rejected.

Previous stable version 1.9.10+

There are 9 requests for accepting a patch into the 1.9 branch from the last week; 6 of them have been accepted and graduated to quality assurance; 3 of them have been rejected during the integration review.

Quotes of the week

“This is my last commit to CVS. Goodbye old friend – you’ve served us well.”
– Martin Dougiamas says goodbye to CVS that held the history of changes for 9 years of Moodle development.

“The point is to fix more than you break.”
– Petr Å koda’s simple recipe for an acceptable commit (originally said in a context of upcoming major version release but still applies)

Wind of change

Last week was the first one running under new development procedures. I helped to set up git repositories moodle.git and integration.git that are key part of the new “pull” model of code contribution. Although I am not member of the integration team, I was trying to assist guys in their work, observing the process and discussing the patch workflow.
It is too early for any strict and definitive rules. However, the whole process is getting the final shape. Let us illustrate the key points of it on an example of MDL-25647. The patch fixing the described issue is pretty trivial. In the previous model, I would modify my local checkout and commit the change into CVS central repository immediately. So we can call that “push” model.
In the current “pull” model, I do not have write access to the official Moodle repository any more. Actually almost nobody has – even Moodle HQ developers lost this permission. Instead, I have a clone of moodle.git repository at my local computer. I created a local branch forked off the remote master branch (master branch is where the current Moodle 2.0 development happens) and named it MDL-25647_mnet_peer_setting_not_saved:

$ git checkout -b MDL-25647_mnet_peer_setting_not_saved origin/master

I committed the fix into the branch. By running

$ git cherry -v origin/master MDL-25647_mnet_peer_setting_not_saved

I can make sure that the branch contains just the commit I want to be included in Moodle core (the line starts with the plus sign). I use `git diff`, `git log -p` or `git show` to see the change introduced by the commit.
Once I am happy with the change, I have to publish the branch in some public place so that integrators can pull the change from there and incorporate it into Moodle. I use my repository at github.com for these purposes but I could use any other git hosting site (like legendary repo.or.cz or gitorious.com) or my own git server if I had one. It really does not matter once the repository grants public read access (I could actually export the commit into a file and email it to Moodle or send via tracker – but the public repo method is preferred by Moodle integrators). As I already have github.com repository aliased to “github” remote at my notebook, I simply run

$ git push github MDL-25647_mnet_peer_setting_not_saved

Now I need to ask Moodle integration team to pull from that branch. This time, I had to create an issue in Moodle tracker PULL project which led to PULL-18. But there is an ongoing discussion that we could use single project workflow so that the information about the pull request would be attached to MDL issue directly.
And that’s all! I just needed to wait until Monday when the integration window opens. But that is not a work stopper as I would create other topic branches for all other things and work on them. So the delay is not a blocker and it gives me some extra time to change my mind, for example.
On Monday, integrators go thru the list of submitted branches and review them. They use the information I provided in PULL request and run the following commands to fetch from my public repository into their local clone of integration.git and review the change I propose:

$ git fetch git://github.com/mudrd8mz/moodle.git MDL-25647_mnet_peer_setting_not_saved
$ git log -p ..FETCH_HEAD

If they accept the change, they would just merge it

$ git merge FETCH_HEAD

If they ralize they overlooked something and change their mind immediately after merge, they can undo it by

$ git reset --hard ORIG_HEAD

The same procedure is used for all other pull requests so at the end of the day, the master branch at integration.git contains all accepted patches (of course, git preserves authorship information, commit messages, commit timestamps etc). The repository is then deployed at quality assurance test site and QA team make sure that the issue is fixed from the user’s point of view. After QA tests pass, the integration.git is pushed to moodle.git and a snapshot of it is exported into CVS mirror (that should happen on Wednesdays). So the result is backward compatible with legacy weekly builds.
When moodle.git is updated, I fetch new changes from it and use the following command again:

$ git cherry -v origin/master MDL-25647_mnet_peer_setting_not_saved

This time, the commit line starts with the minus sign indicating that the commit has been included upstream (note that this check would work even if the patch got into upstream via cherry-picking or via email and therefore its commit hash would be different). So it is time to delete the topic branch:

$ git push github :MDL-25647_mnet_peer_setting_not_saved
$ git branch -d MDL-25647_mnet_peer_setting_not_saved

More coming soon as a part of a git course for Moodle contributors I am currently working on. Stay tuned!

Post scriptum

I found gitster’s journal excellent place for learning about git

Latest stable version 1.9.9+

There are 3 commits into the stable branch from the last development week (Tue 28 to Mon 4). ♦ Andrew Zoltay spotted a bug in login form on sites with “Use HTTPS for logins” setting on and provided a patch, included by Petr Skoda (MDL-24225). ♦ Two other commits are just trivial code cleanups. Rossiani Wijaya removed whitespace in the code she committed week before. ♦ Andrew Davis fixed a table comment in one of XMLDB files, spotted by Eloy Lafuente.

Moodle 2.0 RC1

There are 106 commits into the future release branch from the last week. The main community site http://moodle.org has been upgraded to 2.0 engine during the weekend and helped the core developers to discover some forgotten bugs and incompatible customizations.

Quote of the week

“Raarrrrrrrr fixed way old bug from dml conversion”
– Sam Hemelryk really enjoys bug fixing rampage

Parsing uploaded string files in AMOS

I was working on a new AMOS feature that allows users (language pack maintainers and translation contributors) to upload their translation to AMOS and include it in the main repository or offer it for inclusion. The first supported format of such file is standard Moodle string file format, which is valid PHP code defining associative array $string. The problem was that for obvious security reasons, I can not just let anonymous users to upload whatever PHP code and execute it. Imagine what would happen if hacker came and uploaded a file like

<?php
global $CFG;
$string['something'] = $CFG->dbpass;

If I just executed such file via include(), the user would get access to sensitive server configuration data. So it was clear I have to write my own parser that extracts the array from the file without actually executing the file.

I initially tried to solve it by searching for some patterns in the file contents using regular expressions. That worked pretty well in simple cases but unit tests (of course I started with them) failed quickly for complex samples that included commented lines, block comments or strings that contained the interesting patterns themselves.

<?php
// $string['lalala'] = 'Knock knock';
/*
$string['grrrr'] = 'Who\'s there?';
*/
$string['nasty'] = '$string[\'nasty\'] = \'Funny heh?\';';

So I realized regexps are not suitable for this kind of task. The other approach how to deal with comment blocks would be stream processing of the file contents, with a lot of flags like “inside a comment”, “after line comment mark” or “waiting for variable name”. But that was evident reinventing of the wheel which would have a square shape at the end of the day, anyway. PHP itself has to do this boring job when analysing the source so I just had to learn how to use its results.

Wise may already know I ended with tokenizer extension. The tokenizer functions provide an interface to the PHP tokenizer embedded in the Zend Engine. Using these functions you may write your own PHP source analyzing or modification tools without having to deal with the language specification at the lexical level. The parser method I have finally implemented calls token_get_all() to get array of tokens found in the uploaded file and picks patterns that are considered as valid string definition.

For the purpose of uploading the string file into AMOS, valid definition is something like ‘T_VARIABLE $string followed by [ followed by T_CONSTANT_ENCAPSED_STRING followed by ] followed by assignment = followed by T_CONSTANT_ENCAPSED_STRING followed by semicolon’. All other tokens like T_WHITESPACE, T_INLINE_HTML, T_COMMENT or T_DOC_COMMENT are just ignored.

As you can see this means that code like

$string['greeting'] = 'Hello ' . 'world';

is considered as syntax error for AMOS import even though it is valid PHP
code. But I am sure it OK as there is no real reason to support all thinkable
ways of string definiton (like heredoc etc).

Latest stable version 1.9.9+

There are five commits into the stable branch from the last development week (from Tue Sep 21 to Mon Sep 27). ♦ Rossiani Wijaya fixed a bug in Forum module that allowed students to delete their posts even after $CFG->maxeditingtime (MDL-24258). Rossie also applied a patch suggested by Ken Gibson that deals with a bug in Internet Explorer 8. This browser does not support links to empty anchors like <a id="gohere"></a> and caused that section links from Course sections block (and Course contents extension, too) did not work in it. Credit goes to Matthew Davidson for discovering this IE8 behaviour (MDL-18232). ♦ Andrea Bicciolo committed a fix prepared by Daniele Cordella that fixes a regression in Database module, caused by recent work on MDL-24033. ♦ Iñaki Arenaza continues with fixes of CAS authentication module that crashes if there is no LDAP server set up (MDL-16168). Iñaki also applied a correction by Jose Garcia that fixes CAS synchronization with Microsoft Active Directory.

Moodle 2.0 RC1

There are 110 commits into the main branch from the last week. Development is now focused on fixing bugs and polishing the code style. More than 50% of total 176 QA tests already passed the second QA testing cycle, more testers are welcome to confirm the rest.
Eloy Lafuente’s recent work on Question bank backup landed in CVS and he is currently in a heavy combat with their restore. Backup and restore of Questions is not trivial job. By the design, Moodle backup and restore (now stored in *.mbz files) works best with whole course backup. But questions generally live in their own contexts outside the course, they can be used from multiple activities across courses, there is a complex machinery of access rights etc. These factors make the procedure of their backup and restore pretty complex.

Quotes of the week

“I sort of think at this point the best solution is to install an orbital death laser which will be automatically targeted at any server that ever runs MySQL, however I will try to come out with a plan B just in case that one doesn’t work out.”
– Sam Marshall encountered a feature of MySQL: you cannot update a table and select from it in a subquery.

CVS is fine if you aren’t using git.
– Martin Dougiamas originally said this in a different context but I like the sentence as it is :-p

Who to blame for Moodle 2.0

Moodle project is approaching 2.0 release. Let us look back on who contributed to the HEAD branch during this development cycle.
The work on Moodle 2.0 officially started on 10th October 2007 when the branch MOODLE_19_STABLE was forked and the HEAD branch was renamed to ’2.0 dev’. Today, there are 8097 files in the source code tree, containing 1.9 millions lines of code (if they all were considered as text files – some of them are binary files like pictures).
For the purpose of this report, I used moodle.git mirror and a couple of shell lines combining common UNIX data processing tools (like sed, awk, cut, sort and uniq) to go through all these files and executing ‘git blame’. This command displays the content of the file and (among other information) the name of the developer who most recently modified every single line in it. We can roughly estimate a developer’s contribution by counting all the lines reported for them by git-blame. Note that this method does not say anything about the amount of the real work – the most recent modification could by a fix of a stupid typo or even just a removal of trailing whitespace. Also, if a developer just commits some other’s work, git-blame will consider them as the author. Which is not that bad actually as the person is really the one who is responsible for that code being part of Moodle. While having in mind what this really represent, let us look at the aggregated figures.
The following table contains the list of top 20 contributors into Moodle 2.0 according the number of most recently modified lines of code. The sum of numbers of lines modified by these people represents around 97% of all lines in the current Moodle 2.0 tree.

At first, I was surprised by the amount of ‘my’ lines. Then I realized that while working on new format of Moodle strings in 2.0, I re-committed all the language files from legacy /lang/en_utf8/ folder into new lang/en/ folders. Therefore I am the one reported as the most recent editor of most of 15000 strings in Moodle and git blames me instead of the real string authors. Let us take it as an example that similar statistics should be always interpreted very carefully.

Post scriptum

Yesterday, I had my last John Smith’s Extra Smooth beer can and so the last relic of my holiday trip to Wales has gone.

Latest stable version 1.9.9+

There are 15 commits into the latest stable branch from the last development week (from Tue Sep 14 to Mon 20 Sep) and the current weekly build seems to contain several important fixes. ♦ Andrew Davis fixed two critical bug in the Gradebook. The first one caused that the calculated total course grades in the user report were incorrect when all users were selected. The first student’s grade was displayed instead of the valid values (MDL-22841). The second bug, spotted by Chris Bandy, was that Moodle actually never recorded the id of the user making a change when archiving it in the grades history (MDL-23928). ♦ Iñaki Arenaza fixed a bug in CAS authentication plugin. Now there is no need to have an LDAP server to fetch user data from and you can fetch your CAS users into Moodle from other sources (like a text file), too. Credit goes to Matthew Turney for the patch (MDL-16168). ♦ Martin Dougiamas fixed a problem reported by Stefan Schramm. Forum module put invalid In-Reply-To and References headers in mails for the initial thread posts. That led to broken sorting of emails in mail clients with mail threads support (MDL-22551). ♦ Sam Hemelryk fixed the Assignment module so that it still allows students to view their submission even after they do not have the capability to submit it any more (MDL-23848). ♦ Andrea Bicciolo fixed a small regression of his recent patch in the Database module (MDL-24033). ♦ Pierre Pichet committed Oleg Sychev’s patch that replaces one !empty() call with the correct isset() in the form for editing a question with extra_question_fields so the zero value is accepted now (MDL-24241). ♦ Eloy Lafuente made new Dzongkha language pack available for Moodle installation. ♦ I have applied a patch provided by Hubert Chathi that fixes addGroupRule() in form elements containing square brackets and fixed client side validation for the Location field in the Resource so that the field is required now.

Moodle 2.0 RC1

There are 403 commits into the main development branch from the last week. Most of them are results of general code clean up made by Petr Å koda. Petr used code analysis feature of his currently favourite IDE PhpStorm and discovered many malicious typos, forgotten global variable declarations and other mistakes in Moodle 2.0 code.
Even without an official announcement, many community members noticed that 2.0 development cycle reached the Release Candidate 1 milestone. The development team considers the branch feature-complete. See more details in the Release notes.
Hot gossip, Martin Dougiamas confirmed that Moodle development will definitely switch from CVS to git in a very near future. Even though some doubt about it, I can’t imagine any development work without this tool any more. Yes, I admit that mastering git takes some time but I can just recommend it to all developers and contributors – and especially to those making their own customizations of the code.

Quotes of the week

“In any case, the important point to learn is to never use $PAGE for deciding contexts as far as it causes all the ‘remote’ operations to fail.”
– Eloy Lafuente reminds us that global $PAGE is supposed to be used only for output rendering purposes and that abusing it at lower levels leads to problems.

“Software development is meant to be fun. If it isn’t, the process is wrong.”
– Pete McBreen in his Software Craftsmanship: The New Imperative

Latest stable version 1.9.9+

There is just one commit into the stable branch from the last development week (from Tue Aug 17 to Mon Aug 23). Sam Marshall fixed a bug in a library that handles displaying of side blocks. The bug caused that a column space was reserved on the left or right side of the screen if there was an instance of a block even if that type of block was disabled at the given site (MDL-23871).

Future version Moodle 2.0 Preview 4

There are 85 commits into the main development branch from the last week. The branch is very near to the feature freeze point and testers are preparing for the second round of Moodle 2.0 QA testing, which will start together with the first release candidate.
Petr Å koda redesigned the concept of the internal constant CLI_SCRIPT. Until now, this constant has behaved as an autodetection result of whether the script is run via web or command line interface. It used to be used in if-statements to produce HTML output for browsers and plain text output for command line, for example. This issue is now handled properly by the output rendering mechanism and the concept of CLI_SCRIPT is now different. Developer uses this constant to explicitly declare that a script is supposed to be run via CLI only – by calling define('CLI_SCRIPT', true); before including main config.php. And the autodetection in setup.php just makes sure that such a script is really called via CLI. And vice-versa, it is not possible to run a script via CLI if it does not declare CLI_SCRIPT explicitly. This step makes the logic of CLI_SCRIPT constant similar to AJAX_SCRIPT and prevents accidental execution of CLI scripts via web and web scripts via CLI. Therefore it was needed to prepare a CLI version of /admin/cron.php file. So if you run cron.php via CLI, you must execute /admin/cli/cron.php script now. See MDL-23824 for details.

Quotes of the week

| .
– Jordan Tomkinson plays pong game with Penny Leach via Jabber chat room

. |
– Penny Leach strikes the ball back to Jordan

“David is going to surely put this in his blog”
– Penny Leach was right

Mr Moodle, let me introduce you Mrs Statistics

Moodle 2.0 File API uses kind of content-addressable storage to keep course and user files on the server disk. Shortly said, every file is saved into the file pool with a filename that is calculated as SHA1 hash of the file content. If a file is copied (for example when the teacher is cloning the course), there is no need (and actually no way) to duplicate the file stored physically on the disk – just a new record in a special table of files is created.
All physical files are stored in so called file pool – a directory in moodledata. But as almost every file system has some limits in the number of files/sub-directories per each directory, it is necessary to distribute the files into sub-directories in the filepool. And here it comes interesting. The initial implementation of the file pool presented three levels of sub-directories to store each file. So if the SHA1 hash of the file content was e.g. 10a68843c08fe4446839961153812b94a1983c6b, such a file would be store in $CFG->dataroot/filedir/10/a6/88/10a68843c08fe4446839961153812b94a1983c6b
Ashley Holman from NetSpot realized that this distribution of files into three levels of sub-directories in the file pool is quite an overkill. If we expect that SHA1 hash has randomly distributed bit values, the chance of even having more than one file in the sub-directory seems to be around 1/16 millions, so it is pretty unlikely that even two files share the same directory at majority of Moodle sites. So we probably use four file descriptors (three directories plus one normal file) at the harddisk filesystem to keep a single file. That was considered as wasting of OS resources and the issue was filed as MDL-23885.
Eloy Lafuente mentioned a system he was working on in the past, which did not use fixed number of directory levels but was internally scaling itself as needed, so it would use just one directory level on small sites with several dozens of files and more of them on bigger sites with zillions of files. “I can say it has worked perfectly, scaling from 0 to 10 millions of files (today), with all the directories being filled as the system grows along the last 8 years (without waste of directory descriptions),” shared Eloy his experience.
Tim Hunt from the Open University was asked to come with a statistical analysis of the problem: “For N files (where N is an estimate of the most uploaded files any Moodle will ever have), come up with a directory structure for SHA1 hashes, so most folders contain ~1.000 files or sub-directories, and no sub-directory ever goes above 32.000 files (or at least the chance of that is vanishingly small).” Tim posted a nice report of the results and it was decided that there will be just two levels with 8 bits each to group the files in the file pool. Must be said that the structure of the file pool is internal implementation detail of File API layer and no-one is ever supposed to access it directly. Therefore it was not difficult to modify the code according the results of Tim’s research.
It is nice when software development is not just about the programming language.

Post scriptum

var_dump('php_sucks' == 0);

Latest stable version 1.9.9+

There are 6 commits into the stable branch from the last development week (from Tue Aug 10 to Mon Aug 16). ♦ Petr Å koda applied Matt Clarkson’s SQL related patch in external database enrolment plugin (MDL-23667). ♦ Andrew Davis committed a patch into the Gradebook dealing with a situation when a grade item is linked with a scale that does not exist any more (MDL-19070) and backported a recent Gradebook UI improvement suggested by Caroline Moore (MDL-22931). ♦ Tim Hunt fixed a bug in Quiz module overview report spotted by Brandon Quinn. The bug caused that Moodle notification message could get included in the output stream when generating XLS or ODS report (MDL-23161). Tim also committed a patch provided by Oleg Sychev to properly quote HTML special characters in extra question fields during the question XML export (MDL-23198). ♦ Ashley Holman removed a three seconds delay after sending user-to-user message copy via email. This results in significant speed boost when a mass message is sent to multiple recipients. It seems to me that the delay was intended there to prevent SMTP queue floods and I am personally worried a bit about possible functionality regressions here (MDL-23755).

Future version Moodle 2.0 Preview 4

There were 110 patches committed into the main development branch during the last development week. A single one QA test (MDLQA-245) of all 145 current test cases for the first round of Moodle 2.0 does not pass yet and there are 15 non resolved blockers in the tracker.
Petr Å koda started up his personal crusade against MySQL default storage engine – MyISAM. After a long and flamy discussion in Moodle development chat room, he created an issue in the tracker to discourage all Moodle administrators from using this storage engine. “MyISAM is a not a good choice for Moodle, it will thrash your data sooner or later – this is usually discovered much later during upgrades when it is too late,” says Petr in the tracker. Eloy Lafuente, the Knight in Shining Armor and our database guru, does not agree with Petr’s radical attitude: “I can see the point about recommending to switch to InnoDB (or any other ACID engine), but Moodle has worked under MyISAM for ages and IMO should continue working, at least until 2.1,” he objects.
As a result, InnoDB or XtraDB storage engines are now being used by default for new Moodle 2.0 installations, unless the administrator explicitly claims their will to use MyISAM. A migration tool from MyISAM to InnoDB was added to Moodle to ease the procedure for existing installations. See more details in MDL-23682.

Quotes of the week

“mysqldump is a joke”
– Petr Å koda blames MyISAM for a high chance to produce non-consistent dumps without actual stopping the access to the database for all other clients.

“pg_dump is a utility for backing up a PostgreSQL database. It makes consistent backups even if the database is being used concurrently. pg_dump does not block other users accessing the database (readers or writers).”
– PostgreSQL documentation

“Adding a handful of fields to Moodle core tables is simple in the same way that lighting the fuse on a sticky of dynamite is simple – you don’t want to hang around to look after it.”
– Tim Hunt does not recommend to modify Moodle core tables without trying all other ways of customizations

Playing hide-and-seek with mform elements

During a review of my own code, I realized I constantly forget to declare the value type of hidden elements in my mforms. Especially in those cases where the hidden element is defined as in

$mform->addElement('hidden', 'id', $current->id);

Such usage of hidden elements as carriers of known static values (which is typical usage for them in fact) may give us false impression that the set value is sort of constant. Wrong. Hidden elements are ordinary form elements, they just do not have any visible widget. But for an experienced user (did someone say hacker?) it is easy to set their value manually before submitting the form. So the values of hidden element must be validated as if they were normal input fields, there is nothing special about them. And setting the allowed data type is a first step of the validation. For the example above, something like

$mform->setType('id', PARAM_INT);

should be added into the form definiton() method.

Latest stable version 1.9.9+

There are 4 commits into the stable branch from the last development week (from Tue Jul 20 to Mon Jul 26) ♦ Dan Marsden removed a possibility to open SCORM player in a new browser window manually because it led to problems with communication between the SCO and the server and no grades were reported back (MDL-21333) ♦ Sam Hemelryk fixed HTML rendering issue in quiz module (MDL-20724) ♦ Petr Skoda improved isguestuser() core function which detects if the given user has logged in via “Login as guest” button. Such user has hard-coded username ‘guest’. But in MNet environments, just user name is not enough to uniquely identify user as there may be accounts with the same user name on different MNet peers. Therefore we must check the user’s mnethostid attribute and make sure it is really our local account to prevent reported problems with multiple guest accounts (MDL-22871) ♦ David Mudrak spotted two typos in the assignment module’s recent activity report. These typos accidentally eliminated each other so that the reporting of the recent activity for students in case of separate groups mode could not work but no error or false positive activity were displayed (e5bc356).

Future version Moodle 2.0 Preview 4

There are 284 commits into the main development branch from the last week. The list of Moodle 2.0 release blockers contains 19 unresolved issues. Thanks to our community QA testers, dozens of other issues have been discovered and fixed.

Quotes of the week

“That’s the second time in as many days I have brought up something and Petr has gone and removed it- I’d better watch out what I highlight next! ”
– Mary Cooch discovers dark side of QA testing

May git be with you

While my recent work on MNet remote enrolment  services for Moodle 2.0, I could not praise git more. Whenever dealing with some MNet development, one usually has to work on two separate instances of Moodle – server side and client side (aka service provider and service subscriber). This is roughly how I used git during the development. Let us say I have moodle.git clone in my ~/public_html/moodle20 directory. It is the checkout I use for everyday development tasks. I started with creating a new branch at the top of the latest upstream Moodle version. At the end of the day, this branch will contain all the commits to be pushed into Moodle CVS repository.

$ cd ~/public_html/moodle20/
$ git fetch
$ git branch mnet-integration origin/cvshead
$ git checkout mnet-integration

Then I prepared checkouts for the two clear installations of Moodle. Due to the lack of invention, let us call them A and B.

$ cd ~/public_html/
$ git clone moodle20 moodle20a
$ git clone moodle20 moodle20b

Thanks to the new CLI installer in Moodle 2.0, it was not actually needed to leave the pleasant dark shell window.

$ cd ~/public_html/moodle20a/
$ php admin/cli/install.php

And the work could start. After I set up both installations, I connected them into MNet network. Moodle A served as a course with user accounts and Moodle B was the one with courses. Simply said, remote enrolment services allows the administrator at Moodle A to enrol their users into remote courses at Moodle B. To make this work within the new enrolment framework in Moodle 2.0, changes had to be done on both of the sides – service provider (B in our scenario as it is the one with the courses) and service subscriber (A as the one with the users).

After a while, I had several commits in both checkouts. When I had a commit in A that I needed in B too, I just run

$ cd ~/public_html/moodle20b/
$ git fetch ../moodle20a
$ git cherry-pick 63891a8

When I finished, it was time to prepare commits to be sent into CVS. I went back to the repository with the integration branch and run

$ cd ~/public_html/moodle20/
$ git fetch ../moodle20a
$ gitk HEAD..FETCH_HEAD
$ git merge FETCH_HEAD
$ git fetch ../moodle20b
$ git merge FETCH_HEAD

git is clever enough to know that some commits from moodle20a were already included in moodle20b due to the cherry picking. Now I could go back to A and B and repeat the merging as many time as needed. So I ended with commits log like this. Sorry for different names of branches at the picture against the ones in the text. You can see how branches can be merged here, there and back again.

When I finished and was ready to publish my work, I rebased the integration branch against the up-to-date upstream head. Once the integration branch is rebased, the merging from the cloned branches can not work any more and would lead to conflicts, problems and duplicated commits. I rebased the branch accidentally so I know that git manual is really right about this. Rebasing is actually re-applying a sequence of commits. During the rebasing, commits order can be modified, commits can be meld into single commits etc. This allowed me to prepare a set of patches to be send into CVS using git cvsexportcommit command.

Post scriptum

When in Rome, do as the Romans. When in Moodle, use forums.