David Mudrak's blog

Latest stable version 1.9.9+

There are 4 commits into the stable branch from the last development week (from Tue Jul 20 to Mon Jul 26) ♦ Dan Marsden removed a possibility to open SCORM player in a new browser window manually because it led to problems with communication between the SCO and the server and no grades were reported back (MDL-21333) ♦ Sam Hemelryk fixed HTML rendering issue in quiz module (MDL-20724) ♦ Petr Skoda improved isguestuser() core function which detects if the given user has logged in via “Login as guest” button. Such user has hard-coded username ‘guest’. But in MNet environments, just user name is not enough to uniquely identify user as there may be accounts with the same user name on different MNet peers. Therefore we must check the user’s mnethostid attribute and make sure it is really our local account to prevent reported problems with multiple guest accounts (MDL-22871) ♦ David Mudrak spotted two typos in the assignment module’s recent activity report. These typos accidentally eliminated each other so that the reporting of the recent activity for students in case of separate groups mode could not work but no error or false positive activity were displayed (e5bc356).

Future version Moodle 2.0 Preview 4

There are 284 commits into the main development branch from the last week. The list of Moodle 2.0 release blockers contains 19 unresolved issues. Thanks to our community QA testers, dozens of other issues have been discovered and fixed.

Quotes of the week

“That’s the second time in as many days I have brought up something and Petr has gone and removed it- I’d better watch out what I highlight next! ”
– Mary Cooch discovers dark side of QA testing

May git be with you

While my recent work on MNet remote enrolment  services for Moodle 2.0, I could not praise git more. Whenever dealing with some MNet development, one usually has to work on two separate instances of Moodle – server side and client side (aka service provider and service subscriber). This is roughly how I used git during the development. Let us say I have moodle.git clone in my ~/public_html/moodle20 directory. It is the checkout I use for everyday development tasks. I started with creating a new branch at the top of the latest upstream Moodle version. At the end of the day, this branch will contain all the commits to be pushed into Moodle CVS repository.

$ cd ~/public_html/moodle20/
$ git fetch
$ git branch mnet-integration origin/cvshead
$ git checkout mnet-integration

Then I prepared checkouts for the two clear installations of Moodle. Due to the lack of invention, let us call them A and B.

$ cd ~/public_html/
$ git clone moodle20 moodle20a
$ git clone moodle20 moodle20b

Thanks to the new CLI installer in Moodle 2.0, it was not actually needed to leave the pleasant dark shell window.

$ cd ~/public_html/moodle20a/
$ php admin/cli/install.php

And the work could start. After I set up both installations, I connected them into MNet network. Moodle A served as a course with user accounts and Moodle B was the one with courses. Simply said, remote enrolment services allows the administrator at Moodle A to enrol their users into remote courses at Moodle B. To make this work within the new enrolment framework in Moodle 2.0, changes had to be done on both of the sides – service provider (B in our scenario as it is the one with the courses) and service subscriber (A as the one with the users).

After a while, I had several commits in both checkouts. When I had a commit in A that I needed in B too, I just run

$ cd ~/public_html/moodle20b/
$ git fetch ../moodle20a
$ git cherry-pick 63891a8

When I finished, it was time to prepare commits to be sent into CVS. I went back to the repository with the integration branch and run

$ cd ~/public_html/moodle20/
$ git fetch ../moodle20a
$ gitk HEAD..FETCH_HEAD
$ git merge FETCH_HEAD
$ git fetch ../moodle20b
$ git merge FETCH_HEAD

git is clever enough to know that some commits from moodle20a were already included in moodle20b due to the cherry picking. Now I could go back to A and B and repeat the merging as many time as needed. So I ended with commits log like this. Sorry for different names of branches at the picture against the ones in the text. You can see how branches can be merged here, there and back again.

When I finished and was ready to publish my work, I rebased the integration branch against the up-to-date upstream head. Once the integration branch is rebased, the merging from the cloned branches can not work any more and would lead to conflicts, problems and duplicated commits. I rebased the branch accidentally so I know that git manual is really right about this. Rebasing is actually re-applying a sequence of commits. During the rebasing, commits order can be modified, commits can be meld into single commits etc. This allowed me to prepare a set of patches to be send into CVS using git cvsexportcommit command.

Post scriptum

When in Rome, do as the Romans. When in Moodle, use forums.

Latest stable version 1.9.8+

There are 7 commits into the stable branch from the last week. Tim Hunt committed David Binney’s patch fixing a question bank SQL query at Oracle (MDL-21828). Iñaki Arenaza fixed an old issue when he committed his patches that significantly increase the quality of images produced by TeX filter. If the binaries of latex, dvips and convert are available at the system, Moodle can generate nice PNG formulas. Otherwise, “ugly” GIFs are created by mimetex (MDL-10197). Another Iñaki’s committed patch allows to use HTML editor when editing login page instructions (MDL-19053) and yet another one fixes small typo in CAS, DB and LDAP authentication plugins (MDL-18689).

Unstable development version 2.0dev

There are 160 commits at the main development branch from the last week, many of them done by Petr Škoda during the weekend strings clean-up. Patrick Malley added a new test theme called Boxxie based on his Greenie theme for Moodle 1.9. On Wed Apr 7 00:31:20 2010 UTC, the last ever run of Eloy’s installer_builder generated and committed strings for Moodle installer and was turned off. In the near feature, it will be replaced with a new script taking the data from our new strings repository called AMOS. So long installer_builder, and thanks for all the strings!

Quotes of the week

“I like git because it might actually force us to fix our sloppy/missing review processes before things hit CVS”
– Petr Škoda guards the code against eager commits – get alarmed!

“NOBUG: Bombing BOM: BOM bombed”
– Eloy Lafuente and his bombastic commit message

Milliseconds count

At the moment, I have 3,740,067  records (yes, 3.74 millions of records) in the new database repository of Moodle strings. The repository (basically one huge SQL table) contains one record per every modification of every single string that happened since Moodle 1.6 in any language. It takes approximately two hours to re-populate the table from our git mirrors of Moodle core and Moodle langs CVS repositories. There are some critical queries that are used quite frequently – for example to take a snapshot of the most recent string values for the given component, language and branch. That is, for example, “what is currently saved in lang/en/moodle.php at the main development branch?” (or what was there at the given timestamp). Writing this query was a big lesson for me. Not only it was actually for the first time I ever used an SQL subselect. Having so many records, the query optimising became very important as I refused to wait dozens of seconds to actually see some results (and I can imagine translators  would not be happy with such delays, too).

Basically, all Moodle strings and their history is now stored in one database table containing fields branch (integer code for Moodle branch, like 1900 representing MOODLE_19_STABLE), lang (language code like ‘es’), component (the string domain – or the file where the string is defined, like ‘forum’), stringid (the key is $string array), text (the string itself) and finally timestamp (since when the change described by this record  is valid). There are other fields defined in the table – to hold the author of the change, commit message, deletion flag and others. There is composited index created for fields  (component, language, branch).

This is the query I was using during the optimisation and performance comparison. It returns the most recent version from several branches, several languages and several components – which is exactly what I need for the new translation interface.

SELECT r.id, r.stringid, r.text, r.timemodified, r.deleted
  FROM  mdl_amos_repository r
  JOIN (SELECT branch,lang,component,stringid,
               MAX(timemodified) AS timemodified
          FROM mdl_amos_repository
         WHERE branch IN (2000,1900)
               AND lang IN ('en','cs')
               AND component IN ('moodle','workshop')
             GROUP BY  branch,lang,component,stringid) j
    ON (r.branch = j.branch
        AND r.lang =  j.lang
        AND r.component = j.component
        AND r.stringid = j.stringid
        AND  r.timemodified = j.timemodified)
  WHERE r.branch IN (2000,1900)
        AND r.lang IN ('en','cs')
        AND r.component IN ('moodle','workshop')
  ORDER BY  r.component, r.stringid, r.lang, r.branch;

The basic idea of the query is to get know (in subselect) which one is the most recent version of every string. Then (via the table self join) get some additional information about such string. I was experimenting with various variants of the query -like putting those WHERE conditions into the inner subselect or the outer main select. None of them was good enough until Petr Škoda gave me a world-saving hint: put the same WHERE conditions into both inner subselect and outer main select! The inner conditions help to filter the rows that are going to be aggregated by GROUP BY. The outer conditions help to join results quickly.

Using this, I managed to get to times like 1500ms at my laptop and 200ms at the testing server, both using PostgreSQL. First attempts to run the same query on MySQL led to significantly worse figures. I am going to do more tests to be sure that both engines have similar resources/conditions available. That massive usage of IN() is supposed to be a challenge for MySQL…

Post scriptum

I fell in love running Moodle 2.0 upgrading script via command line interface – see admin/cli/*.php scripts

(1) make coffee – unfortunately this does not work as Linux command yet on my machine. I should find some time to implement http://tldp.org/HOWTO/Coffee.html

(2) Start-up Vimperator and triage my email inbox, using a variant of GTD method.

(3) Connect to Moodle developers and Moodle commits Jabber rooms

(4) $ cd public_html/moodle-head

(5) $ git fetch

Download the recent commits from our git mirror of the main CVS repository.

(6) $ gitk master..origin

This gives me an overview on what happened in the code since the last time I merged.

(7) $ git checkout master && git pull

Fast-forward my local master branch so it contains the recent commits.

(8) $ git checkout fix/somebug-MDL-xxxxx

I love branching. I do every development (even a simple bug fix) on a separate branch.

(9) $git cvsexportcommit -c -p -u -v <hash>

Once the fix is committed on my branch, I export it to CVS. Thanks to my set-up everything happens really quickly. If the fix is done on a stable branch I have to move the merged-status tag manually yet in the CVS checkout, which is a bit annoying. Once we move to git (note not “If we moved to git” :-p) thing will become even easier.

It was one of that discussions when people prefer leaving the chat room before they say something they would regret later. The Pandora’s box was opened by an innocent question on the recommended procedure to customize a contributed Moodle plugin. The discussion turned into a hot topic on what the current source code management system (CVS) gives to us and what we and our contributors could gain (and lose) from switching the Moodle CONTRIB repository to git.

Imagine you have clients who pay you for implementing a solution based on a contributed Moodle plugin. The quality of the Moodle contributed code varies and some sort of the code review and clean-up is strongly recommended before using it at production server. The contributed code may contain critical security holes and such plugin may easily become an open gate to your server with a big bright sign “Welcome Attackers!” So you have reviewed the code, fixed several obvious bugs and improved the features a bit as requested by your client. And – now what?

Of course you want to deploy your changes to the client’s server – as soon as it is tested  (or at least it does not produce any PHP warnings ) And you, as a follower of the open source spirit, want to contribute your changes back to upstream. So you send your patches to the original component’s maintainer to be included in CVS. Sounds good. In a perfect world it would work with no problems. But the humans are not dead yet and things tend to get complicated. Also, some objections against the real benefits of this procedure were raised. Let me list some issues that may influence this contributing back to upstream.

1. The maintainer does not respond. Such things happen. You can ask the CVS administrator to give you write access and become a maintainer. Do not do this if you know/feel you would not have time to do it responsibly. Remember the lesson #5 from the Eric Raymond’s famous essay: “When you lose interest in a program, your last duty to it is to hand it off to a competent successor.” It’s better if it’s known there is no maintainer than if people think they can rely on you. On the other hand, do it if a silent voice inside your mind whispers that it’s the job you always wanted to do. It’s both nice and good thing to spend your life on.
2. The maintainer does not agree with your improvements. Then it’s time to discuss things. Summarize the issue and open the topic in Moodle forums. Listen to the maintainer’s arguments. Maybe he/she has some further plans with the module? Who else can profit from your changes? Try to not to fork the product unless there is a chance to reach a consensus.
3. The rubbish can get into the code after you reviewed it. Yes, it can. You must remember the state of the code when it was reviewed. Source code management software calls such signs as “tags”. You shall tag a certain point of the source code history and review exactly that point. Then you know what version is sort of safe to be deployed. Also, you can focus on the changes that came after this point later when you review the code again.
4. The maintainer does not understand your fixes. Some say that simply sending the patches to the maintainers is useless as they do not spend any effort on learning from it. I personally do not agree with this. However, I can see the importance of educating the contributors. If the contributed code contains a security bug which is exploited, who is blamed? The administrators? No,  they have always been against this community thingie. The original author? No, nobody knows him/her. Moodle itself? Sure – it’s all Moodle’s fault! And that’s not fair.

All these have nothing to do with the SCM software itself. It is about responsibility, communication, constructive peer-reviews and developer etiquette. However, I think that the features and the characteristics of the SCM may significantly help with sorting out the issues discussed above or with not running into them at all.

Switching Moodle CONTRIB area into a cluster of separate git repositories emerged as an idea during the discussion. The nature of git – a distributed versioning system – seems to perfectly fit into the nature of our contrib, the hive of really interesting features and ideas. Contributors may come with their own trees of source code and the comparison of such two trees is easy. If the maintainer stops responding to the community questions and proposals, his/her tree becomes quickly obsolete and the community may start to prefer some other tree (if such is available). The pull-based model of the the maintainer-contributor interaction requires the maintainer pays attention to the patches to be included and hopefully learns from them. On the other hand, the contributors have to keep their trees up-to-date with the recent versions of the upstream code if they wish it is included in the maintainer’s tree. The point is, with git, everybody can easily become a maintainer of his/her own clone of the product. They have to gain the respect of the community members by keeping the high quality of their version. And remember – respect is everything.

I do not think we should switch to git-only at the moment. Let us give git a chance and see what  Moodle contributors think. To help them with their first steps, I summarized my personal experiences with using git in Moodle development into a sort of tutorial at http://docs.moodle.org. You are welcome to review it and/or put comments with your own solutions.

Recently I spent many hours reading git user’s manual and experiencing git-merge and git-rebase. Thanks to Nigel’s and Penny’s comments, I hope I can see differences between these. Hopefuly I’ll be able to maintain my adminlang repo effectively.

Still, I would like to find what procedure to take when porting a fix from one branch to another. Is cherry-pick intented for this?

Also, I found an interesting note about using git-format-patch | git-am to massive port from a branch to another.

Git is cool Can’t wait Moodle switch ti it as well…