Recently, we released the latest development to the ManageWiki project, ManageWikiPermissions. This feature implements new technology into ManageWiki which, to date, we've been unable to use to create performance improvements with and of course allows yet another integral part of a MediaWiki to be customisable by individual wikis quickly.

History of ManageWiki

Let's start this off by discussing the history of ManageWiki. When @John and @Southparkfan were planning the details of a new wikifarm, one of the key features they wanted Miraheze to have was it's own independently built and designed wiki management system. The advantage of having this was we'd be able easily design an efficient system and have no constraints later down the line as Miraheze grows as we would have 100% home developed software powering the creation, management and handling of all wikis linked centrally. The first commit for ManageWiki was on July 8th 2016, just less than a year after Miraheze began on July 22nd 2015. Obviously the initial release had very future features only being able to manage a wikis sitename, language and closure status. Over time, few features were added such as private status, activity and category until the introduction of extension management on May 11th 2018 and settings management on May 15th 2018.

Screen grab of the GitHub insights page showing ManageWiki's activity since November 2016.

So, How Was On-Wiki Permissions Management Implemented Given MediaWiki Doesn't Do It?

This was actually a good question I ponded months before I even started on this project. There were a few problems thought about such as;

1. How would this be implemented on wiki cleanly?
2. How would this be implemented in the backend?
3. Will this be efficient?
4. What are the drawbacks of doing this?
5. Are there any risks of doing this?

For the answer to questions 2 and 3, wait patiently for the end because this was a massive problem to overcome and one we struggled with. For the rest, because we had developed our own wiki creation and management software, this was actually relatively easy to manipulate and work with as we had good knowledge of the backend of the existing code base and the freedom to rewrite it how we wanted.

So, for question 1, we already had a nice frontend implementation thanks to the amazing experimentation of CentralAuth with Special:GlobalGroupPermissions. The UI was based on simple HTML tables (while we prefer OOUI) that were easy to manipulate and build on to, so we could easily extend to suite our needs.

The major drawbacks and risks we had with this feature was always going to be performance and ensuring sensitive rights and data couldn't be accessed by way of ManageWikiPermissions. Performance is something we address later on in this post but sensitive data we could handle by introducing easy to configure blacklists around groups and userrights. The introduction of these blacklists means we can allow all the amazing features of local user rights management from the creation of any group and assigned of rights while ensuring we have small oversight over what can and can not be assigned and made. These configuration parameters work by checking group names when they are being edited (disallowing any sensitive names we don't want to allow) and removing all the special userrights away from the array we use to build the tables you use to add new rights to groups.

You Mentioned a Performance Issue?

Yes! The initial deployment of ManageWikiPermissions was far from successful mainly for performance reasons. The load on our database server increased significantly (CPU usage jumped from around 40% to 177%) with no caching in place. The reason for this is every single user group created a query on every page load for every wiki. The average number of usergroups on a wiki is roughly 10. Since each group creates a new database query we've never had before, the average web request would generate 10 new MySQL queries. Taking into consideration last week, we had around 67,000 visits across the wikifarm. This would then generate an astonishing 670,000 new queries.

This is a considerable increase in work for our databases and one we had to alleviate quickly before we could deploy the feature globally. So we turned to caching.

Initially, implementing caching on each server in the form of memcached was attempted however this brought along issues in term of increasing load on MediaWiki servers and per-server cache rather than an easily management farm-wide cache. So on we went, looking for a different solution. Then the idea of breaking down key caching down to a single small key per wiki and file storage cache (thanks to the power of SSDs!) came up and implementing it using technology called CDB (or Constant DataBase). With the implementation of this, we would be able to store the whole permissions meta data on files on the servers and have them generated on the fly when changes are registered or when it's appropriate for a cache key to expire. Our attempt to implement CDB worked rather well after a few final bugs were resolved earlier today. After this change was pushed, CPU usage on database servers dropped from the 177% to 44%, overall this constitutes a 4% increase in processing power for such a massive social and technical change to how Miraheze works.

The above image shows the significant drop in CPU usage on db4 when fixes to caching in ManageWikiPermissions were pushed earlier today.

Okay, So Initially You Mentioned This Technology Was Tried Before?

Right, Kind-of. There was an implemention of CDB in CreateWiki added in this commit however it was never used in production and was removed in this commit as there was no intention to start using it. Earlier than the first CDB change though, we trialied using JSON syntax to serve our database list and wiki meta data however this made the processing of wikis incredibly slow and was reverted. While the idea of CDB never initially took off, now it has and we may investigate seeing if there is any performance gains of using this method over plain text lists.

While This Is All Nice, Has This Changed Anything Moving Forward?

Almost certainly it has! This has allowed us to experiment with in-house modular extension to ManageWiki without having to force changes to the current database schema and means we can add things that no one has to use unless they chose to in an upstream setting. Further, this adds a nice backend for us to change to support other more complex settings such as Namespaces (T3434) as it allows us to cache core MediaWiki settings that we have to populate immediately and can't overwrite later safely without issues.

ManageWiki as a project still has a long way to go before it is able to support every setting MediaWiki has to offer and before we're able to support everything we could possibly want to do on Miraheze. The development of both CreateWiki and ManageWiki will continue long into 2019 when it comes to core changes and core additions however I feel both projects are stable for third party use. Anyone wishing to use either piece of software (or our other ones such as WikiDiscover and MatomoAnalytics) are encouraged to contact me by email at john [at] miraheze.org and I will happily support anyone in using the software.

To keep up to date on the developments of all of our software, below are links to projects both on Phabricator and GitHub for the software!

• CreateWiki: CreateWiki, GitHub
• ManageWiki: ManageWiki, GitHub
• WikiDiscover: WikiDiscover, GitHub
• MatomoAnalytics: MatomoAnalytics, GitHub