fromMarch 2011
Article:

The Infrastructure Behind a Redesigned Drupal.org

0

The redesigned Drupal.org has been running strong since its launch back in October of 2010. The redesign is a big departure from the previous Drupal.org. It added a new theme and a number of new pages including a customizable user dashboard, implemented a multi-site search, and integrated Drupal.org with its sub-sites. The design and functionality was the result of a massive collaboration between paid contractors, community leaders and hundreds of volunteer community members.

>

The infrastructure behind drupal.org is always growing and adapting. The redesign provided the infrastructure team with an opportunity to leverage a number of new and existing technologies to keep Drupal.org and the contributors involved running efficiently.

The development process and Hudson

Before the redesign, the use of Hudson was still in its infancy with much of the infrastructure team. Hudson was being used as a replacement for existing cron jobs across the infrastructure, and was beginning to allow us to control and automate more complex routines.

The redesign had a need for individual development environments for the redesign, and no easy way to automate the builds. We created Hudson jobs to take sanitized database snapshots of Drupal.org sites, but we still had no automated way of creating development environments. An automated way to create a sanitized environment reduces the barrier to entry for community members interested in contributing.

We were able to use Hudson to tie together the deployment of an entire development environment . The new Hudson job allowed us to create development environments by entering the name of the environment and a comment describing what the environment was for. The job would handle the rest: create a database and database user, checkout the latest redesign codebase from bazaar, import the latest copy of the sanitized database, configure settings.local.php, and set up a new virtual host in httpd.

Solr and stagingsolr

Solr has become increasingly integrated into Drupal.org. We now have a highly available master/slave Solr cluster built using a combination of Varnish and Tomcat. The cluster efficiently handles searching on all of the Drupal.org sites, and Solr’s ability to scale allows us to index all of the sites in a single Solr core.

Using Hudson allowed us to provide a Solr core that was available for redesign contributors to download and install in their local development environments.

Load testing and tuning

Load testing was one of the most important steps of the redesign. With the number of new features being added to Drupal.org, it was important that the community was able to test and identify problem areas before launch. Beta.drupal.org was provided as a place for the community to test the new functionality and features, recording issues in the Drupal.org redesign issue queue. Using Hudson, we created a loadtest development environment, database and Solr configuration to mimic that of beta.drupal.org. We then added another Hudson job to control running a JMeter load test suite. This job made use of the Performance plugin in Hudson which aggregates data from the load test after completion and creates graphs and tables based on the results.

JMeter was used to test new areas of the site as both authenticated users and anonymously. The test suite was designed to test new areas of the site, including Solr multi-site searching, the homepage, the dashboard, get started pages, download and extend, and module categories. Hudson was used to flush logs in MySQL, and analyze MySQL’s slow log with Maatkit mk-query-digest. The combination of mk-query-digest and the output of the Performance plugin were both available via Hudson’s load test job.

We used these tests to identify the problem areas, and got to work tuning code and SQL queries for optimal performance. As performance improved, testing was moved from the loadtest environment to the production infrastructure. Tests on the production infrastructure were run in the same way with JMeter and Hudson. The devel module played a huge part in profiling slow loading pages as well.

Once all of the problem areas were identified and fixed, we were confident that we had a stable and performant product that was ready for release.

Highly available NFS

The redesign also gave us the opportunity to overhaul one of the remaining single points of failure in the production infrastructure: NFS. NFS is now hosted on a highly available cluster using DRBD and heartbeat. DRBD allows us to easily mirror a block device over the network, commonly described as allowing a RAID1 mirror over a network. DRBD enables us to keep both nodes’ NFS exports in sync. Heartbeat controls a virtual IP address, and if failure is detected on “nfs1”, points the virtual IP to “nfs2” without downtime. The new NFS cluster also gave us a huge increase of available space, with plenty of room to grow.

In The End

We were able to create an automated set of tools that enabled the distributed team of redesign contributors to effectively and efficiently write and test their changes. Many of these tools are still in use, and provide an easy way to create new development environments for anyone interested in working on Drupal.org. After the redesign, the Drupal.org infrastructure is stronger than ever, with a more dependable infrastructure and a number of new tools available for testing and QA.

DRBD: http://www.drbd.org/
Heartbeat: http://www.linux-ha.org/wiki/Heartbeat
Hudson: http://hudson-ci.org/
Hudson performance plugin: http://wiki.hudson-ci.org/display/HUDSON/Performance+Plugin
JMeter: http://jakarta.apache.org/jmeter/
Maatkit: http://www.maatkit.org/
Solr: http://lucene.apache.org/solr/
Varnish: http://www.varnish-cache.org/

Advertisement

From our blog

Entity Storage, the Drupal 8 Way

In Drupal 7 the Field API introduced the concept of swappable field storage.

The Drupal 6 to 8 Upgrade Challenge - Part 2

Having concluded the readiness assessment, we turn next to migrating the content and configuration. In reality, there’s little chance that we would migrate anything but the blogs from our old site. For the sake of giving Migrate in Core a workout with real conditions, however, we’re going to upgrade with core’s Migrate Drupal module rather than rebuilding.

The Drupal 6 to 8 Upgrade Challenge - Part 1

Nathaniel Catchpole , the Drupal 8 release maintainer and Tag1 Senior Performance Engineer, suggested that Drupal shops everywhere could support the

DrupalCon Austin

The entertainment industry is not for the faint of heart.

Drupal Watchdog Joins the Linux New Media Family
Drupal Watchdog 6.01 is the first issue published by Linux New Media.

Drupal Watchdog 6.01 is the first issue published by Linux New Media. Come see the Drupal Watchdog team at DrupalCon 2016!

Drupal Watchdog was founded in 2011 by Tag1 Consulting as a resource for the Drupal community to share news and information. Now in its sixth year, Drupal Watchdog is ready to expand to meet the needs of this growing community.

Drupal Watchdog will now be published by Linux New Media, aptly described as the Pulse of Open Source.

Welcome to DrupalCon Barcelona - The Director's Cut

For all you schedule-challenged CEOs – and ADHD coders – this Abbreviated Official Director’s Cut is just what the doctor ordered.

Welcome to DrupalCon - The Barcelona Edition

Did we have fun in Barcelona?
OMG, yes!

Did we eat all the tapas on the menu and wash them down with pitchers of sangria?
Yes indeed!

Recursive Closures and How to Get Rid of Them

This came up while manipulating taxonomy children and their children recursively, so it’s as not far from Drupal as you’d think.