Running Drush Commands in Parallel on Over 500 Sites

What is Drush?

Drush, short for “Drupal Shell”, is a powerful command-line interface tool that any Drupal user will recognise, which can create an effortless solution to a number of tasks, aiding with the management and administration of Drupal websites.

With Drush, a user can perform actions such as:

1. Clearing Drupal's cache.
2. Installing, updating, and uninstalling modules and themes.
3. Running database updates.
4. Importing and exporting site configurations.
5. Running cron jobs.

The Issue

Under regular circumstances, when a cache clear or database update is required on a site, simply running a Drush command would rectify the issue with simplicity and efficiency. For example, to clear the cache for a site, you would run:

# D7
drush cc all

# >D7
drush cr

However, in the event of a Drupal multisite setup being made up of over 500 sites, broader issues are likely to arise. By running individual commands one by one on each of the 500 sites, not only will this be an extremely lengthy process, but also highly error-prone.

Zoocha’s Solution

In order to create a solution to this issue, a script was written originally to loop through each of the sites in order to automatically run these commands. However, while this process did not take as long as it would have if it were a manual task, the script still took over 12 hours to complete its run.

Through the use of the Linux built-in background jobs “&” functionality, we were able to produce our own script. This new script allows the user to separate the multisites into 10 individual sites, or “chunks”, whilst running the Drush commands concurrently in the background. Furthermore, by using the “wait” command, the system waits until all of the sites within each chunk have been completed before moving onto the next. Therefore, this will decrease the amount of commands from over 500 in succession, to just over 50 that are needed in order to process the entirety of the multisite, considerably reducing the amount of time required.

Below is a simplified example of the script:

chunk_size=10

# Get the multisites found in the ‘sites’ folder (and make sure they are setup by having a settings.php file).
SITES=$(find sites -mindepth 2 -maxdepth 2 -type f -name settings.php | sed 's|^sites/||g' | sed 's|/settings.php$||g'")

for((i=0; i < ${#SITES[@]}; i+=chunk_size))
do

  # Get the next  group of multisites.
  chunk=( "${SITES[@]:i:chunk_size}" )

  for SITE in "${chunk[@]}"; do {

    echo -e "Processing site: $SITE \n"

    drush -l "$SITE" cc all && echo "Finished site $SITE" &

    drush -l "$SITE" updb -y && echo "Finished site $SITE" &

    drush -l "$SITE" fra -y --force && echo "Finished site $SITE" &

  } done

  echo -e "Waiting for parallel commands in this chunk to finish running...\n"

  wait

done

Based on the arguments delivered to the script, checks exclusively on CacheClear, UPDB, or FRA are contained when needed and selected on deployment. 

An issue that we were able to flag and resolve was that the majority of the deployments did not require a full feature revert (FRA) despite being assigned one, and therefore it would take a significantly longer amount of time for the server to load the deployment than required. To resolve this issue, we updated the deployment parameters, which allow a chosen number of features to be reverted:

# Replacing the "fra" command with a list of space separated features
drush -l "$SITE" fr ${FEATURES_SPACE} -y --force

This update then contributed further to the decrease in time spent running the commands in parallel.

Options and Caution

Despite significant trial and error, as well as various benchmarking, we were finally able to come to the decision of using 10 as the number of sites per chunk. While we were keen to keep the number as large as possible in order to save the most amount of time, we found that any more than 10 would create a halt in the server, which would slow almost to a stop, and eventually cause a reboot. However, for more powerful servers it is possible to increase the number of sites to higher than 10, as can one decrease the amount of sites per chunk to 7 if the server was deemed less powerful, which is the case with our non-production server.

Recently, we have also been working on a new version of this solution for a Drupal 9 build using Github Actions which we are keen to share in a future blog post.