Permanent Tourist

The personal website of Mark Howells-Mead

As part of the day job (or even when I’m working on my own website), I need to clone a live website to my local development environment so that I know I have a current version of all the data and files to work on. There are many migration plugins which allow me to do this, but many of them throw a fit when dealing with large amounts of data. So I use a couple of simple command-line scripts to automate the process.

My current website contains over 25 Gb of data and so the server can often run out of memory when a plugin attempts to create a downloadable package. That lead me to write a couple of scripts, which allow me to bring the local development version up-to-date with the live version.

First of all, I’m running a local development version of the website on my own computer using Local: an installable app which allows me to create and run WordPress on my own machine. I can’t recommend this strongly enough, as it’s installed in minutes. Setting up a new website using Local – including a local database, a WordPress installation, a local domain and a free SSL certificate – takes less than a minute.

Once the local site is up and running, I right-click on the website in Local and choose Open Site Shell. This opens an SSH connection to the running website and I can then run the following commands on the command line.

Make sure that these commands are run in the webroot folder, or the files will be copied to the wrong directory.

The first command is to get the database. This assumes that WP CLI – the command line interface for WordPress – is installed on the server. (In the example, at ­/bin/wp.) If you’re using Cyon hosting in Switzerland, there are instructions for installing WP CLI in this (German language) blog post.

(In these examples, replace the uppercase words with the values which are appropriate to your site and server.)

ssh $USER@$SERVER -A "cd $WEBROOT && ­/bin/wp db export - | gzip" > live.sql.gz && gunzip live.sql.gz && wp db import live.sql && wp search-replace $LIVEDOMAIN $DEVDOMAIN && rm live.sql

Once you have the database, you will also need to copy the files before trying to open the website in a browser. By using rsync, only files which are new (or newer) than those already existing locally will be copied down. This saves a lot of time if you’ve run the commands before.

Make sure that the ending slashes are correct as per this example, or the files will be copied to the wrong place. Local will have already created local themes, plugins and uploads folders, but you might need to create the languages folder manually before you start trying to copy files down.

rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/themes/ ./wp-content/themes -delete
rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/plugins/ ./wp-content/plugins -delete
rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/languages/ ./wp-content/languages -delete

If your site is using “must-use” plugins, then you’ll need to copy them down…

rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/mu-plugins/ ./wp-content/mu-plugins -delete

…and you’ll obviously need the uploads.

rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/uploads/ ./wp-content/uploads -delete

The number of files in the uploads folder can be very large the first time you run the command, depending on how large your website is. If you don’t want to copy all of the files down, you can modify the paths accordingly. You might need to manually create the target folder locally (e.g. wp-content/uploads/2020) if it doesn’t exist.

rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/uploads/2020/ ./wp-content/uploads/2020 -delete

Once you’ve run these commands, your local version will be a precise copy of the live website and you can start work.

A final note: if you’re familiar with bash scripts, then it’s easy to pack these commands into a single file and run them from the command line in one go using the command sh



ssh $USER@$SERVER -A "cd $WEBROOT && $WPCLI db export - | gzip" > live.sql.gz && gunzip live.sql.gz && wp db import live.sql && wp search-replace $LIVEDOMAIN $DEVDOMAIN && rm live.sql
rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/themes/ ./wp-content/themes -delete
rsync -azP -e "ssh" $USER@SERVER:$WEBROOT/wp-content/plugins/ ./wp-content/plugins -delete
rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/languages/ ./wp-content/languages -delete
rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/mu-plugins/ ./wp-content/mu-plugins -delete
rsync -azP -e "ssh" $USER@$SERVER:$WEBROOT/wp-content/uploads/ ./wp-content/uploads -delete

4 responses to “How to clone a WordPress website with WP CLI and rsync”

  1. Jan Tack avatar
    Jan Tack

    Very helpful, Thanks a lot!

    I wonder why you don’t also use the –delete option in the rsync calls so that files in the target system are also deleted if they no longer exist in the source directory?

    1. Mark Howells-Mead avatar

      Thanks for the suggestion, Jan. I’ve added this option to the code examples.

  2. Madhu V avatar

    Mark: This is a crisply written and helpful post. Thank you! Do you have tips for how you push from your local dev back to the live site? The rsync bit to get the files back up is simple. But I am uncertain about how to deal with the SQL portion. Is there any issue with simply flipping the remote and local SQL commands? Will that do it?

    1. Mark Howells-Mead avatar

      That’s kind of you, thanks. Yes, you should be able to swap the paths around. Export the file locally and upload it, then use the remote WP CLI to import it and use search-replace on things like the domain.

Leave a Reply

Your email address will not be published. Required fields are marked *

For security, use of Google’s reCAPTCHA service is required which is subject to the Google Privacy Policy and Terms of Use.

This site uses Akismet to reduce spam. Learn how your comment data is processed.