Trustworthy Bash to the rescue

  • Sharebar

Linux’s Bash is trustworthy and awesome!!!

Today was a very strange day as I was trying to clear/refresh varnish’s cache. PHP’s cUrl was not working as fast as I wanted if it bothered to work at all. JMeter wasn’t working, siege was not working and netbeans was not working (this had nothing to do with anything but couldn’t help but mention it).

So in the middle of so much unworkingness I decided to turn to my always trustworthy friend: bash, and create a single script to purge and refresh varnish’s cache. I already had my external IP set as allowed in the varnish ACL, so make sure you do the same before proceeding.

Why did you write the script in bash and not python, ruby or vala

Simply because I like bash more than those other languages… if I could write web sites in Bash, I would. Linux systems are very powerful and bash is in essence the main power source of every Linux system (this my interpretation but it is almost 100% true).

Purge Varnish cache with bash and curl

I have a Magento module that generates JMeter test plans for a given site. It also stands to reason that I’d also generate a siege file for stress testing, but as jmeter and siege has failed me I realize how much I depended on them to do some critical tasks. I also tried to run siege directly on a clients server with a cron job, only to realize that it was not installed (seriously???).

To break my dependency on both JMeter and siege I created the following bash script:

#!/bin/bash

curl="curl -sSL" 
#path to the url that generates the url list
generator="url_to_generate_file"
#grab a copy of the file
file_path="domain:/path_to/url_list.txt ."

#generate the file and grab it locally, it will be save in the same path as this script
$curl $generator && scp $file_path

#comma separated list of emails
emails="user@domain.com"

#list of urls
URLS=$(cat url_list.txt)

echo "-----------------------------
Purging old URLS from cache
-----------------------------"
counter=1
for url in $URLS; 
do 
    echo "$counter: cleaning url $url" 
    let counter=$counter+1;
    `$curl -X PURGE $url -o /dev/null`
done
echo "---------------------------------------
Populating cache with new url content
---------------------------------------"
counter=1
for url in $URLS; 
do 
    echo "$counter: ReCaching url $url" 
    let counter=$counter+1;
   `$curl -X REFRESH $url -o /dev/null`
done

echo "---- emailing ---"

mail -s "Urls processed" $emails < url_list.txt

Another version could be something like this:

#!/bin/bash

curl="curl -sSL" 
#path to the url that generates the url list
generator="url_to_generate_file"
#grab a copy of the file
file_path="domain:/path_to/url_list.txt ."

#generate the file and grab it locally, it will be save in the same path as this script
$curl $generator && scp $file_path

#comma separated list of emails
emails="user@domain.com"

#list of urls
URLS=$(cat url_list.txt)

echo "-----------------------------
Purging old URLS from cache
-----------------------------"
counter=1
for url in $URLS; 
do 
    echo "$counter: cleaning and refreshing url $url" 
    let counter=$counter+1;
    `$curl -X PURGE $url -o /dev/null`
    `$curl -X REFRESH $url -o /dev/null`
done
echo "---- emailing ---"

mail -s "Urls processed" $emails < url_list.txt

This way you don’t have to interact with the URLs array multiple times.

Save that file as “varnish” and then give it permissions to execute

chmod +x varnish

Now you can run it every time:

./varnish

Now in one of my controllers for this module I have the following piece of code:

    /**
     * Creates a siege test plan (as in a bunch of urls to try)
     */
    public function buildSiegeTestAction(){

          $io = new Varien_Io_File();
          $io->setAllowCreateFolders(true);
          $io->open(array('path' => $this->getPath()));
          $io->streamOpen($this->getFilename());
          $helper = Mage::helper('testplans');
          //$urls is now an array with a bunch of urls
          $urls = implode("\n", $helper->getAllUrls());
          $io->streamWrite($urls);
          $io->streamClose();
          $io->close();
    }

As you probably figured out by now, my bash script is calling that action in the sites and grabbing the file it generates. The beauty of the script is that it can be ran on any hosting provider that runs magento because cURL is one of the requirements unlike siege or JMeter.

This could also be accomplish using varnishadm, however I prefer not to use it because it needs superuser permissions.

How is this useful

Well every client architecture is different. For instance, if the current architecture is configured to have a local DNS server for the backend servers (meaning every time that you request an url internally it will be routed directly to the backend server, bypassing the varnish server), then the only way to clear the cache is from the outside or by editing the /etc/hosts file on every backend server. Also I find PHP’s cURL implementation to be slow and very memory intensive when you have 4000+ urls.

VN:F [1.9.22_1171]
Rating: 10.0/10 (1 vote cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)
Trustworthy Bash to the rescue, 10.0 out of 10 based on 1 rating
 Trustworthy Bash to the rescue

About Luis Tineo

Husband, Father, performance improvement junkie, biker and video gamer, Linux user and in my day job I'm a Systems Architect at Blue Acorn.

Tags: , , ,