Magento developers Unleash the power of automation

  • Sumo

Magento developers unleash the power of automation

Deployments are always a pain, especially if there are holidays around because magento developers don’t usually unleash the power of automation even if there are tools in place to help them do so. This is not to say that everyone has to have the same workflow but I hope that with this example and pain of mine you can see why automating deployments is such a great idea.

Black Friday…the measuring stick

Black Friday came and went and left me exhausted and a bit disappointed because well some sites failed and by fail I don’t mean there were down 2 minutes but a bit over 10-15 minutes. Any other day that wouldn’t have been… acceptable but on Black Friday that is HELL!!! Good grief!!!

Normally you’d say Vagrant… except that we are talking about production sites. You could say: wield the power of GIT and revert to the latest commit and that would have been wonderful if the problem was so obvious.

Some of the cool things that helped me through such a wonderful time were 2 tools that led me to automate a couple of menial tasks while I occupied myself playing my CSI role researching why the sites were down to begin with. These tools made the top Magento developers tools which you can find magento developers toolboox v2.0 and Magento developers toolbox and of course I am talking about n98-magerun and magentify.

Whatever Christian Münch and Alistair Stead, which deserve all the kudos even for this post, were drinking I want some of that maybe it’d wake up my brain from its long slumber.

What’s so cool about these 2 tools?

Because of the clients confidentiality I cannot mention or reveal their name or their hosting providers, that is just not cool. So I will make up a story for it.

Imagine that you own a cool Gaming store and of course Sony awarded you with the best PS Vita deal in the market. Assassins Creed Liberation for $149.99, simply awesome. I am even online to get one for myself.

You’ve gotten yourself some beefy servers(8 in this case – 5 webs and 3 dbs in a cluster, you have 2 webs sharing resources with memcache and solr) plus because you heard Varnish Cache does wonders you decided to add 2 varnish servers to take care of the frontlines. We are using magento EE so we also got full page caching at the application level. You got a top-notch Magento hosting provider with all servers being dedicated. You don’t mind the costs because this holiday season may even set you for life.

You did your homework the best you could to prepare for the worse and more importantly you got some great talent taking care of the optimizations… let’s say Fabrizio, Colin and me (Colin and Fabrizio were not really involved – I use them here because they are extremely good with server optimization).

10 webservers to handle anything that comes your way and 3 performance junkies to get the most out them. You are set let’s get down to business.

So we used modman, some composer magic and even thew in Jam to handle dependencies for us.

Wow… so far so good.

We started using memcache so we could seamlessly integrate Nginx, varnish and magento because of some SSL issues. But planned on replacing it later with Redis.

The eye of the storm

We sieged the fort with 1000 concurrent users per second and not a server gets a load higher than 3… after all we used haproxy and stunnel to make sure we didn’t lose a single customer. Black Friday is here nothing out of the ordinary, we rolled out the last changes and test once more. Everything is good so we decided to get a few beers to celebrate and heck even play some arcade games in the process.

And just when I am about to start eating that Turkey Leg… customer service gets overloaded and we are told that some customers were losing their session state, losing their carts and we confirmed it. Good bye Turkey leg…

How did this happen? We were careful, methodical and even put Sherlock Holmes to shame because we were like surgeons at work with our laptops, combining magic and science with every keystroke.

We redeployed all files, while manually checking all local.xml, .htaccess files and clear cache everywhere. While checking the local.xml we noticed some servers weren’t configured to use memcache for session storage and some of them were not configured to pass the remote headers correctly. Bring on the nightmare, because we are lost.

After some digging we found out some servers had some hardware failure and the hosting provider restored the files as fast as they could from a backup except that we deployed these new changes today so they had an old version. We had 2 solutions start manually deploying the files, they are not in git or automate it. And even if you had them in git with a different extension you had to manually symlink them.

The devil is on the details

So that long story to say manually deploy some files and there was a hardware failure so you could see how these two tools could shine.

With Capistrano (magentify in our case) you can automate deployments and Alistair already has some magento related tasks. If you add n98-magerun well now you got everything you could hope for in a magento setup. So just for the sake of me adding some value to the tools let’s call their fusion capn98.

To use magentify, you’d do something like this in Ubuntu:

sudo gem install magentify

In magento’s root folder or somewhere else you want:

magentify .

You’d then edit your config/deploy file to match your requirements.

n98-magerun is a phar file… so you just need to make sure you have it somewhere you can use it globally.

Let’s see how we did this:

1-) Colin is tasked with creating templates for the configurations for varnish, mysql, nginx, memcache and rolling out a strategy to replace memcache with redis.
2-) Fabrizio is tasked with configuring Magentify to clear varnish, nginx and memcache
3-) I am tasked with installing n98-magerun in all webservers and creating a good local.xml and .htaccess file and place them in a good location, outside of the servers if possible. Configure our user to use sudo without password.

All this is going to happen while the site is live and not in maintenance mode.

Collin then gives me the changes for the local.xml to reflect the changes to redis. We add the modules to the repo and deploy:

cap deploy

The modules are in place but not active yet.

Customer services gets a new shipment of products so we need to reindex.

Once I am done and Collin is done (we got off easy here) – Fabrizio then does the following:

  task :refresh_servers, :roles => [:web, :app] do
    run "sudo ~/"
    run "sudo ~/"
    run "sudo ~/"
    run "cd #{current_path}/ && n98-magerun.phar cache:clean"
    run "cd #{current_path}/ && n98-magerun.phar index:reindex:all" #prob not smart to do this in 5 servers at the time, this is an example
  task :fix_local_files, :roles => [:web, :app] do
    run "cd #{current_path}/app/etc/ && rsync -avz -e ssh user@host:/~/path_to_file/ local.xml"
    run "cd #{current_path} && rsync -avz -e ssh user@host:/~/path_to_file/ .htaccess"
after   'deploy', 'mage:fix_local_files'
after   'deploy', 'mage:refresh_servers'

Then we went and got You, so you could do this:

cap deploy

and see how with that single command from your computer you’ve fixed 10 servers at the same time and took the technicality away from deployments.

But because the party isn’t over yet, we realized (just assume our user was deleted) that we are locked out of the admin, so I do the following:

  task :create_admin_user, :roles => [:web, :app] do
    run "cd #{current_path}/ && n98-magerun.phar admin:user:create demo password Demo DEmo"

And then:

cap mage:create_admin_user

And problems are solved.

You can see how simple combining magentify and n98-magerun is and how much they can help you.

So from now and on push your changes from the repository, select the branch you want to deploy from and simply do:

cap deploy

and make sure that the local.xml and .htaccess file are updated, the cache is cleared (you prob don’t want to do this on every deployment) and the site is re-indexed. Whatever task you may need simply add it to the hooks capistrano offers.

Now notice that the great thing here is that every magento developer can unleash and wield the power of automation by just using these 2 tools.

VN:F [1.9.22_1171]
Rating: 9.5/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)
Magento developers Unleash the power of automation, 9.5 out of 10 based on 2 ratings

Author: Luis Tineo

Husband, Father, performance improvement junkie, biker and video gamer, Linux user and in my day job I'm a Software Engineer at BuyerQuest.

Share This Post On
  • Aaron James

    Really entertaining reading here Luis :). The tooling in the Magento ecosystem has definitely matured over the past 18 months. The work Alistair has done on the testing and capistrano side, Christian and Alex with Magerun, really has made ‘mageops’ a lot easier than it was.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Thanks Aaron – I agree the MageOps are really making things easier even for the non tactical forces. This eventually will help the community so much because Magento work will be a lot easier!

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  • fhgh

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)