Saturday, July 28, 2012

Connect to a wireless network through the command line

Sometimes you decide that you don't want to use any GUI to connect to your wireless network, especially when it doesn't seem to work.


One of the main benefits of Linux and any other Unix based operating system is its ability to empower users to do things on their own.
So tonight when the latest Ubuntu updates broke my laptop's wireless, I decided to write a little script to fix it.


You will need to know the following:

  • ifconfig
  • iwlist
  • iwconfig
  • dhclient
Find your network interface, unless you know it (my laptop is always wlan0)
ifconfig
ifconfig NETWORK_ID up
Find the network you want to connect to:
iwlist network_interface scan

Let' configure your network card
iwconfig NETWORK_ID essi "NAME" key s:password
or
iwconfig network_interface essi "NAME" key hex_password
Now all we need is to get the IP from the DHCP server:
dhclient network_interface

You could have it all in a script:
#!/bin/bash
ifconfig wlan0 up
iwconfig wlan0 essi "$1" key s:$2
dhclient wlan0
Save it as wireless_up and then chmod a+x wireless_up.
Now you just need to either do ./wireless_up network_id my_password or bash wireless_up network_id my_password in the same directory. You could also move the script to a global location or add the directory where you have it as part of your local path. The choice is yours.

Friday, July 27, 2012

Dirt cheap way to make magento fly

There are several ways to accomplish the same task and some are more elegant than others.
This one makes any Magento community edition fly and is cheap and quick (read no Magento compliant - because it modifies the templates)
In other words while this trick can help you boost your Magento store server response I am not too proud of it.

Why is that a big deal? I am not a huge fan or changing templates, even if they are "allowed".
But let's forget about that and get down to business
Your actual cache helper class
class Performance_CacheHelper_Helper_Data extends Mage_Core_Helper_Data {

    //###########################################
    //Cache related functions and variables
    //###########################################
    /**
     * cache key
     * @var string|mixed 
     */
    protected $key;

    /**
     *
     * @var Mage_Core_Model_Cache 
     */
    protected $_cache;

    /**
     * data container
     * @var string|mixed 
     */
    protected $data;
    protected $tags = array(__CLASS__);

    public function canUseCache() {
        //verify if the cache is enabled
        return Mage::app()->useCache('block_html');
    }

    /**
     * Gets the cache object
     * @return Mage_Core_Model_Cache 
     */
    protected function _getCacheObject() {
        if (!$this->_cache) {
            $this->_cache = Mage::app()->getCache();
        }
        return $this->_cache;
    }

    public function getData() {
        return $this->data;
    }

    public function setData($data) {
        $this->data = $data;

        return $this;
    }

    public function getKey() {
        return $this->key;
    }

    public function setKey($key) {
        $this->key = $key;
        return $this;
    }

    /**
     * saves data in a serialize format to cache under the name of this class if the cache can be used
     * @return Performance_CacheHelper_Helper_Data 
     */
    public function saveDataInCache() {
        if ($this->canUseCache()) {
            $cookie = Mage::getModel('core/cookie');
            $this->_getCacheObject()->save(
                    serialize($this->data), $this->key, $this->tags, $cookie->getLifetime());
        }
        return $this;
    }

    /**
     *
     * @param string $key
     * @return mixed
     */
    public function getDataFromCache($key) {
        $this->key = $key;
        return $this->_getCachedData();
    }

    /**
     * gets the data saved in cache, if it finds it then it unserializes itF
     * @param string|mixed $key
     * @return bool | string
     */
    protected function _getCachedData($key = null) {
        if ($key !== null) {
            $this->key = $key;
        }
        /**
         * clear the data variable 
         */
        $this->data = false;
        //ensure cache can be used
        if ($data = $this->_getCacheObject()->load($this->key)) {
            $this->data = unserialize($data);
        }
        return $this->data;
    }

}

In your current template:
                
         //this code has been modified so the highligthing works right                   
        $helper = Mage::helper('cachehelper');
        $cachingEnabled = array('catalog');
        $key = md5(Mage::helper('core/url')->getCurrentUrl());
        $currentModule = Mage::app()->getRequest()->getModuleName();
        if (in_array($currentModule, $cachingEnabled)){
         if ($content = $helper->getDataFromCache($key)){
         //nothing do $content now has the data
         }else{
             $content = $this->getChildHtml('content');
             $helper->setKey($key)->setData($content)->saveDataInCache();
            
        }else{
         $content = $this->getChildHtml('content');
        }

  
siege -t1m -c5 -b -d -i http://mage.dev/apparel
Examples with File Based cache
Without the "dirt cheap trick"
Transactions: 324 hits
Availability: 100.00 %
Elapsed time: 59.94 secs
Data transferred: 2.13 MB
Response time: 0.92 secs
Transaction rate: 5.41 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 4.95
Successful transactions: 324
Failed transactions: 0
Longest transaction: 2.32
Shortest transaction: 0.63

With the "dirt cheap trick"
Transactions: 475 hits
Availability: 100.00 %
Elapsed time: 59.56 secs
Data transferred: 3.14 MB
Response time: 0.62 secs
Transaction rate: 7.98 trans/sec
Throughput: 0.05 MB/sec
Concurrency: 4.97
Successful transactions: 475
Failed transactions: 0
Longest transaction: 1.45
Shortest transaction: 0.37

Examples with memcache
Without the "dirt cheap trick"
Transactions: 329 hits
Availability: 100.00 %
Elapsed time: 59.65 secs
Data transferred: 2.17 MB
Response time: 0.90 secs
Transaction rate: 5.52 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 4.95
Successful transactions: 329
Failed transactions: 0
Longest transaction: 1.60
Shortest transaction: 0.68

With the "dirt cheap trick"
Transactions: 507 hits
Availability: 100.00 %
Elapsed time: 59.47 secs
Data transferred: 3.35 MB
Response time: 0.58 secs
Transaction rate: 8.53 trans/sec
Throughput: 0.06 MB/sec
Concurrency: 4.97
Successful transactions: 507
Failed transactions: 0
Longest transaction: 1.91
Shortest transaction: 0.40

Pretty awesome right? Don't think siege is a good testing tool and this is not a good indication? What about jmeter?
Here is the test plan I used:
      
         
            
               
               false
               false
               
                  
               
               
            
            
               
                  
                     false
                     1
                  
                  10
                  1
                  1340804136000
                  1340804136000
                  false
                  continue
                  
                  
               
               
                  
                     
                        
                     
                     mage.dev
                     
                     
                     
                     
                     
                     /apparel
                     GET
                     false
                     true
                     true
                     false
                     
                     
                     
                     false
                     
                  
                  
                     
                        
                           
                              Accept-Language
                              en-us,en;q=0.5
                           
                           
                              Accept
                              text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                           
                           
                              User-Agent
                              Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
                           
                           
                              Accept-Encoding
                              gzip, deflate
                           
                        
                     
                     
                  
                  
                     
                        
                     
                     mage.dev
                     
                     
                     
                     
                     
                     /furniture/living-room
                     GET
                     false
                     true
                     true
                     false
                     
                     
                     
                     false
                     
                  
                  
                     
                        
                           
                              Accept-Language
                              en-us,en;q=0.5
                           
                           
                              Accept
                              text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                           
                           
                              User-Agent
                              Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
                           
                           
                              Accept-Encoding
                              gzip, deflate
                           
                        
                     
                     
                  
                  
                     
                        
                     
                     mage.dev
                     
                     
                     
                     
                     
                     /furniture/bedroom
                     GET
                     false
                     true
                     true
                     false
                     
                     
                     
                     false
                     
                  
                  
                     
                        
                           
                              Accept-Language
                              en-us,en;q=0.5
                           
                           
                              Accept
                              text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                           
                           
                              User-Agent
                              Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
                           
                           
                              Accept-Encoding
                              gzip, deflate
                           
                        
                     
                     
                  
               
               
                  false
                  
                     saveConfig
                     
                        
                        true
                        true
                        true
                        
                        true
                        true
                        true
                        true
                        false
                        true
                        true
                        false
                        false
                        true
                        false
                        false
                        false
                        false
                        false
                        0
                        true
                     
                  
                  
               
               
            
         
      
      

Here are the results, sorry about the formatting:


Without the "dirt cheap trick"

sampler_label

aggregate_report_count

average

aggregate_report_min

aggregate_report_max

aggregate_report_stddev

aggregate_report_error%

aggregate_report_rate

aggregate_report_bandwidth

average_bytes

/apparel

10

1710

1274

2180

290.2423125597

0

4.5724737083

199.8358553384

44753

/furniture/living-room

10

896

618

1272

254.8664159908

0

7.5757575758

227.1173650568

30699

/furniture/bedroom

10

747

565

953

137.6

0

7.3637702504

204.8623895434

28488

TOTAL

30

1117

565

2180

484.8805133455

0

7.2904009721

246.6680589307

34646.6666666667


With the "dirt cheap trick"

sampler_label

aggregate_report_count

average

aggregate_report_min

aggregate_report_max

aggregate_report_stddev

aggregate_report_error%

aggregate_report_rate

aggregate_report_bandwidth

average_bytes

/apparel

10

675

434

904

147.3737086457

0

6.1199510404

268.3156269125

44895

/furniture/living-room

10

741

581

950

107.7506844526

0

5.4436581383

164.0678160724

30862.6

/furniture/bedroom

10

631

389

827

157.5221889132

0

7.3691967576

206.1388863301

28644.4

TOTAL

30

682

389

950

146.3524057427

0

10.8577633008

369.0013685306

34800.6666666667


As you can see this is trick helps a lot but it doesn't compete with Magento's FPC or Varnish and it still needs lots of polishing. The good think is that is doesn't require holepunching or lots of Magento knowledge. Now a few pointers... this is not the full implementation, as this code doesn't work with load balancers and there is not a way to clear it here. But implementing both is trivial.

Thursday, July 26, 2012

PHP Offloading... Nginx helps so much!

So let's continue our attempt to improve Magento's performance and this time around let's focus on the web server itself.
There are a bunch of web servers out there but the most known are, a least to me: Apache, Nginx and Lighttpd
My favorite, and of course I am biased here, is Nginx and it approach to serving content. As you may all know Nginx is an event-based web server and the main advantage of this asynchronous approach is scalability. 
Because at the end of the day what we are going to have is a complete static page (after it has been cached once) being served from the web server.

Apache is out of the question because nginx is faster at serving static files and consumes much less memory for concurrent requests. As Nginx is event-based it doesn't need to spawn new processes or threads for each request, so its memory usage is very low and that is the key. 

We could use lighttpd (it's even more compatible with apache) but the cpu and memory fingerprint is bigger than nginx and we are all about performance aren't we?

So here are my recommended configurations for both Nginx and fast-cgi, of course suggestions are welcome:
server {
    listen 80 default;
    listen 443 default ssl;
    ssl_certificate /etc/pki/tls/certs/2014-www.example.com.pem;
    ssl_certificate_key /etc/pki/tls/certs/2014-www.example.com.pem;
    server_name www.example.com example.com; ## Domain is here twice so server_name_in_redirect will favour the www
    root /var/www/vhosts/example.com;
 
    location / {
        index index.html index.php;
        try_files $uri $uri/ @handler; 
        expires 30d; ## Assume all files are cachable
    }
 
    ## These locations would be hidden by .htaccess normally
    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }
 
#    location /var/export/ { 
#        auth_basic           "Restricted"; ## Message shown in login window
#        auth_basic_user_file htpasswd; ## See /etc/nginx/htpassword
#        autoindex            on;
#    }
 
    location  /. { ## Disable .htaccess and other hidden files
        return 404;
    }
 
    location @handler { ## Magento uses a common front handler
        rewrite / /index.php;
    }
 
    location ~ .php/ { ## Forward paths like /js/index.php/x.js to relevant handler
        rewrite ^(.*.php)/ $1 last;
    }
 
    location ~ .php$ { ## Execute PHP scripts
        if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss
 
        expires        off; ## Do not cache dynamic content
 fastcgi_read_timeout 120;
 fastcgi_buffer_size 128k;
 fastcgi_buffers 4 256k;
 fastcgi_busy_buffers_size 256k;
        fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_param  HTTPS $fastcgi_https;
        fastcgi_param  PHP_VALUE "memory_limit = 341M";
 fastcgi_param  PHP_VALUE "max_execution_time = 18000";
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
        fastcgi_param  MAGE_RUN_TYPE store;
        #fastcgi_param  MAGE_IS_DEVELOPER_MODE true; //enable developer mode?
        include        fastcgi_params; ## See /etc/nginx/fastcgi_params
    }
}
; Start a new pool named 'www'.
[www]

listen = /var/run/php-fpm/php-fpm.sock
user = apache
group = apache
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 50
pm.max_requests = 500

pm.status_path = /phpfpm-status
request_terminate_timeout = 5m
request_slowlog_timeout = 2m
slowlog = /var/log/php-fpm/www-slow.log

php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on

Contents of /etc/nginx/fastcgi_params

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

Tuesday, July 24, 2012

PHP offloading for better performance

So you recently launched your PHP application (read Magento) just to realize that it is really slow even though you have state of the art hardware, software and it is configured to kick ass. You decided to install varnish to help full page caching and your CDN, which it only has gzipped merged and minified assets, to serve content faster now your site seems to be flying but New Relic still reveals so many bottlenecks specially for the fillers you needed to keep some dynamic content rocking.

You decide to run promotion that would bring 20K plus customers and decided to add more servers, virtual machines and the cloud can be so awesome. Soon to realize that your site is now really slow specially when checking out and some time later your site is DOWN!!! WHY!?!!!
Because your reverse proxy is not serving all of the customers let alone is not serving the whole content from cache as it has to fetch some from the server, more on this later. So believe it or not, your server is still getting hammered and might be even worse than before.

Reverse proxies are really awesome but if applied to a inefficient application, it will come back to hunt you and be your worst nightmare.
Caching is supposed to be good but nowadays it has become the best way to hide all of the snafus and foobars in the application.

So what is the problem? For once almost everyone does development one way through the programming language they know and rarely, if any, venture or dare to think of a possible world outside of that realm. Aside from the copy paste dogma of course. 
One problem is that most people tend to try to keep the dynamic content in the page through a hybrid combination of fresh content (retrieved from the server) and cached content. Most caching systems have developed sophisticated algorithm to make it work which adds a new layer of complexity.
Let's go back to your PHP application, one of the areas that usually needs to be hole-punched is the header because it contains the cart widget, the account (log-in or out) and wishlist links. But do we really need to hole-punched it? Nop!

PHP offloading is a very useful technique to improve performance, it refers to efficently distribute the workload among all the servers or other programming languages in the sytem. Also known as Operation Driven Development (ODD) where the type of operation dictates in which server or programming language the task it will take place.

So let's take the cart widget and see how we could do it without having to even think about hole-punching:

Every time an operation that involves the cart object takes place, this one gets saved. This means when you remove, add or update an item in your shopping cart you get this event checkout_cart_save_after so why don't we create a cookie that contains off all the items information?

Snippets below, I assume you do Magento development:


In your module etc/config.xml add:
 
        
            
                
                    
                        phpoffloading/observer
                        cartUpdate
                    
                
            
        
        
            
                
                    phpoffloading.xml
                
            
        
    
Replace phpoffloading with the name of your model class.
Now in your observer add:
    /**
     * Process all the cart / quote updates to ensure we update the cookie correctly
     * @param type $observer 
     */
    public function cartUpdate($observer) {
        // using singleton instead of the event to use the same function for different events
        $cart = Mage::getSingleton('checkout/cart');
        $quote = $cart->getQuote();

        /**
         * @var $totals needed for the subtotal 
         */
        $totals = $quote->getTotals();
        /**
         * @var $checkouthelper used for translation and price formatting 
         */
        $checkoutHelper = Mage::helper('checkout');
        $items = (int) $cart->getSummaryQty();
        if ($items > 0) {
            $cartInfo = array(
                'subtotal' => $checkoutHelper->formatPrice($totals['subtotal']->getValue()),
                'items' => $items,
                'itemsmsg' => $items == 1 ? "item" : "items",
                'viewcart' => $checkoutHelper->__('View Cart')
            );
            $cartInfo = json_encode($cartInfo);
             $this->setCookie('cart_cookie', $cartInfo);
        } else {
            $this->deleteCartCookie();
        }
        return $this;
    }
In your template:
You have no items in your cart.
You can have this javascript either on the same template or add it as an external file:

Every time a page gets loaded now instead of going to the server the content will remain in the browser and you don't need to request content from the server again. This is a very small change how much improvement do I get? Say you go from 750ms to 2ms in the server response time what would you say? This is because varnish 1-) doesn't need to understand what ESI is and 2-) all of the content is served from cache. This doesn't apply solely to varnish but to Magento's own full page caching solution.


This approach works fine for: wishlist, logged in/out but it doesn't work quite well with recently viewed items. For that we'll do a follow up article.

Monday, July 23, 2012

FireFox vs Virtualized environments

Firefox rocks no doubt about it... but using more memory than virtualbox just to run magento is not cool at all!

This is why I really dislike using FireFox and love using SwiftFox. How come Firefox is taking 2GB of ram, that is as much as my virtualized Ubuntu Desktop and more than my virtualized ubuntu server. In other words with that much memory I could be running a few more computers.


Swiftfox is a highly optimized firefox which aims to be lightweight while still being robust, powerful and most importantly, compatible with all of the FifeFox addons.