Working with Stunnel, Varnish, Nginx and Node.js

  • SumoMe

Let’s have some fun working with Stunnel, Varnish, Nginx and Node.js with socket.io

Overview

At one point I was thinking we could use node.js to serve static content, an engineered solution for a simple problem.

Because as we have done in the past we can implement full page caching with nginx very easy and varnish essentially serves static content.

We’ll be making use of Stunnel, Varnish, and Nginx in addition to Node.js with socket.io.
Varnish has all sorts of useful caching functions, as we have seen before.

So just for the fun of it let’s do this super and over architected setup, here is what need and how we are going to use it:

Stunnel listens on port 443. It terminates SSL connections and passes traffic to Varnish on port 80.
Varnish listens on port 80 and splits the traffic to Nginx on 8080 and Node.js on port 8081.

This arrangement will be created on an Ubuntu 12.04 server using vagrantup and chef – you will have to adjust package installation and configuration file locations accordingly if working with another branch of Unix.

Use Vagrant

You know vagrant is part of my toolbox as a magento developer and this post will show you why.

For those who haven’t installed it already, let’s install vagrant first:

apt-get install virtualbox-ose
apt-get install rubygems
gem install vagrant

Let’s get the ubuntu boxes:

vagrant box add precise64 http://files.vagrantup.com/precise64.box
vagrant init precise64
vagrant up

Essentially that means get the preconfigured box precise64.bx and download it.

To install chef in Ubuntu I recommend this tutorial because if I could understand it, well anyone can.

Showing how to create chef receipes and using them with Vagrant requires more than just a paragraph so to configure the receipes this tutorial is pretty neat. And just to use a fancy word… well it is out of “scope”

Install the Packages manually

In case you don’t want to use vagrant at first, so it can install everything for you, here is how we can do it.

We’ll install Node.js – but I want to assume you can set up to run as a service on port 8081 – and dive right into the rest of the setup.

First add the Node.js repository to Ubuntu

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update

Now let’s install all of our required packages at once:

sudo apt-get install nodejs npm stunnel varnish nginx

Install socket.io

npm install socket.io

Configure Stunnel

Create a new configuration file https-proxy.conf in the /etc/stunnel/ directory:
each separate *.conf file that directory will be used as the basis for a separate Stunnel process when the Stunnel service starts.

cert = /etc/ssl/example.pem
[https]
accept  = 443
connect = 80

One important part to note is that your SSL key and certificate must be concatenated into a single file. The order is important: key then certificate. So example.pem is actually:

cat private/ssl-cert-example.key certs/ssl-cert-example.pem >example.pem

Now let’s make that Stunnel starts at boot

vim /etc/default/stunnel4
# Change to one to enable stunnel automatic startup
ENABLED=1

Configure Varnish

Alter these lines in /etc/default/varnish to tell Varnish to run on port 80, which we don’t need if we want varnish in switch mode, rather than the default 6081, you will see varnish has different ways to do this my favorite is this one:

## Alternative 3, Advanced configuration
#
# See varnishd(1) for more information.
#
# # Main configuration file. You probably want to change it :)
VARNISH_VCL_CONF=/etc/varnish/example_varnish.vcl
#
# # Default address and port to bind to
# # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
# # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80
#
# # Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
#
# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=5
#
# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=10
#
# # Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120
#
# # Cache file location
VARNISH_STORAGE_FILE=/var/lib/varnish/$INSTANCE/varnish_storage.bin
#
# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=1G
#
# # File containing administration secret
VARNISH_SECRET_FILE=/etc/varnish/secret
#
# # Backend storage specification
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
#
# # Default TTL used when the backend does not specify one
VARNISH_TTL=1w
#
# # DAEMON_OPTS is used by the init script.  If you add or remove options, make
# # sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -t ${VARNISH_TTL} \
             -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
             -s ${VARNISH_STORAGE} \
             -S ${VARNISH_SECRET_FILE}"

The following is an example configuration file to replace the default /etc/varnish/default.vcl, which we renamed example_varnish.vcl in case you are running multiple varnish instances.

It is kept deliberately short and simple:

include  "/etc/varnish/default.d/backends.setup.vcl";
sub vcl_recv {
	//set the backends
	include "/etc/varnish/default.d/set.backend.vcl";
    #the rest of configuration here
}
Contents of include “/etc/varnish/default.d/set.backend.vcl”;
  set req.backend = default;
  set req.grace = 30s;
  # Pass the correct originating IP address for the backends
  if (req.restarts == 0) {
    if (req.http.X-Forwarded-For) {
      set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
      set req.http.X-Forwarded-For = client.ip;
    }
  }
  set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
  if (req.http.Upgrade ~ "(?i)websocket") {
    set req.backend = node;
  }
  else if (req.url ~ "^/socket.io") {
    set req.backend = node;
  }
  else if (req.url ~ "^/another_node_path/") {
    set req.backend = node;
  }
Contents of include “/etc/varnish/default.d/backends.setup.vcl”;
# Nginx.
backend default {
  .host = "127.0.0.1";
  .port = "8080";
  .connect_timeout = 5s;
  .first_byte_timeout = 15s;
  .between_bytes_timeout = 15s;
  .max_connections = 400;
}
# Node.js.
backend node {
  .host = "127.0.0.1";
  .port = "8081";
  .connect_timeout = 1s;
  .first_byte_timeout = 2s;
  .between_bytes_timeout = 15s;
  .max_connections = 400;
}

Note that this is a fairly simple usage of Varnish. And it is assumed the reader will add the remaining configurations for varnish and their needs.

Configure Nginx

The assumption is that Nginx will be set up here to serve both normal website content but not static content, we also want some sort of caching from nginx as well.

So visit this tutorial for more on nginx and full page caching.

Restart Everything and Test

Create a little bash script:

vim restart
#!/bin/bash
if [[ $EUID -ne 0 ]]; then
  echo "You must be a root user" 2>&1
  exit 1
else
  service stunnel restart
  service varnish restart
  service nginx restart
  service php-fpm restart
  service memcached restart
fi

Make that script executable:

chmod +x restart

Restart the services after you are done with configuring your services:

sudo ./restart

Conclusion

Now test your Node.js / Socket.IO application – it should work just fine.

This may look over engineered and most likely it is… however this was a fun project and believe me this taught me some cool tricks about stunnel and nginx.

You can accomplish all this just using nginx and varnish just help with loads, at the end of the day you will find yourself knowing at lot more.

A side effect is that you can start serving pages with node.js and socket.io and utilize all of the resources at hand. In other words your JavaScript developers won’t need to learn PHP anymore 😀

Also if you happen to be using Apache2 and you have static content or are planning on it, this might be easier:

In your .htaccess file:

RewriteCond %{REQUEST_METHOD} !POST
RewriteCond %{DOCUMENT_ROOT}/static/%{SERVER_NAME}/$1/index-https.html -f
RewriteRule ^(.*) "/static/%{SERVER_NAME}/$1/index-https.html" [L]

What do you think?

VN:F [1.9.22_1171]
Rating: 9.0/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)
Working with Stunnel, Varnish, Nginx and Node.js, 9.0 out of 10 based on 2 ratings

Author: Luis Tineo

Husband, Father, performance improvement junkie, biker and video gamer, Linux user and in my day job I'm a Systems Architect at Blue Acorn.

Share This Post On
  • Gautham Ganapathy

    I tried this out with apache instead of nginx. However, in Chrome (v23) and Firefox (v18), I got the problem where, due to connection reuse, the browser sent non-websocket meant for apache to node using the same TCP connection used for the websocket connection. Any idea on how to get around this? Setting the websocket connection to (pipe) does not help either. Neither did http://www.exratione.com/2012/08/websockets-over-ssl-stunnel-varnish-nginx-nodejs/

    Cheers
    Gautham

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • letas

      I’ve never seen this problem before, anything in the logs that you could share, usually it is a lot simpler to troubleshoot if you follow the trace of the request,

      Regards,

      Luis

      VA:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  • http://twitter.com/argyleink Adam Argyle

    Love it! I haven’t tried any caching techniques with Nginx yet. I’m going to use this post as a base for a vagrant base I want in my toolbelt. Thanks for sharing!

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • letas

      You are welcome – I am glad I could help!

      VA:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)