Software Engineering Consulting & Managed Information Systems. Experts in Ruby on Rails web development.

AWS Elasticbeanstalk makes it very simple to create an environment with load balanced instances running a web server proxy in front of your custom app. This is as easy as a couple of clicks from the web interface, or just eb create & eb deploy from the Elastic Beanstalk Command Line Interface (EB CLI).

As long as you aren’t serving WebSocket, this all just works right out of the box without any custom configuration. That is a pretty powerful ops setup for a couple of commands! However, if you want to serve WebSocket traffic, you’ll need to edit the configuration of your proxy server (Nginx in my case) to allow the http upgrade.

Previously, you also had to configure you Elastic Load Balancer to get around the fact that it didn’t really support WebSockets. But, the new Application Load Balancers (now available on Elasticbeanstalk environments) have native support for WebSockets.

But, to support WebSockets between your Apache/Nginx Web server proxy and your app, you might need to edit your Web server’s configuration file based upon what Elasticbeanstalk provides as a default template. At least this was the case for me using the provided default Nginx configuration.

The popular posts I found floating around on how to modify your Elasticbeanstalk Nginx configuration suggest using the .ebextensions (as described in my last post) with a files section to create a custom Nginx.conf file that ends up completely replacing the basic Nginx configuration that Elasticbeanstic provides in 00_elastic_beanstalk_proxy.conf.

This approach amounts to creating a config file in .ebextensions with content like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38 files:
  "/etc/nginx/conf.d/01_nginx_websocket.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      upstream nodejs {
        server 127.0.0.1:8081;
        keepalive 256;
      }

      server {
        listen 8080;

        if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
            set $year $1;
            set $month $2;
            set $day $3;
            set $hour $4;
        }
        access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
        access_log  /var/log/nginx/access.log  main;

        location / {
          proxy_pass  http://nodejs;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
          proxy_http_version 1.1;
          proxy_set_header  Host  $host;
          proxy_set_header  X-Real-IP $remote_addr;
          proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        }

      gzip on;
      gzip_comp_level 4;
      gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

      }

..and then a commands section to either remove or rename the auto generated 00_elastic_beanstalk_proxy.conf file so that the new file is used instead.

While this approach works fine, the problem I have is that it amounts to including a whole new copy of an Nginx configuration, hardcoded into a separate custom configuration section and ignoring what AWS Elasticbeanstalk provides as a template. Essentially, the only thing we really need to replace is proxy_set_header Connection ""; in the default 00_elastic_beanstalk_proxy.conf with lines #26-27 from above.

Whenever I make configuration changes to default provided files, like the Elasticbeanstalk provided 00_elastic_beanstalk_proxy.conf, I prefer an approach that respects future changes that Elasticbeanstalk might make to its default configuration decisions. In other words, if we are replacing the entire file with our very slightly modified version, we prevent our instances from ever getting changes that AWS might decide are important for its Elasticbeanstalk setups. This might be things related to default interaction between its load balancers and web servers, time outs, health checks, etc.

If possible, I prefer an approach that respects future changes automatically instead having to remember a year or two from now that “oh yeah, I’m replacing that file on every deployment and that’s why my server isn’t automatically working with those new load balancers.”

The approach I decided to take was figuring out how to use the default 00_elastic_beanstalk_proxy.conf file as is, but just replace the:

proxy_set_header Connection "";

with what I needed to allow WebSockets (i.e. HTTP upgrades):

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

First, just to make sure I was on the right path, I ssh’d into my instance and manually edited the 00_elastic_beanstalk_proxy.conf file, restated Nginx and sure enough WebSockets connections were flowing. But, I discovered that upon next deployment my changes to 00_elastic_beanstalk_proxy.conf were overwritten.

After a little hunting around, I discovered that Elasticbeanstalk uses templates located in: /tmp/deployment/config/ to build their default configuration files. In this case, the template I was looking at was: #etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf

Sure enough, right in the header of that file, it gives me a tip on how to proceed:

1
2
3
4
5
6
7 # Elastic Beanstalk managed configuration file
# Some configuration of nginx can be by placing files in /etc/nginx/conf.d
# using Configuration Files.
# http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers.html
#
# Modifications of nginx.conf can be performed using container_commands to modify the staged version
# located in /tmp/deployment/config/etc#nginx#nginx.conf

According to the comment, it seems I should use container_commands in order to replace the standard proxy_set_header Connection ""; in the /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf with the couple of lines I need for for WebSockets.

To do that, I created a file called nginx_websocket_upgrade.conf (name was arbitrary) in .ebextensions with the following content:

1
2
3
4
5
6
7 container_commands:
  01_nginx_websocket_support:
    command: |
      sed -i '/\s*proxy_set_header\s*Connection/c \
              proxy_set_header Upgrade $http_upgrade;\
              proxy_set_header Connection "upgrade";\
          ' /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf

And on the next eb deploy WebSocket connections were flowing!

To verify that it was actually the result of this change that allowed WebSockets to start flowing, I did a quick eb ssh and took a look at /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf, and verified that it contained the changes that the new container_command made to the template file.

Hat tip on getting the sed command right by finding this gist here: https://gist.github.com/adamgins/f99635447a1239289460 that included the syntax to replace the respective proxy_set_header.