Jekyll2016-10-26T16:05:04-04:00https://scopestar.com/blog//Scopestar BlogWe design, build and support Custom Software, provide Software Engineering Consulting Services, and manage Cloud Information Systems.
Enable Websockets on Elasticbeanstalk Nginx Proxy2016-10-21T00:00:00-04:002016-10-21T00:00:00-04:00https://scopestar.com/blog/aws/elasticbeanstalk/websockets/2016/10/21/enable-websockets-on-elasticbeanstalk-nginx-proxy<p>AWS Elasticbeanstalk makes it very simple to create an environment with load balanced instances running a web server proxy in front of your custom app. This is as easy as a couple of clicks from the web interface, or just <code class="highlighter-rouge">eb create</code> & <code class="highlighter-rouge">eb deploy</code> from the <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3.html">Elastic Beanstalk Command Line Interface (EB CLI)</a>.</p>
<p>As long as you aren’t serving <a href="/blog#WebSockets">WebSocket</a>, this all just works right out of the box without any custom configuration. That is a pretty powerful ops setup for a couple of commands! However, if you want to serve WebSocket traffic, you’ll need to edit the configuration of your proxy server (Nginx in my case) to allow the http upgrade.</p>
<p>Previously, you also had to configure you <a href="https://aws.amazon.com/elasticloadbalancing/classicloadbalancer/">Elastic Load Balancer</a> to get around the fact that it didn’t really support WebSockets. But, the new <a href="https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/">Application Load Balancers</a> (now available on Elasticbeanstalk environments) have native support for WebSockets.</p>
<p>But, to support WebSockets between your Apache/Nginx Web server proxy and your app, you might need to edit your Web server’s configuration file based upon what Elasticbeanstalk provides as a default template. At least this was the case for me using the provided default Nginx configuration.</p>
<p>The popular posts I found floating around on how to modify your Elasticbeanstalk Nginx configuration suggest using the .ebextensions (as described in my last <a href="/blog/aws/elasticbeanstalk/2016/10/16/installing-zeromq-on-elasticbeanstalk.html">post</a>) with a <code class="highlighter-rouge">files</code> section to create a custom Nginx.conf file that ends up completely replacing the basic Nginx configuration that Elasticbeanstic provides in <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code>.</p>
<p>This approach amounts to creating a config file in <code class="highlighter-rouge">.ebextensions</code> with content like:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38</pre></td><td class="code"><pre><span class="s">files</span><span class="pi">:</span>
<span class="s2">"</span><span class="s">/etc/nginx/conf.d/01_nginx_websocket.conf"</span><span class="pi">:</span>
<span class="s">mode</span><span class="pi">:</span> <span class="s2">"</span><span class="s">000644"</span>
<span class="s">owner</span><span class="pi">:</span> <span class="s">root</span>
<span class="s">group</span><span class="pi">:</span> <span class="s">root</span>
<span class="s">content</span><span class="pi">:</span> <span class="pi">|</span>
<span class="no">upstream nodejs {</span>
<span class="no">server 127.0.0.1:8081;</span>
<span class="no">keepalive 256;</span>
<span class="no">}</span>
<span class="no">server {</span>
<span class="no">listen 8080;</span>
<span class="no">if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {</span>
<span class="no">set $year $1;</span>
<span class="no">set $month $2;</span>
<span class="no">set $day $3;</span>
<span class="no">set $hour $4;</span>
<span class="no">}</span>
<span class="no">access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;</span>
<span class="no">access_log /var/log/nginx/access.log main;</span>
<span class="no">location / {</span>
<span class="no">proxy_pass http://nodejs;</span>
<span class="no">proxy_set_header Upgrade $http_upgrade;</span>
<span class="no">proxy_set_header Connection "upgrade";</span>
<span class="no">proxy_http_version 1.1;</span>
<span class="no">proxy_set_header Host $host;</span>
<span class="no">proxy_set_header X-Real-IP $remote_addr;</span>
<span class="no">proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span>
<span class="no">}</span>
<span class="no">gzip on;</span>
<span class="no">gzip_comp_level 4;</span>
<span class="no">gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;</span>
<span class="no">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></figure>
<p>..and then a <code class="highlighter-rouge">commands</code> section to either remove or rename the auto generated <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code> file so that the new file is used instead.</p>
<p>While this approach works fine, the problem I have is that it amounts to including a whole new copy of an Nginx configuration, hardcoded into a separate custom configuration section and ignoring what AWS Elasticbeanstalk provides as a template. Essentially, the only thing we really need to replace is <code class="highlighter-rouge">proxy_set_header Connection "";</code> in the default <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code> with <code class="highlighter-rouge">lines #26-27</code> from above.</p>
<p>Whenever I make configuration changes to default provided files, like the Elasticbeanstalk provided <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code>, I prefer an approach that respects future changes that Elasticbeanstalk might make to its default configuration decisions. In other words, if we are replacing the entire file with our very slightly modified version, we prevent our instances from ever getting changes that AWS might decide are important for its Elasticbeanstalk setups. This might be things related to default interaction between its load balancers and web servers, time outs, health checks, etc.</p>
<p>If possible, I prefer an approach that respects future changes automatically instead having to remember a year or two from now that “oh yeah, I’m replacing that file on every deployment and that’s why my server isn’t automatically working with those new load balancers.”</p>
<p>The approach I decided to take was figuring out how to use the default <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code> file as is, but just replace the:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>proxy_set_header Connection "";
</code></pre>
</div>
<p>with what I needed to allow WebSockets (i.e. HTTP upgrades):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
</code></pre>
</div>
<p>First, just to make sure I was on the right path, I ssh’d into my instance and manually edited the <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code> file, restated Nginx and sure enough WebSockets connections were flowing. But, I discovered that upon next deployment my changes to <code class="highlighter-rouge">00_elastic_beanstalk_proxy.conf</code> were overwritten.</p>
<p>After a little hunting around, I discovered that Elasticbeanstalk uses templates located in: <code class="highlighter-rouge">/tmp/deployment/config/</code>
to build their default configuration files. In this case, the template I was looking at was: <code class="highlighter-rouge">#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf</code></p>
<p>Sure enough, right in the header of that file, it gives me a tip on how to proceed:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1
2
3
4
5
6
7</pre></td><td class="code"><pre><span class="c1"># Elastic Beanstalk managed configuration file</span>
<span class="c1"># Some configuration of nginx can be by placing files in /etc/nginx/conf.d</span>
<span class="c1"># using Configuration Files.</span>
<span class="c1"># http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers.html</span>
<span class="c1">#</span>
<span class="c1"># Modifications of nginx.conf can be performed using container_commands to modify the staged version</span>
<span class="c1"># located in /tmp/deployment/config/etc#nginx#nginx.conf</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></figure>
<p>According to the comment, it seems I should use <code class="highlighter-rouge">container_commands</code> in order to replace the standard <code class="highlighter-rouge">proxy_set_header Connection "";</code> in the <code class="highlighter-rouge">/tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf</code> with the couple of lines I need for for WebSockets.</p>
<p>To do that, I created a file called <code class="highlighter-rouge">nginx_websocket_upgrade.conf</code> (name was arbitrary) in <code class="highlighter-rouge">.ebextensions</code> with the following content:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1
2
3
4
5
6
7</pre></td><td class="code"><pre><span class="ss">container_commands:
</span><span class="mo">01</span><span class="ss">_nginx_websocket_support:
command: </span><span class="o">|</span>
<span class="n">sed</span> <span class="o">-</span><span class="n">i</span> <span class="s1">'/\s*proxy_set_header\s*Connection/c \
proxy_set_header Upgrade $http_upgrade;\
proxy_set_header Connection "upgrade";\
'</span> <span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">deployment</span><span class="o">/</span><span class="n">config</span><span class="o">/</span><span class="c1">#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></figure>
<p>And on the next <code class="highlighter-rouge">eb deploy</code> WebSocket connections were flowing!</p>
<p>To verify that it was actually the result of this change that allowed WebSockets to start flowing, I did a quick <code class="highlighter-rouge">eb ssh</code> and took a look at <code class="highlighter-rouge">/etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf</code>, and verified that it contained the changes that the new <code class="highlighter-rouge">container_command</code> made to the template file.</p>
<p>Hat tip on getting the sed command right by finding this gist here: https://gist.github.com/adamgins/f99635447a1239289460 that included the syntax to replace the respective proxy_set_header.</p>{"nick_name"=>"Scott", "full_name"=>"Scott Stewart", "avatar"=>"https://s.gravatar.com/avatar/7d92081717240f3a783b571e2e45a02f?s=80", "github_username"=>"scott-stewart", "twitter_username"=>"scott_stewart", "linkedin_username"=>"scottpstewart"}AWS Elasticbeanstalk makes it very simple to create an environment with load balanced instances running a web server proxy in front of your custom app. This is as easy as a couple of clicks from the web interface, or just eb create & eb deploy from the Elastic Beanstalk Command Line Interface (EB CLI).Installing Zeromq on Elasticbeanstalk2016-10-16T00:00:00-04:002016-10-16T00:00:00-04:00https://scopestar.com/blog/aws/elasticbeanstalk/2016/10/16/installing-zeromq-on-elasticbeanstalk<p>I’ve been using <a href="https://aws.amazon.com/opsworks/">AWS Opsworks</a> with <a href="https://www.chef.io">Chef</a> for most projects these days, but started looking at <a href="https://aws.amazon.com/elasticbeanstalk/">Elasticbeanstalk</a> again on a recent project. It took a little bit to get my head around the <a href="https://aws.amazon.com/elasticbeanstalk/">Elasticbeanstalk</a> way of doing stuff like server setup, configuration, etc. but it seems like there are a lot more configuration options available then when I first used it years ago.</p>
<p>If you aren’t already familiar with it, <a href="https://aws.amazon.com/elasticbeanstalk/">Elasticbeanstalk</a> looks in a folder called <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html"><code class="highlighter-rouge">.ebextensions</code></a> for configuration commands, files, etc. The files and the directives inside them get processed in alphabetical order and so a nn_purpose.config file naming convention seems to the norm.</p>
<p>The project I’m deploying is a node.js project that uses the <a href="https://github.com/JustinTulloss/zeromq.node">zmq</a> node library. That means I first need to get <a href="http://zeromq.org">Zeromq</a> packages installed on my instances before I can have the <code class="highlighter-rouge">eb deploy</code> attempt run <code class="highlighter-rouge">npm install</code> on the node dependencies.</p>
<p>According to the Elasticbeanstalk <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-commands">Advanced Environment Configuration documentation</a>:</p>
<blockquote>
<p>You can use the commands key to execute commands on the EC2 instance. The commands are processed in alphabetical order by name, and they run before the application and web server are set up and the application version file is extracted.</p>
</blockquote>
<p>Sounds like exactly what we need!</p>
<p>This made the solution pretty simple; create a file called <code class="highlighter-rouge">packages.config</code> inside <code class="highlighter-rouge">.ebextensions</code>. My name here for the file was arbitrary other than I’ll use this for any commands or directive associated with installing additional packages.</p>
<p>Through a little trial and error, I found the latest <a href="http://zeromq.org">Zeromq</a> packages for <a href="https://aws.amazon.com/amazon-linux-ami/">Amazon Linux AMI</a> by enabling the <a href="https://aws.amazon.com/premiumsupport/knowledge-center/ec2-enable-epel/">Extra Packages for Enterprise Linux</a>.</p>
<p>The contents of my <code class="highlighter-rouge">.ebextensions/packages.config</code> ended up as:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>commands:
01_zmq:
command: "yum --enablerepo=epel -y install zeromq3"
02_zmq_dev:
command: "yum --enablerepo=epel -y install zeromq3-devel"
</code></pre>
</div>
<p>Then a <code class="highlighter-rouge">git add</code> and <code class="highlighter-rouge">git commit</code>, and on my next <code class="highlighter-rouge">eb deploy</code> the <a href="http://zeromq.org">Zeromq</a> packages were installed and the instance was able to complete the <code class="highlighter-rouge">npm install</code> including the <a href="https://github.com/JustinTulloss/zeromq.node">zmq</a> node bindings.</p>{"nick_name"=>"Scott", "full_name"=>"Scott Stewart", "avatar"=>"https://s.gravatar.com/avatar/7d92081717240f3a783b571e2e45a02f?s=80", "github_username"=>"scott-stewart", "twitter_username"=>"scott_stewart", "linkedin_username"=>"scottpstewart"}I’ve been using AWS Opsworks with Chef for most projects these days, but started looking at Elasticbeanstalk again on a recent project. It took a little bit to get my head around the Elasticbeanstalk way of doing stuff like server setup, configuration, etc. but it seems like there are a lot more configuration options available then when I first used it years ago.