Overview

Dynamic Servers

Enabling dynamic servers in resin.properties

To enable dynamic servers, you can modify the resin.properties with three values: the triad hub servers for the cluster, enabling the elastic flag, and the cluster to join.

• Configure at least one static server in app_servers. Configuring three static servers is better.
• Enable dynamic servers with elastic_cloud_enable.
• Select the cluster to join with home_cluster.

...
app_servers          : 192.168.1.10:6800

elastic_cloud_enable : true

home_cluster         : app
...

A similar pattern works for other clusters like the 'web' cluster. If you're enabling dynamic web servers, configure web_servers for the hub and home_cluster with 'web'.

If you use a custom resin.xml configuration, you can add additional clusters following the same pattern. The standard resin.xml shows how the app_servers are defined. If you follow the same pattern, your custom 'mytier' cluster can use properties like mytier_servers and home_cluster 'mytier'.

Starting an dynamic server

The new server requires a copy of the site's resin.xml and resin.properties. In a cloud environment, you can clone the virtual machine with the properties preconfigured. At least one server must be configured in the hub, because the new server needs the IP address of the hub servers to join the cluster.

The main operation is a "--elastic-server" which forces Resin to create a dynamic server, and "--cluster app-tier" which tells the new server which cluster to join.

unix> resinctl start --elastic-server --cluster app-tier

As usual, you can name the server with "--server my-name". By default, a name is generated automatically.

You can omit the --elastic-server and --cluster if you define the home_cluster and if the server is not one of the static servers. (Resin searches for local IP address in the addresses of the static servers if you don't specify a --server. If it finds a match, Resin will start it as a static server.)

After the server shuts down, the cluster will remember it for 15 minutes, giving the server time to restart before removing it from the cluster.

Deploying Applications to a Cluster

Remote Deployment

In order to use Resin clustered deployment from a remote system, you must first enable remote deployment on the server, which is disable by default for security. You do this using the following Resin configuration:

remote_cli_enable : true

admin_user     : my-admin
admin_password : {SSHA}G3UOLv0dkJHTZxAwmhrIC2CRBZ4VJgTB

In the example above, both the remote admin service and the deployment service is enabled. Note, the admin authenticator must be enabled for any remote administration and deployment for obvious security reasons. To keep things simple, we used a clear-text password above, but you should likely use a password hash instead.

When deploying remotely, you will need to include the user and password.

unix> resinctl deploy --user my-admin --password my-password test.war

Deployed production/webapp/default/test from /home/caucho/ws/test.war
  to http://192.168.1.10:8080/hmtp

Deployment details and architecture

The output exposes a few important details about the underlying remote deployment implementation for Resin that you should understand. Remote deployment for Resin uses Git under the hood. In case you are not familiar with it, Git is a newish version control system similar to Subversion. A great feature of Git is that it is really clever about avoiding redundancy and synchronizing data across a network. Under the hood, Resin stores deployed files as nodes in Git with tags representing the type of file, development stage, virtual host, web application context root name and version. The format used is this:

stage/type/host/webapp[-version]

In the example, all web applications are stored under webapp, we didn't specify a stage or virtual host in the Ant task so default is used, the web application root is foo and no version is used since one was not specified. This format is key to the versioning and staging featured we will discuss shortly.

As soon as your web application is uploaded to the Resin deployment repository, it is propagated to all the servers in the cluster - including any dynamic servers that are added to the cluster at a later point in time after initial propagation happens.

When you deploy an application, it's always a good idea to check that all servers in the cluster have received the correct web-app. Because Resin's deployment is based on a transactional version-control system, you can compare the exact version across all servers in the cluster using the /resin-admin web-app page.

The following screenshot shows the /hello1 webapp deployed across the cluster using a .war deployment in the webapps/ directory. Each version (date) is in green to indicate that the hash validation matches for each server. If one server had a different content, the date would be marked with a red 'x'.

The details for each Ant and Maven based clustered deployment API is outlined here.

HTTP Load Balancing and Failover

resin.properties load balancing configuration

For load balancing with the standard resin.xml and resin.properties, the web_servers defines the static servers for the web (load-balancing) tier, and the app_servers defines the static servers for the application (backend) tier. The standard resin.xml configuration will proxy requests from the web tier to the app tier.

web_servers : 192.168.1.20:6820
web.http : 80

app_servers : 192.168.1.10:6820 \
              192.168.1.11:6820 \
              192.168.1.22:6820
app.http : 8080              

resin.xml load balancing configuration

Dispatching requests with load balancing in Resin is accomplished through the <resin:LoadBalance> tag placed on a cluster. In effect, the <resin:LoadBalance> tag turns a set of servers in a cluster into a software load-balancer. This makes a lot of sense in light of the fact that as the traffic on your application requires multiple servers, your site will naturally be split into two tiers: an application server tier for running your web applications and a web/HTTP server tier talking to end-point browsers, caching static/non-static content, and distributing load across servers in the application server tier.

The best way to understand how this works is through a simple example configuration. The following resin.xml configuration shows servers split into two clusters: a web-tier for load balancing and HTTP, and an app-tier to process the application:

<resin xmlns="http://caucho.com/ns/resin"
          xmlns:resin="urn:java:com.caucho.resin">

  <cluster-default>
    <resin:import path="${__DIR__}/app-default.xml"/>
  </cluster-default>

  <cluster id="app-tier">
    <server id="app-a" address="192.168.0.10" port="6800"/>
    <server id="app-b" address="192.168.0.11" port="6800"/>

    <host id="">
      <web-app id="" root-directory="/var/resin/htdocs"/>
    </host>
  </cluster>

  <cluster id="web-tier">
    <server id="web-a" address="192.168.0.1" port="6800">
      <http port="80"/>
    </server>

    <proxy-cache memory-size="256M"/>

    <host id="">
    
      <resin:LoadBalance regexp="" cluster="app-tier"/>

    </host>
  </cluster>

</resin>

In the configuration above, the web-tier cluster server load balances across the app-tier cluster servers because of the cluster attribute specified in the <resin:LoadBalance> tag. The <proxy-cache> tag enables proxy caching at the web tier. The web-tier forwards requests evenly and skips any app-tier server that's down for maintenance/upgrade or restarting due to a crash. The load balancer also steers traffic from a single session to the same app-tier server (a.k.a sticky sessions), improving caching and session performance.

Each app-tier server produces the same application because they have the same virtual hosts, web-applications and Servlets, and use the same resin.xml. Adding a new machine just requires adding a new <server> tag to the cluster. Each server has a unique name like "app-b" and a TCP cluster-port consisting of an <IP,port>, so the other servers can communicate with it. Although you can start multiple Resin servers on the same machine, TCP requires the <IP,port> to be unique, so you might need to assign unique ports like 6801, 6802 for servers on the same machine. On different machines, you'll use unique IP addresses. Because the cluster-port is for Resin servers to communicate with each other, they'll typically be private IP addresses like 192.168.1.10 and not public IP addresses. In particular, the load balancer on the web-tier uses the cluster-port of the app-tier to forward HTTP requests.

When no explicit server is specified, Resin will search for local IP addresses that match the configured servers and start all of them. This local IP searching lets Resin's service start script work without modification. When dynamic servers are enabled, and no local server matches, Resin will start a new dynamic server to join the home_cluster cluster.

unix> resinctl start-all

All three servers will use the same resin.xml, which makes managing multiple server configurations pretty easy. The servers are named by the server id attribute, which must be unique, just as the <IP,port>. When you start a Resin instance, you'll use the server-id as part of the command line:

192.168.0.10> resinctl -server app-a start
192.168.0.11> resinctl -server app-b start
192.168.0.1> resinctl -server web-a start

Since Resin lets you start multiple servers on the same machine, a small site might start the web-tier server and one of the app-tier servers on one machine, and start the second server on a second machine. You can even start all three servers on the same machine, increasing reliability and easing maintenance, without addressing the additional load (although it will still be problematic if the physical machine itself and not just the JVM crashes). If you do put multiple servers on the same machine, remember to change the port to something like 6801, etc so the TCP binds do not conflict.

In the /resin-admin management page, you can manage all three servers at once, gathering statistics/load and watching for any errors. When setting up /resin-admin on a web-tier server, you'll want to remember to add a separate <web-app> for resin-admin to make sure the <rewrite-dispatch> doesn't inadvertantly send the management request to the app-tier.

Sticky/Persistent Sessions

To understand what sticky sessions are and why they are important, it is easiest to see what will happen without them. Let us take our previous example and assume that the web tier distributes sessions across the application tier in a totally random fashion. Recall that while Resin can replicate the HTTP session across the cluster, it does not do so by default. Now lets assume that the first request to the web application resolves to app-a and results in the login being stored in the session. If the load-balancer distributes sessions randomly, there is no guarantee that the next request will come to app-a. If it goes to app-b, that server instance will have no knowledge of the login and will start a fresh session, forcing the user to login again. The same issue would be repeated as the user continues to use the application from in what their view should be a single session with well-defined state, making for a pretty confusing experience with application state randomly getting lost!

One way to solve this issue would be to fully synchronize the session across the load-balanced servers. Resin does support this through its clustering features (which is discussed in detail in the following sections). The problem is that the cost of continuously doing this synchronization across the entire cluster is very high and relatively unnecessary. This is where sticky sessions come in. With sticky sessions, the load-balancer makes sure that once a session is started, any subsequent requests from the client go to the same server where the session resides. By default, the Resin load-balancer enforces sticky sessions, so you don't really need to do anything to enable it.

Resin accomplishes this by encoding the session cookie with the host name that started the session. Using our example, the hosts would generate cookies like this:

On the web-tier, Resin decoded the session cookie and sends it to the appropriate server. So baaX8ZwooOz will go to app-b. If app-b fails or is down for maintenance, Resin will send the request a backup server calculated from the session id, in this case app-a. Because the session is not clustered, the user will lose the session but they won't see a connection failure (to see how to avoid losing the session, check out the following section on clustering).

Manually Choosing a Server

For testing purposes you might want to send requests to a specific servers in the app-tier manually. You can easily do this since the web-tier uses the value of the jsessionid to maintain sticky sessions. You can include an explicit jsessionid to force the web-tier to use a particular server in the app-tier.

Since Resin uses the first character of the jsessionid to identify the server to use, starting with 'a' will resolve the request to app-a. If www.example.com resolves to your web-tier, then you can use values like this for testing:

1. http://www.example.com/test-servlet;jsessionid=aaaXXXXX
2. http://www.example.com/test-servlet;jsessionid=baaXXXXX
3. http://www.example.com/test-servlet;jsessionid=caaXXXXX
4. http://www.example.com/test-servlet;jsessionid=daaXXXXX
5. http://www.example.com/test-servlet;jsessionid=eaaXXXXX
6. etc.

You can also use this fact to configure an external sticky load-balancer (likely a high-performance hardware load-balancer) and eliminate the web tier altogether. In this case, this is how the Resin configuration might look like:

<resin xmlns="http://caucho.com/ns/resin">
  <cluster id="app-tier">
    <server-default>
      <http port='80'/>
    </server-default>

    <server id='app-a' address='192.168.0.1' port="6800"/>
    <server id='app-b' address='192.168.0.2' port="6800"/>
    <server id='app-c' address='192.168.0.3' port="6800"/>

  </cluster>
</resin>

Each server will be started as -server a, -server b, etc to grab its specific configuration.

Socket Pooling, Timeouts, and Failover

For efficiency, Resin's load balancer manages a pool of sockets connecting to the app-tier servers. If Resin forwards a new request to an app-tier server and it has an idle socket available in the pool, it will reuse that socket, improving performance and minimizing network load. Resin uses a set of timeout values to manage the socket pool and to handle any failures or freezes of the backend servers. The following diagram illustrates the main timeout values:

• load-balance-connect-timeout: the load balancer timeout for the connect() system call to complete to the app-tier (5s).
• load-balance-idle-time: load balancer timeout for an idle socket before closing it automatically (5s).
• load-balance-recover-time: the load balancer connection failure wait time before trying a new connection (15s).
• load-balance-socket-timeout: the load balancer timeout for a valid request to complete (665s).
• keepalive-timeout: the app-tier timeout for a keepalive connection
• socket-timeout: the app-tier timeout for a read or write (65s)

When an app-tier server is down due to maintenance or a crash, Resin will use the load-balance-recover-time as a delay before retrying the downed server. With the failover and recover timeout, the load balancer reduces the cost of a failed server to almost no time at all. Every recover-time, Resin will try a new connection and wait for load-balance-connect-timeout for the server to respond. At most, one request every 15 seconds might wait an extra 5 seconds to connect to the backend server. All other requests will automatically go to the other servers.

The socket-timeout values tell Resin when a socket connection is dead and should be dropped. The web-tier timeout load-balance-socket-timeout is much larger than the app-tier timeout socket-timeout because the web-tier needs to wait for the application to generate the response. If your application has some very slow pages, like a complicated nightly report, you may need to increase the load-balance-socket-timeout to avoid the web-tier disconnecting it.

Likewise, the load-balance-idle-time and keepalive-timeout are a matching pair for the socket idle pool. The idle-time tells the web-tier how long it can keep an idle socket before closing it. The keepalive-timeout tells the app-tier how long it should listen for new requests on the socket. The keepalive-timeout must be significantly larger than the load-balance-idle-time so the app-tier doesn't close its sockets too soon. The keepalive timeout can be large since the app-tier can use the keepalive-select manager to efficiently wait for many connections at once.

resin:LoadBalance Dispatching

<resin:LoadBalance> is part of Resin's rewrite capabilities, Resin's equivalent of the Apache mod_rewrite module, providing powerful and detailed URL matching and decoding. More sophisticated sites might load-balance based on the virtual host or URL using multiple <resin:LoadBalance> tags.

In most cases, the web-tier will dispatch everything to the app-tier servers. Because of Resin's proxy cache, the web-tier servers will serve static pages as fast as if they were local pages.

In some cases, though, it may be important to send different requests to different backend clusters. The <resin:LoadBalance> tag can choose clusters based on URL patterns when such capabilities are needed.

The following rewrite example keeps all *.png, *.gif, and *.jpg files on the web-tier, sends everything in /foo/* to the foo-tier cluster, everything in /bar/* to the bar-tier cluster, and keeps anything else on the web-tier.

<resin xmlns="http://caucho.com/ns/resin"
          xmlns:resin="urn:java:com.caucho.resin">

  <cluster-default>
     <resin:import path="classpath:META-INF/caucho/app-default.xml"/>
  </cluster-default>
      
  <cluster id="web-tier">
    <server id="web-a">
      <http port="80"/>
    </server>

    <proxy-cache memory-size="64m"/>

    <host id="">
      <web-app id="/">

        <resin:Dispatch regexp="(\.png|\.gif|\.jpg)"/>

        <resin:LoadBalance regexp="^/foo" cluster="foo-tier"/>

        <resin:LoadBalance regexp="^/bar" cluster="bar-tier"/>

      </web-app>
    </host>
  </cluster>

  <cluster id="foo-tier">
    ...
  </cluster>

  <cluster id="bar-tier">
    ...
  </cluster>
</resin>

Triple Redundant Clustering (the Triad)

Cluster Heartbeat

As part of Resin's health check system, the cluster continually checks that all servers are alive and responding properly. Every 60 seconds, each server sends a heartbeat to the triad, and each triad server sends a heartbeat to all the other servers.

When a server failure is detected, Resin immediately detects the failure, logging it for an administrator's analysis and internally prepares to failover to backup servers for any messaging for distributed storage like the clustered sessions and the clustered deployment.

When the server comes back up, the heartbeat is reestablished and any missing data is recovered.

Clustered Sessions

always-save-session

Resin's distributed sessions need to know when a session has changed in order to save/synchronize the new session value. Although Resin does detect when an application calls HttpSession.setAttribute, it can't tell if an internal session value has changed. The following Counter class shows the issue:

package test;

public class Counter implements java.io.Serializable {
  private int _count;

  public int nextCount() { return _count++; }
}

Assuming a copy of the Counter is saved as a session attribute, Resin doesn't know if the application has called nextCount. If it can't detect a change, Resin will not backup/synchronize the new session, unless always-save-session is set on the <session-config>. When always-save-session is true, Resin will back up the entire session at the end of every request. Otherwise, a session is only changed when a change is detected.

...
<web-app id="/foo">
...
<session-config>
  <use-persistent-store/>
  <always-save-session/>
</session-config>
...
</web-app>

Serialization

Resin's distributed sessions relies on Hessian serialization to save and restore sessions. Application objects must implement java.io.Serializable for distributed sessions to work.

No Distributed Locking

Resin's clustering does not lock sessions for replication. For browser-based sessions, only one request will typically execute at a time. Because browser sessions have no concurrency, there's really no need for distributed locking. However, it's a good idea to be aware of the lack of distributed locking in Resin clustering.

Distributed Caching

@CacheResult

@CacheResult enables caching for method results. It uses the method parameters as a cache key, and stores the method result in the cache. On the next method call, the enhanced method will look for the saved result in the cache, and return it, saving the effort of the method.

1. add a @CacheResult annotation to the method you want to cache
2. use Java Dependency Injection (CDI) to get the bean

<beans/>

package org.example.mypkg;

import javax.cache.annotation.CacheResult;

public class MyBean {
  @CacheResult
  public String doLongOperation(String key)
  {
    ...
  }
}

package org.example.mypkg;

import javax.cache.annotation.CacheResult;

public class MyServlet extends GenericServlet {
  @Inject MyBean _bean;

  public void service(ServletRequest req, ServletResponse res)
    throws IOException, ServletException
  {
    PrintWriter out = res.getWriter();
 
    String result = _bean.doLongOperation("test");
  
    out.println("test: " + result);
  }
}