Deploying components in a failover architecture

A Caplin Platform installation is typically deployed with multiple component instances to provide resilience against hardware, software, and network failures. To help achieve this, the hardware and software components can be arranged in processing units called "failover legs".

In normal operation, all the components in a single failover leg – typically Caplin Liberator, Caplin Transformer, Integration Adapters, and the Bank’s internal systems – work together to provide the system’s functionality. If a component fails (or a connection to it, or the machine on which it runs), the operations provided by that component are taken up by an alternative copy of the component running in a different failover leg. This transfer of operation is called "failover".

The Caplin Platform Deployment Framework provides configuration that allows failover components to be deployed to the framework.

Download the presentations for this course.

Objectives

In this tutorial you will deploy another leg, it means another copy of the Deployment Framework containing the same components as the original leg. You will define the config to allow them to communicate with each other.

If you have developed your Pricing, Trading and Permissioning Integration Adapters earlier in the course and have not done so already, use the CIS to export each of these Integration Adapters as a "Complete Integration Adapter blade" (as opposed to configuration only). Redeploy the blades, and configure the primary host using the command: ./dfw hosts <adapter name> localhost

Start up the Deployment Framework with ./dfw start (This will start up Liberator, Transformer and the 3 adapters and you will not need to run them from Eclipse)

Check that all is still working as before before proceeding with this tutorial.

Create the second leg

Follow the steps below:

  1. Stop your Deployment Framework. In the global_config/environment.conf file, change the FAILOVER property from DISABLED to ENABLED. Restart the Deployment Framework and open the CMC. You should see the following network of components.

    tutorial failover CMC no 2nd leg

  2. Notice that the Deployment Framework is expecting to find an additional failover for each of the 5 DataSource applications (Liberator, Transformer, and the 3 Integration Adapters). The running components make up the first (primary) leg.

  3. Stop the Deployment Framework again.

  4. Update the file global_config/hosts-defns.conf with the hostnames of the server machines that will host the secondary failover components. In our case, the two legs (primary and secondary) will be deployed on the same machine, localhost. This needs to be done for Liberator, Transformer and each of the Integration Adapters.

  5. Copy the whole Deployment Framework folder somewhere else on your system. This will be the secondary leg.

  6. Adjust the configuration on the secondary server machine as follows. In the global_config/environment.conf file:

    1. Change the definition of NODE from PRIMARY to SECONDARY.

    2. Change the definition of THIS_LEG from 1 to 2.

    3. Change the definition of OTHER_LEG from 2 to 1.

  7. Start both legs.

  8. Go to the status page to check that both Transformers are available.

  9. In the CMC and connect all the elements of both legs. (Right-click in the CMC and "Add new DataSource".) The JMX port numbers to use are the same as for the primary leg for each component except they start with 2 instead of 1, e.g. 21001 instead of 11001 for the Liberator second (failover) leg. You will now notice that each connection from a component has been duplicated. There is one connection to the primary leg component and a new connection to the secondary leg component.

    tutorial failover CMC with 2nd leg

  10. Use the Liberator Diagnostics page to request a price instrument e.g. /FX/GBPUSD. The price updates should appear. This is using the primary leg server.

  11. Now stop the Transformer running in your primary leg server. Use the following commands:

    cd <Deployment Framework Primary Leg Path>
    ./servers/Transformer/etc/transformer stop

  12. In the CMC panel you will see that the Transformer of the primary leg is down and that one of the secondary leg is used instead.

    tutorial failover CMC transformer1 down

  13. Use the Diagnostics page to request a price instrument and you will see that messages are still passing through from the Pricing Adapter, through to the secondary leg Transformer and on to Liberator.

  14. Next stop the Pricing DataSource of the primary leg:

    cd <Deployment Framework Primary Leg Path>
    ./active-blades/PricingAdapter/DataSource/etc/datasource stop

  15. See that the connection is established with the secondary leg PricingAdapter in the CMC and the price updates are still coming through the failover PricingAdapter to the Diagnostics page client.