Migrating to scalable data services

Enable Discovery’s scalable data services to allow data services to recognise new DataSource instances added to a deployment at runtime.

Requirements

The instructions on this page assume that you have already completed the first four stages in the migration path below:

Overview

In a traditional data service, providing peers are defined by lists of static labels (remote-label). Any peer added at runtime that is not listed in the data service’s configuration is not recognised by the data service as a provider.

In a scalable data service, providing peers are defined by regular expression (remote-label-regex). The regular expression is evaluated at runtime for each subscription request, and any peer available at runtime with a label that matches the regular expression is recognised by the data service as a provider.

In a data service’s configuration, providing peers are divided into one or more source groups, each further divided by priority (primary, secondary, …​).

Data serviceSource groupPriorityProviding peerRequests are load-balancedacross peers.Priorities are defined indescending order of priority;primary priority first. Requests are routed to thehighest priority that has atleast one available peer.Requests are routed toeach source group.

Scalable data services support many different arrangements of providing peers (see Scalable data services for examples), but the simplest arrangement, and the one most suitable for deployments to container-orchestrated environments, uses a single-priority source group:

liberator1add-data-serviceservice-name MyDataServiceinclude-pattern ^/MYSUBJECTadd-source-groupadd-priorityremote-label-regex^transformer[0-9]+end-priorityend-source-groupend-data-servicetransformer1transformer2transformerN

For the sake of clarity, the instructions on this page assume that you are migrating data services to a single-priority source group, as illustrated in the example above. This arrangement is not suitable for all use cases. Please review the examples illustrated in Scalable data services, and contact Caplin Support for guidance.

The instructions on this page also assume that your data services don’t require source affinity or service rebalancing. Both features are supported in scalable data services. See sections Source affinity and Service rebalancing on the Scalable data services page for details.

To migrate a deployment to scalable data services, you may need to make changes to the core configuration of components. How much configuration you need to change, and where you need to change it, depends on whether your deployment’s configuration is under manual control or under the management of the Deployment Framework.

Manually configured deployments

Follow the steps below:

  1. Locate the add-data-service configuration items in your deployment:

    List configuration files containing add-data-service
    $ grep -RIl --include='*.conf' 'add-data-service'

  2. Within each data service (add-data-service) in your deployment, convert each source group (add-source-group) to use a single priority (add-priority) containing one remote-label-regex configuration item.

    add-data-service
    …
    add-source-group
        …
        add-priority
            remote-label-regex pattern
        end-priority
    end-source-group
    …
end-data-service
    Example 1. Converting a data service with failover

    Consider the fictional data service below:

    add-data-service
    service-name MyDataService
    include-pattern ^/MYSUBJECT
    add-source-group
        add-priority
            remote-label myadapter-1
            remote-label myadapter-2
            remote-label myadapter-3
        end-priority
        add-priority
            remote-label myadapter-4
            remote-label myadapter-5
            remote-label myadapter-6
        end-priority
    end-source-group
end-data-service

    To specify providing peers by regular expression, make the following changes:

    add-data-service
    service-name MyDataService
    include-pattern ^/MYSUBJECT
    add-source-group
-         add-priority
-             remote-label myadapter-1
-             remote-label myadapter-2
-             remote-label myadapter-3
-         end-priority
-         add-priority
-             remote-label myadapter-4
-             remote-label myadapter-5
-             remote-label myadapter-6
-         end-priority
+         add-priority
+             remote-label-regex ^myadapter-[0-9]+
+         end-priority
    end-source-group
end-data-service

Deployment Framework deployments

Follow the steps below:

  1. If your deployment uses any of the Transformer services below, activate both the service’s standard blade and its complementary Discovery blade:

    Standard blade Complementary blade Minimum version

    AlertsService

    AlertsServiceDiscovery

    7.1.0

    ChartingService

    ChartingServiceDiscovery

    7.1.1

    RefinerService

    RefinerServiceDiscovery

    7.1.4

    WatchlistService

    WatchlistServiceDiscovery

    7.1.1

  2. If your deployment uses any of the Transformer built-in services below, activate both the service’s standard blade and its complementary Discovery blade:

    Standard blade Complementary blade Minimum version

    PersistenceServiceClient

    PersistenceServiceClientDiscovery

    7.1.15

  3. If any other Transformer service blades and adapter blades in your deployment have complementary configuration blades for Discovery, activate them now.

  4. For all remaining blades in your deployment, follow the steps below:

    1. Locate add-data-service configuration items in the blade’s configuration:

      List configuration files containing add-data-service
      $ grep -RIl --include='*.conf' 'add-data-service' active_kits/blade_name global_config/overrides/blade_name

    2. Within each data service (add-data-service), convert each source group (add-source-group) to use a single priority (add-priority) containing one remote-label-regex configuration item.

      add-data-service
    …
    add-source-group
        …
        add-priority
            remote-label-regex pattern
        end-priority
    end-source-group
    …
end-data-service
      Example 2. Converting a data service with failover

      Consider the fictional data service below:

      add-data-service
    service-name MyDataService
    include-pattern ^/MYSUBJECT
    add-source-group
        add-priority
            remote-label myadapter-1
            remote-label myadapter-2
            remote-label myadapter-3
        end-priority
        add-priority
            remote-label myadapter-4
            remote-label myadapter-5
            remote-label myadapter-6
        end-priority
    end-source-group
end-data-service

      To specify providing peers by regular expression, make the following changes:

      add-data-service
    service-name MyDataService
    include-pattern ^/MYSUBJECT
    add-source-group
-         add-priority
-             remote-label myadapter-1
-             remote-label myadapter-2
-             remote-label myadapter-3
-         end-priority
-         add-priority
-             remote-label myadapter-4
-             remote-label myadapter-5
-             remote-label myadapter-6
-         end-priority
+         add-priority
+             remote-label-regex ^myadapter-[0-9]+
+         end-priority
    end-source-group
end-data-service

Next steps

You’ve reached the end of the migration path.

  • Deploy Discovery server

  • Connect DataSources to Discovery

  • Migrate to Discovery licensing

  • Migrate to peer discovery

  • Migrate to scalable data services

If you’re interested in migrating Caplin Platform to Kubernetes, read Example deployment to Kubernetes.