Split Synchronizer

By default, Split’s SDKs keep segment and feature flag data synchronized as users navigate across disparate systems, treatments, and conditions. However, some languages, do not have a native capability to keep a shared local cache of this data to properly serve treatments. For these cases, we built the Split Synchronizer service.

This tool coordinates the sending and receiving of data to a remote datastore that all of your processes can share to pull data for the evaluation of treatments. Out of the box, Split supports Redis as a remote datastore, and so the Split Synchronizer uses Redis as the cache for your SDKs when evaluating treatments. It also posts impression and event data and metrics generated by the SDKs back to Split’s servers, for exposure in the user interface or sending to the data integration of your choice. The Synchronizer service runs as a standalone process in dedicated or shared servers and it does not affect the performance of your code, or Split’s SDKs.

Supported SDKs

The Split Synchronizer works with most of the languages that Split supports.

• PHP SDK
• Python SDK
• .NET SDK
• Node.js SDK
• Ruby SDK
• GO SDK
• Java SDK

If you are looking for a language that is not listed here, contact the support team at support@split.io to discuss your options.

Overall Architecture

The service performs five actions:

• Fetch feature flags: Retrieve the feature flag definitions.
• Fetch segments: Retrieve your set segments lists.
• Post impressions: Send to Split servers the generated impressions by the SDK.
• Post telemetry: Send to Split servers different metrics of the SDK, such as latencies.
• Post events: Send to Split servers the generated events by the SDK .track method.

Architecture

Setup

Docker (recommended)

The service is available via Docker or command line and its source code is available at https://github.com/splitio/split-synchronizer.

• Pull the image: docker pull splitsoftware/split-synchronizer
• Run as:

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY="your-sdk-key" \
 -e SPLIT_SYNC_REDIS_HOST=<your-redis-host> \
 -e SPLIT_SYNC_REDIS_PORT=<your_redis_port> \
 splitsoftware/split-synchronizer

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY="your-sdk-key" \
 -e SPLIT_SYNC_REDIS_SENTINEL_REPLICATION="true" \
 -e SPLIT_SYNC_REDIS_SENTINEL_MASTER="MASTER_SERVICE_NAME" \
 -e SPLIT_SYNC_REDIS_SENTINEL_ADDRESSES="SENTINEL_HOST_1:SENTINEL_PORT_1,SENTINEL_HOST_2:SENTINEL_PORT_2" \
 splitsoftware/split-synchronizer

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY="your-sdk-key" \
 -e SPLIT_SYNC_REDIS_CLUSTER_MODE="true" \
 -e SPLIT_SYNC_REDIS_CLUSTER_NODES="CLUSTER_NODE_1:CLUSTER_PORT_1,CLUSTER_NODE_2:CLUSTER_PORT_2,CLUSTER_NODE_3:CLUSTER_PORT_3" \
 splitsoftware/split-synchronizer

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY="your-sdk-key" \
 -e SPLIT_SYNC_REDIS_CLUSTER_MODE="true" \
 -e SPLIT_SYNC_REDIS_CLUSTER_NODES="Cluster Entry Host" \
 splitsoftware/split-synchronizer

Command line

To install and run the service from command line, follow the steps below depending of your platform.

Linux

On Linux systems, the Synchronizer service install script is invoked with this.

curl -L -o install_linux.bin 'https://downloads.split.io/synchronizer/install_split_sync_linux.bin' && chmod 755 install_linux.bin && ./install_linux.bin

OSX

On OSX systems, the Synchronizer service install script is invoked with this.

curl -L -o install_osx.bin 'https://downloads.split.io/synchronizer/install_split_sync_osx.bin' && chmod 755 install_osx.bin && ./install_osx.bin

Windows

On Microsoft Windows systems, follow these steps:

1. Download the app from https://downloads.split.io/synchronizer/split_sync_windows.zip.

2. Unzip the downloaded file.

3. Run it!

Run the service

To run the service, paste the snippet below into your command line terminal and add in your SDK-Key.

Linux/Mac

split-sync -apikey "your_sdk_key"

Windows

Open the cmd terminal or the PowerShell terminal, go to (cd) unzipped Split Synchronizer folder, and type:

split-sync.exe -apikey "your_sdk_key"

Recommended configuration for production

You can run the service with the simple steps above, but the system is more stable in your production environment when you run the job with a scheduling system. We recommend starting the synchronizer via supervisord, a daemon that launches other processes and ensures they stay running.

To use supervisord, make sure that it is installed on your machine. You can get help on the installation at the official Supervisord documentation.

When supervisord is installed into your project, copy and paste the program below anywhere into the supervisord.conf file that should now be in your project.

[program:splitio_sync]
command=/usr/local/bin/split-sync -config /path/to/your/config.file.json
process_name = SplitIO
numprocs = 1
autostart=true
autorestart=true
user = your_user
stderr_logfile=/var/log/splitio.err.log
stderr_logfile_maxbytes = 1MB
stdout_logfile=/var/log/splitio.out.log
stdout_logfile_maxbytes = 1MB

Advanced configuration

The Synchronizer service has a number of knobs for configuring performance. Each knob is tuned to a reasonable default, however, you can override the default values by changing a splitio.config.json file or by setting your customer values as parameters of -config in the command line option. In this section, we lay out all the different knobs you can configure for performance, Redis, and logging.

The splitio.config.json file provided via the -config option lets you control how often the synchronizer fetches data from Split servers. You can create a sample JSON file automatically with default values by running this command.

./split-sync -write-default-config "/home/someuser/splitio.config.json"

{
  "apikey": "YOUR_SDK_KEY",
  "ipAddressEnabled": true,
  "initialization": {
    "timeoutMS": 10000,
    "forceFreshStartup": false
  },
  "storage": {
    "type": "redis",
    "redis": {
      "host": "localhost",
      "port": 6379,
      "db": 0,
      "username": "",
      "password": "",
      "prefix": "",
      "network": "tcp",
      "maxRetries": 0,
      "dialTimeout": 5,
      "readTimeout": 10,
      "writeTimeout": 5,
      "poolSize": 10,
      "sentinelReplication": false,
      "sentinelAddresses": "",
      "sentinelMaster": "",
      "clusterMode": false,
      "clusterNodes": "",
      "keyHashTag": "",
      "enableTLS": false,
      "tlsServerName": "",
      "caCertificates": null,
      "tlsSkipNameValidation": false,
      "tlsClientCertificate": "",
      "tlsClientKey": ""
    }
  },
  "sync": {
    "splitRefreshRateMs": 60000,
    "segmentRefreshRateMs": 60000,
    "impressionsMode": "optimized",
    "advanced": {
      "streamingEnabled": true,
      "httpTimeoutMs": 30000,
      "internalTelemetryRateMs": 3600000,
      "telemetryPushRateMs": 60000,
      "impressionsFetchSize": 0,
      "impressionsProcessConcurrency": 0,
      "impressionsProcessBatchSize": 0,
      "impressionsPostConcurrency": 0,
      "impressionsPostSize": 0,
      "impressionsAccumWaitMs": 0,
      "eventsFetchSize": 0,
      "eventsProcessConcurrency": 0,
      "eventsProcessBatchSize": 0,
      "eventsPostConcurrency": 0,
      "eventsPostSize": 0,
      "eventsAccumWaitMs": 0
    }
  },
  "admin": {
    "host": "0.0.0.0",
    "port": 3010,
    "username": "",
    "password": "",
    "secureChecks": false
  },
  "integrations": {
    "impressionListener": {
      "endpoint": "",
      "queueSize": 100
    },
    "slack": {
      "webhook": "",
      "channel": ""
    }
  },
  "logging": {
    "level": "info",
    "output": "stdout",
    "rotationMaxFiles": 10,
    "rotationMaxSizeKb": 0
  },
  "healthcheck": {
    "app": {
      "storageCheckRateMs": 3600000
    }
  }
}

{
  "apikey": "YOUR_SDK_KEY",
  "ipAddressEnabled": true,
  "initialization": {
    "timeoutMS": 10000,
    "forceFreshStartup": false
  },
  "storage": {
    "type": "redis",
    "redis": {
      "host": "localhost",
      "port": 6379,
      "db": 0,
      "username": "",
      "password": "",
      "prefix": "",
      "network": "tcp",
      "maxRetries": 0,
      "dialTimeout": 5,
      "readTimeout": 10,
      "writeTimeout": 5,
      "poolSize": 10,
      "sentinelReplication": true,
      "sentinelAddresses": "SENTINEL_HOST_1:SENTINEL_PORT_1, SENTINEL_HOST_2:SENTINEL_PORT_2,SENTINEL_HOST_3:SENTINEL_PORT_3",
      "sentinelMaster": "MASTER_SERVICE_NAME",
      "clusterMode": false,
      "clusterNodes": "",
      "keyHashTag": "",
      "enableTLS": false,
      "tlsServerName": "",
      "caCertificates": null,
      "tlsSkipNameValidation": false,
      "tlsClientCertificate": "",
      "tlsClientKey": ""
    }
  },
  "sync": {
    "splitRefreshRateMs": 60000,
    "segmentRefreshRateMs": 60000,
    "impressionsMode": "optimized",
    "advanced": {
      "streamingEnabled": true,
      "httpTimeoutMs": 30000,
      "internalTelemetryRateMs": 3600000,
      "telemetryPushRateMs": 60000,
      "impressionsFetchSize": 0,
      "impressionsProcessConcurrency": 0,
      "impressionsProcessBatchSize": 0,
      "impressionsPostConcurrency": 0,
      "impressionsPostSize": 0,
      "impressionsAccumWaitMs": 0,
      "eventsFetchSize": 0,
      "eventsProcessConcurrency": 0,
      "eventsProcessBatchSize": 0,
      "eventsPostConcurrency": 0,
      "eventsPostSize": 0,
      "eventsAccumWaitMs": 0
    }
  },
  "admin": {
    "host": "0.0.0.0",
    "port": 3010,
    "username": "",
    "password": "",
    "secureChecks": false
  },
  "integrations": {
    "impressionListener": {
      "endpoint": "",
      "queueSize": 100
    },
    "slack": {
      "webhook": "",
      "channel": ""
    }
  },
  "logging": {
    "level": "info",
    "output": "stdout",
    "rotationMaxFiles": 10,
    "rotationMaxSizeKb": 0
  },
  "healthcheck": {
    "app": {
      "storageCheckRateMs": 3600000
    }
  }
}

{
  "apikey": "",
  "ipAddressEnabled": true,
  "initialization": {
    "timeoutMS": 10000,
    "forceFreshStartup": false
  },
  "storage": {
    "type": "redis",
    "redis": {
      "host": "localhost",
      "port": 6379,
      "db": 0,
      "username": "",
      "password": "",
      "prefix": "",
      "network": "tcp",
      "maxRetries": 0,
      "dialTimeout": 5,
      "readTimeout": 10,
      "writeTimeout": 5,
      "poolSize": 10,
      "sentinelReplication": false,
      "sentinelAddresses": "",
      "sentinelMaster": "",
      "clusterMode": true,
      "clusterNodes": "CLUSTER_NODE_1:CLUSTER_PORT_1, CLUSTER_NODE_2:CLUSTER_PORT_2,CLUSTER_NODE_3:CLUSTER_PORT_3",
      "keyHashTag": "",
      "enableTLS": false,
      "tlsServerName": "",
      "caCertificates": null,
      "tlsSkipNameValidation": false,
      "tlsClientCertificate": "",
      "tlsClientKey": ""
    }
  },
  "sync": {
    "splitRefreshRateMs": 60000,
    "segmentRefreshRateMs": 60000,
    "impressionsMode": "optimized",
    "advanced": {
      "streamingEnabled": true,
      "httpTimeoutMs": 30000,
      "internalTelemetryRateMs": 3600000,
      "telemetryPushRateMs": 60000,
      "impressionsFetchSize": 0,
      "impressionsProcessConcurrency": 0,
      "impressionsProcessBatchSize": 0,
      "impressionsPostConcurrency": 0,
      "impressionsPostSize": 0,
      "impressionsAccumWaitMs": 0,
      "eventsFetchSize": 0,
      "eventsProcessConcurrency": 0,
      "eventsProcessBatchSize": 0,
      "eventsPostConcurrency": 0,
      "eventsPostSize": 0,
      "eventsAccumWaitMs": 0
    }
  },
  "admin": {
    "host": "0.0.0.0",
    "port": 3010,
    "username": "",
    "password": "",
    "secureChecks": false
  },
  "integrations": {
    "impressionListener": {
      "endpoint": "",
      "queueSize": 100
    },
    "slack": {
      "webhook": "",
      "channel": ""
    }
  },
  "logging": {
    "level": "info",
    "output": "stdout",
    "rotationMaxFiles": 10,
    "rotationMaxSizeKb": 0
  },
  "healthcheck": {
    "app": {
      "storageCheckRateMs": 3600000
    }
  }
}

Methods to configure the Split synchronizer

You can configure the Split synchronizer service using the command line or by directly editing the above mentioned JSON configuration file.

split-sync -config "/etc/splitio.config.json" -log-level "debug" -redis-pass "somePass" 

CLI Configuration options and its equivalents in JSON & Environment variables

Listener

The Split Synchronizer provides an impression listener that bulks post impressions to a user-defined HTTP endpoint.

The endpoint should expect a POST request, containing a JSON body with the following format.

{
 "impressions": [
  {
   "testName": "feature1",
   "keyImpressions": [
    {
     "keyName": "user1",
     "treatment": "on",
     "time": 1502754901182,
     "changeNumber": -1,
     "label": ""
    },
    {
     "keyName": "user2",
     "treatment": "off",
     "time": 1502754876144,
     "changeNumber": -1,
     "label": ""
    }
   ]
  }
 ],
 "sdkVersion": "php-5.2.2",
 "machineIP": "208.63.222.7",
 "MachineName": ""
}

Currently, the configuration options are available in the integrations.impressionListener section of the JSON configuration file detailed in Advanced configuration.

Using a network proxy

If you need to use a network proxy, configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY. The internal HTTP client reads those variables and uses them to perform the server request.

$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

Admin tools

Endpoints

The split-sync service has a set of endpoints and a dashboard to let DevOps and infra team monitor its status and cached data in real-time. By default the port is 3010 and for security reason supports HTTP Basic Authentication configured by the user.

/info/ping

A ping endpoint to monitor the service status. If the service is running, it sends the text response pong and the HTTP status code 200.

/info/version

Returns the split-sync version in JSON format.

{
  "version" : "1.1.0"
}

/info/uptime

Returns the uptime string representation in JSON format.

{
  "uptime" : "5d 3h 36m 39s"
}

/info/config
Returns a JSON object describing the current configuration of the Synchronizer.

{
  "config": {
    "apikey": "*",
    "ipAddressEnabled": true,
    "initialization": {
      "timeoutMS": 10000,
      "forceFreshStartup": false
    },
    "storage": {
      "type": "redis",
      "redis": {
        "host": "localhost",
        "port": 6379,
        "db": 0,
        "username": "",
        "password": "",
        "prefix": "",
        "network": "tcp",
        "maxRetries": 0,
        "dialTimeout": 5,
        "readTimeout": 10,
        "writeTimeout": 5,
        "poolSize": 10,
        "sentinelReplication": false,
        "sentinelAddresses": "",
        "sentinelMaster": "",
        "clusterMode": false,
        "clusterNodes": "",
        "keyHashTag": "",
        "enableTLS": false,
        "tlsServerName": "",
        "caCertificates": null,
        "tlsSkipNameValidation": false,
        "tlsClientCertificate": "",
        "tlsClientKey": ""
      }
    },
    "sync": {
      "splitRefreshRateMs": 60000,
      "segmentRefreshRateMs": 60000,
      "impressionsMode": "optimized",
      "advanced": {
        "streamingEnabled": true,
        "httpTimeoutMs": 30000,
        "internalTelemetryRateMs": 3600000,
        "telemetryPushRateMs": 60000,
        "impressionsFetchSize": 0,
        "impressionsProcessConcurrency": 0,
        "impressionsProcessBatchSize": 0,
        "impressionsPostConcurrency": 0,
        "impressionsPostSize": 0,
        "impressionsAccumWaitMs": 0,
        "eventsFetchSize": 0,
        "eventsProcessConcurrency": 0,
        "eventsProcessBatchSize": 0,
        "eventsPostConcurrency": 0,
        "eventsPostSize": 0,
        "eventsAccumWaitMs": 0
      }
    },
    "admin": {
      "host": "0.0.0.0",
      "port": 3010,
      "username": "",
      "password": "",
      "secureChecks": false
    },
    "integrations": {
      "impressionListener": {
        "endpoint": "",
        "queueSize": 100
      },
      "slack": {
        "webhook": "",
        "channel": ""
      }
    },
    "logging": {
      "level": "info",
      "output": "stdout",
      "rotationMaxFiles": 10,
      "rotationMaxSizeKb": 0
    },
    "healthcheck": {
      "app": {
        "storageCheckRateMs": 3600000
      }
    }
  }
}

/health/application
Returns a JSON object describing whether the synchronizer is healthy or not.

{
  "healthy": true,
  "healthySince": "2021-11-20T19:04:46.528242-03:00",
  "items": [
    {
      "name": "Splits",
      "healthy": true,
      "lastHit": "2021-11-20T19:04:49.079956-03:00"
    },
    {
      "name": "Segments",
      "healthy": true,
      "lastHit": "2021-11-20T19:04:49.268349-03:00"
    },
    {
      "name": "Storage",
      "healthy": true
    }
  ]
}

/health/dependencies
Returns a JSON object describing whether the servers the synchronizer depends on are healthy or not.

{
  "serviceStatus": "healthy",
  "dependencies": [
    {
      "service": "https://telemetry.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528262-03:00"
    },
    {
      "service": "https://auth.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528264-03:00"
    },
    {
      "service": "https://sdk.split.io/api/version",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528265-03:00"
    },
    {
      "service": "https://events.split.io/api/version",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528266-03:00"
    },
    {
      "service": "https://streaming.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528266-03:00"
    }
  ]
}

Admin Dashboard

Split-sync has a web admin UI out of the box that exposes all available endpoints. Browse to /admin/dashboard to see it.

The dashboard is organized in four sections for ease of visualization:

• Dashboard: Tile-sorted summary information, including these metrics:
  • Uptime: Uptime metric
  • Healthy Since: Time passed without errors
  • Logged Errors: Total count of error messages
  • SDKs Total Hits: Total SDKs requests
  • Backend Total Hits: Total backend requests between split-sync and Split servers
  • Cached Feature flags: Number of feature flags cached in memory
  • Cached Segments: Number of segments cached in memory
  • Impressions Queue Size: shows the total amount of Impressions stored in Redis (only Producer Mode).
  • Impressions Lambda: shows the eviction rate for Impressions (only Producer Mode).
  • Events Queue Size: shows the total amount of Events stored in Redis (only Producer Mode).
  • Events Lambda: shows the eviction rate for Events (only Producer Mode).
  • SDK Server: displays the status of Split server for SDK.
  • Events Server: displays the status of Split server for Events.
  • Streaming Server*: displays the status of Split streaming service
  • Auth Server*: displays the status of Split server for initial streaming authentication
  • Telemetry Server*: displays the status of Split server for telemetry capturing.
  • Storage: (only Sync mode) displays the status of the storage.
  • Sync: displays the status of the Synchronizer.
  • Last Errors Log: List of the last 10 error messages
• SDK stats: Metrics numbers and a latency graph, measured between SDKs requests integration and proxy.
• Data inspector: Cached data showing feature flags and segments; filters to find keys and feature flag definitions.
• Queue Manager: expose sizes of Impressions and Events queues.

Service shutdown

The split-sync service can catch a kill sig command and start a graceful shutdown, flushing all cached data progressively. Additionally, you can perform graceful stop and force stop (kill -9) with one click from the admin dashboard.

If you have configured a Slack channel and the Slack Webhook URL, an alert is sent to the channel when and initialization or shutdown is performed.