Split synchronizer & proxy

By default, Split’s SDKs keep segment and split data synchronized as users navigate across disparate systems, treatments, and conditions. However, some languages, do not have a native capability to keep a shared local cache of this data to properly serve treatments. For these cases, we built the Split Synchronizer service.

Split synchronizer can run in two different modes:

• Synchronizer mode (default): Coordinates the sending and receiving of data to a remote datastore that all of your processes can share to pull data for the evaluation of treatments. Out of the box, Split supports Redis as a remote datastore, and so the Split Synchronizer uses Redis as the cache for your SDKs when evaluating treatments. It also posts impression data and metrics generated by the SDKs back to Split’s servers, for exposure in the web console or sending to the data integration of your choice.
• Proxy mode: Use proxy mode to reduce connection latencies from the SDKs to the Split server in a way that is completely transparent to the SDKs and when a single connection from a private network to the oustide is required for security reasons.

The Synchronizer service runs as a standalone process in dedicated or shared servers and it does not affect the performance of your code, or Split’s SDKs.

Supported SDKs

The Split Synchronizer works with most of the languages that Split supports.

• PHP SDK
• Python SDK
• .NET SDK
• Node.js SDK
• Ruby SDK
• GO SDK
• Java SDK (Proxy mode only)

If you are looking for a language that is not listed here, contact the support team at support@split.io to discuss your options.

Overall Architecture

The service performs five actions:

• Fetch splits: Retrieve the split definitions.
• Fetch segments: Retrieve your set segments lists.
• Post impressions: Send to Split servers the generated impressions by the SDK.
• Post metrics: Send to Split servers different metrics of the SDK, such as latencies.
• Post events: Send to Split servers the generated events by the SDK .track method.

Synchronizer mode

Proxy mode

Setup

Docker (recommended)

The service is available via Docker or command line and its source code is available at https://github.com/splitio/split-synchronizer.

• Pull the image: docker pull splitsoftware/split-synchronizer
• Run as:

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_API_KEY="your-sdk-api-key" \
 -e SPLIT_SYNC_REDIS_HOST=<your-redis-host> \
 -e SPLIT_SYNC_REDIS_PORT=<your_redis_port> \
 splitsoftware/split-synchronizer

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_API_KEY="your-api-key" \
 -e SPLIT_SYNC_REDIS_SENTINEL_REPLICATION="on" \
 -e SPLIT_SYNC_REDIS_SENTINEL_MASTER="MASTER_SERVICE_NAME" \
 -e SPLIT_SYNC_REDIS_SENTINEL_ADDRESSES="SENTINEL_HOST_1:SENTINEL_PORT_1,SENTINEL_HOST_2:SENTINEL_PORT_2" \
 splitsoftware/split-synchronizer

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_API_KEY="your-api-key" \
 -e SPLIT_SYNC_REDIS_CLUSTER_MODE="on" \
 -e SPLIT_SYNC_REDIS_CLUSTER_NODES="CLUSTER_NODE_1:CLUSTER_PORT_1,CLUSTER_NODE_2:CLUSTER_PORT_2,CLUSTER_NODE_3:CLUSTER_PORT_3" \
 splitsoftware/split-synchronizer

docker run --rm --name split-synchronizer-proxy \
 -p 3000:3000 \
 -p 3010:3010 \
 -e SPLIT_SYNC_API_KEY="your-api-key" \
 -e SPLIT_SYNC_PROXY="on" \
 -e SPLIT_SYNC_PROXY_SDK_APIKEYS="123456,qwerty" \
 splitsoftware/split-synchronizer

Command line

To install and run the service from command line, follow the steps below depending of your platform.

Linux

On Linux systems, the Synchronizer service install script is invoked with this.

curl -L -o install_linux.bin http://downloads.split.io/synchronizer/install_linux.bin && chmod 755 install_linux.bin && ./install_linux.bin

OSX

On OSX systems, the Synchronizer service install script is invoked with this.

curl -L -o install_osx.bin http://downloads.split.io/synchronizer/install_osx.bin && chmod 755 install_osx.bin && ./install_osx.bin

Windows

On Microsoft Windows systems, follow these steps:

1. Download the app from http://downloads.split.io/synchronizer/split-sync-win.zip.

2. Unzip the downloaded file.

3. Run it!

Run the service

To run the service, paste the snippet below into your command line terminal and add in your API-Key.

Linux/Mac

split-sync -api-key "split_sdk_api_key"

split-sync -api-key "your_api_key" -proxy -proxy-apikeys "sdk_api_key_1,sdk_api_key_2"

Windows

Open the cmd terminal or the PowerShell terminal, go to (cd) unzipped Split Synchronizer folder, and type:

split-sync.exe -api-key "split_sdk_api_key"

split-sync.exe -api-key "your_api_key" -proxy -proxy-apikeys "sdk_api_key_1,sdk_api_key_2"

Recommended configuration for production

You can run the service with the simple steps above, but the system is more stable in your production environment when you run the job with a scheduling system. We recommend starting the synchronizer via supervisord, a daemon that launches other processes and ensures they stay running.

To use supervisord, make sure that it is installed on your machine. You can get help on the installation at the official Supervisord documentation.

When supervisord is installed into your project, copy and paste the program below anywhere into the supervisord.conf file that should now be in your project.

[program:splitio_sync]
command=/usr/local/bin/split-sync -config /path/to/your/config.file.json
process_name = SplitIO
numprocs = 1
autostart=true
autorestart=true
user = your_user
stderr_logfile=/var/log/splitio.err.log
stderr_logfile_maxbytes = 1MB
stdout_logfile=/var/log/splitio.out.log
stdout_logfile_maxbytes = 1MB

Advanced configuration

The Synchronizer service has a number of knobs for configuring performance. Each knob is tuned to a reasonable default, however, you can override the default values by changing a splitio.config.json file or by setting your customer values as parameters of -config in the command line option. In this section, we lay out all the different knobs you can configure for performance, Redis, and logging.

The splitio.config.json file provided via the -config option lets you control how often the synchronizer fetches data from Split servers. You can create a sample JSON file automatically with default values by running this command.

./split-sync -write-default-config "/home/someuser/splitio.config.json"

{
 "apiKey": "YOUR API KEY",
 "proxy": {
  "port": 3000,
  "adminPort": 3010,
  "adminUsername": "",
  "adminPassword": "",
  "dashboardTitle": "",
  "persistInFilePath": "",
  "impressionsMaxSize": 10485760,
  "eventsMaxSize": 10485760,
  "auth": {
   "sdkAPIKeys": [
    "SDK_API_KEY"
   ]
  }
 },
 "redis": {
  "host": "localhost",
  "port": 6379,
  "db": 0,
  "password": "",
  "prefix": "",
  "network": "tcp",
  "maxRetries": 0,
  "dialTimeout": 5,
  "readTimeout": 10,
  "writeTimeout": 5,
  "poolSize": 10
 },
 "sync": {
  "admin": {
   "adminPort": 3010,
   "adminUsername": "",
   "adminPassword": "",
   "dashboardTitle": ""
  }
 },
 "log": {
  "verbose": false,
  "debug": false,
  "stdout": false,
  "file": "/tmp/split-agent.log",
  "fileMaxSizeBytes": 2000000,
  "fileBackupCount": 3,
  "slackChannel": "",
  "slackWebhookURL": ""
 },
 "impressionListener": {
  "endpoint": ""
 },
 "splitsRefreshRate": 60,
 "segmentsRefreshRate": 60,
 "impressionsPostRate": 20,
 "impressionsPerPost": 5000,
 "impressionsThreads": 1,
 "eventsPostRate": 60,
 "eventsPerPost": 10000,
 "eventsThreads": 1,
 "metricsPostRate": 60,
 "httpTimeout": 60
}

{
 "apiKey": "YOUR API KEY",
 "proxy": {
  "port": 3000,
  "adminPort": 3010,
  "adminUsername": "",
  "adminPassword": "",
  "dashboardTitle": "",
  "persistInFilePath": "",
  "impressionsMaxSize": 10485760,
  "eventsMaxSize": 10485760,
  "auth": {
   "sdkAPIKeys": [
    "SDK_API_KEY"
   ]
  }
 },
 "redis": {
  "prefix": "",
  "sentinelReplication": true,
  "sentinelMaster": "MASTER_SERVICE_NAME",
  "sentinelAddresses": "SENTINEL_HOST_1:SENTINEL_PORT_1, SENTINEL_HOST_2:SENTINEL_PORT_2,SENTINEL_HOST_3:SENTINEL_PORT_3"
 },
 "sync": {
  "admin": {
   "adminPort": 3010,
   "adminUsername": "",
   "adminPassword": "",
   "dashboardTitle": ""
  }
 },
 "log": {
  "verbose": false,
  "debug": false,
  "stdout": false,
  "file": "/tmp/split-agent.log",
  "fileMaxSizeBytes": 2000000,
  "fileBackupCount": 3,
  "slackChannel": "",
  "slackWebhookURL": ""
 },
 "impressionListener": {
  "endpoint": ""
 },
 "splitsRefreshRate": 60,
 "segmentsRefreshRate": 60,
 "impressionsPostRate": 60,
 "impressionsPerPost": 1000,
 "impressionsThreads": 1,
 "eventsPostRate": 60,
 "eventsPerPost": 10000,
 "eventsThreads": 1,
 "metricsPostRate": 60,
 "httpTimeout": 60
}

{
 "apiKey": "YOUR API KEY",
 "proxy": {
  "port": 3000,
  "adminPort": 3010,
  "adminUsername": "",
  "adminPassword": "",
  "dashboardTitle": "",
  "persistInFilePath": "",
  "impressionsMaxSize": 10485760,
  "eventsMaxSize": 10485760,
  "auth": {
   "sdkAPIKeys": [
    "SDK_API_KEY"
   ]
  }
 },
 "redis": {
  "prefix": "",
  "clusterMode": true,
  "clusterNodes": "CLUSTER_NODE_1:CLUSTER_PORT_1, CLUSTER_NODE_2:CLUSTER_PORT_2,CLUSTER_NODE_3:CLUSTER_PORT_3"
 },
 "sync": {
  "admin": {
   "adminPort": 3010,
   "adminUsername": "",
   "adminPassword": "",
   "dashboardTitle": ""
  }
 },
 "log": {
  "verbose": false,
  "debug": false,
  "stdout": false,
  "file": "/tmp/split-agent.log",
  "fileMaxSizeBytes": 2000000,
  "fileBackupCount": 3,
  "slackChannel": "",
  "slackWebhookURL": ""
 },
 "impressionListener": {
  "endpoint": ""
 },
 "splitsRefreshRate": 60,
 "segmentsRefreshRate": 60,
 "impressionsPostRate": 60,
 "impressionsPerPost": 1000,
 "impressionsThreads": 1,
 "eventsPostRate": 60,
 "eventsPerPost": 10000,
 "eventsThreads": 1,
 "metricsPostRate": 60,
 "httpTimeout": 60
}

Methods to Configure the Split service

You can configure the Split synchronizer service using the command line or by directly editing the above mentioned JSON configuration file.

split-sync -config "/etc/splitio.config.json" -log-debug -redis-pass "somePass" 

Common configuration (synchronizer and proxy mode)

Synchronizer configuration

The Synchronizer mode adds four new parameters to post impressions and events accurately and for its Redis dependency.

Configure Redis

Custom set Redis parameters by plugging them into the Redis section in the JSON configuration file or as command line parameters.

Configure Redis Sentinel

Custom set Redis Sentinel parameters by plugging them in to the Redis section in the JSON configuration file or as command line parameters (you can add the others parameter options listed in previous section without considering host and port).

Configuring Redis Cluster

Custom set Redis Cluster parameters by plugging them in to the Redis section in the JSON configuration file or as command line parameters (you can add the others parameters options listed in previous section also without considering host and port).

Proxy mode configuration

Set Synchronizer proxy mode parameters in the proxy section of the JSON file or as command line parameters.

Listener

The Split Synchronizer provides an impression listener that bulks post impressions to a user-defined HTTP endpoint.

The endpoint should expect a POST request, containing a JSON body with the following format.

{
 "impressions": [
  {
   "testName": "feature1",
   "keyImpressions": [
    {
     "keyName": "user1",
     "treatment": "on",
     "time": 1502754901182,
     "changeNumber": -1,
     "label": ""
    },
    {
     "keyName": "user2",
     "treatment": "off",
     "time": 1502754876144,
     "changeNumber": -1,
     "label": ""
    }
   ]
  }
 ],
 "sdkVersion": "php-5.2.2",
 "machineIP": "208.63.222.7",
 "MachineName": ""
}

Currently, the configuration options are available in the impressionListener section of the JSON configuration file detailed in Advanced configuration.

Logging

Set the logging parameters in the log section of the JSON file or as command line parameters.

Using a network proxy

If you need to use a network proxy, configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY. The internal HTTP client reads those variables and uses them to perform the server request.

$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

Admin tools

Endpoints

The split-sync service has a set of endpoints and a dashboard to let DevOps and infra team monitor its status and cached data in real-time. By default the port is 3010 and for security reason supports HTTP Basic Authentication configured by the user.

/admin/ping

A ping endpoint to monitor the service status. If the service is running, it sends the text response pong and the HTTP status code 200.

/admin/version

Returns the split-sync version in JSON format.

{
  "version" : "1.1.0"
}

/admin/uptime

Returns the uptime string representation in JSON format.

{
  "uptime" : "5d 3h 36m 39s"
}

/admin/stats

Returns stats, latencies, and counters of service behavior.

{
  "counters" : {
    "backend::request.ok" : 403760,
    "request.error" : 303,
    "request.ok" : 4530342
  },
  "latencies" : {
    "/api/segmentChanges/*" : [
      2317783,
      36,
      32,
      773024,
      7756,
      252273,
      20617,
      9588,
      9371,
      4054,
      580,
      131,
      14,
      2,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0
    ]
  }
}

/admin/healthcheck
Returns a JSON object describing whether the synchronizer is healthy or not.

PROXY
{
  "events": {
    "healthy": true,
    "message": "Events service working as expected"
  },
  "healthySince": {
    "date": "07-18-2019 22:17:25",
    "time": "0d 20h 28m 44s"
  },
  "sdk": {
    "healthy": true,
    "message": "SDK service working as expected"
  },
  "proxy" :  {
    "healthy" : true,
    "message" : "Proxy service working as expected"
  },
  "uptime": "0d 20h 28m 44s"
}

SYNC
{
  "events": {
    "healthy": true,
    "message": "Events service working as expected"
  },
  "healthySince": {
    "date": "07-18-2019 22:17:25",
    "time": "0d 20h 28m 44s"
  },
  "sdk": {
    "healthy": true,
    "message": "SDK service working as expected"
  },
  "storage": {
    "healthy": true,
    "message": "Storage service working as expected"
  },
  "sync": {
    "healthy": true,
    "message": "Synchronizer service working as expected"
  },
  "uptime": "0d 20h 28m 44s"
}

/admin/userConfig
Returns a JSON object describing the current configuration of the Synchronizer.

PROXY
{
  "admin": {
    "adminPort": 3010,
    "adminUsername": "",
    "adminPassword": "",
    "dashboardTitle": "Title"
  },
  "apiKey": "2uae...r7na",
  "eventsPerPost": 10000,
  "eventsThreads": 1,
  "eventsPostRate": 30,
  "httpTimeout": 60,
  "impressionListener": {
    "endpoint": ""
  },
  "impressionsPerPost": 1000,
  "impressionsPostRate": 30,
  "impressionsThreads": 1,
  "log": {
    "verbose": false,
    "debug": true,
    "stdout": false,
    "file": "",
    "fileMaxSizeBytes": 2000000,
    "fileBackupCount": 3,
    "slackChannel": "slack-channel",
    "slackWebhookURL": ""
  },
  "metricsPostRate": 360,
  "mode": "ProxyMode",
  "proxy": {
    "port": 3000,
    "adminPort": 3010,
    "adminUsername": "",
    "adminPassword": "",
    "dashboardTitle": "",
    "persistInFilePath": "",
    "impressionsMaxSize": 10485760,
    "eventsMaxSize": 10485760,
    "auth": {
      "sdkAPIKeys": [
        "SDK_API_KEY"
      ]
    }
  },
  "redis": null,
  "redisMode": null,
  "segmentsRefreshRate": 30,
  "splitRefreshRate": 30
}

SYNC
{
  "admin": {
    "adminPort": 3010,
    "adminUsername": "",
    "adminPassword": "",
    "dashboardTitle": "Title"
  },
  "apiKey": "2uae...r7na",
  "eventsPerPost": 10000,
  "eventsThreads": 1,
  "eventsPostRate": 30,
  "httpTimeout": 60,
  "impressionListener": {
    "endpoint": ""
  },
  "impressionsPerPost": 1000,
  "impressionsPostRate": 30,
  "impressionsThreads": 1,
  "log": {
    "verbose": false,
    "debug": true,
    "stdout": false,
    "file": "",
    "fileMaxSizeBytes": 2000000,
    "fileBackupCount": 3,
    "slackChannel": "slack-channel",
    "slackWebhookURL": ""
  },
  "metricsPostRate": 360,
  "mode": "ProducerMode",
  "proxy": null,
  "redis": {
    "host": "localhost",
    "port": 6379,
    "db": 0,
    "password": "",
    "prefix": "{SPLITIO}test:",
    "network": "tcp",
    "maxRetries": 0,
    "dialTimeout": 5,
    "readTimeout": 10,
    "writeTimeout": 5,
    "poolSize": 10,
    "sentinelReplication": false,
    "sentinelAddresses": "",
    "sentinelMaster": "",
    "clusterMode": true,
    "clusterNodes": "NODE1,NODE2,NODE3",
    "keyHashTag": ""
  },
  "redisMode": "Cluster",
  "segmentsRefreshRate": 30,
  "splitRefreshRate": 30
}

SYNC Endpoints

/admin/events/queueSize & /admin/impressions/queueSize
Returns a JSON with the amount of elements stored in redis.

{
  "queueSize": Number
}

/admin/events/drop & /admin/impressions/drop
Discard Events or Impressions

Method: POST
Data In
    - Name size
      Validation: 1..X
      Optional
      QueryParam
RESPONSE:
    - 200 if no error occurs
    - 400 if wrong size parameter is passed
    - 5XX if server error

/admin/events/flush & /admin/impressions/flush
Flush Events or Impressions by sending them to the Split Servers

Method: POST
Data In
    - Name size
      Validation: 1..25000
      Optional
      QueryParam
RESPONSE:
    - 200 if no error occurs
    - 400 if wrong size parameter is passed
    - 5XX if server error

Admin Dashboard

Split-sync has a web admin UI out of the box that exposes all available endpoints. Browse to /admin/dashboard to see it.

Proxy Mode Main

Sync Mode Main

The dashboard is organized in four sections for ease of visualization:

• Dashboard: Tile-sorted summary information, including these metrics:
  • Uptime: Uptime metric
  • Healthy Since: Time passed without errors
  • Logged Errors: Total count of error messages
  • SDKs Total Hits: Total SDKs requests
  • Backend Total Hits: Total backend requests between split-sync and Split servers
  • Cached Splits: Number of splits cached in memory
  • Cached Segments: Number of segments cached in memory
  • Impressions Queue Size: shows the total amount of Impressions stored in Redis (only Producer Mode).
  • Impressions Lambda: shows the eviction rate for Impressions (only Producer Mode).
  • Events Queue Size: shows the total amount of Events stored in Redis (only Producer Mode).
  • Events Lambda: shows the eviction rate for Events (only Producer Mode).
  • SDK Server: displays the status of Split Server for SDK.
  • Events Server: displays the status of Split Server for Events.
  • Storage: (only Sync mode) displays the status of the storage.
  • Sync: displays the status of the Synchronizer.
  • Last Errors Log: List of the last 10 error messages
• SDK stats: Metrics numbers and a latency graph, measured between SDKs requests integration and proxy.
• Split stats: Metrics numbers and a latency graph, measured between proxy requests integration with Split servers.
• Data inspector: Cached data showing splits and segments; filters to find keys and split definitions.
• Queue Manager: expose sizes of Impressions and Events queues. Allows to drop/flush Events or Impressions. Displays the eviction rate for Impressions and Events.

Service shutdown

The split-sync service can catch a kill sig command and start a graceful shutdown, flushing all cached data progressively. Additionally, you can perform graceful stop and force stop (kill -9) with one click from the admin dashboard.

If you have configured a Slack channel and the Slack Webhook URL, see the Logging section. An alert is sent to the channel when and initialization or shutdown is performed.