Amazon S3 – Split Help Center

Overview

Amazon Simple Storage Service (Amazon S3) is an object storage service offered by Amazon as part of Amazon Web Services (AWS) that offers the ability to store and retrieve any amount of data, at any time, from anywhere on the web. With this integration you can easily and reliably send high volumes of event data to Split, as well as export your impression data.

Send events to Split

Prepare your S3 Bucket and event files

To connect your AWS S3 bucket to Split, you need:

An S3 bucket that will contain files with supported file types and encodings.
An AWS account with the ability to grant Split permission to read the folder with data in the bucket as well as read/put permission for the folder where the status is expected to be posted.

File restrictions:

Compression examples: gzip, bzip2, snappy, or LZO compression.
File format: parquet only

Schema:

- environmentID: [String] Identifier for the environment in Split
- trafficTypeID: [String] Identifier for the traffic type in Split
- eventTypeID: [String] Name for the type of event
- key: [String] Identifier for the "user" or "system" triggering or associated with the event
- timestamp: [Long] The time when the event took place, in milliseconds past epoch
- value: [Double, Nullable] A numeric value associated with this event
- properties: [Map<String,String>, Nullable] Arbitrary properties data providing context to the event

(To send the same event into multiple environments within Split, you must duplicate the events with different environment IDs.)

Create Amazon S3 bucket

(Follow this step only if you do not already have an S3 bucket. If you already have one created, find your Bucket ARN and move to the next step.)

Sign in to the AWS Management Console and open the Amazon S3 console.
Click Create Bucket.
Create a unique Bucket name. All other settings are optional.
Click Save. Note your Bucket ARN.

Navigate to the Amazon SNS console. Click Create Topic.
Under Details > Type, select Standard.
Create a Name for your SNS topic. (Example: "split-events")
Click Save. Note the SNS Topic ARN.
Navigate to your SNS topic and click Edit.
Navigate to Access policy.
Edit the JSON snippet by adding the following snippet as a new statement (do not replace the full snippet.) Be sure to update the JSON snippet with your SNS Topic ARN, your AWS Account ID, and your Bucket ARN.

{
      "Sid": "S3 Events Notification",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "{YourSNSTopicARN}",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "{YourAwsAccountID}"
        },
        "ArnLike": {
          "aws:SourceArn": "{YourBucketArn}"
        }
     }
}

Click Save.

Configure S3 event notifications

In the Amazon S3 console, navigate to your S3 bucket.
Under Event notifications, click Create event notification.
Provide an Event name.
Under Suffix, add the suffix for your compression type (example: .parquet.gz or .snappy.parquet.)
(Optional) You may also specify a Prefix like "events/".
Under Event types, select All object create events.
Under Destination > Destination, select SNS topic. Then, under Specify SNS topic, select Choose from your SNS topics. Find your SNS topic created in the previous section. (Optional: you can also enter the SNS topic ARN if you cannot find your topic in the dropdown list.)
Click Save changes.

Configure integration (inbound) in Split

In the Split UI, navigate to admin settings, click integrations, select your workspace, and navigate to the marketplace.
Click add next to Amazon S3. (You can also click Warehouse under Categories to filter to Amazon S3.)
Under Send events to Split, and click Add configuration.
Under Select environment, choose which Split environments that will receive S3 data.
Under Paste S3 bucket name, provide the name of your unique S3 bucket from above. (Note: we will create it for you if the directory specified does not exist.)
Under Paste SNS topic ARN, provide the SNS topic ARN from above.
Under Paste status folder location, provide the file path for status files.
Click Save.
Navigate back to your Amazon S3 integration. Note that Split has generated two fields: Notification endpoint URL and Role ARN.

Update S3 bucket policy for Split access (inbound)

In the Amazon S3 console, navigate to your S3 bucket.
Under Edit bucket policy > Bucket policy, replace the JSON with the snippet below. Note that you should update the Role ARN with the value generated by Split, as well as your bucket name as appropriate.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "{role-ARN}"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::{bucketName}",
                "arn:aws:s3:::{bucketName}/{filePrefix}/*"
            ]
        }, 
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "{role-ARN}"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::{bucketName}/{statusPrefix}/*"
            ]
        }, 
    ]
}

Navigate to the Amazon SNS console, and find your SNS Topic created above. Click Create Subscription.
Under Details, provide your Topic ARN.
Under Protocol choose HTTPS.
Under Endpoint, provide the Notification endpoint URL generated by Split.
Click Create subscription. On the confirmation page, confirm that Check status is “confirmed” and Protocol is “https.”

Send Split impressions to S3

Prepare your S3 bucket

To connect your AWS S3 bucket to Split, you need:

An S3 bucket destination already created.
An AWS account with the ability to grant Split permission to read,put, and remove data in the bucket. (We will only remove transactional files created during the transfer.)

Configure integration (outbound) in Split

In the Split UI, navigate to admin settings, click integrations, select your workspace, and navigate to the marketplace.
Click add next to Amazon S3. (You can also click Warehouse under Categories to filter to Amazon S3.)
Under Send Impressions to S3, click Add configuration
Under Select environment, choose which Split environments that will receive S3 data.
Under Paste S3 bucket name, provide the name of your unique S3 bucket name.
(Optional) Under Provide the write location in your S3 bucket, put the S3 prefix you’d like the impressions files to be saved to. If no value is provided, files will be saved to the root directory of your provided S3 bucket.
Under Select file format, select your desired format for your impressions files.
Under Select file compression type, select your desired file compression format for your impression files.
- Note: the dropdown will only display compression methods that are compatible with your chosen file format.
Click Save. Note your Role ARN generated by Split.

Update S3 bucket policy for Split access (outbound)

In the Amazon S3 console, navigate to your S3 bucket.
Under Permissions > Edit bucket policy > Bucket policy, replace the JSON with the snippet below. Be sure to replace the following parameters in the snippet:
- {role-ARN-to-be-provided-to-you-by-Split} with the Role ARN generated by Split above.
- {bucketName} with your S3 bucket name
- {writeLocationPrefix} with the write location prefix as appropriate. If you left the write location section blank in the Split configuration step, and intend to write to the root directory, then you must update the following line:
  - “Resource": "arn:aws:s3:::{bucketName}/{writeLocationPrefix}/*" > "Resource": "arn:aws:s3:::{bucketName}/*”.
(Optional) If you later update the integration to include a write location prefix, or change the write location in any way, then we recommend changing the policy to the more restricted line provided below to include the {writeLocationPrefix}/*

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "{role-ARN-to-be-provided-to-you-by-Split}"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::{bucketName}"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "{role-ARN-to-be-provided-to-you-by-Split}"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::{bucketName}/{writeLocationPrefix}/*"
        }
    ]
}

(Optional) SSE_KMS Support

If your S3 bucket is encrypted using an AWS KMS key (SSE-KMS), you need to update the policy to allow Split's AWS ARN to encrypt data using the KMS key you have chosen.) If your S3 bucket is using an Amazon S3 key for encryption (SSE-S3), then the following is not necessary.

In the Amazon S3 console, navigate to the Key Management Service (KMS) dashboard.
Navigate to the KMS key being used for your encrypted S3 bucket.
Navigate to the KMS key policy and make sure the following actions are included in your existing key policy:

{
    "Version": "2012-10-17",
    "Id": "cross-account-policy",
    "Statement": [
        {
            "Sid": "KMSOwner",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{your-aws-account-id}:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "KMSCrossAccount",
            "Effect": "Allow",
            "Principal": {
                "AWS": "{role-ARN-to-be-provided-to-you-by-Split}"
            },
            "Action": [
                "kms:DescribeKey",
                "kms:GenerateDataKey",
    "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}