Setting up audit log streaming

CDF can stream audit logs continuously to cloud storage you own and control. Once streaming is configured, new audit log entries are delivered to your storage destination as they are generated, giving you a queryable, long-term copy that you can feed into a security information and event management (SIEM) or analysis tool. Setting up a stream is a two-part process: you create the storage destination and credentials in your cloud account, then share a configuration JSON securely with your Cognite representative, who completes the setup on the CDF side.

Azure Blob Storage

Create a storage account and container

In Microsoft Azure, create a storage account and a container to receive the audit log files. See Introduction to Azure Blob Storage in the Microsoft documentation.

Generate a SAS token

From the Azure portal:

On the Home page, select Storage accounts.
Under Name, select the storage account you want to use.
Under Data storage, select Containers.
Select the container you want to use.
In the left sidebar, under Settings, select Shared access tokens.
Open the Permissions dropdown and select Create, Read, Add, List, and Write. Deselect all other options.
Set an expiry date that complies with your secret rotation policy.
Select Generate SAS token and URL.

Copy the value of the Blob SAS URL field. The URL has this format:

https://{STORAGE_ACCOUNT}.blob.core.windows.net/{CONTAINER}?{SAS_TOKEN}

Build the configuration JSON

Create a JSON using the values from your SAS URL:

{
  "TARGET_URL": "abfss://{CONTAINER}/audit-logs",
  "ORG": "{CUSTOMER_CDF_ORG_NAME}",
  "AZURE_STORAGE_ACCOUNT_NAME": "{STORAGE_ACCOUNT}",
  "AZURE_STORAGE_SAS_TOKEN": "{SAS_TOKEN}"
}

The audit-logs segment in TARGET_URL is the path within the container. You can request a different path when you contact your Cognite representative.

Share the configuration with Cognite

Send the JSON to your Cognite representative through a secure channel.

Amazon S3

Create an S3 bucket

Create a bucket and block all public access to it. See Creating, configuring, and working with Amazon S3 buckets in the AWS documentation.

Create an IAM policy

Create a policy that allows CDF to write to the bucket. Copy the following JSON and replace EXAMPLE-BUCKET with the name of your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::EXAMPLE-BUCKET",
      "Condition": {
        "StringLike": {
          "s3:prefix": "audit-logs/*"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::EXAMPLE-BUCKET/audit-logs/*"
    }
  ]
}

The audit-logs segment in s3:prefix is the path within the bucket. You can request a different path when you contact your Cognite representative.See Creating IAM policies in the AWS documentation.

Create an IAM user and access key

Create an IAM user and attach the policy you created directly to the user.
Open the user and select Create access key.
Copy the Access key ID and Secret access key.

For more information, see Understanding and getting your AWS credentials in the AWS documentation.

Build the configuration JSON

Create a JSON using the values from your bucket and IAM user:

{
  "AWS_ACCESS_KEY": "{YOUR_IAM_USER_ACCESS_KEY}",
  "AWS_SECRET_ACCESS_KEY": "{YOUR_IAM_USER_ACCESS_KEY_SECRET}",
  "AWS_REGION": "{YOUR_S3_BUCKET_REGION}",
  "ORG": "{CUSTOMER_CDF_ORG_NAME}",
  "TARGET_URL": "s3://{NAME_OF_YOUR_BUCKET}/audit-logs/"
}

Share the configuration with Cognite

Send the JSON to your Cognite representative through a secure channel.

Google Cloud Storage

Create a Cloud Storage bucket

Create a new bucket in your Google Cloud project and enable public access prevention on it. See Create buckets in the Google Cloud documentation.

Create a custom IAM role

The built-in Storage Object User role includes delete permissions that audit log delivery does not need. Create a custom role with only the permissions required:

Permission	Purpose
`storage.buckets.get`	Access the bucket
`storage.objects.create`	Write new log files
`storage.objects.get`	Read existing Delta log entries
`storage.objects.list`	List objects in the table path

See Create and manage custom roles in the Google Cloud documentation.

Create a service account and grant access

Create a service account in your Google Cloud project. See Create service accounts in the Google Cloud documentation.
On the Cloud Storage bucket, grant the custom role you created to the service account. Grant the role at the bucket level, not the project level.
Open the service account, go to the Keys tab, and select Add key → Create new key.
Select JSON as the key type and download the key file.

Build the configuration JSON

Create a JSON in the following format. Set GCS_SERVICE_ACCOUNT_KEY to the entire contents of the downloaded key file as a single-line string:

{
  "ORG": "{CUSTOMER_CDF_ORG_NAME}",
  "TARGET_URL": "gs://{NAME_OF_YOUR_BUCKET}/audit-logs",
  "GCS_SERVICE_ACCOUNT_KEY": "{SERIALIZED_JSON_KEY}"
}

The audit-logs segment in TARGET_URL is the path within the bucket. You can request a different path when you contact your Cognite representative.

Share the configuration with Cognite

Send the JSON to your Cognite representative through a secure channel.

Administration

Setting up audit log streaming

Azure Blob Storage

Amazon S3

Google Cloud Storage

Further reading

​Azure Blob Storage

​Amazon S3

​Google Cloud Storage

​Further reading

Azure Blob Storage

Amazon S3

Google Cloud Storage

Further reading