# AWS S3 *** ### Overview LightBeam Spectra users can connect various data sources to the LightBeam application and these data sources will be continuously monitored for PII, PHI data. **Example**: AWS S3, Google Drive, OneDrive, SharePoint, etc. *** ### Connecting AWS S3 Data Source 1. Login to your LightBeam Instance.
2. Click on **DATASOURCES** on the Top Navigation Bar.
3. Click on **“Add a data source”.**

Figure 1. Add Data Source

4. Search for “**AWS S3**”.

Figure 2. Search AWS S3

5. Click on **AWS S3**.

Figure 2.1 AWS S3

6. Fill in the requested information and click on **Next**. #### Basic Information 1. **Data Source Name:** This is the unique name given to the data source. 2. **Description:** This is an optional field needed to describe the use of this data source. 3. **Primary Owner:** Email address of the person responsible for this data source which will get alerts by default. 4. **Entity Creation:** LightBeam Spectra detects and associates attributes based on the context and identifies whose data it is; these are called entities. **Example:** Jane Doe is an entity for whom LightBeam Spectra might have detected Name and SSN in a monitored data source. 5. **Source of Truth:** LightBeam Spectra would have monitored data sources that contain data acting as a single point of truth and that can be used for looking up entities/attributes which help to identify if the other attributes/entities found in any other data source are accurate or not. A Source of Truth data set would create entities based on the attributes found in the data. 6. **Location:** The location of the data source. 7. **Purpose:** The purpose of the data being collected/processed. 8. **Stage**: The stage of the data source. **Example**: Source, Processing, Archival, etc.

Figure 3. LightBeam AWS S3 - Basic Information

{% hint style="info" %} **Note**: To get the AWS S3 connection details please check [#appendix](#appendix "mention"). {% endhint %} 7. **Data Source Configuration** \ \ LightBeam uses the "**Live Scan**" approach, which tracks changes made to objects in buckets and makes use of **AWS EventBridge** to provide real-time updates of these changes. Each bucket must have the EventBridge service enabled for this to work. If it isn't already enabled, LightBeam will do so automatically.
1\. Under **Authentication Method**, choose: **a. Access Key/Secret Key** (default) **b. IAM Role** (*Only for AWS EKS deployments*)\ \ \ 2\. In the **"Scan Data"** section, specify how frequently LightBeam should scan your S3 buckets: * **Numerical Input**: Enter a value (e.g., `10`) or use ▲/▼ arrows to adjust. * **Unit Selector**: Choose `Seconds`, `Minutes`, `Hours`, or `Days` by dropdown. **Examples**: * `Every 30 Minutes` * `Every 2 Hours` * `Every 1 Day`

Figure 4. LightBeam AWS S3 - Configuration

**a. For Access Key/Secret Key**: Please ensure that appropriate permissions to do this are configured with these credentials. * **Access keys:** Access keys are long-term credentials for an **IAM user** or the **AWS account root user.** You can use access keys to sign programmatic requests to the AWS CLI or AWS API (directly or using the AWS SDK).
* **Secret access keys:** Secret Access keys are like your password. AWS does not allow retrieval of a secret access key after its initial creation. This applies to both **root secret access keys** and **AWS Identity** **and Access Management (IAM) user secret** access keys.
**b. For IAM Role**: * Ensure LightBeam is deployed on an AWS EKS cluster. * Verify the EKS node group’s IAM role has the required policy (see Appendix: IAM Role Setup). * No credentials needed – authentication is role-based.
8. Click on **Test Connection**. \
9. Verify that you get the message **Connection Success!** on the screen. Click on **Next**. \
10. In this step, you can choose either of two scan setting options – i) Scan all buckets ii) Scan selected buckets iii) Scan selected folders To choose **option (i)**, select **Scan all Buckets,** and click on **Validate And Save**.

Figure 5.1(a) Scan all buckets - Registration of S3 Buckets

This will allow for the registration of the AWS S3 buckets.
### Registration of S3 Buckets Registration of S3 buckets is a two-step process: 1. **Validating the bucket:** Certain modifications are made to the client's S3 buckets by LightBeam Spectra. After these adjustments take effect, LightBeam Spectra starts scanning the buckets in real-time. 2. **Downloading JSON file:** Following the validation of the buckets, an automated **JSON file** download will reveal a history of any modifications made to the user's buckets, including the configuration both before and after the modifications. This will help the user to track the changes made by LightBeam. To choose **option (ii)**, select **Scan selected Buckets**. Now enter the names of the buckets that you would like to scan in the **Search** box individually. Select the buckets by ticking the checkboxes next to them.

Figure 5.2(a) LightBeam AWS S3 - Scan selected buckets

To choose **option (iii)**, select **Scan selected folders**. Here we need to enter name folder within bucket which we want to scan in format **s3://\/\.**

Figure 5.3(a) Scan selected folders - Registration of S3 Buckets

This will start the identical procedure outlined in the [Registration of S3 Buckets](#registration-of-s3-buckets) section. If these modifications fail for any of the reasons listed in the [Failure in Validating Scan Settings](#failure-in-validating-scan-settings) section. 4. Once the required buckets is selected, click on **Save** Now that the AWS S3 datasource is connected to LightBeam, we can begin viewing the dashboard and other pages of the onboarded datasource.
*** ### Failure in Validating Scan Settings A validation failure for the AWS S3 buckets can occur in the following cases: 1. **Failure in enabling AWS Eventbridge:** If the Eventbridge service is not functioning properly, it may impact the ability to verify the scan parameters, which could result in incorrect or incomplete scans. There can be various reasons why AWS Eventbridge may fail to launch, including: * *Network connectivity issues* * *Insufficient permissions or access to the service* * *Incorrect configuration of Eventbridge settings* * *Resource constraints, such as low memory or disk space* * *Service outages or maintenance by AWS* * *Software bugs or compatibility issues.* *** ## Appendix ### Steps to get AWS S3 credentials #### Create IAM User **(Access Key/Secret Key)** To onboard AWS S3 datasource we need the **AWS Access key** and **AWS Secret key** of the IAM user with the following permissions: ``` { "Statement": [ { "Sid": "Stmt", "Action": [ "servicequotas:GetServiceQuota" ], "Effect": "Allow", "Resource": "arn:aws:servicequotas:*:*:events/L-244521F2" }, { "Sid": "Stmt100000", "Effect": "Allow", "Action": [ "events:ListEventBuses", "events:ListRules" ], "Resource": "arn:aws:events:*:*:*/*" }, { "Sid": "Stmt1673701659146", "Action": [ "events:CreateEventBus", "events:DeleteEventBus", "events:DeleteRule", "events:DescribeEventBus", "events:DescribeEventSource", "events:DescribeRule", "events:DisableRule", "events:EnableRule", "events:ListEventBuses", "events:ListEventSources", "events:PutRule", "events:PutTargets", "events:RemoveTargets" ], "Effect": "Allow", "Resource": [ "arn:aws:events:*:*:*/lightbeam-*-s3-rule-*", "arn:aws:events:*:*:*/lightbeam-*-s3-bus" ] }, { "Action": [ "sns:TagResource", "sqs:DeleteMessage", "sns:DeleteTopic", "sqs:UntagQueue", "sqs:ReceiveMessage", "sqs:ListQueueTags", "sqs:RemovePermission", "sns:Subscribe", "sns:Unsubscribe", "sqs:ChangeMessageVisibility", "sqs:SetQueueAttributes", "sqs:GetQueueUrl", "sqs:ChangeMessageVisibility", "sns:CreateTopic", "sns:GetTopicAttributes", "sns:SetTopicAttributes", "sqs:SendMessage", "sqs:GetQueueAttributes", "sqs:TagQueue", "s3:PutBucketNotification", "sqs:ListDeadLetterSourceQueues", "sqs:AddPermission", "sqs:PurgeQueue", "sqs:DeleteQueue", "sqs:CreateQueue", "s3:GetBucketLocation" ], "Effect": "Allow", "Resource": [ "arn:aws:sqs:*:*:lightbeam-*-s3-events", "arn:aws:sns:*:*:lightbeam-*-s3-events" ], "Sid": "VisualEditor0" }, { "Action": [ "sqs:ListQueues", "sns:Unsubscribe" ], "Effect": "Allow", "Resource": [ "arn:aws:sqs:*:*:lightbeam-*-s3-events", "arn:aws:sns:*:*:lightbeam-*-s3-events" ], "Sid": "VisualEditor1" }, { "Sid": "Stmt1673716987802", "Action": [ "s3:ListAllMyBuckets", "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:GetBucketNotification", "s3:GetBucketOwnershipControls", "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectAttributes", "s3:GetObjectTagging", "s3:GetObjectVersion", "s3:GetObjectVersionAcl", "s3:GetObjectVersionAttributes", "s3:GetObjectVersionTagging", "s3:ListBucket", "s3:PutBucketNotification" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::*", "arn:aws:s3:::*/*" ] }, { "Sid": "KMS", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey*" ], "Resource": [ "arn:aws:kms:*:*:*" ] } ], "Version": "2012-10-17" } ``` The IAM permissions are limited write and read access to [`EventBridge`](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html), [`SQS`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html), [`SNS`](https://docs.aws.amazon.com/sns/latest/dg/welcome.html), [`S3`](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html). We require **read** and **write** permissions for the following reasons: ***`EventBridge:`**We use EventBridge EventBus to consume real-time change events from S3 and route them to SNS topics.* ***`S3:`**We require read permissions for all files and write permissions to modify the bucket notification configuration for powering real-time sync of data to LightBeam.* ***`SQS:`** We create new SQS queues to consume the real-time change events from S3.* ***`SNS:`**We use a fan-out approach to subscribe SNS topics to SQS queues so multiple actors in the system can consume these events.* These SNS topics, SQS topics etc are created as part of datasource registration by LightBeam backend. These resources may incur additional cost on AWS. **Note**: For External and Open access files evaluation we need following permissions to be added also: ``` [ { "Sid": "S3SecurityAuditing", "Effect": "Allow", "Action": [ "s3:GetBucketPublicAccessBlock", "s3:GetBucketPolicy", "s3:GetAccountPublicAccessBlock" ], "Resource": "*" }, { "Sid": "OrganizationDiscovery", "Effect": "Allow", "Action": [ "organizations:ListAccounts", "organizations:DescribeOrganization" ], "Resource": "*" }, { "Sid": "IAMRoleAuditing", "Effect": "Allow", "Action": [ "iam:ListRoles", "iam:ListRolePolicies", "iam:GetRolePolicy", "iam:ListAttachedRolePolicies", "iam:GetPolicy", "iam:GetPolicyVersion" ], "Resource": "*" }, { "Sid": "STSIdentity", "Effect": "Allow", "Action": [ "sts:GetCallerIdentity" ], "Resource": "*" } ] ``` #### Create **IAM Role (EKS Node Group)** Follow these steps to attach the required policy to your EKS node group’s IAM role: 1. Navigate to the EKS Cluster.
2. Click on **Compute -> Node** group. Select the node group on which Lightbeam is running.
3\. Open the **IAM Role** attached to the node group.
4: Click on **Add permissions -> Create inline policy**
5: Copy the JSON payload. ``` { "Statement": [ { "Sid": "Stmt", "Action": [ "servicequotas:GetServiceQuota" ], "Effect": "Allow", "Resource": "arn:aws:servicequotas:*:*:events/L-244521F2" }, { "Sid": "Stmt100000", "Effect": "Allow", "Action": [ "events:ListEventBuses", "events:ListRules" ], "Resource": "arn:aws:events:*:*:*/*" }, { "Sid": "Stmt1673701659146", "Action": [ "events:CreateEventBus", "events:DeleteEventBus", "events:DeleteRule", "events:DescribeEventBus", "events:DescribeEventSource", "events:DescribeRule", "events:DisableRule", "events:EnableRule", "events:ListEventBuses", "events:ListEventSources", "events:PutRule", "events:PutTargets", "events:RemoveTargets" ], "Effect": "Allow", "Resource": [ "arn:aws:events:*:*:*/lightbeam-*-s3-rule-*", "arn:aws:events:*:*:*/lightbeam-*-s3-bus" ] }, { "Action": [ "sns:TagResource", "sqs:DeleteMessage", "sns:DeleteTopic", "sqs:UntagQueue", "sqs:ReceiveMessage", "sqs:ListQueueTags", "sqs:RemovePermission", "sns:Subscribe", "sns:Unsubscribe", "sqs:ChangeMessageVisibility", "sqs:SetQueueAttributes", "sqs:GetQueueUrl", "sqs:ChangeMessageVisibility", "sns:CreateTopic", "sns:GetTopicAttributes", "sns:SetTopicAttributes", "sqs:SendMessage", "sqs:GetQueueAttributes", "sqs:TagQueue", "s3:PutBucketNotification", "sqs:ListDeadLetterSourceQueues", "sqs:AddPermission", "sqs:PurgeQueue", "sqs:DeleteQueue", "sqs:CreateQueue", "s3:GetBucketLocation" ], "Effect": "Allow", "Resource": [ "arn:aws:sqs:*:*:lightbeam-*-s3-events", "arn:aws:sns:*:*:lightbeam-*-s3-events" ], "Sid": "VisualEditor0" }, { "Action": [ "sqs:ListQueues", "sns:Unsubscribe" ], "Effect": "Allow", "Resource": [ "arn:aws:sqs:*:*:lightbeam-*-s3-events", "arn:aws:sns:*:*:lightbeam-*-s3-events" ], "Sid": "VisualEditor1" }, { "Sid": "Stmt1673716987802", "Action": [ "s3:ListAllMyBuckets", "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:GetBucketNotification", "s3:GetBucketOwnershipControls", "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectAttributes", "s3:GetObjectTagging", "s3:GetObjectVersion", "s3:GetObjectVersionAcl", "s3:GetObjectVersionAttributes", "s3:GetObjectVersionTagging", "s3:ListBucket", "s3:PutBucketNotification" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::*", "arn:aws:s3:::*/*" ] }, { "Sid": "KMS", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey*" ], "Resource": [ "arn:aws:kms:*:*:*" ] } ], "Version": "2012-10-17" } ```
6. Scroll down and click on "**Next**".
7. Enter the name as "`lightbeam-s3-policy`" and click on "**Create policy**".
*** ## About LightBeam LightBeam automates Privacy, Security, and AI Governance, so businesses can accelerate their growth in new markets. Leveraging generative AI, LightBeam has rapidly gained customers’ trust by pioneering a unique *privacy-centric* and *automation-first* approach to security. Unlike siloed solutions, LightBeam ties together sensitive data cataloging, control, and compliance across structured and unstructured data applications providing 360-visibility, redaction, self-service DSRs, and automated ROPA reporting ensuring ultimate protection against ransomware and accidental exposures while meeting data privacy obligations efficiently. \ \ LightBeam is on a mission to create a secure privacy-first world helping customers automate compliance against a patchwork of existing and emerging regulations. For any questions or suggestions, please get in touch with us at: .