Installer Guides

AWS

Installer Guide for AWS

Setup Overview

  • There are 2 scenarios involved when deploying the LightBeam application:

  1. EKS cluster is already present on the customer account.

  2. EKS cluster is to be deployed by Lightbeam.

Note: This pre-setup is required only for when EKS cluster is to be deployed by Lightbeam.

  • In this case, the user needs to first create 2 IAM/SSO users with a required set of permissions.

  • The first user/role will have a larger set of permissions for deploying the EKS cluster, while the other user/role will be having EKS related permissions only to access the cluster post-deployment.

LightBeam Installation:

Before proceeding with the steps in this guide, ensure you have gone through the LightBeam Installation process. Familiarize yourself with the installation prerequisites and steps to ensure a smooth setup.

1.0 Deployment Architecture

Figure 1. EKS Cluster Network Diagram

Figure 2. LightBeam Microservices and Connection to Data Sources

2.0 Create and Set up a Jump Box VM

Create a t2.micro instance (Ubuntu OS) to use that as a jump box to configure and access the EKS cluster.

Note: Verify that the deployed Jump Box VM has internet access (confirm that firewall rules allow outbound traffic).

Set up the Jump Box VM

  • Install unzip package:

sudo apt-get update
sudo apt-get install unzip

  • Install kubectl cli:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mv 
kubectl version

sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform=1.3.7

  • Install aws cli:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

  • Setup python:

sudo cp /usr/bin/python3 /usr/bin/python
sudo apt install python3-pip

  • Install docker:

sudo apt-get -y remove docker docker-engine docker.io containerd runc
sudo apt-get update -y
sudo apt-get install -y apt-transport-https   ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu    $(lsb_release -cs) stable"

3.0 Creating LightBeam EKS Deployment Users

AWS_REGION: This refers to the AWS Region where the EKS cluster will be established.

AWS_ACCOUNT_ID: This is the AWS Account ID where the EKS cluster will be created.

EKS_CLUSTER_NAME: This is the chosen name for the EKS cluster.

Creating EKS admin user (Used only for EKS Cluster Deployment)

  1. Log in to the AWS Console, and using the Service search, navigate to IAM Management Console (amazon.com). On the left tab, click on Policies->Create policy.

  2. Select the JSON tab, and paste the following JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:ListRoleTags",
                "iam:RemoveRoleFromInstanceProfile",
                "iam:CreateRole",
                "iam:CreatePolicy",
                "iam:GetPolicy",
                "iam:CreateOpenIDConnectProvider",
                "iam:GetPolicyVersion",
                "iam:GetOpenIDConnectProvider",
                "iam:ListPolicyVersions",
                "iam:DeleteOpenIDConnectProvider",
                "iam:DeletePolicy",
                "iam:AttachRolePolicy",
                "iam:AddRoleToInstanceProfile",
                "iam:DetachRolePolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:ListPolicies",
                "iam:GetRole",
                "iam:DeleteRole",
                "iam:CreateInstanceProfile",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole",
                "iam:DeleteInstanceProfile",
                "iam:GetInstanceProfile",
                "iam:ListRoles",
                "iam:ListInstanceProfiles",
                "iam:CreateServiceLinkedRole",
                "eks:UpdateClusterConfig",
                "eks:DeleteFargateProfile",
                "eks:DescribeFargateProfile",
                "eks:ListTagsForResource",
                "eks:DescribeNodegroup",
                "eks:ListNodegroups",
                "eks:DeleteCluster",
                "eks:CreateFargateProfile",
                "eks:DeleteNodegroup",
                "eks:UpdateNodegroupConfig",
                "eks:DescribeCluster",
                "eks:ListClusters",
                "eks:UpdateClusterVersion",
                "eks:UpdateNodegroupVersion",
                "eks:ListUpdates",
                "eks:CreateCluster",
                "eks:UntagResource",
                "eks:CreateNodegroup",
                "eks:ListFargateProfiles",
                "eks:DescribeUpdate",
                "eks:TagResource",
                "logs:ListLogDeliveries",
                "logs:CreateLogStream",
                "logs:TagLogGroup",
                "logs:CreateLogGroup",
                "logs:CreateLogDelivery",
                "logs:ListTagsLogGroup",
                "logs:DescribeLogStreams",
                "logs:DeleteLogStream",
                "logs:DeleteLogDelivery",
                "logs:DescribeLogGroups",
                "logs:UntagLogGroup",
                "logs:DeleteLogGroup",
                "logs:PutLogEvents",
                "logs:UpdateLogDelivery",
                "logs:PutRetentionPolicy",
                "logs:GetLogGroupFields",
                "sts:AssumeRole",
                "sts:GetCallerIdentity",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:RunInstances",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:DescribeVpcs",
                "ec2:DescribeTags",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeRouteTables",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeImages",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeAccountAttributes",
                "ec2:DeleteTags",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteKeyPair",
                "ec2:CreateTags",
                "ec2:CreateSecurityGroup",
                "ec2:CreateLaunchTemplateVersion",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateKeyPair",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:ReplaceRoute",
                "ec2:ModifyVpcAttribute",
                "ec2:ModifySubnetAttribute",
                "ec2:DisassociateRouteTable",
                "ec2:DetachInternetGateway",
                "ec2:DeleteVpc",
                "ec2:DeleteSubnet",
                "ec2:DeleteRouteTable",
                "ec2:DeleteRoute",
                "ec2:DeleteInternetGateway",
                "ec2:CreateVpc",
                "ec2:CreateSubnet",
                "ec2:CreateRouteTable",
                "ec2:CreateRoute",
                "ec2:CreateInternetGateway",
                "ec2:AttachInternetGateway",
                "ec2:AssociateRouteTable",
                "ec2:DescribeVpcAttribute",
                "ec2:AllocateAddress",
                "ec2:DescribeVpcClassicLink",
                "ec2:DescribeVpcClassicLinkDnsSupport",
                "ec2:DescribeAddresses",
                "ec2:ReleaseAddress",
                "ec2:CreateNatGateway",
                "ec2:DescribeNatGateways",
                "ec2:DeleteNatGateway",
                "eksCreateAddon",
                "eks:DeleteAddon",
                "eks:DescribeAddon",
                "eks:DescribeAddonConfiguration",
                "eks:DescribeAddonVersions",
                "eks:UpdateAddon"
            ],
            "Resource": "*"
        }
    ]
}

  1. Give an appropriate name to the policy and click on Save.

  2. Now, on the left panel of the IAM Management Console (amazon.com), click Users->Add Users.

  3. Give it an appropriate username, select only Programmatic Access, and click on Next.

  4. Click on the Existing Policies tab and attach the policy created in Step 3. Also, attach the VPCFullAccess managed policy to the user and save it.

  5. On the Create page, ensure that you copy the Access Key Id and the Secret Access Key.

Alternatively, instead of creating a user with programmatic access, you have the option to create a role:

  1. Go to the IAM Management Console (amazon.com).

  2. On the left tab, click on Roles -> Create role.

  3. Select "AWS account" as the trusted entity and ensure the "This account" radio button is selected.

  4. Optionally enable Multi-Factor Authentication (MFA) if required by your organization's security policy.

  5. While adding permissions to this role, select the policy you created in the previous step (Step 3) and the "VPCFullAccess" managed policy.

  6. To assume this role, use the AWS CLI command aws sts assume-role:

aws sts assume-role --role-arn arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME_CHOSEN> --role-session-name <SESSION_NAME>

This command will allow you to obtain temporary credentials for the role, including Access Key ID, Secret Access Key, and Token. Ensure you export these AWS credentials before proceeding with the installation step.

This means that the jump box, a separate EC2 instance, can directly use the IAM role to access the EKS cluster without assuming the role separately.

To do this, instead of selecting "AWS account" in Step 3, select "AWS service", and select "EC2" under "Common use cases". Next, find the jump box instance in the EC2 console, select it, and click "Actions" -> "Security" -> "Modify IAM role". Find the IAM role in the drop-down menu, and click "Update IAM role".

Creating LightBeam EKS access user

  1. Log in to the AWS Console, and using the Service search, navigate to IAM Management Console (amazon.com). On the left tab, click Policies->Create policy.

  2. Select the JSON tab, and paste the following JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
               "eks:ListTagsForResource",
                "eks:DescribeNodegroup",
                "eks:ListNodegroups",
                "eks:DescribeCluster",
                "eks:ListClusters",
                "eks:UpdateClusterVersion",
                "eks:UpdateNodegroupVersion",
                "eks:ListUpdates",
                "eks:UntagResource",
                "eks:ListFargateProfiles",
                "eks:DescribeUpdate",
                "eks:TagResource"
            ],
            "Resource": "arn:aws:eks:<AWS_REGION>:<AWS_ACCOUNT_ID>:cluster/<EKS_CLUSTER_NAME>"
        }
    ]
}

  1. Give an appropriate name to the policy and click Save.

  2. On the left panel of the IAM Management Console (amazon.com), click on Users -> Add Users.

  3. Give it an appropriate username, select only Programmatic Access, and click on Next.

  4. Click on the Existing Policies tab and attach the policy created in Step 3.

  5. On the Create page, ensure that you copy the Access Key Id and the Secret Access Key.

  1. Go to the IAM Management Console (amazon.com).

  2. On the left tab, click on Roles -> Create role.

  3. Select "AWS account" as the trusted entity and ensure the "This account" radio button is selected.

  4. Optionally enable Multi-Factor Authentication (MFA) if required by your organization's security policy.

  5. While adding permissions to this role, select the policy you created in the previous step (Step 3) which includes the necessary permissions for accessing the EKS cluster.

  6. To assume this role, use the AWS CLI command aws sts assume-role:

aws sts assume-role --role-arn 
arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME_CHOSEN> 
--role-session-name <SESSION_NAME>

This command will allow you to obtain temporary credentials for the role, including Access Key ID, Secret Access Key, and Token. Ensure you export these AWS credentials before proceeding with the installation step.

Another option instead of assuming the role is to attach the role to the jump box:

This means that the jump box, which is a separate EC2 instance, can directly use the IAM role to access the EKS cluster without assuming the role separately.

To do this, instead of selecting "AWS account" in Step 3, select "AWS service", and select "EC2" under "Common use cases". Next, find the jump box instance in the EC2 console, select it, and click "Actions" -> "Security" -> "Modify IAM role". Find the IAM role in the drop down menu, and click "Update IAM role".

4.0 EKS cluster creation

A. AWS Cloud Credentials Setup

Option 1: Using EKS Admin User

  1. Connect to the jump box VM created above.

  2. This can be done using `aws configure` with EKS admin user credentials or setting Env Variables as below with EKS admin user credentials:

export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""

Option 2: Using EKS Admin Role

  1. Connect to the jump box VM created above.

  2. Assume the EKS admin role by running the following AWS CLI command:

aws sts assume-role --role-arn
 arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME_CHOSEN> 
--role-session-name <SESSION_NAME>

  1. Export the temporary AWS credentials as environment variables in your terminal session by running the following commands:

export AWS_ACCESS_KEY_ID="<TEMP_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<TEMP_SECRET_ACCESS_KEY>"
export AWS_SESSION_TOKEN="<TEMP_SESSION_TOKEN>"

B. Start Provisioning the EKS Cluster using Terraform script

Configuration:

  • We need to provision 3 node EKS clusters and each node instance type is m6i.2xlarge which has 8 CPU and 32 GB memory on each node.

  • Copy eks-cluster.zip shared by LightBeam to the Jump Host and unzip the file.

  • Run following commands:

    • terraform init

    • terraform apply

Note: Verify that the deployed EKS cluster has internet access.

Once the EKS cluster configuration is complete we can set up EKS Access Users and continue with LightBeam application deployment.

We need to update the EKS auth-config, so that the EKS Access User can then seamlessly access the cluster.

  • To update the auth config:

kubectl edit -n kube-system configmap/aws-auth

  • The auth-config file opens up, where you need to add the following YAML content.

(Ensure that you use the Lightbeam EKS access user ARN)

mapUsers: |
    - userarn: <arn:aws:iam::<AWS_ACCOUNT>:user/LightbeamEKSUser>
      username: <LightbeamEKSUser>
      groups:
      - system:masters

  • If you created a role instead of the user with the Lightbeam EKS Access policy, add the following to the mapRole section:

- rolearn: arn:aws:iam::<AWS_ACCOUNT>:role/<EKS_ACCESS_ROLE_NAME_CHOSEN>
username: lb-eks-access-user
groups:
    - system:masters

5.0 Access the LightBeam cluster using EKS Access user

As we plan to deactivate the Deployment user credentials, it is important we verify accessing the cluster using the Lightbeam EKS user, so, export the second user credentials using the following commands executed on the Jump Box VM:

export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""

To access the cluster (control plane) :

aws eks --region <AWS_REGION> update-kubeconfig --name <EKS_CLUSTER_NAME>

Verify EKS access:

kubectl get nodes

If you see the cluster node details, it ensures that the cluster is accessible.

6.0 Admin User Deletion/ Credential deactivation (optional)

  1. Once the application is accessible, the auth-config is updated and the cluster access is verified, the EKS Deployment user can either be deleted or the credentials for the same can be deactivated.

  2. To deactivate the credentials, navigate to the EKS Deployment User, and under Security Credentials Tab, under Access Keys, use the Make Inactive option to inactivate the key.

7.0 LightBeam Installation

Follow the installation instructions in the LightBeam Installer guide.

Use the --eks flag in the installer command to specify the platform used is AWS EKS.

8.0 LightBeam Upgrade

  1. The LightBeam team shares the latest release LightBeam chart which needs to be copied on the Jump Box.

  2. Upgrade the LightBeam chart:

NOTE: Dockerhub registry access is needed from all worker nodes of the Kubernetes cluster.

The --spectra flag specifies the spectra deployment. Use the --privacy_ops flag to specify the privacy ops deployment.

export DOCKER_USERNAME="lbcustomers" DOCKER_REGISTRY_PASSWORD="<DOCKER_REGISTRY_TOKEN>" KBLD_REGISTRY_HOSTNAME="docker.io" KBLD_REGISTRY_USERNAME="lbcustomers" KBLD_REGISTRY_PASSWORD="<DOCKER_REGISTRY_TOKEN>"
./installer/lb-install.sh --upgrade --spectra --values charts/lightbeam/values.yaml --final_values charts/lightbeam/values.yaml

One can verify the upgrade by running kubectl to get pods -n lightbeam and checking the status of all pods.

As part of the upgrade, the following actions are taken:

  • Pausing datasource scanning and processing.

  • Updating the latest database schema.

  • Updating all LightBeam services to the respective release versions. As part of this step, all the latest release version Docker images from the Dockerhub registry are downloaded on respective worker nodes.

  • A post-upgrade job is executed which resumes scans for all data sources and starts processing, also verifies all services health.

About LightBeam

LightBeam automates Privacy, Security, and AI Governance, so businesses can accelerate their growth in new markets. Leveraging generative AI, LightBeam has rapidly gained customers’ trust by pioneering a unique privacy-centric and automation-first approach to security. Unlike siloed solutions, LightBeam ties together sensitive data cataloging, control, and compliance across structured and unstructured data applications providing 360-visibility, redaction, self-service DSRs, and automated ROPA reporting ensuring ultimate protection against ransomware and accidental exposures while meeting data privacy obligations efficiently. LightBeam is on a mission to create a secure privacy-first world helping customers automate compliance against a patchwork of existing and emerging regulations.

For any questions or suggestions, please get in touch with us at: [email protected].

PreviousPlatform SpecificNextMicrosoft Azure

Last updated