AWS Auto Discovery
1. Introduction
AWS Auto Discovery streamlines the process of integrating your AWS resources with LightBeam. This feature automatically detects and registers supported AWS services, eliminating the need for manual data source registration. It also provides visibility into unsupported resources, helping you maintain an up-to-date inventory of your AWS environment.
2. Supported Services
2.1 Fully Supported Services
Amazon S3 (Simple Storage Service)
Amazon RDS (Relational Database Service)
MySQL
PostgreSQL
Microsoft SQL Server
Oracle
Amazon Redshift
AWS Glue
Amazon DynamoDB
Amazon FSx (File Systems)
2.2 Discoverable Services (Limited Support)
Amazon EC2 (Elastic Compute Cloud)
Amazon EFS (Elastic File System)
Amazon S3 Glacier
AWS Backup
Amazon CloudWatch
3. Onboarding Process
3.1 Create an IAM Role in AWS
To enable LightBeam to access your AWS resources, you need to create an IAM role with the necessary permissions. For detailed steps for creating the IAM role go to Appendix.
3.2 Accessing the Onboarding Screen
Navigate to the "Datasources" header in the main navigation.
Click on the "Cloud Platforms" tab.
Select AWS from the left sidebar menu.
Click on the Onboard Now button.
Alternative Method:
Scroll to the bottom of the Datasources page to find a list of supported cloud platforms.
Click on the AWS icon to start the onboarding process.
3.4 Entering AWS Account Details
After selecting "AWS" as the cloud platform,
enter the following AWS account details:
NameDescription(Optional)Primary OwnerCo-owner(Optional)AWS Account IDAccess Key IDSecret Access KeySession Key(Optional)
After entering the
Secret Key, the system automatically runs an internal test connection API to validate the credentials.Once you see the message
Connection Verified, proceed to the next step.
3.5 Configuring Discovery Settings
Set the frequency for scanning resources:
Options include daily, weekly, or monthly scans.
Click "Save" to confirm your settings and initiate the discovery process.
4. Resource Discovery Process
After saving, an in progress message will appear:
Resource discovery is in progress, and this process may take some time.During this process:
A Terraform workflow is triggered.
This workflow creates necessary IAM roles and policies.
The created roles and policies are assigned to the appropriate service account.
The process typically takes about 5 minutes for a single AWS account.
3.6 Reviewing Discovered Resources
Refresh the page to view the results.
You'll see your AWS account listed.
Discovered resources are displayed in a table format with the following details:
Data source name and type
Account name
Region
Owner
Status (Registered or Unregistered)
Date added
Naming Convention for Discovered Resources:
LightBeam uses a specific naming convention for automatically discovered AWS resources. This convention helps in easily identifying and categorizing different types of resources across your AWS account.
1. Data Sources with One Instance per Account
For resources that typically have one instance per AWS account, such as S3 and DynamoDB:
Format: <Data source type>_<Account Name>
Examples:
S3_Lightbeam
DynamoDB_Lightbeam
2. Data Sources with One Instance per Region
For resources that are limited to one instance per region, such as AWS Glue:
Format: <Data source type>_<region>_<Account Name>
Example:
Glue_us-east-1_Lightbeam
3. Data Sources with Multiple Instances per Region
For resources that can have multiple instances within a single region:
Format: The name will be the same as the cluster or instance name in AWS.
Example:
If you have an RDS instance named "production-db" in AWS, it will appear as "production-db" in LightBeam.
4. Handling Duplicate Names
In cases where duplicate names are detected:
The first instance will use the standard naming convention.
Subsequent instances with the same name will have suffixes appended: *1, *2, and so on.
Example:
First instance: MyDatabase
Second instance with the same name: MyDatabase*1
Third instance with the same name: MyDatabase*2
This naming convention helps in maintaining consistency and clarity across your discovered AWS resources in LightBeam. It allows for easy identification of resource types, associated AWS accounts, and, where applicable, the AWS region.
Initial status:
All discovered resources are marked as "
Unregistered" with an orange status icon.Each resource has a "Register" button for manual registration.
Discovery scope:
Regional services:
RDS,Redshift,Glue,FSxOrganization-level services:
S3,DynamoDB
Search Bar: Enables quick resource lookup using keywords.
Filter Options: Allows refinement of results based on specific criteria.
Status Tabs: Toggles between "Supported" and "Unsupported" resources.
4. Registration Process
4.1 Automatic Registration
Pre-selected services are registered automatically upon discovery.
4.2 Manual Registration (Example: Redshift)
Click on the "Register" button next to the Redshift datasource.
Add basic details for the Redshift data source and click on Next.
Since IAM roles are already set up, the Host, Port and Frequency of Scanning fields are prefilled.
Enter the Username and Password.
Click "Test Connection" to verify access.
The rest of the workflow is the same as described in AWS Redshift document.
4.3 Post-Registration
The registered data source will appear in your list of data sources.
In the Cloud Platforms view, the status for AWS S3 will update to "Sync On".
You can click on the data sourc name to access its dashboard.
5. Operational Procedures
5.1 Manual Sync
Users can trigger a manual sync to update the status of discovered resources:
Navigate to the Cloud Platforms dashboard
Click the "Manual Sync" button
Wait for the sync process to complete
5.2 Error Handling
Resources with errors during discovery or registration are flagged with appropriate status indicators
Users can attempt to resolve issues and re-run the registration process
5.3 Additional Views
An "All Datasources" section is available with two tabs:
Scanning: Shows data sources currently being scanned.
Unregistered: Displays discovered but not yet registered data sources.
6. Security Considerations
High-privileged credentials used during onboarding are discarded after initial setup
IAM roles with least-privilege permissions are created for ongoing operations
All communications with AWS APIs use secure, encrypted connections
7. Limitations and Future Enhancements
Some discoverable services (e.g., EC2, CloudWatch) have limited metadata available
Future versions may include enhanced support for currently limited services
Additional AWS services may be added to the list of fully supported services in upcoming releases
Required IAM Permissions
For LightBeam AWS Auto Discovery to function properly, the AWS IAM user needs the following permissions. Create a policy with this JSON:
1. Create IAM Policy
Navigate to Console Home page
Click on IAM
Click on "Policies" in left sidebar.
Click the "Create policy" button.
Select "JSON" editor tab.
Configure the policy by erasing the existing code in editor.
Type the required JSON policy.
Click on "Next".
Give a name to the policy, for example: "
lightbeam-aws-discovery-policy-test1".Click on "Create policy".
2. User Creation
Navigate to IAM → Users
Click "Create user" button
Set User Details
Username: "lightbeam-aws-discovery-user-test1"
Click "Next"
Set Permissions
Select "Attach policies directly"
Search for "lightbeam-aws-discovery-policy-test1".
Check the box next to the policy.
Click "Next".
Click "Create user"
3. Access Key Generation
Access User Credentials
Search for created user "lightbeam-aws-discovery-user-test1"
Click on created user "lightbeam-aws-discovery-user-test1".
Click on "Security credentials" tab.
Start Access Key Creation
Locate "Access keys" section.
Click "Create access key" button.
Configure Access Key Purpose
Select "Command Line Interface (CLI)" use case.
Check the box "I understand the above recommendation and want to proceed to create an access key."
Create and Save Credentials
Click on Create access key
4. Securely Save AWS Access Keys
After creating the access key, you have two options to save the credentials:
Download .csv File Method
Click "Download .csv file" button
File will contain:
Access Key ID
Secret Access Key
Store this file securely
Manual Copy Method
Access Key ID:
Copy directly from interface by clicking on the icon 📑 next to Access Key
Secret Access Key:
Click "Show" to reveal
Copy directly from interface by clicking on the icon 📑
Record both values securely.
About LightBeam
LightBeam automates Privacy, Security, and AI Governance, so businesses can accelerate their growth in new markets. Leveraging generative AI, LightBeam has rapidly gained customers’ trust by pioneering a unique privacy-centric and automation-first approach to security. Unlike siloed solutions, LightBeam ties together sensitive data cataloging, control, and compliance across structured and unstructured data applications providing 360-visibility, redaction, self-service DSRs, and automated ROPA reporting ensuring ultimate protection against ransomware and accidental exposures while meeting data privacy obligations efficiently. LightBeam is on a mission to create a secure privacy-first world helping customers automate compliance against a patchwork of existing and emerging regulations.
Last updated
