Installer Guides

GCP Auto Discovery

1. Introduction

GCP Auto Discovery is an advanced solution designed to streamline the process of discovering, registering, and managing Google Cloud Platform (GCP) resources across multiple services. This technical document provides a comprehensive guide to using GCP Auto Discovery, covering everything from initial setup to ongoing resource management.

2. Supported Services

2.1 Fully Supported Services

GCP Auto Discovery offers complete support for the following services:

  • BigQuery

  • Google Cloud Storage (GCS)

  • Google Cloud SQL

  • Google Compute Engine (GCE)

2.2 Discoverable Services (Limited Support)

The following services can be discovered but have limited support:

  • Google Cloud Firestore

  • Google Cloud Spanner

  • Google Cloud Dataproc

3. Onboarding Process

3.1 Accessing the Onboarding Screen

  1. Navigate to the "Datasources" header in the main navigation.

  2. Click on the "Cloud Platforms" tab.

  1. Select GCP from the left sidebar menu.

  2. Click on the Onboard Now button.

Alternative Method:

  • Scroll to the bottom of the Datasources page to find a list of supported cloud platforms.

  • Click on the Google Cloud icon to start the onboarding process.

3.2 Entering GCP Account Details

  1. Next, enter the following GCP account details:

  • Name for the GCP platform (required)

  • Description (optional)

  • Primary Owner (required)

  • Co-owner (optional)

  • GCP Credentials

    • Service Account Key (required)

  1. After entering the Service Account Key, the system automatically runs an internal test connection API to validate the credentials.

  2. Once you see the message Connection Verified, proceed to the next step.

3.3 Configuring Discovery Settings

  1. Project Selection:

    • Tick the checkbox from the list to select the GCP projects for which you want to discover and register resources.

    • Tick Select All to choose all available projects for discovery and registration.

  1. Set the frequency for scanning resources:

  • Options include daily, weekly, or monthly scans.

  1. Click "Save" to confirm your settings and initiate the discovery process.

3.4 Resource Discovery Process

  1. After saving, an "in progress" message will appear: "Resource discovery is currently in progress, and this process may take some time."

  2. The process typically takes about 5 minutes.

3.5 Reviewing Discovered Resources

  1. Refresh the page to view the results.

  2. You'll see the projects with supported resources listed.

  3. To see the projects with unsupported resources, click on the Unsupported tab.

    Here you will see the count of unsupported resources in each project.

  4. To register a new datasource:

    • Click on the project containing the datasource.

4. Registration Process (example: BigQuery)

  1. Click on the "Register" button next to the BigQuery datasource.

The next steps in the datasource registration process are the same as described in the BigQuery document.

4.1 Post-Registration

  • The registered data source will appear in your list of data sources.

  • In the Cloud Platforms view, the status for BigQuery will update to "Sync On".

  • You can click on the data source name to access its dashboard.

5. Modifying Scan Settings

After onboarding BigQuery or other resources, you can modify scan settings and manage projects:

  • Remove previously onboarded projects:

    • This action removes all discovered cloud resources.

    • Any registered data sources under the project will also be removed.

  • Updating project scan settings:

    • Triggers a new discovery workflow in the backend.

    • There may be a slight delay in reflecting changes in the UI.

    • The scan settings update immediately, but the discovered accounts list may take time to refresh.

5.1 Deletion Process

When modifying scan settings or removing projects:

  • A background process starts deleting records.

  • A workflow initiates to delete the respective project and discovered resources.

  • If any data sources are registered, they are also removed.

Appendix:

Setting Up GCP Access for LightBeam Auto Discovery

1. Create Custom Role

  1. Access GCP Console

  2. In left navigation, click "IAM & Admin"

  1. Click "Roles" from the left menu

  1. Click "CREATE ROLE" at the top of the page

  1. Enter role details:

  • Title: gcp-lb-test1

  • ID: Will auto-generate (e.g., CustomRole585)

  • Description (optional)

  • Launch stage: "Alpha"

  1. Click "ADD PERMISSIONS" button

  1. In search bar, add these permissions one by one:

cloudsql.instances.get
compute.instances.get
compute.regions.list
compute.zones.list
datastore.databases.list 
spanner.instances.list

For each permission:

  • Type in search bar

  • Check the box next to permission

  • Click "ADD" button

  1. Verify all permissions are listed under "Assigned permissions"

  1. Click "CREATE" button

Create Service Account

  1. In "IAM & Admin", click "Service Accounts" from left menu

  1. Click "+ CREATE SERVICE ACCOUNT" at the top

  1. Enter service account details:

  • Service account name: gcp-test-1

  • Service account ID: Will auto-generate

  • Description (optional): Purpose of the account

  1. Click "CREATE AND CONTINUE"

  1. In "Grant this service account access to project":

  • Click "Select a role" dropdown

  1. Add following roles one by one:

a. BigQuery Admin:

  • Filter by "BigQuery"

  • Select "BigQuery Admin"

  • Click "ADD ANOTHER ROLE"

b. Pub/Sub Admin:

  • Filter by "Pub/Sub"

  • Select "Pub/Sub Admin"

  • Click "ADD ANOTHER ROLE"

c. Storage Admin:

  • Filter by "Storage"

  • Select "Storage Admin"

  • Click "ADD ANOTHER ROLE

d. Custom role:

  • Search "gcp-lb-test1"

  • Select created custom role

  1. Click "CONTINUE"

  1. Click "DONE"

Generate Service Account Key

  1. In IAM list, find gcp-test-1

  1. Click on service account name to view details

  1. Select "KEYS" tab

  1. Click "ADD KEY" dropdown

  • Select "Create new key"

  1. In key creation dialog:

  • Select "JSON" (recommended)

  1. Click "CREATE"

  1. JSON key file downloads automatically:

  • Format: lightbeam-privacy-[KEY-ID].json

  • Save in secure location

  • This file cannot be recovered if lost

  • Required for LightBeam configuration

About LightBeam

LightBeam automates Privacy, Security, and AI Governance, so businesses can accelerate their growth in new markets. Leveraging generative AI, LightBeam has rapidly gained customers’ trust by pioneering a unique privacy-centric and automation-first approach to security. Unlike siloed solutions, LightBeam ties together sensitive data cataloging, control, and compliance across structured and unstructured data applications providing 360-visibility, redaction, self-service DSRs, and automated ROPA reporting ensuring ultimate protection against ransomware and accidental exposures while meeting data privacy obligations efficiently. LightBeam is on a mission to create a secure privacy-first world helping customers automate compliance against a patchwork of existing and emerging regulations.

For any questions or suggestions, please get in touch with us at: [email protected].

PreviousAWS Auto DiscoveryNextDatabases and Datalakes

Last updated