LightBeam Documentation
Installer GuidesData SourcesPlaybooksInsightsPrivacyOpsGovernance
  • 💡What is LightBeam?
  • 🚀Getting Started
    • ⚙️Installer Guides
      • Pre-Requisites / Security Configurations
        • Firewall Requirements
        • Securing LightBeam on EKS with AWS Certificate Manager on Elastic Load Balancer
        • Configure HTTPS for LightBeam Endpoint FQDN Standalone deployment
        • Using Custom Certificates with LightBeam
        • Securing LightBeam on GKE with Google Certificate Manager and GCE Ingress
      • Core
        • LightBeam Deployment Instructions
        • LightBeam Installer
        • Web App Deployment
        • LightBeam Diagnostics
        • LightBeam Cluster Backup & Restore using Velero
      • Platform Specific
        • AWS
        • Microsoft Azure
        • Google Cloud (GKE)
        • Standalone Virtual Machine
        • Deployment on an Existing Managed Kubernetes Cluster
        • Azure Marketplace Deployment
      • Integration and Setup
        • Setting Up AWS PrivateLink for RDS-EKS Interaction
        • Twingate and LightBeam Integration Guide
        • Data Subject Request Web Application Server
        • Generate CSR for LightBeam
  • 🧠Core Features
    • 🔦Spectra AI
      • 🔗Data Sources
        • Cloud Platforms
          • AWS Auto Discovery
          • GCP Auto Discovery
        • Databases and Datalakes
          • PostgreSQL
          • Aurora (PostgreSQL)
          • Snowflake
          • MS SQL
          • MySQL
          • Aurora (MySQL)
          • BigQuery
          • AWS Redshift
          • Oracle
          • DynamoDB
          • MongoDB
          • CosmosDB (PostgreSQL)
          • CosmosDB (MongoDB)
          • CosmosDB (NoSQL)
          • Looker
          • AWS Glue
          • Databricks
          • SAP HANA
          • CSV Files as a Datasource
        • Messaging
          • Gmail
          • Slack
          • MS Teams
          • MS Outlook
        • Developer Tools
          • Zendesk
          • ServiceNow
          • Jira
          • GitHub
          • Confluence
        • File Repositories
          • NetDocuments
          • AWS S3
          • Azure Blob
          • Google Drive
          • OneDrive
          • SharePoint
          • Viva Engage
          • Dropbox
          • Box
          • SMB
        • CRM
          • Hubspot
          • Salesforce
          • Automated Data Processing (ADP)
          • Marketo
          • Iterable
          • MS Dynamics 365 Sales
          • Salesforce Marketing Cloud
      • 🔔PlayBooks
        • What is LightBeam Playbooks?
        • Policy and Alerts
          • Types of Policies
          • How to create a rule set
            • File Extension Filter
          • Configuring Retention Policies
          • Viewing Alerts
          • Sub Alerts
            • Reassigning Sub-Alerts
            • Sub-alert States
          • Levels of Actions on Alerts
          • User Roles and Permissions
            • Admin View
            • Alert Owner View
            • Onboarding New Users
              • User Management
              • Okta Integration
              • Alert Assignment Settings
              • Email Notifications
            • Planned Enhancements
          • Audit Logs
          • No Scan List
          • Permit List
          • Policy in read-only mode
      • 📊Insights
        • Entity Workflow
        • Document Classification
        • Attribute Management Overview
          • Attributes Page View
          • Attribute Sets
          • Creating Custom Attribute
          • Attributes List
        • Template Builder
        • Label Management
          • MIP Integration
          • Google Labels Integration
      • 🗃️Reporting
        • Delta Reporting
        • Executive Report
        • LightBeam Lens
      • Scanning and Redaction of Files
        • On-demand scanning
      • How-to Guides
        • Leveraging LightBeam insights for structured data sources
      • LightBeam Dashboard Outlay
      • Risk Score
    • 🏛️PrivacyOps
      • Data Subject Request (DSR)
        • What is DSR?
        • Accessing the DSR Module
        • DSR Form Builder (DPO View)
          • Creating a New DSR Form
            • Using a Predefined Template
            • Creating a Custom Form
          • Form Configuration
          • Form Preview and Publishing
          • Multi-Form Management
          • Messaging Templates
        • Form Submission & Email Verification (Data Subject View)
        • DSR Management Dashboard (DPO View)
        • Processing DSR Requests
          • Data Protection Officer (DPO) Workflow
          • Self Service Workflow (Direct Validation)
          • Data Source Owner (DSO) Workflow
        • DSR Report
      • 🚧Consent Management
        • Overview
        • Consent Logs
        • Preference Centre
        • Settings
      • 🍪Cookie Consent
        • Dashboard
        • Banners
        • Domains
        • Settings
        • CMP Deployment Guide for Google Tag Manager
        • FAQs
      • 🔏Privacy Impact Assessment (PIA)
        • PIA Templates
        • PIA Assessment Workflow
        • Collaborator View
        • Process Owner Login View (With Collaborator)
        • Filling questionnaire without collaborator
        • Submitting the assessment for DPO review
        • DPO review process
        • Marking the assessment as reviewed
        • Editing and resubmitting assessments after DPO review
        • Revoke review request
        • Edit Reviewer
        • PIA Reports
      • ⏺️Records of Processing Activity (RoPA)
        • Creating a RoPA Template
          • How to clone a template
          • How to use a template
        • How to create a process
          • Adding Process Details
          • Adding Data Elements
          • Adding Data Subjects
          • Adding Data Retention
          • Adding Safeguards
          • Adding Transfers
          • Adding a Custom Section
          • Setting a Review Schedule
          • Data Flow Diagram
        • How to add a collaborator
        • Overview Section
        • Generating a RoPA Report Using LightBeam
        • Collaborator working on a ticket
    • 🛡️Governance
      • Access
        • Dashboard
        • Users
        • Groups
        • Objects
        • Active Directory Settings
        • Access Governance at a Data Source Level
        • Policies and Alerting
        • Access Governance Statistics
        • Governance Module Dashboard
      • Privacy At Partners
  • 📊Tools & Resources
    • 🔀API Documentation
      • API to Create Reports for Structured Datasource
    • ❓Onboarding Assessments
      • Structured Datasource Onboarding Questionnaire
        • MongoDB/CosmosDB Questionnaire
        • Oracle Datasource Questionnaire
      • SMB Questionnaire
    • 🛠️Administration
      • Audit Logs
      • SMTP
        • Basic and oAuth Configuration
      • User Management
        • SAML Identity Providers
          • Okta
            • LightBeam Okta SAML Configuration Guide
          • Azure
            • Azure AD SAML Configuration for LightBeam
          • Google
            • Google IDP
        • Local User Management
          • Adding a User to the LightBeam Dashboard
          • Reset Default Admin Password
  • 📚Support & Reference
    • 📅Release Notes
      • LightBeam v2.2.0
      • Reporting Release Notes
      • Q1 2024 Key Enhancements
      • Q2 2024 Key Enhancements
      • Q3 2024 Key Enhancements
      • Q4 2024 Key Enhancements
    • 📖Glossary
Powered by GitBook
On this page
  • Overview
  • Onboarding BigQuery Data Source
  • APPENDIX
  • About LightBeam
  1. Core Features
  2. Spectra AI
  3. Data Sources
  4. Databases and Datalakes

BigQuery

Connecting BigQuery to LightBeam

PreviousAurora (MySQL)NextAWS Redshift

Last updated 8 months ago


Overview

LightBeam Spectra users can connect various data sources to the LightBeam application and these data sources will be continuously monitored for PII, PHI data.

Examples: BigQuery, SMB, MySQL, PostgreSQL, etc.


Onboarding BigQuery Data Source

  1. Login to your LightBeam Instance.

  2. Click on DATASOURCES on the Top Navigation Bar.

  3. Click on “Add a data source”.

    Figure 1. Add Data Source
    1. Search for “BigQuery”.

  1. Fill in the details as shown below and click Next:

Basic Information

  1. Instance Name: This is the unique name given to the data source.

  2. Description: This is an optional field needed to describe the use of this data source.

  3. Assign owner: Email address of the person responsible for this data source which will get alerts by default.

  4. Entity Creation: LightBeam Spectra detects and associates attributes based on the context and identifies whose data it is; these are called entities. Example: Jane Doe is an entity for whom LightBeam Spectra might have detected Name and SSN in a monitored data source.

  5. Source of Truth: LightBeam Spectra includes monitored data sources that serve as a single point of truth. These sources are utilized for looking up entities/attributes to verify the accuracy of attributes/entities discovered in other data sources. By using a Source of Truth dataset, entities are formulated based on the attributes present in the data.

  6. Location: The location of the data source.

  7. Purpose: The purpose of the data being collected/processed.

  8. Stage: The stage of the data source. Example: Source, Processing, Archival, etc.

  1. Insert the credentials as shown below:

  1. Verify that you get the message Test Connection Success on the screen. Click on Next.

  1. In this step, select specific project(s) that you have to scan from the drop-down list.

Now we are ready to connect to the test database and proceed.

Click on Register.

This will show you the following message:

Click on Okay.

Now you can browse the updated datasource.


APPENDIX

Creating a new role in BigQuery

Role: A role is a named set of access privileges that can be granted to users or other roles. These privileges determine what actions a user can perform and on which database objects.

  1. In the Google Cloud console, go to IAM & Admin -> Roles.

  1. Click on Create Role.

  1. Fill in the role name.

Assign the following permissions to the role:

bigquery.bireservations.get 

bigquery.capacityCommitments.get 
bigquery.capacityCommitments.list 

bigquery.connections.get 
bigquery.connections.getIamPolicy 
bigquery.connections.list 
bigquery.connections.use 

bigquery.datasets.get 
bigquery.datasets.getIamPolicy 

bigquery.jobs.create 
bigquery.jobs.get 
bigquery.jobs.list 
bigquery.jobs.listAll 
bigquery.jobs.listExecutionMetadata 

bigquery.models.export 
bigquery.models.getData 
bigquery.models.getMetadata 
bigquery.models.list 

bigquery.readsessions.create 
bigquery.readsessions.getData 
bigquery.readsessions.update 

bigquery.reservationAssignments.list 
bigquery.reservationAssignments.search 
bigquery.reservations.get 
bigquery.reservations.list 

bigquery.routines.get 
bigquery.routines.list 

bigquery.rowAccessPolicies.getFilteredData 

bigquery.tables.createSnapshot 
bigquery.tables.export 
bigquery.tables.get 
bigquery.tables.getData 
bigquery.tables.getIamPolicy 
bigquery.tables.list 

resourcemanager.projects.get
  1. Navigate to Service Accounts and then click on Create Service Account.

  1. Enter the required details, such as the name of the service account. Click Create And Continue.

  1. Grant the role you created in the previous step to this service account and click on Continue And Save.

  1. Once the service account has been created, select it. Navigate and click on Keys.

  1. Choose Create New Key.

  1. For Key Type, choose JSON and then click Create.

10. Download the JSON file and convert it to base64 by using the following command

cat <path of downloaded file> | base64

Now use the JSON key to connect the datasource.


About LightBeam

LightBeam automates Privacy, Security, and AI Governance, so businesses can accelerate their growth in new markets. Leveraging generative AI, LightBeam has rapidly gained customers’ trust by pioneering a unique privacy-centric and automation-first approach to security. Unlike siloed solutions, LightBeam ties together sensitive data cataloging, control, and compliance across structured and unstructured data applications providing 360-visibility, redaction, self-service DSRs, and automated ROPA reporting ensuring ultimate protection against ransomware and accidental exposures while meeting data privacy obligations efficiently. LightBeam is on a mission to create a secure privacy-first world helping customers automate compliance against a patchwork of existing and emerging regulations.

Check the tickboxes next to the project you would like to add. To add or remove datasets within the project, you can use the '' or '' signs.

For any questions or suggestions, please get in touch with us at: .

🧠
🔦
🔗
➕
➖
support@lightbeam.ai
Figure 2. BigQuery Data Source
Figure 3. BigQuery Configuration - Basic Info
Figure 4. BigQuery Configuration - Connection details
Figure 5. BigQuery Configuration - Test Connection Success
Figure 6. BigQuery Configuration - Select project
Figure 6.1 BigQuery Configuration - Select project
Figure 7. BigQuery - Register Datasource
Figure 8. Create a new role.
Figure 8.1 Create a new role
Figure 8.2 Create a new role - Assign Permissions
Figure 8.3 Create a new role - Create service account
Figure 8.4 Create a new role - Service account details
Figure 8.4 Create a new role - Service account created
Figure 8.5 Create a new role - Click on Keys
Figure 8.6 Create a new role - Create New Key
Figure 8.7 Create a new role - Select Key Type