Skip to main content

Connecting Fivetran and Databricks on Skypoint AIP

Overview​

Skypoint provides built-in connectors and the flexibility to integrate with external tools like Fivetran, dbt, Power BI, and Tableau to seamlessly import data from user databases. With Fivetran, users can securely transfer data from multiple sources to Databricks on Skypoint AIP within a robust cloud network. This guide will walk you through the steps to set up your data pipeline.

Find the architectural outline of the data pipeline as follows:

Alt text

Prerequisite​

  • You have an active account with Fivetran and Skypoint AIP & credentials to access them.

Setup Guide to Connect Fivetran and Databricks on Skypoint AIP​

The setup involves two key steps:

  • Integrating your Data Source with Fivetran

  • Connect Fivetran to Databricks on Skypoint AIP

Step 1: Integrate your Data Source with Fivetran​

Fivetran offers a comprehensive range of connectors to integrate with your databases.

  1. Follow the Fivetran setup guide to configure the connectors, required to set up connection with your data source.

  2. Prepare the following prerequisites for your source and destination to streamline the setup:

    • Valid database credentials

    • An active Fivetran account

    • Skypoint AI account with access to the Studio.

Step 2: Connect Fivetran to Databricks on Skypoint AIP​

Actions on Skypoint AI Studio​

  • Log in to your Skypoint AI Studio account.

  • Go to Lakehouse > Workspaces.

  • Click the Show Fields button to display connection strings.

Alt text

  • Copy the following credentials:

    • Server Hostname
    • HTTP Path
    • Personal Access Token

Alt text

Actions on Fivetran​

  • Log in to your Fivetran account.

  • From the left panel, navigate to Destinations.

Alt text

  • Click Add Destination.

Alt text

  • Select Databricks as your destination.
note

This selection is based on the fact that Skypoint Lakehouse is built on Databricks, ensuring seamless integration, enhanced compatibility, and optimal performance.

Alt text

  • Name the destination by following the naming convention for the destination as follows:

skypointaip<Tenant_name>__<Instance_name>_lakehouse_bronze

Components:

  • skypoint_aip: Identifies the Skypoint AIP platform.
  • <Tenant_name>: Placeholder for the tenant's name.
  • <Instance_name>: Placeholder for the instance name.
  • lakehouse_bronze: Specifies the Bronze layer in the Lakehouse architecture.

Alt text

  • Key in the required credentials by copying from Skypoint AI Studio.

Alt text

  • Under Authentication Type, select Personal Access Token from the dropdown menu.

  • Key in the personal access token in the field below.

  • Click Save & Test to verify the connection.

Alt text

Congratulations! You’ve successfully connected Fivetran with Databricks on Skypoint AIP. Your data pipeline is ready to sync securely!​

Post-Setup Verification​

  • The new connection will appear on the Destinations page.

  • Click on the connection to view its details.

  • Use the Actions menu to:

    • Test the connection

    • Rename the destination

    • Edit connection details

Alt text

Next step​

After completing the data import, start the Master Data Management (MDM) - Stitch process to develop a unified view of your customers.