Skip to main content

Transformations

Overview​

Skypoint platform integrates with dbt Core for data transformation. dbt (data build tool) is an open-source command-line tool that helps build, test, and maintain data infrastructure. dbt allows users to define their data models using SQL that can be run against a data warehouse or other data storage system and build a maintainable and scalable data infrastructure that can be easily updated and extended over time.

The transformations are defined using SQL code stored in .sql files and organized into a directory structure within a dbt project. You can transform your data by writing SQL SELECT statements (also known as "dbt models") in a Git repository. You can create relationships and dependencies between models, and then materialize those models as tables.

Data transformation in Lakehouse SQL​

Skypoint's Modern Data Stack Platform (MDSP) allows tenants to create transformed tables based on the table imported or generated by the platform and use them for analysis or other downstream uses. It follows multi-tenant architecture with each tenant consisting of one or more instances, allowing one dbt git repository per tenant and one branch of the repository per instance. This allows users to add/modify models in a pre-configured branch that isolates each instance's models.

Skypoint platform is integrated with dbt Core and built on top of the Lakehouse SQL warehouse to provide a centralized environment for collaborative data transformation. Skypoint-powered transformations allow data teams to organize and analyze data without sacrificing data quality or analytical flexibility. This can involve a wide range of activities, including data cleaning, data standardization, data aggregation, data enrichment, and more.

Features of Skypoint integration with dbt Core are as follows:

  • Offer a managed and cloud-hosted service to run the transformation queries and create tables in your Lakehouse
  • Integrate with any git repository that hosts a valid dbt project
  • Run models when data changes or on a periodic schedule
  • Manage schedules within Skypoint Cloud
  • View the current status and historical logs of each dbt transformation within Skypoint