Datamart

We’ve already discussed that no kexes are accessible from outside of your Bizzflow project repository with the exception of datamarts. Datamarts are in fact a secure copies of your output kexes from transformations. Since we only have a single transformation resulting in a single table, let’s create just a single datamart for our table. This datamart can easily be used to be connected to from any visualisation or database tool.

Configuration

In your project repository, there should be datamarts.json file. Right now, it should contain an empty array []:


datamarts.json

[]

This is what a datamart definition looks like:

{
  "id": "{datamart_configuration_id}",
  "out_kex": "{out_kex_name}",
  "dm_kex": "{dm_kex_name}",
  "allowed_tables": [
    /* List of allowed tables */
  ]
}

datamart_configuration_id is an id that will be used to refer to our datamart configuration later on.

Via out_kex we specify which out stage kex to copy our data from. dm_kex specifies the name of our target datamart kex (it will be created for us if not existent).

allowed_tables is a non-required parameter. We can namely specify which tables from out_kex should be copied. By default, our datamart will contain all the tables from the specified out_kex. If you need to narrow this down to keep some tables from being accessible via your datamart, use allowed_tables parameter.

Putting it together

It seems easy enough, doesn’t it? This is what our datamarts.json should look like now:


datamarts.json

[
  {
    "id": "main",
    "out_kex": "out_main",
    "dm_kex": "dm_main"
  }
]

Commit your changes, run 90_update_project DAG. Your DAGs should now look like this:

Datamart in Airflow UI
Datamart in Airflow UI

Go ahead and trigger the 60_Datamrt_main DAG. After it finishes, go to your Storage Console via Consoles -> Storage Console. After refreshing kexes in the storage console, you should be able to find our datamart:

Datamart in storage console
Datamart in storage console

Getting the credentials

In your project, go to IAM & Admin -> Service accounts.

GCP IAM
GCP IAM

Find service account with our kex’s name (dm-main) and click the three dots at the end of the line and select Manage keys.

GCP Service account
GCP Service account

Click Add key -> Create new key and confirm with Create on the next screen.

IAM new key
IAM new key
IAM create key
IAM create key

A JSON key file should be downloaded to your computer. You can use this key to connect to the datamart the same way we connected to our sandbox before.

To connect to any 3rd-party software (viztools and other), please consult the maintainer’s documentation.

Keep it moving

This was quick, easy, I am really proud of you. Let’s keep the things moving with our last chapter.