Datamart

We’ve already discussed that no kexes are accessible from outside of your Bizzflow project repository with the exception of datamarts. Datamarts are in fact a secure copies of your output kexes from transformations. Since we only have a single transformation resulting in a single table, let’s create just a single datamart for our table. This datamart can easily be used to be connected to from any visualisation or database tool.

Configuration

In your project repository, there should be datamarts.json file. Right now, it should contain an empty array []:


datamarts.json

[]

This is what a datamart definition looks like:

{
  "id": "{datamart_configuration_id}",
  "out_kex": "{out_kex_name}",
  "dm_kex": "{dm_kex_name}",
  "allowed_tables": [
    /* List of allowed tables */
  ]
}

datamart_configuration_id is an id that will be used to refer to our datamart configuration later on.

Via out_kex we specify which out stage kex to copy our data from. dm_kex specifies the name of our target datamart kex (it will be created for us if not existent).

allowed_tables is a non-required parameter. We can namely specify which tables from out_kex should be copied. By default, our datamart will contain all the tables from the specified out_kex. If you need to narrow this down to keep some tables from being accessible via your datamart, use allowed_tables parameter.

Putting it together

It seems easy enough, doesn’t it? This is what our datamarts.json should look like now:


datamarts.json

[
  {
    "id": "main",
    "out_kex": "out_main",
    "dm_kex": "dm_main"
  }
]

Commit your changes, run 90_update_project DAG. Your DAGs should now look like this:

Datamart in Airflow UI
Datamart in Airflow UI

Go ahead and trigger the 60_Datamrt_main DAG. After it finishes, go to your Storage page in Flow UI. After refreshing storage you should be able to find our datamart:

Datamart in Flow UI
Datamart in Flow UI

Getting the credentials

Go to the Vault page of Flow UI, select Sandbox & Datamarts section and click your datamart name. You will see the credentials needed to access said datamart.

To connect to any 3rd-party software (viztools and other), please consult the maintainer’s documentation.

Keep it moving

This was quick, easy, I am really proud of you. Let’s keep the things moving with our last chapter.