Datamart

Created: May 14, 2021, Updated: March 28, 2023

We’ve already discussed that no kexes are accessible from outside of your Bizzflow project repository with the exception of datamarts. Datamarts are in fact a secure copies of your output kexes from transformations. Since we only have a single transformation resulting in a single table, let’s create just a single datamart for our table. This datamart can easily be used to be connected to from any visualisation or database tool.

Configuration#

In your project repository, there should be datamarts.json file. Right now, it should contain an empty array []:

datamarts.json

[]

This is what a datamart definition looks like:

{
  "id": "{datamart_configuration_id}",
  "out_kex": "{out_kex_name}",
  "dm_kex": "{dm_kex_name}",
  "allowed_tables": [
    /* List of allowed tables */
  ]
}

datamart_configuration_id is an id that will be used to refer to our datamart configuration later on.

Via out_kex we specify which out stage kex to copy our data from. dm_kex specifies the name of our target datamart kex (it will be created for us if not existent).

allowed_tables is a non-required parameter. We can namely specify which tables from out_kex should be copied. By default, our datamart will contain all the tables from the specified out_kex. If you need to narrow this down to keep some tables from being accessible via your datamart, use allowed_tables parameter.

Putting it together#

It seems easy enough, doesn’t it? This is what our datamarts.json should look like now:

datamarts.json

[
  {
    "id": "main",
    "out_kex": "out_main",
    "dm_kex": "dm_main"
  }
]

Commit your changes, run 90_update_project DAG. Your DAGs should now look like this:

Go ahead and trigger ▶ the 60_Datamrt_main DAG. After it finishes, go to your Storage page in Flow UI. After refreshing storage you should be able to find our datamart:

Getting the credentials#

Go to the Vault page of Flow UI, select Sandbox & Datamarts section and click your datamart name. You will see the credentials needed to access said datamart.

To connect to any 3rd-party software (viztools and other), please consult the maintainer’s documentation.

Keep it moving#

This was quick, easy, I am really proud of you. Let’s keep the things moving with our last chapter.