Datamart

We’ve already discussed that no kexes are accessible from outside of your Bizzflow project repository with the exception of datamarts. Datamarts are in fact a secure copies of your output kexes from transformations. Since we only have a single transformation resulting in a single table, let’s create just a single datamart for our table. This datamart can easily be used to be connected to from any visualisation or database tool.

Configuration#

In your project repository, there should be datamarts.json file. Right now, it should contain an empty array []:

datamarts.json

[]

This is what a datamart definition looks like:

{
  "id": "{datamart_configuration_id}",
  "out_kex": "{out_kex_name}",
  "dm_kex": "{dm_kex_name}",
  "allowed_tables": [
    /* List of allowed tables */
  ]
}

datamart_configuration_id is an id that will be used to refer to our datamart configuration later on.

Via out_kex we specify which out stage kex to copy our data from. dm_kex specifies the name of our target datamart kex (it will be created for us if not existent).

allowed_tables is a non-required parameter. We can namely specify which tables from out_kex should be copied. By default, our datamart will contain all the tables from the specified out_kex. If you need to narrow this down to keep some tables from being accessible via your datamart, use allowed_tables parameter.

Putting it together#

It seems easy enough, doesn’t it? This is what our datamarts.json should look like now:

datamarts.json

[
  {
    "id": "main",
    "out_kex": "out_main",
    "dm_kex": "dm_main"
  }
]

Commit your changes, run 90_update_project DAG. Your DAGs should now look like this:

Go ahead and trigger ▶ the 60_Datamrt_main DAG. After it finishes, go to your Storage Console via Consoles -> Storage Console. After refreshing kexes in the storage console, you should be able to find our datamart:

Getting the credentials#

In your project, go to IAM & Admin -> Service accounts.

Find service account with our kex’s name (dm-main) and click the three dots at the end of the line and select Manage keys.

Click Add key -> Create new key and confirm with Create on the next screen.

A JSON key file should be downloaded to your computer. You can use this key to connect to the datamart the same way we connected to our sandbox before.

To connect to any 3rd-party software (viztools and other), please consult the maintainer’s documentation.

Keep it moving#

This was quick, easy, I am really proud of you. Let’s keep the things moving with our last chapter.