Project Structure

Created: May 20, 2021, Updated: January 26, 2024

This chapter contains all the information about what your Bizzflow project should look like in your git repository.

A Bizzflow project is basically a directory structure containing configuration files (JSON or YAML and SQL files). Typical project structure should look somewhat like this:

🤓

Only one config format is allowed, you can’t combine both JSON and YAML in one project. The format is determined by the existence of project.yaml or project.json. Please note, if you encounter a file called e.g. datamarts.json in this guide, it can be called datamarts.yaml in your project with little to no difference, based on your decision of format.

.
├── dags/
|   └── custom_dag.py
├── extractors/
    ├── hubspot.json
    └── root_mysql.json
├── transformations/
    └── main/
      ├── 00_init_invoices.sql
      ├── 05_init_deals.sql
      ├── 10_process_invoices.sql
      └── 20_process_deals.sql
├── datamarts.json
├── orchestrations.json
├── project.json
├── step.json
├── transformations.json
├── notifications.json
├── datasets.json
├── writers/
    └── hubspot.json
└── .bizzflow_project

A Bizzflow project

Below you will find each of the directory’s and file’s description in order to understand what the structure means for your project.

dags/

dags is a directory that is empty by default. It is a place you can put your custom Airflow DAGs to and they will appear in your Airflow UI.

🤓

Please make sure your custom DAGs are compatible with your current running version of Airflow and Python.

Usually, the installed versions are:

Airflow 1.10.15
Python 3.6

But we are planning to upgrade Airflow’s major version to 2 and Python’s minor to 3.9 in a foreseeable future.

extractors/

This directory will contain all your extractors' configuration files. They are always JSON / YAML files with keys specified in a separate Extractors chapter.

transformations/

With combination of transformations.json (or transformations.yaml) file this directory makes it possible for you to add transformations to your project. While the configuration file contains transformation’s metadata, this is the directory where you will put the transformation SQL files into. See more in Transformations.

datamarts.json

See Datamarts for more information.

orchestrations.json

See Orchestrations for more information.

project.json

See General Project Configuration for more information.

step.json

See Step for more information.

transformations.json

With combination of transformations/ directory this file makes it possible for you to add transformations to your project. While the configuration file contains transformation’s metadata, the directory will contain the transformation SQL files. See more in Transformations.

datasets.json

See Datasets for more information.

notifications.json

See Notifications for more information.

writers/

This directory will contain all your writers' configuration files. They are always JSON / YAML files with keys specified in a separate Writers chapter.

.bizzflow_project

This is a file that servers for Bizzflow to recognize the repository as a Bizzflow project. You do not need to worry about this file at all.

Project Structure

A Bizzflow project#

dags/#

extractors/#

transformations/#

datamarts.json#

orchestrations.json#

project.json#

step.json#

transformations.json#

datasets.json#

notifications.json#

writers/#

.bizzflow_project#