Bizzflow

Component Configuration

Created: January 5, 2022, Updated: March 28, 2023

Since toolkit version 1.2.0 you may specify a custom component to be run instead of standard extractors, transformations and writers.

By default, Bizzflow’s extractor, writer and transformation configurations accept parameter type (as seen in both respective pages Extractors and Writers) that tells Bizzflow to use official Bizzflow’s components. If you find yourself in a situation, when you need a component that currently does not exist and you are experienced enough to create a custom extractor or writer, you can make use of advanced component configuration and tell Bizzflow to use your component instead.

For guidelines on how to create your custom component, see For Developers section of this wiki.

Component Configuration Reference

ParameterRequiredValueDescription
type✓"git", "docker" or "local"Component source type
repository✓stringComplete uri to your component’s git repository (only applicable if type = "git")
checkout✕stringBranch, tag name or commit SHA to be checked out before building component (only applicable if type = "git")
image✓stringFull docker image path (only applicable if type = "docker")
registry✓stringDocker registry (only applicable if type = "docker")
username✕stringUsername to be used for authorization in either docker registry or git repository (only applicable for type = "docker" or "git")
password✕stringPassword to be used for git authorization in either docker registry or git repository (only applicable for type = "docker" or "git")
path✓stringPath within project’s repository relative to repository’s root (the directory where your project.json is), (only applicable if type = "local")

Component Configuration Example 1: Private git repository

Following is an example of what a component configuration would look like, when your custom component exists within a private git repository.

extractors/custom.json

{
  "type": "ex-custom-component",
  // you need to specify type for tables' naming conventions
  "component_source": {
    "type": "git",
    "repository": "https://gitlab.com/ijustfarted/custom-component",
    "checkout": "latest",
    "username": "ijustfarted",
    "password": "#!#:gitlab-password"
  },
  "config": {
    // this object will be passed to your
    // custom component's path '/config/config.json'
  }
}

.

extractors/custom.yaml

type:
  ex-custom-component
  # you need to specify type for tables' naming conventions
component_source:
  type: git
  repository: https://gitlab.com/ijustfarted/custom-component
  checkout: latest
  username: ijustfarted
  password: "#!#:gitlab-password"

config:
  # This object will be passed to your
  # custom component's path '/config/config.json'

Component Configuration Example 2: Public docker image

This is an example configuration that should work well for cases where your custom component is a docker image built and pushed into docker registry.

extractors/custom-docker.json

{
  "type": "ex-custom-component",
  "component_source": {
    "type": "docker",
    "image": "registry.gitlab.com/ijustfarted/custom-component:latest",
    "registry": "registry.gitlab.com"
  },
  "config": {
    // this object will be passed to your
    // custom component's path '/config/config.json'
  }
}

extractors/custom-docker.yaml

type: ex-custom-component
component_source:
  type: docker
  image: registry.gitlab.com/ijustfarted/custom-component:latest
  registry: registry.gitlab.com
config:
  # This object will be passed to your
  # custom component's path '/config/config.json'

Component Configuration Example 3: Local component

This is the easiest use-case, when your component’s Dockerfile resides within your Bizzflow project’s repository.

The example presumes you have already written a working extractor with a Dockerfile with both the Dockerfile and the context in ./my-extractor subdirectory in your project repository.

extractors/custom-local.json

{
  "type": "ex-custom-component",
  "component_source": {
    "type": "local",
    "path": "./my-extractor/"
  },
  "config": {
    // this object will be passed to your
    // custom component's path '/config/config.json'
  }
}

extractors/custom-local.yaml

type: ex-custom-component
component_source:
  type: local
  path: ./my-extractor/
config:
  # This object will be passed to your
  # custom component's path '/config/config.json'

Component Configuration Example 4: Docker transformation

In order to process your data other way than using SQL in your Data Warehouse, Bizzflow offers docker transformations. You can use any language capable of running within a docker image using "type": "docker" in your transformations.json (or transformations.yaml) configuration file.

transformations.json

[
  // ...
  {
    "id": "arima",
    "type": "docker",
    "in_kex": ["in_arima"],
    "out_kex": "out_arima",
    "source": "",
    "component_source": {
      "type": "docker",
      "image": "ijustfarted/py-arima:latest",
      "registry": "", // you can keep empty for DockerHub images
      "username": "ijustfarted",
      "password": "#!#:dockerhub-ijustfarted"
    }
  }
  // ...
]

transformations.yaml

# ...
- id: arima
  type: docker
  in_kex:
    - in_arima
  out_kex: out_arima
  source: ""
  component_source:
    type: docker
    image: ijustfarted/py-arima:latest
    registry: "" # you can keep empty for DockerHub images
    username: ijustfarted
    password: "#!#:dockerhub-ijustfarted"
# ...
← Step