March 30th 2021

Posted March 30, 2021 ‐ 1 min read

Announcing Microsoft Azure support and lots of more minor feature updates.

New features

  • Microsoft Azure support
    • Support for Azure cloud, with Azure SQL and Azure Blob storage
  • Snowflake optimizations
    • it’s possible to clone table instead of copying - it’s much faster but not allowing step processing
    • implement much time/resource effective way to describe tables, schemas
    • adding of __timestamp column to extracted tables is also much faster - especially for tables with many rows
  • Notification
    • new notification settings - two types of notifications (error / warning - for “continue on error” tasks)
      • notifications are sent for tasks in orchestration only by default
    • default notification settings can be overridden in orchestration settings or by DAG run config see in Bizzflow wiki

Bug fixes

  • Fix continue on error
    • continue_on_error in orchestration settings now works properly, see more in Bizzflow wiki
  • Do not create sandbox DAGs for docker transformation
    • we do not have any environment for docker sandbox, so do not create DAG for it, at least until we find out a way how we want to support docker sandboxing

Other stuff to mention

  • Major refactoring
    • major refactoring of executors - try to abstract as much as reasonable
    • reorganize code structure mainly for managers - more folders
    • “minor” refactoring of managers code
    • all these changes make future development easier