March 30^th 2021

Announcing Microsoft Azure support and lots of more minor feature updates.

New features

• Microsoft Azure support
  • Support for Azure cloud, with Azure SQL and Azure Blob storage
• Snowflake optimizations
  • it’s possible to clone table instead of copying - it’s much faster but not allowing step processing
  • implement much time/resource effective way to describe tables, schemas
  • adding of __timestamp column to extracted tables is also much faster - especially for tables with many rows
• Notification
  • new notification settings - two types of notifications (error / warning - for “continue on error” tasks)
    • notifications are sent for tasks in orchestration only by default
  • default notification settings can be overridden in orchestration settings or by DAG run config see in Bizzflow wiki

Bug fixes

• Fix continue on error
  • continue_on_error in orchestration settings now works properly, see more in Bizzflow wiki
• Do not create sandbox DAGs for docker transformation
  • we do not have any environment for docker sandbox, so do not create DAG for it, at least until we find out a way how we want to support docker sandboxing

Other stuff to mention

• Major refactoring
  • major refactoring of executors - try to abstract as much as reasonable
  • reorganize code structure mainly for managers - more folders
  • “minor” refactoring of managers code
  • all these changes make future development easier