Changelog#

0.8.5#

Breaking Changes

  • Python 3.5 is no longer under test.
  • Engine and ExecutorConfig have been deleted in favor of Executor. Instead of the @executor decorator decorating a function that returns an ExecutorConfig it should now decorate a function that returns an Executor.

New

  • The python built-in dict can be used as an alias for Permissive() within a config schema declaration.
  • Use StringSource in the S3ComputeLogManager configuration schema to support using environment variables in the configuration (Thanks @mrdrprofuroboros!)
  • Improve Backfill CLI help text
  • Add options to spark_df_output_schema (Thanks @DavidKatz-il!)
  • Helm: Added support for overriding the PostgreSQL image/version used in the init container checks.
  • Update celery k8s helm chart to include liveness checks for celery workers and flower
  • Support step level retries to celery k8s executor

Bugfixes

  • Improve error message shown when a RepositoryDefinition returns objects that are not one of the allowed definition types (Thanks @sd2k!)
  • Show error message when $DAGSTER_HOME environment variable is not an absolute path (Thanks @AndersonReyes!)
  • Update default value for staging_prefix in the DatabricksPySparkStepLauncher configuration to be an absolute path (Thanks @sd2k!)
  • Improve error message shown when Databricks logs can't be retrieved (Thanks @sd2k!)
  • Fix errors in documentation fo input_hydration_config (Thanks @joeyfreund!)

0.8.4#

Bugfix

  • Reverted changed in 0.8.3 that caused error during run launch in certain circumstances
  • Updated partition graphs on schedule page to select most recent run
  • Forced reload of partitions for partition sets to ensure not serving stale data

New

  • Added reload button to dagit to reload current repository
  • Added option to wipe a single asset key by using dagster asset wipe <asset_key>
  • Simplified schedule page, removing ticks table, adding tags for last tick attempt
  • Better debugging tools for launch errors

0.8.3#

Breaking Changes

  • Previously, the gcs_resource returned a GCSResource wrapper which had a single client property that returned a google.cloud.storage.client.Client. Now, the gcs_resource returns the client directly.

    To update solids that use the gcp_resource, change:

    context.resources.gcs.client

    To:

    context.resources.gcs

New

  • Introduced a new Python API reexecute_pipeline to reexecute an existing pipeline run.
  • Performance improvements in Pipeline Overview and other pages.
  • Long metadata entries in the asset details view are now scrollable.
  • Added a project field to the gcs_resource in dagster_gcp.
  • Added new CLI command dagster asset wipe to remove all existing asset keys.

Bugfix

  • Several Dagit bugfixes and performance improvements
  • Fixes pipeline execution issue with custom run launchers that call executeRunInProcess.
  • Updates dagster schedule up output to be repository location scoped

0.8.2#

Bugfix

  • Fixes issues with dagster instance migrate.
  • Fixes bug in launch_scheduled_execution that would mask configuration errors.
  • Fixes bug in dagit where schedule related errors were not shown.
  • Fixes JSON-serialization error in dagster-k8s when specifying per-step resources.

New

  • Makes label optional parameter for materializations with asset_key specified.
  • Changes Assets page to have a typeahead selector and hierarchical views based on asset_key path.
  • dagster-ssh
    • adds SFTP get and put functions to SSHResource, replacing sftp_solid.

Docs

  • Various docs corrections

0.8.1#

Bugfix

  • Fixed a file descriptor leak that caused OSError: [Errno 24] Too many open files when enough temporary files were created.
  • Fixed an issue where an empty config in the Playground would unexpectedly be marked as invalid YAML.
  • Removed "config" deprecation warnings for dask and celery executors.

New

  • Improved performance of the Assets page.