Changelog#

0.9.14#

New

  • Steps down stream of a failed step no longer report skip events and instead simply do not execute.
  • dagit-debug can load multiple debug files.
  • dagit now has a Debug Console Logging feature flag accessible at /flags .
  • Telemetry metrics are now taken when scheduled jobs are executed.
  • With memoized reexecution, we now only copy outputs that current plan won't generate
  • Document titles throughout dagit

Community Contributions

  • [dagster-ge] solid factory can now handle arbitrary types (thanks @sd2k!)
  • [dagster-dask] utility options are now available in loader/materializer for Dask DataFrame (thanks @kinghuang!)

Bugfixes

  • Fixed an issue where run termination would sometimes be ignored or leave the execution process hanging
  • [dagster-k8s] fixed issue that would cause timeouts on clusters with many jobs
  • Fixed an issue where reconstructable was unusable in an interactive environment, even when the pipeline is defined in a different module.
  • Bugfixes and UX improvements in dagit

Experimental

  • AssetMaterializations now have an optional “partition” attribute

0.9.13#

Bugfixes

  • Fixes an issue using build_reconstructable_pipeline.
  • Improved loading times for the asset catalog in Dagit.

Documentations

  • Improved error messages when invoking dagit from the CLI with bad arguments.

0.9.12#

Breaking Changes

  • Dagster now warns when a solid, pipeline, or other definition is created with an invalid name (for example, a Python keyword). This warning will become an error in the 0.9.13 release.

Community Contributions

  • Added an int type to EventMetadataEntry (Thanks @ChocoletMousse!)
  • Added a build_composite_solid_definition method to Lakehouse (Thanks @sd2k!)
  • Improved broken link detection in Dagster docs (Thanks @keyz!)

New

  • Improvements to log filtering on Run view in Dagit
  • Improvements to instance level scheduler page
  • Log engine events when pipeline termination is initiated

Bugfixes

  • Syntax errors in user code now display the file and line number with the error in Dagit
  • Dask executor no longer fails when using intermediate_storage
  • In the Celery K8s executor, we now mark the step as failed when the step job fails
  • Changed the DagsterInvalidAssetKey error so that it no longer fails upon being thrown

Documentation

  • Added API docs for dagster-dbt experimental library
  • Fixed some cosmetic issues with docs.dagster.io
  • Added code snippets from Solids examples to test path, and fixed some inconsistencies regarding parameter ordering
  • Changed to using markers instead of exact line numbers to mark out code snippets

0.9.10#

Breaking Changes

  • [dagster-dask] Removed the compute option from Dask DataFrame materialization configs for all output types. Setting this option to False (default True) would result in a future that is never computed, leading to missing materializations

Community Contributions

New

  • Console log messages are now streamlined to live on a single line per message
  • Added better messaging around $DAGSTER_HOME if it is not set or improperly setup when starting up a Dagster instance
  • Tools for exporting a file for debugging a run have been added:
    • dagster debug export - a new CLI entry added for exporting a run by id to a file
    • dagit-debug - a new CLI added for loading dagit with a run to debug
    • dagit now has a button to download the debug file for a run via the action menu on the runs page
  • The dagster api grpc command now defaults to the current working directory if none is specified
  • Added retries to dagster-postgres connections
  • Fixed faulty warning message when invoking the same solid multiple times in the same context
  • Added ability to specify custom liveness probe for celery workers in kubernetes deployment

Bugfixes

  • Fixed a bug where Dagster types like List/Set/Tuple/Dict/Optional were not displaying properly on dagit logs
  • Fixed endless spinners on dagit --empty-workspace
  • Fixed incorrect snapshot banner on pipeline view
  • Fixed visual overlapping of overflowing dagit logs
  • Fixed a bug where hanging runs when executing against a gRPC server could cause the Runs page to be unable to load
  • Fixed a bug in celery integration where celery tasks could return None when an iterable is expected, causing errors in the celery execution loop.

Experimental

  • [lakehouse] Each time a Lakehouse solid updates an asset, it automatically generates an AssetMaterialization event
  • [lakehouse] Lakehouse computed_assets now accept a version argument that describes the version of the computation
  • Setting the “dagster/is_memoized_run” tag to true will cause the run to skip any steps whose versions match the versions of outputs produced in prior runs.
  • [dagster-dbt] Solids for running dbt CLI commands
  • Added extensive documentation to illuminate how versions are computed
  • Added versions for step inputs from config, default values, and from other step outputs

0.9.9#

New

  • [Databricks] solids created with create_databricks_job_solid now log a URL for accessing the job in the Databricks UI.
  • The pipeline execute command now defaults to using your current directory if you don’t specify a working directory.

Bugfixes

  • [Celery-K8s] Surface errors to Dagit that previously were not caught in the Celery workers.
  • Fix issues with calling add_run_tags on tags that already exist.
  • Add “Unknown” step state in Dagit’s pipeline run logs view for when pipeline has completed but step has not emitted a completion event

Experimental

  • Version tags for resources and external inputs.

Documentation

  • Fix rendering of example solid config in “Basics of Solids” tutorial.