Changelog#

1.3.14 (core) / 0.19.14 (libraries)#

New#

• DynamicPartitionsDefinition and SensorResult are no longer marked experimental
• DagsterInstance now has a get_status_by_partition method, which returns the status of each partition for a given asset. Thanks renzhe-brian!
• DagsterInstance now has a get_latest_materialization_code_versions method, which returns the code version of the latest materialization for each of the provided (non-partitioned) assets.
• The error message for when an asset illegally depends on itself is now more informative.
• Further performance improvements for the Asset Daemon.
• Performance improvements in the asset graph view for large asset graphs.
• Pandas 2.x is now supported in all dagster packages.
• build_asset_context has been added as an asset focused replacement for build_op_context.
• build_op_context now accepts a partition_key_range parameter.
• New AssetSelection.upstream_source_assets method allows selecting source assets upstream of the current selection.
• AssetSelection.key_prefixes and AssetSelection.groups now accept an optional include_sources parameter.
• The AutoMaterialize evaluations UI now provides more details about partitions and waiting on upstream assets.
• [dbt] The DbtCli resource is no longer marked experimental.
• [dbt] The global_config parameter of the DbtCli resource has been renamed to global_config_flags
• [dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now work with the DbtCli resource.
• [dbt] The manifest argument of the @dbt_assets decorator now additionally can accept a Path argument representing a path to the manifest file or dictionary argument representing the raw manifest blob.
• [dbt] When invoking DbtCli.cli from inside a @dbt_assets-decorated function, you no longer need to supply the manifest argument as long as you provide the context argument.
• [dbt] The DbtManifest object can now generate schedules using dbt selection syntax.

dbt_manifest.build_schedule(
  job_name="materialize_dbt_models",
  cron_schedule="0 0 * * *",
  dbt_select="fqn:*"
)

• [dbt] When invoking DbtCli.cli and the underlying command fails, an exception will now be raised. To suppress the exception, run the DbtCli.cli(..., raise_on_error=False).
• [ui] You can now alphabetically sort your partitions on the asset partitions page
• [ui] A button in the “Run is materializing this asset” and “Run failed to materialize this asset” banners provides direct access to the relevant run logs

Bugfixes#

• Fixed a bug that caused asset metadata to not be available available on the OutputContext when using with_attributes or AssetsDefinition.from_graph.
• Previously, if a partitioned asset at the root of the graph had more missing partitions than its AutoMaterializePolicy’s max_materializations_per_minute parameter, those older partitions would not be properly discarded from consideration on subsequent ticks. This has been fixed.
• Fixed a bug that caused AutoMaterializePolicy.lazy() to not materialize missing assets that were downstream of assets without an AutoMaterializePolicy.
• In rare cases, the AssetDaemon could hit an exception when using a combination of freshness policies and observable source assets. This has been fixed.
• Previously, string type annotations (most commonly via modules containing from __future__ import annotations) would cause errors in most cases when used with Dagster definitions. This has been fixed for the vast majority of cases.
• AssetExecutionContext has returned to being a type alias for OpExecutionContext.
• [ui] Date filtering on the runs page now takes your timezone into consideration
• [ui] Fixed a bug where selecting partitions in the launchpad dialog cleared out your configuration
• [ui] In the run Gantt chart, executed steps that follow skipped steps no longer render off the far right of the visualization.
• [ui] Cancelling a running backfill no longer makes canceled partitions un-selectable on the job partitions page and backfill modal, and cancellation is shown in gray instead of red.

Breaking Changes#

• [experimental] The internal time_window_partition_scope_minutes parameter of the AutoMaterializePolicy class has been removed. Instead, max_materializations_per_minute should be used to limit the number of runs that may be kicked off for a partitioned asset.

Deprecations#

• [dbt] DbtCliResource has been deprecated in favor of DbtCli.
• The python package dagit has been deprecated in favor of a new package dagster-webserver.
• OpExecutionContext.asset_partition_key_range has been deprecated in favor of partition_key_range.

Community Contributions#

• The databricks_pyspark_step_launcher will no longer error when executing steps that target a single partition of a DynamicPartitionsDefinition (thanks @weberdavid!).
• Increased timeout on readinessProbe for example user code images, which prevents breakages in certain scenarios (thanks @leehuwuj)!
• Avoid creation of erroneous local directories by GCS IO manager (thanks @peterjclaw)!
• Fixed typo in intro docs (thanks @adeboyed)!
• Fix typo in bigquery docs (thanks @nigelainscoe)!
• Fix typing on run tag validation (thanks @yuvalgimmunai)!
• Allow passing repositoryCredentials arn as config to ecs run launcher (thanks @armandobelardo)!

Experimental#

• The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval.
• [dbt] DbtCliTask has been renamed to DbtCliInvocation
• [dbt] The get_asset_key_by_output_name and get_node_info_by_output_name methods of DbtManifest have been renamed toget_asset_key_for_output_name and get_node_info_for_output_name, respectively.
• [ui] A new feature flag allows you to switch Asset DAG rendering to a tighter horizontal layout, which may be preferable in some scenarios

Documentation#

• Many public methods that were missing in the API docs are now documented. Updated classes include DagsterInstance, *MetadataValue, DagsterType, and others.
• dagster-pandera now has an API docs page.
• Deprecated methods in the API docs now are marked with a special badge.

1.3.13 (core) / 0.19.13 (libraries)#

Bugfixes#

• Fixes a bug in dagster project from-example that was preventing it from downloading examples correctly.

1.3.12 (core) / 0.19.12 (libraries)#

New#

• The --name argument is now optional when running dagster project from-example.
• An asset key can now be directly specified via the asset decorator: @asset(key=...).
• AssetKey now has a with_prefix method.
• Significant performance improvements when using AutoMaterializePolicys with large numbers of partitions.
• dagster instance migrate now prints information about changes to the instance database schema.
• The dagster-cloud-agent helm chart now supports setting K8s labels on the agent deployment.
• [ui] Step compute logs are shown under “Last Materialization” in the asset sidebar.
• [ui] Truncated asset names now show a tooltip when hovered in the asset graph.
• [ui] The “Propagate changes” button has been removed and replaced with “Materialize Stale and Missing” (which was the “Propagate changes” predecessor).

Bugfixes#

• [ui] Fixed an issue that prevented filtering by date on the job-specific runs tab.

• [ui] “F” key with modifiers (alt, ctrl, cmd, shift) no longer toggles the filter menu on pages that support filtering.

• [ui] Fix empty states on Runs table view for individual jobs, to provide links to materialize an asset or launch a run for the specific job, instead of linking to global pages.

• [ui] When a run is launched from the Launchpad editor while an editor hint popover is open, the popover remained on the page even after navigation. This has been fixed.

• [ui] Fixed an issue where clicking on the zoom controls on a DAG view would close the right detail panel for selected nodes.

• [ui] Fixed an issue shift-selecting assets with multi-component asset keys.

• [ui] Fixed an issue with the truncation of the asset stale causes popover.

• When using a TimeWindowPartitionMapping with a start_offset or end_offset specified, requesting the downstream partitions of a given upstream partition would yield incorrect results. This has been fixed.

• When using AutoMaterializePolicys with observable source assets, in rare cases, a second run could be launched in response to the same version being observed twice. This has been fixed.

• When passing in hook_defs to define_asset_job, if any of those hooks had required resource keys, a missing resource error would surface when the hook was executed. This has been fixed.

• Fixed a typo in a documentation URL in dagster-duckdb-polars tests. The URL now works correctly.

Experimental#

• [dagster-dbt] Added methods to DbtManifest to fetch asset keys of sources and models: DbtManifest.get_asset_key_for_model, DbtManifest.get_asset_key_for_source. These methods are utilities for defining python assets as dependencies of dbt assets via @asset(key=manifest.get_asset_key_for_model(...).
• [dagster-dbt] The use of the state_path parameter with DbtManifestAssetSelection has been deprecated, and will be removed in the next minor release.
• Added experimental support for limiting global op/asset concurrency across runs.

Dependencies#

• Upper bound on the grpcio package (for dagster) has been removed.

Breaking Changes#

• Legacy methods of PartitionMapping have been removed. Defining custom partition mappings has been unsupported since 1.1.7.

Community Contributions#

• [dagster-airbyte] Added the ability to specify asset groups to build_airbyte_assets. Thanks @guy-rvvup!

Documentation#

• For Dagster Cloud Serverless users, we’ve added our static IP addresses to the Serverless docs.

1.3.11 (core) / 0.19.11 (libraries)#

New#

• Assets with lazy auto-materialize policies are no longer auto-materialized if they are missing but don’t need to be materialized in order to help downstream assets meet their freshness policies.
• [ui] The descriptions of auto-materialize policies in the UI now include their skip conditions along with their materialization conditions.
• [dagster-dbt] Customized asset keys can now be specified for nodes in the dbt project, using meta.dagster.asset_key. This field takes in a list of strings that are used as the components of the generated AssetKey.

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          asset_key: ["my", "custom", "asset_key"]

• [dagster-dbt] Customized groups can now be specified for models in the dbt project, using meta.dagster.group. This field takes in a string that is used as the Dagster group for the generated software-defined asset corresponding to the dbt model.

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          group: "my_group"

Bugfixes#

• Fixed an issue where the dagster-msteams and dagster-mlflow packages could be installed with incompatible versions of the dagster package due to a missing pin.
• Fixed an issue where the dagster-daemon run command sometimes kept code server subprocesses open longer than it needed to, making the process use more memory.
• Previously, when using @observable_source_assets with AutoMaterializePolicies, it was possible for downstream assets to get “stuck”, not getting materialized when other upstream assets changed, or for multiple down materializations to be kicked off in response to the same version being observed multiple times. This has been fixed.
• Fixed a case where the materialization count for partitioned assets could be wrong.
• Fixed an error which arose when trying to request resources within run failure sensors.
• [dagster-wandb] Fixed handling for multi-dimensional partitions. Thanks @chrishiste

Experimental#

• [dagster-dbt] improvements to @dbt_assets
  • project_dir and target_path in DbtCliTask are converted from type str to type pathlib.Path.
  • In the case that dbt logs are not emitted as json, the log will still be redirected to be printed in the Dagster compute logs, under stdout.

Documentation#

• Fixed a typo in dagster_aws S3 resources. Thanks @akan72
• Fixed a typo in link on the Dagster Instance page. Thanks @PeterJCLaw

1.3.10 (core) / 0.19.10 (libraries)#

New#

• [dagster-dbt] By default, freshness policies and auto materialize policies on dbt assets can now be specified using the dagster field under +meta configuration. The following are equivalent:

Before:

version: 2

models:
  - name: users
    config:
      dagster_freshness_policy:
        maximum_lag_minutes: 60
        cron_schedule: "0 9 * * *"
      dagster_auto_materialize_policy:
        type: "lazy"

After:

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          freshness_policy:
            maximum_lag_minutes: 60
            cron_schedule: "0 9 * * *"
          auto_materialize_policy:
            type: "lazy"

• Added support for Pythonic Config classes to the @configured API, which makes reusing op and asset definitions easier:

  class GreetingConfig(Config):
      message: str
  
  @op
  def greeting_op(config: GreetingConfig):
      print(config.message)
  
  class HelloConfig(Config):
      name: str
  
  @configured(greeting_op)
  def hello_op(config: HelloConfig):
      return GreetingConfig(message=f"Hello, {config.name}!")
  

• Added AssetExecutionContext to replace OpExecutionContext as the context object passed in to @asset functions.

• TimeWindowPartitionMapping now contains an allow_nonexistent_upstream_partitions argument that, when set to True, allows a downstream partition subset to have nonexistent upstream parents.

• Unpinned the alembic dependency in the dagster package.

• [ui] A new “Assets” tab is available from the Overview page.

• [ui] The Backfills table now includes links to the assets that were targeted by the backfill.

Bugfixes#

• Dagster is now compatible with a breaking change introduced in croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.
• Fixed an issue introduced in 1.3.8 which prevented resources from being bound to sensors when the specified job required late-bound resources.
• Fixed an issue which prevented specifying resource requirements on a @run_failure_sensor.
• Fixed an issue where the asset reconciliation sensor failed with a “invalid upstream partitions” error when evaluating time partitions definitions with different start times.
• [dagster-k8s] Fixed an issue where annotations are not included in the Dagster Helm chart for the pod that is created when configuring the Helm chart to run database migrations.
• [ui] Fixed an issue with filtering runs by created date on the Runs page.
• [ui] The “upstream partitions missing” warning no longer appears in the asset backfill dialog if the upstream partitioned asset is a source asset.
• [dagster-dbt] Fixed an issue where asset dependencies for dbt models with ephemeral models in between them would sometimes be improperly rendered.

Community Contributions#

• Added support for setting resources in asset and multi_asset sensors. Thanks @plaflamme!
• Fixed an issue where py.typed was missing in the dagster-graphql package. Thanks @Tanguy-LeFloch!

Experimental#

• Evaluation history for AutoMaterializePolicys will now be cleared after 1 week.
• [dagster-dbt] Several improvements to @dbt_assets:
  • profile and target can now be customized on the DbtCli resource.
  • If a partial_parse.msgpack is detected in the target directory of your dbt project, it is now copied into the target directories created by DbtCli to take advantage of partial parsing.
  • The metadata of assets generated by @dbt_assets can now be customized by overriding DbtManifest.node_info_to_metadata.
  • Execution duration of dbt models is now added as default metadata to AssetMaterializations.

Documentation#

• Added a new tutorial section about using resources.

Dagster Cloud#

• Fixed an issue where overriding the container name of a code server pod using serverK8sConfig.containerConfig.name did not actually change the container name.