Changelog#

0.10.5#

Community Contributions

Add /License for packages that claim distribution under Apache-2.0 (thanks @bollwyvl!)

New

[k8s] Changed our weekly docker image releases (the default images in the helm chart). dagster/dagster-k8s and dagster/dagster-celery-k8s can be used for all processes which don't require user code (Dagit, Daemon, and Celery workers when using the CeleryK8sExecutor). user-code-example can be used for a sample user repository. The prior images (k8s-dagit, k8s-celery-worker, k8s-example) are deprecated.
configured api on solids now enforces name argument as positional. The name argument remains a keyword argument on executors. name argument has been removed from resources, and loggers to reflect that they are anonymous. Previously, you would receive an error message if the name argument was provided to configured on resources or loggers.
[sensors] In addition to the per-sensor minimum_interval_seconds field, the overall sensor daemon interval can now be configured in the dagster.yaml instance settings with:


sensor_settings:
  interval_seconds: 30 # (default)

This changes the interval at which the daemon checks for sensors which haven't run within their minimum_interval_seconds.

The message logged for type check failures now includes the description included in the TypeCheck
The dagster-daemon process now runs each of its daemons in its own thread. This allows the scheduler, sensor loop, and daemon for launching queued runs to run in parallel, without slowing each other down. The dagster-daemon process will shut down if any of the daemon threads crash or hang, so that the execution environment knows that it needs to be restarted.
dagster new-repo is a new CLI command that generates a Dagster repository with skeleton code in your filesystem. This CLI command is experimental and it may generate different files in future versions, even between dot releases. As of 0.10.5, dagster new-repo does not support Windows. See here for official API docs.
When using a grpc_server repository location, Dagit will automatically detect changes and prompt you to reload when the remote server updates.
Improved consistency of headers across pages in Dagit.
Added support for assets to the default SQLite event log storage.

Integrations

[dagster-pandas] - Improved the error messages on failed pandas type checks.
[dagster-postgres] - postgres_url is now a StringSource and can be loaded by environment variable
[helm] - Users can set Kubernetes labels on Celery worker deployments
[helm] - Users can set environment variables for Flower deployment
[helm] - The redis helm chart is now included as an optional dagster helm chart dependency

Bugfixes

Resolved an error preventing dynamic outputs from being passed to composite_solid inputs
Fixed the tick history graph for schedules defined in a lazy-loaded repository (#3626)
Fixed performance regression of the Runs page on dagit.
Fixed Gantt chart on Dagit run view to use the correct start time, repairing how steps are rendered within the chart.
On Instance status page in Dagit, correctly handle states where daemons have multiple errors.
Various Dagit bugfixes and improvements.

Bugfixes

Fixed an issue with daemon heartbeat backwards compatibility. Resolves an error on Dagit's Daemon Status page

New

[dagster] Sensors can now specify a minimum_interval_seconds argument, which determines the minimum amount of time between sensor evaluations.
[dagit] After manually reloading the current repository, users will now be prompted to regenerate preset-based or partition-set based run configs in the Playground view. This helps ensure that the generated run config is up to date when launching new runs. The prompt does not occur when the repository is automatically reloaded.

Bugfixes

Updated the -n/--max_workers default value for the dagster api grpc command to be None. When set to None, the gRPC server will use the default number of workers which is based on the CPU count. If you were previously setting this value to 1, we recommend removing the argument or increasing the number.
Fixed issue loading the schedule tick history graph for new schedules that have not been turned on.
In Dagit, newly launched runs will open in the current tab instead of a new tab.
Dagit bugfixes and improvements, including changes to loading state spinners.
When a user specifies both an intermediate storage and an IO manager for a particular output, we no longer silently ignore the IO manager

Community Contributions

[docs] Update URL to telemetry info (thanks @emilmelnikov (https://github.com/dagster-io/dagster/commits?author=emilmelnikov)!)
[dagster-azure] Fix for listing files on ADL example (thanks @ericct!)

New

[dagstermill] Users can now specify custom tags & descriptions for notebook solids.
[dagster-pagerduty / dagster-slack] Added built-in hook integrations to create pagerduty/slack alerts when solids fail.
[dagit] Added ability to preview runs for upcoming schedule ticks.

Bugfixes

Fixed an issue where run start times and end times were displayed in the wrong timezone in Dagit when using Postgres storage.
Schedules with partitions that weren’t able to execute due to not being able to find a partition will now display the name of the partition they were unable to find on the “Last tick” entry for that schedule.
Improved timing information display for queued and canceled runs within the Runs table view and on individual Run pages in Dagit.
Improvements to the tick history view for schedules and sensors.
Fixed formatting issues on the Dagit instance configuration page.
Miscellaneous Dagit bugfixes and improvements.
The dagster pipeline launch command will now respect run concurrency limits if they are applied on your instance.
Fixed an issue where re-executing a run created by a sensor would cause the daemon to stop executing any additional runs from that sensor.
Sensor runs with invalid run configuration will no longer create a failed run - instead, an error will appear on the page for the sensor, allowing you to fix the configuration issue.
General dagstermill housekeeping: test refactoring & type annotations, as well as repinning ipykernel to solve #3401

Documentation

Improved dagster-dbt example.
Added examples to demonstrate experimental features, including Memoized Development and Dynamic Graph.
Added a PR template and how to pick an issue for the first time contributors

Community Contributions

Reduced image size of k8s-example by 25% (104 MB) (thanks @alex-treebeard and @mrdavidlaing!)
[dagster-snowflake] snowflake_resource can now be configured to use the SQLAlchemy connector (thanks @basilvetas!)

New

When setting userDeployments.deployments in the Helm chart, replicaCount now defaults to 1 if not specified.

Bugfixes

Fixed an issue where the Dagster daemon process couldn’t launch runs in repository locations containing more than one repository.
Fixed an issue where Helm chart was not correctly templating env, envConfigMaps, and envSecrets.

Documentation

Added new troubleshooting guide for problems encountered while using the QueuedRunCoordinator to limit run concurrency.
Added documentation for the sensor command-line interface.