The Python GraphQL client now includes a shutdown_repository_location API call that shuts down a gRPC server. This is useful in situations where you want Kubernetes to restart your server and re-create your repository definitions, even though the underlying Python code hasn’t changed (for example, if your pipelines are loaded programatically from a database)
io_manager_key and root_manager_key is disallowed on composite solids’ InputDefinitions and OutputDefinitions. Instead, custom IO managers on the solids inside composite solids will be respected:
@solid(input_defs=[InputDefinition("data", dagster_type=str, root_manager_key="my_root")]) def inner_solid(_, data): return data @composite_solid def my_composite(): return inner_solid()
Schedules can now be directly invoked. This is intended to be used for testing. To learn more, see https://docs.dagster.io/master/concepts/partitions-schedules-sensors/schedules#testing-schedules
dagster-postgres or dagster-graphql) are now pinned to the same version as the core dagster package. This should reduce instances of issues due to backwards compatibility problems between Dagster packages.Invoking a generator solid now yields a generator, and output objects are not unpacked.
@solid def my_solid(): yield Output("hello") assert isinstance(list(my_solid())[0], Output)
EcsRunLauncher. This creates a new ECS Task Definition and launches a new ECS Task for each run. You can use the new ECS Reference Deployment to experiment with the EcsRunLauncher. We’d love your feedback in our #dagster-ecs Slack channel!dagit with flag --suppress-warnings will now ignore all warnings, such as ExperimentalWarnings.dagster-dbt library with some helpful tips and example code (thanks @makotonium!).dagster-pyspark documentation for providing and accessing the pyspark resource (thanks @Andrew-Crosby!).build_input_context and build_output_context APIs (link).examples/hacker_news!retry_number is now available on SolidExecutionContext, allowing you to determine within a solid function how many times the solid has been previously retried.--version flag when installing the Helm chart, the tags for Dagster-provided images in the Helm chart will now default to the current Chart version. For --version <0.11.13, the image tags will still need to be updated properly to use old chart version.PIPELINE_INIT_FAILURE event type. A failure that occurs during pipeline initialization will now produce a PIPELINE_FAILURE as with all other pipeline failures.get_run_status method on the Python GraphQL client now returns a PipelineRunStatus enum instead of the raw string value in order to align with the mypy type annotation. Thanks to Dylan Bienstock for surfacing this bug!k8s_job_executor, which executes solids in separate kubernetes jobs. With the addition of this executor, you can now choose at runtime between single pod and multi-pod isolation for solids in your run. Previously this was only configurable for the entire deployment - you could either use the K8sRunLauncher with the default executors (in_process and multiprocess) for low isolation, or you could use the CeleryK8sRunLauncher with the celery_k8s_job_executor for pod-level isolation. Now, your instance can be configured with the K8sRunLauncher and you can choose between the default executors or the k8s_job_executor.DagsterGraphQLClient now allows you to specify whether to use HTTP or HTTPS when connecting to the GraphQL server. In addition, error messages during query execution or connecting to dagit are now clearer. Thanks to @emily-hawkins for raising this issue!from dagster import build_hook_context my_hook(build_hook_context(resources={"foo_resource": "foo"}))
from dagster import build_init_resource_context @resource(config_schema=str) def my_basic_resource(init_context): return init_context.resource_config context = build_init_resource_context(config="foo") assert my_basic_resource(context) == "foo"
ScheduleDefinition and SensorDefinition now carry over properties from functions decorated by @sensor and @schedule. Ie: docstrings.ResourceDefinition was not being passed to the ResourceDefinition created by the call to configured.IOManager handle_output implementation that was a generator, it would not be wrapped DagsterExecutionHandleOutputError. Now, it is wrapped.queuedRunCoordinator. See the docs for more information on setup.RetryPolicy now supports backoff and jitter settings, to allow for modulating the delay as a function of attempt number and randomness.build_schedule_context and validate_run_config functions are still in an experimental state. https://docs.dagster.io/master/concepts/partitions-schedules-sensors/schedules#testing-schedulesvalidate_run_config function is still in an experimental state. https://docs.dagster.io/master/concepts/partitions-schedules-sensors/partitions#experimental-testing-a-partition-setdagit.enableReadOnly . When enabled, a separate Dagit instance is deployed in —read-only mode. You can use this feature to serve Dagit to users who you do not want to able to kick off new runs or make other changes to application state.EventMetadata.asset.LOGS_CAPTURED, which explicitly links to the captured stdout/stderr logs for a given step, as determined by the configured ComputeLogManager on the Dagster instance. Previously, these links were available on the STEP_START event.network key on DockerRunLauncher config can now be sourced from an environment variable.get_execution_data method of SensorDefinition and ScheduleDefinition has been renamed to evaluate_tick. We expect few to no users of the previous name, and are renaming to prepare for improved testing support for schedules and sensors.build_sensor_context API. See Testing sensors.dagster.Int ) as type annotations on functions decorated with @solid have been resolved.K8sRunLauncher sometimes hanged while launching a run due to holding a stale Kubernetes client.