You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Introduce a dedicated deployment mode for kedro run command, which could be enabled via a flag (e.g., --deployment) or through project settings. This mode would skip non-essential validation checks that are only useful during pipeline development.
This ticket is a follow-up to the discussion in #4603 (comment)
Currently, there are more than five checks prefixed with _validate_ that verify the validity of the pipeline structure. However, once a pipeline has been developed and tested locally, these checks are less meaningful when running the pipeline repeatedly in a deployment environment without any changes to the pipeline code.
Benefits
Reduced runtime overhead in production
More control for advanced users deploying stable pipelines
Aligns with patterns seen in other tools that offer dev vs. prod execution modes
The text was updated successfully, but these errors were encountered:
I'd also like to see how we handle the following requirement:
as a user I'm deploying my Kedro pipeline to a k8s based orchestrator
part of my Kedro pipeline is data engineering focused and needs Spark, JDK etc.
part of my Kedro pipeline is ML focused and needs Tensorflow/Pytorch
I want to only want to deploy the relevant dependencies to the target containers. Note it's not an issue / very cheap to deploy the same code twice even if only part of it will be executed in different locations.
Description
Introduce a dedicated
deployment mode
forkedro run
command, which could be enabled via a flag (e.g., --deployment) or through project settings. This mode would skip non-essential validation checks that are only useful during pipeline development.This ticket is a follow-up to the discussion in #4603 (comment)
Currently, there are more than five checks prefixed with
_validate_
that verify the validity of the pipeline structure. However, once a pipeline has been developed and tested locally, these checks are less meaningful when running the pipeline repeatedly in a deployment environment without any changes to the pipeline code.Benefits
The text was updated successfully, but these errors were encountered: