Workflow¶

The workflow component is responsible for converting a Gordo config into an Argo workflow which then runs the various components in order to build and serve the ML models.

Normalized Config¶

class gordo.workflow.config_elements.normalized_config.NormalizedConfig(config: dict, project_name: str, gordo_version: Optional[str] = None, model_builder_env: Optional[dict] = None)[source]¶

Bases: object

Handles the conversion of a single Machine representation in config format and updates it with any features which are ‘left out’ inside of globals key or the default config globals held here.

DEFAULT_CONFIG_GLOBALS: Dict[str, Any] = {'evaluation': {'cv_mode': 'full_build', 'metrics': ['explained_variance_score', 'r2_score', 'mean_squared_error', 'mean_absolute_error'], 'scoring_scaler': 'sklearn.preprocessing.MinMaxScaler'}, 'runtime': {'builder': {'remote_logging': {'enable': False}, 'resources': {'limits': {'cpu': 1001, 'memory': 31200}, 'requests': {'cpu': 1001, 'memory': 3900}}}, 'client': {'max_instances': 30, 'resources': {'limits': {'cpu': 2000, 'memory': 4000}, 'requests': {'cpu': 100, 'memory': 3500}}}, 'influx': {'enable': True}, 'prometheus_metrics_server': {'resources': {'limits': {'cpu': 200, 'memory': 1000}, 'requests': {'cpu': 100, 'memory': 200}}}, 'reporters': [], 'server': {'resources': {'limits': {'cpu': 2000, 'memory': 6000}, 'requests': {'cpu': 1000, 'memory': 3000}}}}}¶

SPLITED_DOCKER_IMAGES: Dict[str, Any] = {'runtime': {'builder': {'image': 'gordo-model-builder'}, 'client': {'image': 'gordo-client'}, 'deployer': {'image': 'gordo-deploy'}, 'prometheus_metrics_server': {'image': 'gordo-model-server'}, 'server': {'image': 'gordo-model-server'}}}¶

UNIFIED_DOCKER_IMAGES: Dict[str, Any] = {'runtime': {'builder': {'image': 'gordo-base'}, 'client': {'image': 'gordo-base'}, 'deployer': {'image': 'gordo-base'}, 'prometheus_metrics_server': {'image': 'gordo-base'}, 'server': {'image': 'gordo-base'}}}¶

UNIFYING_GORDO_VERSION: str = '1.2.0'¶

classmethod get_default_globals(gordo_version: str) → dict[source]¶

classmethod prepare_patched_globals(patched_globals: dict) → dict[source]¶

static prepare_runtime(runtime: dict) → dict[source]¶

Workflow Generator¶

Workflow loading/processing functionality to help the CLI ‘workflow’ sub-command.

gordo.workflow.workflow_generator.workflow_generator.default_image_pull_policy(gordo_version: gordo.util.version.Version) → str[source]¶

gordo.workflow.workflow_generator.workflow_generator.get_dict_from_yaml(config_file: Union[str, _io.StringIO]) → dict[source]¶: Read a config file or file like object of YAML into a dict

gordo.workflow.workflow_generator.workflow_generator.load_workflow_template(workflow_template: str) → jinja2.environment.Template[source]¶

Loads the Jinja2 Template from a specified path

Parameters: workflow_template (str) – Path to a workflow template
Returns: Loaded but non-rendered jinja2 template for the workflow
Return type: jinja2.Template

gordo.workflow.workflow_generator.workflow_generator.yaml_filter(data: Any) → str[source]¶

Helpers¶

gordo.workflow.workflow_generator.helpers.patch_dict(original_dict: dict, patch_dictionary: dict) → dict[source]¶

Patches a dict with another. Patching means that any path defines in the patch is either added (if it does not exist), or replaces the existing value (if it exists). Nothing is removed from the original dict, only added/replaced.

Parameters

original_dict (dict) – Base dictionary which will get paths added/changed
patch_dictionary (dict) – Dictionary which will be overlaid on top of original_dict

Examples

>>> patch_dict({"highKey":{"lowkey1":1, "lowkey2":2}}, {"highKey":{"lowkey1":10}})
{'highKey': {'lowkey1': 10, 'lowkey2': 2}}
>>> patch_dict({"highKey":{"lowkey1":1, "lowkey2":2}}, {"highKey":{"lowkey3":3}})
{'highKey': {'lowkey1': 1, 'lowkey2': 2, 'lowkey3': 3}}
>>> patch_dict({"highKey":{"lowkey1":1, "lowkey2":2}}, {"highKey2":4})
{'highKey': {'lowkey1': 1, 'lowkey2': 2}, 'highKey2': 4}

Returns: A new dictionary which is the result of overlaying patch_dictionary on top of original_dict
Return type: dict