Skip to content

Releases: cognitedata/toolkit

v0.2.0

10 Jun 12:21
@doctrino doctrino
v0.2.0
641307a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0

cdf-tk

Added

  • Support for resource type TransformationNotification in the transformations folder.
  • Support for resource type Label in the labels folder.
  • Support for loading nodes with APICall arguments. The typical use case is when node types are part of a
    data model, and the default APICall arguments works well.
  • Support for securityCategories in the auth folder.
  • Added support for resource type DatapointSubscription in the timeseries folder.
  • Support for the Workflow and WorkflowVersion resource type
  • Support for interactive login. The user can now set LOGIN_FLOW=interactive in the .env file
    to use interactive login.
  • Support for specifying selected as paths and parent paths. For example, you can
    now write cognite_modules/core/cdf_apm_base instead of just cdf_apm_base. This is to support
    modules that have the same name but are in different parent directories. In addition, this also better reflects
    the structure of the cognite_modules and custom_modules folder better.
  • If a resource is referring to another resource, the cdf-tk build will now give a warning if the referred resource
    is not found in the same build. For example, if you have a data model and is missing the space, the build command
    will give a warning that the space required by the data model is missing.
  • The cdf-tk build command will now give warnings on duplicated resource. For example, if you have two files with
    the same externalId in the same module, the build command will give a warning that the externalId is duplicated,
    and that only the first file is used.
  • Variables can now have extra spaces between curly braces and the variable name. For example, {{ my_variable }} is now
    a valid variable. Before this change, you would have to write {{my_variable}}.
  • If an environment variable is not found in a resource file, for example, ${CDF_CLUSTER}, when
    running cdf-tk deploy the user will now get a warning message that the variable is missing. Before this change,
    this would pass silently and potentially cause an error when trying to deploy to CDF that was hard to debug.

Changed

  • [BREAKING] cognite-tookit no longer supports common_function_code. The code used by functions must be in each
    function directory. The reason for this is that cognite-toolkit is a tool for governance and deployment of
    modules, it is not for development of functions. The common_function_code was a feature to support easier
    development of functions. It is expected that functions are developed in a separate environment and then
    moved to the cognite_modules folder for deployment and promotion between environments.
  • [BREAKING] In function YAML config externalDataSetId is renamed to dataSetExternalId to be consistent with
    the naming convention used in the rest of the toolkit.
  • [BREAKING] The cdf-tk build will now clean the build directory by default before building the modules to avoid
    unwanted side effects from previous builds. To stop this behavior, use the --no-clean flag.
  • [BREAKING] The _system.yaml is now required to be on the root level when running any cdf-tk command. This
    means that the _system.yaml file must be in the same directory as the config.[env].yaml files. This is to
    support running cdf-tk without the cognite_modules folder.
  • [BREAKING] In the config.[env].yaml files, the modules section is now renamed to variables. This is to
    better reflect the content of this section, which is variables that can be used in the resource files.
  • In addition to cognite_modules and custom_modules, the cognite-toolkit now also support modules as the
    top-level folder for modules. This together with the two changes above, is to have a better support for running
    the cognite-toolkit as a standalone CLI without the cognite_modules.
  • The .gitignore file you get by running cdf-tk init now ignores the /build by default.
  • The dependency cognite-sdk must now be >=7.37.0 to use the cdf-tk.
  • [BREAKING] In functions, the function config file must be in the root function directory. This means
    that, for example, my_module/function/some_folder/function.yaml will no longer be included by
    the cdf-tk build command. Instead, it must be in my_module/function/function.yaml. The motivation
    is to allow arbitrary YAML files as part of the function code.
  • In config.[env].yaml, in the environment section, selected_modules_and_packages is renamed to selected.
    The old names will still work, but will trigger a deprecation warning.
  • The toolkit now only gives a TemplateVariableWarning (Variable my_variable has value <change_me> ...) if
    the variable is used by selected in the config.[env].yaml. This is to avoid unnecessary warnings.
  • The FeaturePrevieWarnings are no longer printed when running cdf-tk deploy or cdf-tk clean. These warnings
    are from the cognite-sdk and can be confusing to the user.
  • In config.[env].yaml, in the environment section, name is no longer required. Instead, the [env] part
    of the config.[env].yaml file is used as the name of the environment. This is to avoid redundancy.
  • If two modules have the same name, the cdf-tk build command will now stop and raise an error. Before this change,
    the cdf-tk build command would continue and overwrite the first module with the second module.
  • The verification of access by the tool is now scoped to the resources that are being deployed instead of
    the entire project. This means that if the user only has access to a subset of the resources in the project,
    the tool will still be able to deploy those resources.

Improved

  • Gives a more informative error message when the authentication segment of a transformation resource file is
    missing a required field.
  • Transformation queries can be inline, i.e. set in either the Transformation query property in the yaml or
    as a separate file. If set in both, an error is raised because it is ambiguous which query to use.
  • In the cdf-tk pull command, if an error occurs, the temporary directory was not removed. This is now fixed.
  • Improved error message when running cdf-tk deploy/clean before running cdf-tk build.
  • When running cdf-tk clean --dry-run the output would show local resources regardless of whether they existed
    in CDF or not. This is now fixed and only resources that exist in CDF are shown in the output.
  • Better error message (no exception raised) if the config file has selected_modules_and_packages, but with no list items.
  • If yaml files are invalid, a link to the API docs for the resource is shown in the error message.

Fixed

  • When running cdf-tk clean or cdf-tk deploy --drop --drop-data there was an edge case that triggered the bug
    ValueError: No capabilities given. This is now fixed.
  • When deploying containers resources with an index, the cdf-tk deploy would consider the resource as changed
    even though it was not. This is now fixed.
  • When running cdf-tk init --upgrade from version 0.1.4 the user would get a ToolkitMigrationError.
    This is now fixed.
  • The toolkit now ensures Transformations and Functions are deployed before Workflows
  • The toolkit now ensures TimeSeries and Groups are deployed before DatapointSubscriptions.
  • Running the build command, cdf-tk build, with Group resources scoped will read to incorrect
    warning such as WARNING [HIGH]: Space 'spaceIds' is missing and is required by: and
    WARNING [HIGH]: DataSet 'ids' is missing and is required by:. This is now fixed.
  • Running the build command, cdf-tk build, with a View resource with a hasData filter would print a
    UnusedParameterWarning: Parameter 'externalId' is not used in section ('filter', 'hasData', 0, 'externalId')..
    This is incorrect and is now fixed to not print this warning.
  • If you had a container with a direct relation property with a required constraint, the cdf-tk build command
    would incorrectly yield a warning that the Parameter 'type' is not used .... This is now fixed.
  • Error message displayed to console on failed cdf-tk deploy command could be modified. This is now fixed.
  • Using display name instead of folder name on a failed cdf-tk deploy or cdf-tk clean command. For example,
    if datapoints subscription was failing the error message would be Failure to load/deploy timeseries as expected,
    now it is Failure to load/deploy timeseries.subscription as expected.
  • Unique display names for all resource types.
  • Fixed bug when deploying extraction pipeline config, when none existed from before:
    There is no config stored for the extraction pipeline.
  • In a function config, if you did not set fileId you would get an error when running cdf-tk deploy,
    Missing required field: 'fileId'.. The fileId is generated automatically when the function is created,
    so it is not necessary to set it in the config file. This is now fixed.
  • If you do cdf-tk init --upgrade, on a pre 0.2.0a3 version, you are met with
    ERROR (ToolkitModuleVersionError): Failed to load previous version, .... This is now fixed.
  • The parameter container.properties.<property>.type.list was required to be set, even thought it is optional
    in the CDF API. This is now fixed.
  • The ExtractionPipelineConfig create, update and delete report numbers were incorrect. This is now fixed.
  • When deploying a FunctionSchedule that requires an update, the cdf-tk would fail with error
    Failed to update functions.schedules. Error 'FunctionSchedulesAPI' object has no attribute 'update'..
    This is now fixed.
  • When calling cdf-tk init --upgrade, the user is met with a Failed to load previous version, ....
    This is now fixed.
  • When running `cdf-tk auth verif...
Read more

Contributors

ronpal, renovate, and 3 other contributors
Loading

v0.2.0b4

06 Jun 12:08
@doctrino doctrino
v0.2.0b4
4f9a404
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0b4 Pre-release
Pre-release

cdf-tk

Added

  • Support for resource type TransformationNotification in the transformations folder.

Changed

  • [BREAKING] In functions, the function config file must be in the root function directory. This means
    that, for example, my_module/functions/some_folder/function.yaml will no longer be included by
    the cdf-tk build command. Instead, it must be in my_module/functions/function.yaml. The motivation
    is to allow arbitrary YAML files as part of the function code.
  • The toolkit now only gives a TemplateVariableWarning (Variable my_variable has value <change_me> ...) if
    the variable is used by selected in the config.[env].yaml. This is to avoid unnecessary warnings.
  • The FeaturePrevieWarnings are no longer printed when running cdf-tk deploy or cdf-tk clean. These warnings
    are from the cognite-sdk and can be confusing to the user.

Fixed

  • When running cdf-tk init --upgrade from version 0.1.4 the user would get a ToolkitMigrationError.
    This is now fixed.

Templates

Added

  • Extended cognite_modules/examples/my_example_module with a TransformationNotification case.

Details

Full Changelog: v0.2.0b3...v0.2.0b4

Contributors

ronpal and doctrino
Loading

v0.2.0b3

04 Jun 10:36
@doctrino doctrino
v0.2.0b3
f09f1fa
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0b3 Pre-release
Pre-release

cdf-tk

Added

  • Support for resource type Label in the labels folder.

Fixed

  • The toolkit now ensures Transformations and Functions are deployed before Workflows
  • The toolkit now ensures TimeSeries and Groups are deployed before DatapointSubscriptions.

Templates

Added

  • Extended cognite_modules/examples/my_example_module with a Label case.

Details

Full Changelog: v0.2.0b2...v0.2.0b3

Contributors

ronpal and doctrino
Loading

v0.2.0b2

03 Jun 09:26
@doctrino doctrino
v0.2.0b2
4947576
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0b2 Pre-release
Pre-release

cdf-tk

Fixed

  • Running the build command, cdf-tk build, with Group resources scoped will read to incorrect
    warning such as WARNING [HIGH]: Space 'spaceIds' is missing and is required by: and
    WARNING [HIGH]: DataSet 'ids' is missing and is required by:. This is now fixed.
  • Running the build command, cdf-tk build, with a View resource with a hasData filter would print a
    UnusedParameterWarning: Parameter 'externalId' is not used in section ('filter', 'hasData', 0, 'externalId')..
    This is incorrect and is now fixed to not print this warning.
  • If you had a container with a direct relation property with a required constraint, the cdf-tk build command
    would incorrectly yield a warning that the Parameter 'type' is not used .... This is now fixed.

templates

No changes to templates.

Detailes

  • 🌞 Rename subpackage load to loaders by @doctrino in #586
  • [CDF-21640]🐛 Bug in checking dependent Groups by @doctrino in #587
  • [CDF-21639, CDF-21644]🧑‍🚒Warning on valid HasData filter by @doctrino in #590
  • [CDF-21645] 👮Warning on a valid container with direct relation constraint. by @doctrino in #592
  • Prepare 0.2.0b2 by @doctrino in #593

Full Changelog: v0.2.0b1...v0.2.0b2

Contributors

doctrino
Loading

v0.2.0b1

30 May 09:53
@doctrino doctrino
v0.2.0b1
a70ae17
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0b1 Pre-release
Pre-release

cdf-tk

Added

  • Support for loading nodes with APICall arguments. The typical use case is when node types are part of a
    data model, and the default APICall arguments works well.

Fixed

  • Error message displayed to console on failed cdf-tk deploy command could be modified. This is now fixed.
  • Using display name instead of folder name on a failed cdf-tk deploy or cdf-tk clean command. For example,
    if datapoints subscription was failing the error message would be Failure to load/deploy timeseries as expected,
    now it is Failure to load/deploy timeseries.subscription as expected.
  • Unique display names for all resource types.
  • Fixed bug when deploying extraction pipeline config, when none existed from before:
    There is no config stored for the extraction pipeline.

Changed

  • In config.[env].yaml, in the environment section, selected_modules_and_packages is renamed to selected.
    The old names will still work, but will trigger a deprecation warning.

Templates

Fixed

  • Removed illegal characters from DatapointSubscriptoin description in
    cognite_modules/examples/my_example_module.

Detailes

Full Changelog: v0.2.0a5...v0.2.0b1

Contributors

ronpal and doctrino
Loading

v0.2.0a5

28 May 09:37
@doctrino doctrino
v0.2.0a5
eacdffd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0a5 Pre-release
Pre-release

cdf-tk

Added

  • If a resource is referring to another resource, the cdf-tk build will now give a warning if the referred resource
    is not found in the same build. For example, if you have a data model and is missing the space, the build command
    will give a warning that the space required by the data model is missing.
  • The cdf-tk build command will now give warnings on duplicated resource. For example, if you have two files with
    the same externalId in the same module, the build command will give a warning that the externalId is duplicated,
    and that only the first file is used.
  • Support for securityCategories in the auth folder.
  • Added support for resource type DatapointSubscription in the timeseries folder.

Fixed

  • In a function config, if you did not set fileId you would get an error when running cdf-tk deploy,
    Missing required field: 'fileId'.. The fileId is generated automatically when the function is created,
    so it is not necessary to set it in the config file. This is now fixed.
  • If you do cdf-tk init --upgrade, on a pre 0.2.0a3 version, you are met with
    ERROR (ToolkitModuleVersionError): Failed to load previous version, .... This is now fixed.
  • The parameter container.properties.<property>.type.list was required to be set, even thought it is optional
    in the CDF API. This is now fixed.
  • The ExtractionPipelineConfig create, update and delete report numbers were incorrect. This is now fixed.

Improved

  • Gives a more informative error message when the authentication segment of a transformation resource file is
    missing a required field.
  • Transformation queries can be inline, i.e. set in either the Transformation query property in the yaml or
    as a separate file. If set in both, an error is raised because it is ambiguous which query to use.
  • In the cdf-tk pull command, if an error occurs, the temporary directory was not removed. This is now fixed.
  • Improved error message when running cdf-tk deploy/clean before running cdf-tk build.

Changed

  • [BREAKING] In function YAML config externalDataSetId is renamed to dataSetExternalId to be consistent with
    the naming convention used in the rest of the toolkit.

Templates

Added

  • Function used to schedule & trigger workflow
  • Extended cognite_modules/examples/my_example_module with a SecurityCategory case.
  • Extended cognite_modules/examples/my_example_module with a DatapointSubscription case.

Removed

  • The parameter fileId is removed from all function configurations
    (cdf_functions_dummy, cdf_data_pipeline_files_valhall, cdf_data_pipeline_timeseries_valhall,
    and my_example_module) as it is no longer required.
  • In all modules with an extraction_pipelines resource, removed dataSetExternalId and name from all
    ExtractionPipelineConfigs as this is not used and thus only causes confusion.
  • In all modules with a function, renamed externalIdDataSet to dataSetExternalId to be consistent with the
    naming convention used in the Cognite API.
  • In module my_example_module, removed interval and isPaused from the Transformation as these are not used.
    These parameters should only be present in a TransformationSchedule.

Changed

  • Removed schedule from annotation function context:files:oid:fileshare:annotation
  • Add use of map_concatin transformation files_oid_fileshare_file_metadata to keep existing files metadata.
  • Switched to using file.uploaded_time instead of file.last_updated_time since update time
    potentially is updated every time the transformation runs, and don't require a reannotation.

What's Changed

Full Changelog: v0.2.0a4...v0.2.0a5

Contributors

ronpal, haakonvt, and 2 other contributors
Loading

v0.1.4

30 Apr 14:52
@doctrino doctrino
v0.1.4
63fcd4a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.4

cdf-tk

Fixed

  • When running cdf-tk deploy with container YAML config from before cognite-sdk<7.37, the user would get an error
    Missing required field: 'list'. for the direct container properties. This is now fixed.
  • When running cdf-tk init --upgrade, the user would get an error IndexError: list index out of range.
    This is now fixed.

Template

No changes to templates.

What's Changed

Full Changelog: v0.1.3...v0.1.4

Contributors

doctrino
Loading

# v0.2.0a4

29 Apr 12:23
@doctrino doctrino
v0.2.0a4
f7b2dfe
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
# v0.2.0a4 Pre-release
Pre-release

cdf-tk

Removed

  • [BREAKING] cognite-tookit no longer supports common_function_code. The code used by functions must be in each
    function directory. The reason for this is that cognite-toolkit is a tool for governance and deployment of
    modules, it is not for development of functions. The common_function_code was a feature to support easier
    development of functions. It is expected that functions are developed in a separate environment and then
    moved to the cognite_modules folder for deployment and promotion between environments.

Changed

  • In config.[env].yaml, in the environment section, name is no longer required. Instead, the [env] part
    of the config.[env].yaml file is used as the name of the environment. This is to avoid redundancy.

Improved

  • When running cdf-tk clean --dry-run the output would show local resources regardless of whether they existed
    in CDF or not. This is now fixed and only resources that exist in CDF are shown in the output.
  • Better error message (no exception raised) if the config file has selected_modules_and_packages, but with no list items.
  • If yaml files are invalid, a link to the API docs for the resource is shown in the error message.

Fixed

  • When deploying a FunctionSchedule that requires an update, the cdf-tk would fail with error
    Failed to update functions.schedules. Error 'FunctionSchedulesAPI' object has no attribute 'update'..
    This is now fixed.
  • When calling cdf-tk init --upgrade, the user is met with a Failed to load previous version, ....
    This is now fixed.
  • When running cdf-tk auth verify --interactive and the user want to create a new group with the necessary
    capabilities, the cdf-tk would successfully create a group, but then raise
    an Error: cognite.client.exceptions.CogniteAPIError: Insufficient access rights. when trying to validate.
    This is now fixed.

Templates

Added

  • Added Workflow with a Function and a Transformation to the cdf_data_pipeline_files_valhall example

Changed

  • The cdf_functions_dummy module now includes codes from the former common_function_code directory.

Fixed

  • In example_pump_data_model, in the Pump.view.yaml the property source used external_id instead of
    externalId.

Details

Full Changelog: v0.1.3...v0.2.0a4

Contributors

ronpal, haakonvt, and 2 other contributors
Loading

v0.1.3

29 Apr 07:14
@doctrino doctrino
v0.1.3
1da5f87
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.3

cdf-tk

Fixed

  • Functions that are deployed with schedules no longer uses a short-lived session (before: failed after ~an hour).
  • When deploying a FunctionSchedule that requires an update, the cdf-tk would fail with error
    Failed to update functions.schedules. Error 'FunctionSchedulesAPI' object has no attribute 'update'..
    This is now fixed.
  • The dependency cognite-sdk must now be >=7.37.0 to use the cdf-tk.

Templates

Fixed

  • Updated module apm_simple_data_model for cognite-sdk>=7.37, i.e., container properties
    of type direct have now list: false explicitly set.
  • In example_pump_data_model, in the Pump.view.yaml the property source used external_id instead of externalId.

Full Changelog: v0.1.2...v0.1.3

Loading

v0.2.0a3

23 Apr 14:36
@doctrino doctrino
v0.2.0a3
b3d5af8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0a3 Pre-release
Pre-release

cdf-tk

Added

  • Support for the Workflow and WorkflowVersion resource type
  • Support for specifying selected_modules_and_packages as paths and parent paths. For example, you can
    now write cognite_modules/core/cdf_apm_base instead of just cdf_apm_base. This is to support
    modules that have the same name but are in different parent directories. In addition, this also better reflects
    the structure of the cognite_modules and custom_modules folder better.

Fixed

  • Functions that are deployed with schedules no longer uses a short-lived session (before: failed after ~an hour).

Changed

  • [BREAKING] The cdf-tk build will now clean the build directory by default before building the modules to avoid
    unwanted side effects from previous builds. To stop this behavior, use the --no-clean flag.
  • [BREAKING] The _system.yaml is now required to be on the root level when running any cdf-tk command. This
    means that the _system.yaml file must be in the same directory as the config.[env].yaml files. This is to
    support running cdf-tk without the cognite_modules folder.
  • [BREAKING] In the config.[env].yaml files, the modules section is now renamed to variables. This is to
    better reflect the content of this section, which is variables that can be used in the resource files.
  • In addition to cognite_modules and custom_modules, the cognite-toolkit now also support modules as the
    top-level folder for modules. This together with the two changes above, is to have a better support for running
    the cognite-toolkit as a standalone CLI without the cognite_modules.
  • The .gitignore file you get by running cdf-tk init now ignores the /build by default.
  • The dependency cognite-sdk must now be >=7.37.0 to use the cdf-tk.

Templates

Fixed

  • Align tag name in asset hierarchy between the 2 example transformations
  • Added default root asset ID to documents for initial annotation
  • Aligned use of asset external ID across contextualization functions
  • Annotation logic with local time stamp for when to reprocess P&ID files
  • Input to P&ID annotation based on list of synonyms for tag
  • Updated module apm_simple_data_model for cognite-sdk>=7.37, i.e., container properties
    of type direct have now list: false explicitly set.

Added

  • Added Transformation for WorkOrder and WorkItems to OID testdata template
  • Added Workflow with a Function and a Transformation to the custom module example

Details

Full Changelog: v0.1.2...v0.2.0a3

Contributors

ronpal, haakonvt, and 2 other contributors
Loading