Release 3.7.0

Hopsworks

Bug

HWORKS-224 hopsworks python SDK opensearch link broken

HWORKS-853 Model export fails in integration tests

HWORKS-862 Remote clients are not available on non das nodes

HWORKS-864 Istio doesn't use the docker images from the registry

HWORKS-865 NullPointerException when monitoring execution of a deleted job

HWORKS-869 Email validation regex doesn't support capital letters

HWORKS-872 Cannot disable oauth group mapping from cluster definition

HWORKS-878 Grafana uses wrong prometheus name

HWORKS-879 Upload will fail if the same file is uploaded again while the first upload is ongoing.

HWORKS-881 Validate SparkJob and Python job application file exists before launching job

HWORKS-882 Disallow exporting a model in the root path of a dataset

HWORKS-890 Restoring RonDB backup breaks because restore-data does not allow restoring in a table with unique indexes

HWORKS-898 PyTorch installation does not have GPU support

HWORKS-899 Broken image links in gh-pages documentation

HWORKS-901 Git installation support should not prepend git+

HWORKS-907 CloudManager should blacklist all instances from automatic removal

HWORKS-908 NPE in ProjectQouataController for a project that failed to get created

HWORKS-910 Endpoint identification alg missing from kafka get_default_config()

HWORKS-950 git pull fails if the user has multiple surnames

HWORKS-954 api_key for cert-operator is created empty

HWORKS-961 Do not turn on MySQL binlog if global replication is not enabled

HWORKS-974 Resource guard in ndb-chef for extracting RonDB does not work

HWORKS-975 OpenSearchApi in hopsworks-api should not rely on ELASTIC_ENDPOINT

HWORKS-976 Replacement of CREATE USER with CREATE USER IF NOT EXISTS is wrong in RonDB backup restore script

HWORKS-996 Removal of rotated one time JWT signing key should happen only on the Primary node

HWORKS-1001 Feature store sharing should also share data validation dataset

HWORKS-1029 Pin nbconvert in docker images

Task

HWORKS-135 Models backend should store metadata in tables instead of opensearch

HWORKS-302 Enhancement Request: Ability to disable access to Anaconda package repository and hide the functionality at a system defined level

HWORKS-707 Store descriptive statistics in NDB instead of HopsFS

HWORKS-802 Python 3.12 support for clients

HWORKS-808 add database/url to parameter in kafka cookbook

HWORKS-854 hops-system should use the minimal env definition without pydoop

HWORKS-855 Remove Kagent ability to do service key rotation

HWORKS-861 Disable Kafka alerting rules when bring your own kafka is enabled

HWORKS-868 Increase yarn.resourcemanager.rmappsecurity.jwt.validity to 1 hour

HWORKS-871 Hopsworks should be able to handle oauth claims in array format

HWORKS-873 Add flag to delete RonDB backup with the same backup-id

HWORKS-877 Helm chart for certs-operator

HWORKS-880 Model registry tests should account for if model is stored on hdfs and also run in PySpark

HWORKS-883 Fix Sonarqube issues in CommandsController

HWORKS-886 OnlineFS should be able to read Kafka configuration from environment variables

HWORKS-887 certs-operator deploy appropriate certificate for Strimzi operator

HWORKS-889 Chef flag to disable unattended upgrades

HWORKS-891 Add K8s GPUs monitoring to Grafana/Jupyter/Job UI

HWORKS-892 Use a single YarnClientWrapper to monitor all jobs

HWORKS-893 Nightly tests should run without installing the requirements.txt

HWORKS-894 Add pytest to python base environment

HWORKS-895 Upgrade xgboost to 2.0.3

HWORKS-897 Upgrade project python environment libraries

HWORKS-900 Upgrade to PyTorch 2.1.2

HWORKS-909 Add hopsworks-api opensearch and kafka workflow tests

HWORKS-912 Upload staging dir does not need to be configurable.

HWORKS-919 Create database on the online feature store on-demand

HWORKS-934 Use more appropriate data types for statistics

HWORKS-944 Pin upper pandas to 2.1.4

HWORKS-946 Global Chef attribute for arbitrary Systemd unit dependencies

HWORKS-949 Add git repo pull to workflow tests

HWORKS-962 Purge binlog files when Global replication is enabled

HWORKS-965 In some OSes DNS resolution does not work from within a Kubernetes Pod

HWORKS-973 Share feature store with project on creation

HWORKS-977 Mount hopsfs by default in jupyter notebooks and python jobs containers

HWORKS-980 Don't set DB storagePolicy for Hive warehouse location

HWORKS-992 Remove code for generating service Renewal JWTs

HWORKS-1050 Run expat as glassfish hopsfs user

Feature Store

Epic

FSTORE-612 Feature Monitoring

Bug

FSTORE-830 Error fetching feature statistics from feature view UI - but statistics exists with a different timestamp

FSTORE-856 Only one OnlineFS instance running

FSTORE-987 Failed to read data when there is a self-join

FSTORE-989 GCS connector-Encryption fields and secrets update issues

FSTORE-992 create_train_validation_test_split fails with unexpected keyword argument 'pit_query_asof'

FSTORE-998 Can't read from a shared feature store

FSTORE-1000 Tags parameter is missing in TrainingDataset class

FSTORE-1034 .select() method should not default to empty list

FSTORE-1035 No error if a user tries to create a feature view without features

FSTORE-1084 Cannot run multiple insert_stream query on the same project by default

FSTORE-1089 Remove copying of application code from databricks integration

FSTORE-1095 JDBC storage connector missing driver option in the documentation

FSTORE-1099 Nightly test test_append_feature failing on float type mistmatch

[FSTORE-1100](https://hopsworks.atlassian.net/br...

Release 3.5.0

Bug

HWORKS-224 hopsworks python SDK opensearch link broken

HWORKS-267 After fail checkout, branch is empty on pull

HWORKS-269 Don't show the same in file status and in UI table

HWORKS-309 Bugs in searching

HWORKS-331 Hopsworks metrics not showing after payara restart

HWORKS-374 opensearch_api.get_default_py_config() returns public IP of host, should return consul fqdn

HWORKS-386 Spark job does not accept space in arguments

HWORKS-469 Dependent conda commands not handled correctly in case of failure

HWORKS-475 hopsworks Python library can't connect in sklearn legacy deployments

HWORKS-492 Servings logs are not shown in Kibana

HWORKS-493 Request batching configuration is broken in the deployment creation form

HWORKS-496 Exception not raised if other than duplicated deployment entry

HWORKS-498 Transformer resources in dict format are not properly deserialized

HWORKS-499 Support for jupyter notebooks as predictor scripts

HWORKS-561 Hopsworks Kafka Authorizer does not allow IDEMPOTENT_WRITE operations

HWORKS-569 Filebeat cannot list nodes which is required for scraping container logs

HWORKS-602 HA cluster could not connect to live logs, rm UI, spark UI, monitor, onlineFS

HWORKS-630 Project creation doesn't fail on invalid project name

HWORKS-633 Failing deployments are shown as Starting when Kubernetes stops restarting pods

HWORKS-636 Inference logging default value is overwritten when updating a deployment using .save()

HWORKS-656 NP trying to force delete a partially created project

HWORKS-693 Server logs not collected when the predictor class fails to initialize

HWORKS-786 Storage connector specs failure for flag disabled case

HWORKS-787 Git repository current commit and branch not being updated

HWORKS-792 Hopsworks job schedule update doesn't update cron expression

HWORKS-809 Project creation fails when the serving api key k8s secret takes too long to be available

HWORKS-815 Judge service should be restarted when docker daemon restarts

HWORKS-817 Log level in RonDB replication scripts is wrong

HWORKS-819 The request URI port is absent in MultiRegionFilter

HWORKS-820 In MultiRegionFilter the Primary region is null if invoked too early

HWORKS-821 Kubernetes does not accept serving names starting with a number

HWORKS-824 Only Primary MySQLs should be able to update RonDB replication metadata table

HWORKS-833 Consul datacenter is hardcoded in Prometheus configuration

HWORKS-834 Undeploy hopsworks on upgrade

HWORKS-837 RonDB native backup script fails when there is no database to backup

HWORKS-848 AlertManager EJB does not take Datacenter into consideration in Multiregion clusters

HWORKS-851 Check in hops-hadoop-chef for creating RonDB undo and log files is wrong

HWORKS-865 NullPointerException when monitoring execution of a deleted job

Task

HWORKS-135 Models backend should store metadata in tables instead of opensearch

HWORKS-164 Add Airflow documentation

HWORKS-198 Remove hopsworks::image recipe

HWORKS-226 python api should throw informative error if user tries to download a folder

HWORKS-258 Add hopsworks python sdk dataset upload/download to workflow testing

HWORKS-262 Remove user email FK from hopsworks schema.

HWORKS-266 Checking out a branch that exist on remote repo does not work

HWORKS-284 Documentation to export cluster logs

HWORKS-302 Enhancement Request: Ability to disable access to Anaconda package repository and hide the functionality at a system defined level

HWORKS-314 Add support for git fetch and reset

HWORKS-422 Exclude old versions of the documentation from being indexed by search engines

HWORKS-491 Replace hsml with hopsworks in the code snippets shown in the UI

HWORKS-494 Support local script path in model.deploy()

HWORKS-511 Add number of GPUs to Python resource usage

HWORKS-520 Make oauth claims configurable

HWORKS-536 Move master encryption password to KMS

HWORKS-559 Users should be able to specify a directory when using the Hopsworks-api to download logs

HWORKS-565 Hopsworks api dataset upload should support parallel chunk upload

HWORKS-571 Run more integration tests in parallel

HWORKS-590 Make jenkins test use HA cluster

HWORKS-591 Show a better log for when pods are killed by Kubernetes with OOM

HWORKS-617 Make load balancer use https

HWORKS-631 Support info exchange between preprocess and postprocess methods in transformers

HWORKS-635 Add unit tests and loadtest for hsml deployments

HWORKS-641 Add loadtests for hsml and model registry

HWORKS-662 Fix versioning and build automation for hopsify

HWORKS-676 Supress CVE-2023-33265 in payara-embedded-web-5.2022.5.jar/META-INF/maven/com.hazelcast/hazelcast/pom.xml:

HWORKS-677 Supress multiple severe CVE in hive-storage-api-2.6.1.2.jar

HWORKS-679 Add option to set python environment

HWORKS-690 Airflow project dags sharing

HWORKS-697 Remove airflow JWT

HWORKS-707 Separate statistics from feature monitoring PR

HWORKS-729 Nightly loadtest cluster should have an admin user that is member of all projects

HWORKS-745 Support python 3.11 for the client APIs

HWORKS-746 Documentation for python environment history and custom commands

HWORKS-751 Add webhook support for Hopsworks alerts

HWORKS-756 Remove cmake dependency in hops-hadoop-chef

HWORKS-759 Show custom commands file that was used to build the environment in the environment history

HWORKS-764 Reduce JWT lifetime in nightly tests

HWORKS-774 Set hopsworks_rest_log_level to DEV in the nightly tests

HWORKS-775 Nightly tests should reload random module

HWORKS-777 Fix integration ...

Release 3.4.0

Hopsworks 3.4.0

Bug

HWORKS-372 explicit provenance missing community stereotype

HWORKS-397 Links to Model Serving API reference are broken

HWORKS-426 Latest versions of managed Kubernetes may not support first-party-jwt

HWORKS-538 dataset sharing acl bug

HWORKS-610 HazelcastSerializationException on JupyterJWTManager

HWORKS-632 Command.from_response_json in hopsworks-api should parse based on items field

HWORKS-638 Kafka 3 removed --zookeeper option

HWORKS-639 Template cert store passwords in onlinefs-kafka.properties

HWORKS-640 Nightly workflow and load tests should not run as admin user

HWORKS-654 extra constraints migration does not respect the order of the columns

HWORKS-663 remote_file does not support not_if condition

HWORKS-665 Bump Zookeeper dependency in Hopsworks to 3.8.2 to fix CVE-2023-34462

HWORKS-666 Configuring HA Glassfish fails when importing certificates on worker nodes

HWORKS-670 Set Flink job staging and checkpoint directories per user

HWORKS-675 Exclude logback dependencies from zookeeper library in Hopsworks

HWORKS-680 Karamel with recipe parallelism restrictions schedules tasks although a previous task has failed

HWORKS-687 ZooKeepers fail to start because they cannot resolve other ZK consul domain name

HWORKS-688 Python library installation error message not latin1_general_cs

HWORKS-689 oauth logout uri is set to redirect uri

HWORKS-711 User deletion fails with FK constraint

HWORKS-731 Model deployment with a wrong predictor script path fails with a non-intuitive error

HWORKS-735 Chef attribute to configure ldap_groups_search_filter

HWORKS-736 ndb-chef download backup should skip host key verification

HWORKS-750 Pin traitlets in project python environment

Task

HWORKS-248 Publish all images to the internal docker registry

HWORKS-343 Add support for accessing a shared feature store using project.get_feature_store()

HWORKS-351 Add support for running generic Docker commands

HWORKS-354 Remove python_kernel variable

HWORKS-489 Register test code using register_assert_rewrite

HWORKS-544 Opensearch docs creation moved to Hopsworks

HWORKS-545 Move Tags and Keyword to Hopsworks db

HWORKS-546 Feature components - remove xattr - use Hopsworks db

HWORKS-547 featurestore search - static reindex

HWORKS-572 Add uptime to status-all-local-service.sh

HWORKS-576 Test upgrade from HA cluster to HA

HWORKS-589 Remove metastore foreign key from cached/streaming feature groups

HWORKS-618 Make integration test use HA cluster def

HWORKS-627 Log more information during Project creation

HWORKS-628 Skip project creation failure alert if a project with the same name exists

HWORKS-637 Add Release Candidate version to spark-chef for hsfs

HWORKS-642 Online feature group logs

HWORKS-646 Move airflow dags to hopsfs

HWORKS-647 Replace Chef with Cinc

HWORKS-667 MySQL X plugin socket should not be at /tmp

HWORKS-672 Remove angular file browser in old UI

HWORKS-673 Remove Hopsworks Kagent monitor

HWORKS-678 Allow configuration of RDRS with user supplied certificates

HWORKS-681 Flink apis for hopsworks python library

HWORKS-682 Include return type and description for model.download()

HWORKS-684 Separate job scheduler from feature monitoring PR

HWORKS-685 Add HOPS_SERVICE_USER account type to Python Environment endpoints

HWORKS-686 Zip and push airflow image tar to master in jenkins pipeline

HWORKS-696 Add documentation on project quota management

HWORKS-698 Precreate the project root dir in hopsfs

HWORKS-699 Update docker daemon security configuration

HWORKS-700 Remove kagent_disable check in conda::default and template kagent_utils there

HWORKS-701 Add date filter to python environment history

HWORKS-702 Automate setup of RonDB Global replication

HWORKS-703 Add consul/enabled flag to disable consul operations if consul is not available

HWORKS-704 hopsworks::install should create the hopsmon user if it doesn't exists

HWORKS-705 Chef should not pre-register the hosts

HWORKS-706 Switch opensearch-dashboard to use consul instead of IPs

HWORKS-708 Use git api to delete git repository in the hopsworks python SDK

HWORKS-709 hops-hadoop-chef should not overwrite the tls/enabled parameter if hopsworks::default is missing

HWORKS-710 Hopsworks database should create the ts_1 tablespace for the certificates

HWORKS-712 Add option to tag RonDB management node for onlinefs

HWORKS-713 Create a symlink to hopsfs-mount directory in the default working directory in jupyter terminal

HWORKS-715 Add hops::docker dependency to rdrs and onlinefs recipes

HWORKS-716 Add append2ChefTrustAnchors to elastic::default

HWORKS-717 Remove consul dependency in hops::client

HWORKS-718 Switch elastic clients to use consul fqdn and not ips

HWORKS-719 Add chef code to configure WAN federated consul

HWORKS-720 Remove Kafka waiter in Onlinefs systemd script if external kafka

HWORKS-723 Add rondb_replication to the list of forbidden project names

HWORKS-724 Implement multiregion controller in Hopsworks

HWORKS-726 Delete Chef cached attributes json in beginning of new installation

HWORKS-727 Bump Hops version to 3.2.0.11-RC0

HWORKS-728 Add secondary region attribute in Chef

HWORKS-734 Remove jdbc pool when initializing online feature store database for project

HWORKS-737 Deleted project leads to NPE when processing SearchFSCommand

HWORKS-738 Recipe to setup Judge service

HWORKS-740 Expat - disable provenance on exi...

Release 3.3.0

Hopsworks 3.3.0

Bug

HWORKS-224 hopsworks python SDK opensearch link broken

HWORKS-267 After fail checkout, branch is empty on pull

HWORKS-269 Don't show the same in file status and in UI table

HWORKS-309 Bugs in searching

HWORKS-319 User active projects not updated

HWORKS-331 Hopsworks metrics not showing after payara restart

HWORKS-374 opensearch_api.get_default_py_config() returns public IP of host, should return consul fqdn

HWORKS-379 LDAP users cannot download certificates

HWORKS-386 Spark job does not accept space in arguments

HWORKS-397 Links to Model Serving API reference are broken

HWORKS-459 Audit log caller should be consistent

HWORKS-469 Dependent conda commands not handled correctly in case of failure

HWORKS-475 hopsworks Python library can't connect in sklearn legacy deployments

HWORKS-492 Servings logs are not shown in Kibana

HWORKS-493 Request batching configuration is broken in the deployment creation form

HWORKS-496 Exception not raised if other than duplicated deployment entry

HWORKS-498 Transformer resources in dict format are not properly deserialized

HWORKS-499 Support for jupyter notebooks as predictor scripts

HWORKS-501 Collect logs in Flink UI and distribute custom libraries

HWORKS-521 Resolve ruby test failures

HWORKS-524 HSML ignores knative_domain parameter

HWORKS-525 Python resource monitor for K8s counts all nodes

HWORKS-528 Agent users should never be blocked

HWORKS-533 Nightly jenkins pipeline reporting 500 failures even though we have only 40 tests

HWORKS-539 Storage initializer broken after release of requests 2.29.0

HWORKS-540 Bump guava version to 27.0-jre

HWORKS-541 update CVE suppression - hops updated dependencies

HWORKS-558 Old UI getting 404 trying to get dela client

HWORKS-561 Hopsworks Kafka Authorizer does not allow IDEMPOTENT_WRITE operations

HWORKS-562 Serving hashcode method should handle null cases

HWORKS-563 Bump commons-configuration2 in expat

HWORKS-566 User account creation fails if the serving api key is not available within 3 seconds

HWORKS-568 Expat fails to start with logging class not found exception

HWORKS-569 Filebeat cannot list nodes which is required for scraping container logs

HWORKS-570 Flatbuffer class not found exception when running arrow internal spark

HWORKS-577 Upgrade 3.2 to 3.3 failing on kzookeeper::default

HWORKS-599 Execution status is not being updated

HWORKS-600 Python library installation failure on non DAS nodes

HWORKS-602 HA cluster could not connect to live logs, rm UI, spark UI, monitor, onlineFS

HWORKS-604 Collect glassfish node logs in logstash

HWORKS-605 GitTimeoutCommandsMonitor not working in HA

HWORKS-606 Spark and Flink configuration files are not present in Glassfish config nodes

HWORKS-608 Git commit message might be bigger than VARCHAR(1000)

HWORKS-609 Alertmanager config missing route.repeat_interval

HWORKS-610 HazelcastSerializationException on JupyterJWTManager

HWORKS-611 ApiKey filter should check if user is blocked

HWORKS-612 hops-hadoop-chef should automatically set HopsYARN HA attributes

HWORKS-614 hopsworks.login on a project with shared feature store

HWORKS-616 Upgrade fails on fuse_mnt

HWORKS-619 Python jobs are always set to FAILED even after completion

HWORKS-622 hsml connection fails to close

HWORKS-624 Race condition in LibraryInstaller CondaCommand execution

HWORKS-625 Kafka storage connector fails migration if inode is null

HWORKS-634 Revisit RonDB restore process in Chef recipes

HWORKS-638 Kafka 3 removed --zookeeper option

HWORKS-639 Template cert store passwords in onlinefs-kafka.properties

HWORKS-643 Revoke Glassfish internal certificate when restoring a backup

HWORKS-653 Publish hopsworks-service-discovery module on archiva

HWORKS-654 extra constraints migration does not respect the order of the columns

HWORKS-655 elkadmin certificate is not generated when restoring a backup

HWORKS-657 Setting current_version attribute with patch version will template all the DDL SQL files

HWORKS-660 Consul recipe should install bc package

HWORKS-661 Incorrect glassfish certificates after upgrade to HA environment

HWORKS-663 remote_file does not support not_if condition

HWORKS-666 Configuring HA Glassfish fails when importing certificates on worker nodes

HWORKS-670 Set Flink job staging and checkpoint directories per user

HWORKS-674 Feature store job migration doesn't work if the spark user is not spark

Task

HWORKS-217 Alerting needs to update alertmanager config in local file system

HWORKS-218 Fix upload for clustered payara

HWORKS-225 Create Deployment Groups in payara5 (clustering)

HWORKS-226 python api should throw informative error if user tries to download a folder

HWORKS-258 Add hopsworks python sdk dataset upload/download to workflow testing

HWORKS-266 Checking out a branch that exist on remote repo does not work

HWORKS-284 Documentation to export cluster logs

HWORKS-314 Add support for git fetch and reset

HWORKS-409 cleaner timer bean are too verbose and pollute logs

HWORKS-415 Add project name to the audit logs

HWORKS-422 Exclude old versions of the documentation from being indexed by search engines

HWORKS-428 Add sleep to loadtest to avoid running into issue due to parallelizing test

HWORKS-437 Upgrade Kafka to version 3.4.0

HWORKS-446 Fix programmatic timers for HA

HWORKS-447 Merge ObjectMapper singletons

[HWORKS-448](https://hopsworks.atlassian.net/browse...

Release 3.2.0

Release notes - Feature Store - 3.2.0

Epic

FSTORE-615 HSFS JavaDoc

Bug

FSTORE-22 Failed to insert data to streaming FG

FSTORE-313 Throw intelligible error message when calling save on unsupported external feature groups in python engine

FSTORE-456 Validation time in the future

FSTORE-516 the 'transformationFunction" fields empty in feature_view.json()

FSTORE-540 Fix shared fs integration tests

FSTORE-549 Uploading a large number of rows to onlinefs fails with TOPIC_AUTHORIZATION_FAILED

FSTORE-566 Feature store online metrics page returns dashboard not found

FSTORE-602 get_or_create_feature_group does not validate feature group name

FSTORE-607 Investigate hsfs java resource leaks

FSTORE-636 ExpectationSuite sonarkube smartUpdate transaction

FSTORE-639 Pin SQLalchemy version or upgrade to new 2.0.0 version

FSTORE-653 get_feature_vector is broken

FSTORE-660 Docs: Syntax issue for creating external feature group

FSTORE-661 Creating external feature group in Jupyter Notebook didn't work

FSTORE-682 Fix fg save_expectation_suite

FSTORE-689 Shallow dataframe copy hides TZ bug

FSTORE-693 Insert with validation suite throws date format exception

FSTORE-701 Fix loadtest datavalidation

FSTORE-730 Fix Multiple backfill job started in dataval loadtest

FSTORE-760 Fix the printing of log in the stack trace for loadtest jobs

FSTORE-763 Onlinefs handling avro exception after append

FSTORE-776 Floating Point Issue with Train Test Split Ratios

FSTORE-780 Documentation doens't mention type limitation for event_time

FSTORE-781 Documentation for extenral feature groups still refers to them as on-demand feature groups

FSTORE-794 Offline fg backfill fails due to the DelataStreamer config Class path(s) in HudiEngine.java are not set correctly

FSTORE-804 Resolve OnlineFS getAndCastValue NullPointerException

FSTORE-812 Enable Kafka topic auto creation

FSTORE-814 StackOverflowException if you provide statistics columns in the statistics config

Design Log

FSTORE-559 Test connection feature for storage connectors

FSTORE-572 Design Log: HSFS Javadocs API

FSTORE-609 Design Log for FM Extend Alerts API

FSTORE-610 Design Log for FM UI / UX

Task

FSTORE-41 From the storage connector; there is no way to know if you are actually connected

FSTORE-44 From a notebook the connector does not say either if it worked or failed

FSTORE-343 Separate HSFS java client build for different Frameworks

FSTORE-349 Add advanced tutorials to documentation

FSTORE-353 Extend the README.md files for the electricity prices and NYC Taxi tutorials

FSTORE-373 Remove job when a fg or fv is deleted.

FSTORE-405 Add entity type to fs job name

FSTORE-434 Improve Kafka message delivery callback error handling

FSTORE-483 Hiveserver2 query optimization config

FSTORE-487 Handle each feature group in its own retry loop

FSTORE-490 Add more assert to Feature Store load test

FSTORE-491 Backfilling user story Design Document

FSTORE-492 Change notebooks that use sklearn models to XGBoost and save them as JSON files, instead of as pickled python files

FSTORE-493 Add load test for online feature store to test data integrity and volume

FSTORE-495 CItibike Tutorial

FSTORE-506 update hopsworks tutorials for 3.1 APIs

FSTORE-528 Add single expectation API and 3.1 changes to data validation tutorials

FSTORE-546 Update several Tutorials to 3.1 hopsworks API

FSTORE-547 Update Tutorials to 3.1

FSTORE-554 cleanup training dataset statistics

FSTORE-557 Simplify getRowClass method

FSTORE-558 Attach transformation function to self join query

FSTORE-561 Add load test for concurrent creation and insertion of online feature groups

FSTORE-562 Add load test to integrate benchmark for vector lookup

FSTORE-568 Add meaningful exceptions to the tutorials, that use API keys

FSTORE-569 New version of Bitcoin Tutorial

FSTORE-570 Design log: Kafka Authoriser Update

FSTORE-571 3.0 Some tutorials Fix

FSTORE-573 documentation/guide about on-demand feature computation pipeline

FSTORE-575 write streaming feature computation pipeline example

FSTORE-581 Review feature view documentation

FSTORE-588 Kafka Authoriser Update

FSTORE-589 Document behaviour of get_feature_vector(s) for missing PK values

FSTORE-591 Fix for hops-kafka-authorizer Sonarqube issues

FSTORE-597 Fix for online-fs Sonarqube issues

FSTORE-611 Investigate how to visualise Feature Monitoring

FSTORE-613 Make Kafka Topic partition default configurable for APP upgrade

FSTORE-614 Auto Kafka Topic recreation needs to provide better error message in Hopsworks for APP upgrade

FSTORE-617 Fix java docstrings to build javadocs

FSTORE-618 Expand current java docstrings to cover new hsfs functionalities

FSTORE-619 make the changes to the github action, pom.xml and mkdocs.yml to publish javadocs

FSTORE-621 Make unit tests run on hsfs branch-3.1

FSTORE-623 Flying Duck: Fix integration into deployment process

FSTORE-624 Add examples from hops-examples repo to hopsworks-tutorials repo

FSTORE-627 Create dockerfile and python entrypoint script-snowflake connector

FSTORE-628 Hopsworks backend logic, bash script to start the container and REST API-snowflake

FSTORE-629 Add CI/CD, set up Jenkins to deploy docker image to docker registry and pull on new clusters-snowflake

FSTORE-658 Incomplete doc string of create_external_feature_group

FSTORE-666 Fix modeling notebook in Bitcoin Price Prediction tutorial

[FSTOR...

Release 3.1.0

Release notes - Feature Store - 3.1.0

Epic

FSTORE-472 Metadata Management

FSTORE-474 Feature Flags

FSTORE-475 Storage Connectors

Task

FSTORE-33 Improve HSFS parameter description

FSTORE-35 Permission matrix

FSTORE-42 Not very helpful doc "you can do it via the UI" (for external FG)

FSTORE-43 Documentation sample data has syntax issue (missing comma)

FSTORE-52 Fix typo in GE tutorial

FSTORE-309 Add support for like operator in query filter

FSTORE-311 Support datetime64[ns, TZ] typed columns

FSTORE-315 Upgrading GE version to 0.14.12 to support Jinja2 version 3.0.0

FSTORE-316 Add Timestamp/Timezone Guide

FSTORE-322 Enable update of feature view metadata in python client

FSTORE-325 Make driver field optional in Redshift connector configuration

FSTORE-328 throw intelligible error message if user set primary and partition keys doesn't exist

FSTORE-330 Adapt python client to the change introduced for the single expectation api

FSTORE-332 implement method for get_or_create_feature_view

FSTORE-334 explicit provenance - feature groups, feature views, training datasets

FSTORE-336 Improve error message in get_batch_data

FSTORE-337 Single expectation API & Expectation Suite Smart Update

FSTORE-339 Add Great Expectations Validation Activity

FSTORE-340 Add validation for great expectations kwars for feature names

FSTORE-341 Validation History for an expectation

FSTORE-342 Add java tests to github pipeline

FSTORE-345 Update documentation to reflect supported methods in hsfs engines

FSTORE-346 Fail to create feature view with query having `as_of`

FSTORE-347 Make documentation build strict

FSTORE-348 Refactor Tutorials documentation page

FSTORE-350 Clean up Key Files for Storage Connectors on Delete

FSTORE-352 Clean up hudi delta streamer job config files in resources

FSTORE-354 Write Home Credit Default Risk hopsworks tutorial

FSTORE-358 Add icon to external links in documentation navigation

FSTORE-359 Add return types to transformation functions in 3.1

FSTORE-360 Add "Client install guide" to docs

FSTORE-364 Fail to run DeltaStreamer job

FSTORE-369 Remove feature group details endpoint

FSTORE-371 Cannot modify a FG on the UI

FSTORE-379 Add documentation on how to share a feature store store and how to access it.

FSTORE-380 Expand query documentation

FSTORE-381 Label does not consider prefixed feature name

FSTORE-385 Handle same instance of transformation function for multiple features properly

FSTORE-387 Add feature_store_id and feature_group_id in expectation suite DTO

FSTORE-397 Fail to get batch query in java client

FSTORE-398 Add validation history method to python client

FSTORE-406 Document time related parameter in feature view

FSTORE-407 API Doc should include full package name for the RestAPIError

FSTORE-408 Fix feature group integration tests

FSTORE-411 Sharing featurestore fails

FSTORE-416 Problem with get_training_data

FSTORE-422 PIT join does not return event time column of right feature group

FSTORE-425 throw meaningful error if user tries to set time_travel_format=None in a Python kernel

FSTORE-429 Add data validation tutorial to loadtest

FSTORE-430 Add return type and code example in feature view python client

FSTORE-431 Fix typo in user guide for data validation

FSTORE-433 Accept list in event_time

FSTORE-441 Enable shortening of expectation_config field in ValidationResult

FSTORE-442 Increase kwargs varchar field in expectation table

FSTORE-446 user guide update for 3.1 data validation

FSTORE-447 Improve documentation for Tags

FSTORE-448 Feature view documentation problems

FSTORE-451 Bump onlinefs version to 1.4-SNAPSHOT on master

FSTORE-452 Show entire stack trace when exception in engine occurs

FSTORE-460 The code part of the Data Validation is confusing

FSTORE-467 Make it easier to start the backfill job from python

FSTORE-470 Add tests for running workflows with hudi commits

FSTORE-477 Change Online FS ALTER table algorithm

FSTORE-478 Add workflow tests for pyspark fraud batch and fraud online

FSTORE-481 Unify python and pyspark test code in load tests

FSTORE-482 Workflow test for onlinefs append feature

FSTORE-484 Storage connector feature flag

FSTORE-485 Update Hive Grafana dashbaord to reflect new GC configuration

FSTORE-488 Ensure type consistency between write/read/online/offline python/spark

FSTORE-489 Bump spark driver memory to 2GB and executor memory to 4GB

FSTORE-494 add documentation how to add new filters and/or join new FGs to feature view query

FSTORE-496 Test improvements

FSTORE-497 Change integration test after validation time in future bug fix

FSTORE-498 Throw error when filter feature not exist

FSTORE-499 add python code snippets to API docs

FSTORE-505 Remove old feature store tour files

FSTORE-509 Update expectation suite metadata in backend on setter call.

FSTORE-510 Implement reliable ingestionResult for Validation Report

FSTORE-513 Make .parquet file default format for training datasets

FSTORE-515 Prevent deleting Expectation Suite activity on deletion of the suite

FSTORE-519 Redshift storage connector documentation is out of sync

FSTORE-523 Improve user messages for forbidden actions.

FSTORE-526 Don't cast all types w...

Release 3.0.0

Release notes - Hopsworks - Version 3.0.0

Bug

HOPSWORKS-3255 elastic search - highlight fragment bug

HOPSWORKS-3253 Feature Group Type Enum breaks with Stream Feature Group after upgrade

HOPSWORKS-3252 Disable Kafka Topic deletion for feature group ingestion topics

HOPSWORKS-3246 New kafka topic names not included in the response after updating a deployment

HOPSWORKS-3236 Improve model inference error responses

HOPSWORKS-3235 Use Hudi snapshot query when users doesn't specify as_of in the query param

HOPSWORKS-3233 Timestamp incompatibility Spark/Hive/Hudi

HOPSWORKS-3232 Deleting a model used in existing deployments should not be possible

HOPSWORKS-3228 Bump scikit-learn to v1.1.1

HOPSWORKS-3227 Return valid inference endpoints only

HOPSWORKS-3221 Spark dist files do not respect HopsFS replication factor

HOPSWORKS-3220 HopsKafkaAuthorizer retry and refresh ACLs cache if cannot find any ACLs

HOPSWORKS-3212 Remove pickle file validation in storage-initializer

HOPSWORKS-3209 expectation_suite update is failing

HOPSWORKS-3207 Unexported service log viewer Chef attributes

HOPSWORKS-3204 Feature View can't get query when one of the feature groups was deleted

HOPSWORKS-3197 Feature Store quota enforcement fail with NPE for non-cached Feature Groups

HOPSWORKS-3193 Creating Feature Group from UI without Expectation Suite fails

HOPSWORKS-3191 OAuth not removing group if mapping is empty

HOPSWORKS-3187 Inference with Istio using HSML not enabled in transformers

HOPSWORKS-3185 Api key scopes do not cascade on delete

HOPSWORKS-3183 Online feature vectors are not sorted according to the user input

HOPSWORKS-3171 Rename PYTHON scope to PYTHON_LIBRARIES

HOPSWORKS-3149 Consul should only register the active resource manager

HOPSWORKS-3142 Transformation function does not handle null value

HOPSWORKS-3141 Query filter does not handle DATE type properly

HOPSWORKS-3133 Hopsify log file is in wrong directory

HOPSWORKS-3129 Consul does not detected a failed MySQLd

HOPSWORKS-3126 Should flush new Serving entity to get id

HOPSWORKS-3124 ruby test - always wait for started executions to stop

HOPSWORKS-3121 [FeatureView] Bug report

HOPSWORKS-3111 Hopsworks K8s certificate expiration should not be hardcoded

HOPSWORKS-3097 Make OAuth provider display name should not be mandatory

HOPSWORKS-3093 FileSaver.js package was outdated from Teleborder repository

HOPSWORKS-3090 CVE-2022-26612 - flink client vulnerability

HOPSWORKS-3087 Feature group data preview returns 500 when previewing streaming feature groups

HOPSWORKS-3070 Install custom PyTorch and Torchvision in base python environment

HOPSWORKS-3059 Latest version for monitored library should be null if not available

HOPSWORKS-3058 Hopsfs-mount chown's root directory ownership during upgrade

HOPSWORKS-3048 Python library installation using APIKey fails to get the user of the request

HOPSWORKS-3038 Add lock management in Python search singletons

HOPSWORKS-3031 Uglify failing hopsworks-web build

HOPSWORKS-3019 Filebeats data directory is wrong

HOPSWORKS-3015 flyway validation_rule table broken for 2.3 -> 2.4 -> 2.5 upgrade

HOPSWORKS-3013 Storage Connector Update not working

HOPSWORKS-2994 ModuleNotFoundError when saving code

HOPSWORKS-2993 Update git provider secrets throws an secret already exists exception

HOPSWORKS-2958 Hopsify should use internal https port to register with Hopsworks

HOPSWORKS-2957 Call hops::default from hopsworks::default

HOPSWORKS-2953 init_prepared_statement throws NoneType exception

HOPSWORKS-2952 update hsfs examples with train_split property

HOPSWORKS-2564 Alert manager config backup timer fails

HOPSWORKS-2100 Cannot enable model serving when Models folder already exists

HOPSWORKS-1932 feature store sharing ignores online feature store

Improvement

HOPSWORKS-3251 Add numFeatureViews to FeaturestoreDTO for front-end

HOPSWORKS-3248 Rename On Demand Feature groups to external feature groups in hsfs

HOPSWORKS-3242 Improve handling of different data types in feature groups across online/offline

HOPSWORKS-3240 Stream Feature Group online enable exception only when trying to change the setting

HOPSWORKS-3231 Support custom external IP for istio ingressgateway

HOPSWORKS-3226 Hops Kafka Authorizer improve synchonization

HOPSWORKS-3225 Throw a warning instead of error while creating empty data featuregroup

HOPSWORKS-3222 Improve Logstash CPU utilization

HOPSWORKS-3217 Set default values for Inference batching and add access to Hopsworks var for Job audience

HOPSWORKS-3216 hops-hadoop-chef hops/nn/heap_size should set only the NN heap

HOPSWORKS-3211 Add endpoint for deployment logs

HOPSWORKS-3206 bump hopshadoop version 3.2.0.7 RC0

HOPSWORKS-3195 Refactor inference endpoints details

HOPSWORKS-3190 Upgrade OpenSearch to 1.3.3

HOPSWORKS-3186 [FeatureView] Feature view training dataset API redesign

HOPSWORKS-3184 feature view search

HOPSWORKS-3181 OAuth update user profile not working

HOPSWORKS-3176 Change the format of the Hopsworks audit logs to JSON.

HOPSWORKS-3173 Kafka store number of partitions and replicas in the database

HOPSWORKS-3169 [hops-hadoop-chef] Expose attribute to configure user limits in Yarn Capacity Scheduler

HOPSWORKS-3161 Hive connection should return better error message when accessing an unshared feature group

HOPSWORKS-3158 Print Job and Feature Store URLs within Hopsworks with correct Hostname

[HOPSWORKS-3154](https://hopsworks.atlassian.net/br...

Release 2.5.0

Release notes - Hopsworks - Version 2.5.0

Bug

HOPSWORKS-2983 Projects can't be created without hops-system namespace in Kubernetes

HOPSWORKS-2980 Allow multiple model files in a model artifact

HOPSWORKS-2973 zip/unzip URI encoding calls system python, use hops-system python instead

HOPSWORKS-2965 Remove pathlib, confluent-kafka, hops-petastorm, opencv-python as dependencies for hops-util-py

HOPSWORKS-2963 NullPointerException when listing tags for model

HOPSWORKS-2962 Components in Serving UI are not aligned

HOPSWORKS-2953 init_prepared_statement throws NoneType exception

HOPSWORKS-2952 update hsfs examples with train_split property

HOPSWORKS-2951 hsml should also be monitored for minor releases

HOPSWORKS-2950 Remove user, project and installation section from hopsworks documentation

HOPSWORKS-2939 [tez] Tez master does not send application certificates with the create container request

HOPSWORKS-2936 Pin ipython to 7.31.0

HOPSWORKS-2935 jupyter_spec insufficient spark memory test should set python kernel to false

HOPSWORKS-2924 cloud-metrics-daemon ownership is incorrect

HOPSWORKS-2920 exclude org.slf4j:slf4j-log4j12 from zookeeper

HOPSWORKS-2893 SYNC_BASE_ENV should also check for conflicts

HOPSWORKS-2884 DefaultJobConfiguration update fails for PySpark/Spark JobType

HOPSWORKS-2873 Log4j2 vulnerability mitigation

HOPSWORKS-2869 Improve Spark/PySpark logging

HOPSWORKS-2863 parallel conversion of same notebook - add execution id to make name unique

HOPSWORKS-2856 OnlineFS user removal during project removal can result in collision

HOPSWORKS-2839 Fix batch serving vectors sql syntax

Release 2.4.0

Release notes - Hopsworks - Version 2.4.0

Bug

HOPSWORKS-2770 Feature Group clear content endpoint rethinking

HOPSWORKS-2748 Kibana and Grafana consul health checks are wrong

HOPSWORKS-2738 Artifact files should not have glassfish as owner

HOPSWORKS-2730 Proxy servlet is leaking resources

HOPSWORKS-2717 Avro fails with schema names starting with numeric characters

HOPSWORKS-2710 Wrong permissions for Airflow subdirectories

HOPSWORKS-2696 Cannot store features of tinyint offline type as avro serialization fails

HOPSWORKS-2682 node_exporter is not added to kagent services file if kagent is not enabled

HOPSWORKS-2669 Nullpointerexception when trying to get an environment that doesn't exists

HOPSWORKS-2665 Use Hopsworks internal port when calling generate_service_jwt in chef

HOPSWORKS-2663 Fix docker installation on ubuntu 18.04

HOPSWORKS-2662 Fix angular-material version

HOPSWORKS-2659 [hopsworks-chef] Chef guard has incorrect syntax

HOPSWORKS-2656 Incorrect payload when creating sklearn servings with new artifacts

HOPSWORKS-2644 Non HOPS_ADMIN users should not be able to access the cluster settings.

HOPSWORKS-2642 EditableSelect listening to keyboard event on window.

HOPSWORKS-2641 Project page not loading after login

HOPSWORKS-2640 Cannot start Jupyter if Kafka is offline

HOPSWORKS-2638 elastic clear cached scrolling contexts

HOPSWORKS-2620 Add shared by and accepted by to dataset sharing requests

HOPSWORKS-2213 feature group commit endpoints doesn't fall back to pagination defaults

Improvement

HOPSWORKS-2768 Add private_ips attribute in consul-chef metadata.rb

v2.3.0

Release notes - Hopsworks - Version 2.3.0

Bug

HOPSWORKS-2663 Fix docker installation on ubuntu 18.04

HOPSWORKS-2639 NPE with no Databricks Clusters in Feature Store Integration

HOPSWORKS-2634 Feature order not respected in training dataset prepared statement

HOPSWORKS-2631 SYNC_BASE_ENV operation should also show error logs in case of failure

HOPSWORKS-2628 OnlineFS schema of feature groups with appended features misses old features

HOPSWORKS-2626 MySQL timezone issue

HOPSWORKS-2625 Add status to dataset projectsSharedWith

HOPSWORKS-2619 Add missing JVM imports in Livy for Spark 3

HOPSWORKS-2618 NPE when force deleting a Project

HOPSWORKS-2614 Training Dataset from Python with default connector fails

HOPSWORKS-2609 Unshare a pending feature store fails

HOPSWORKS-2594 Attaching Jupyter config when running notebook with Spark or Python kernel with git backend fails

HOPSWORKS-2583 Experiment mnist notebook should work in airgapped environment

HOPSWORKS-2582 Spark driver cores is incorrectly set to number of configured executor cores

HOPSWORKS-2577 Python Feature Group Ingestion fails with additional write arguments

HOPSWORKS-2574 Feature validation rule Level.Warning defaults to Error

HOPSWORKS-2569 Allow capital letters keywords

HOPSWORKS-2567 Nullpointerexception when retrieving validations for on-demand feature groups

HOPSWORKS-2556 Jupyter should use default DockerJobConfiguration for Experiments and Spark

HOPSWORKS-2555 KAFKA_BROKERS not set in Python kernel

HOPSWORKS-2554 oauth2 get user bug

HOPSWORKS-2553 Fix kfserving airgapped deployment

HOPSWORKS-2546 Add RonDB services to kagent even if disabled

HOPSWORKS-2538 Secrets API not setting projectId and visibility in certain cases

HOPSWORKS-2536 Online FS Avro schema constructor decimals should take precision and scale

HOPSWORKS-2526 add fg to prepared statement only if it contains features other than pk and targets.

HOPSWORKS-2520 Installing python libraries with no version and from git should use --no-cache docker build option

HOPSWORKS-2517 Job executions not filtered by submission time

HOPSWORKS-2515 Set spark.serializer to Kryo for Databricks integration

HOPSWORKS-2510 Updating validationType on feature group response contains old value

HOPSWORKS-2503 Pin containerd version in ubuntu

HOPSWORKS-2495 HSFS connection cannot not read JWT in Python jobs

HOPSWORKS-2490 Disable Python Library Updates Monitor for snapshot versions

HOPSWORKS-2386 Decimal types are converted to varbinary on the online feature grouip

HOPSWORKS-2349 Hive InodeHelper doesn't work if path contains slahs (/)

HOPSWORKS-2343 Validate storage connector type before creating a training dataset

Improvement

HOPSWORKS-2653 Flyway dir already exists error in hopsworks::install

HOPSWORKS-2645 Support feature validation for on-demand FGs in hsfs

HOPSWORKS-2643 Add option to enable hopsfs/rm audit logs

HOPSWORKS-2636 Add option to specify Kubernetes node labels from cluster definition

HOPSWORKS-2615 Add option to redirect stdout and stderr of Docker jobs

HOPSWORKS-2607 Set NotValidBefore of issued X.509 a few minutes earlier

HOPSWORKS-2605 Configuration parameter for Kubernetes maximum retry number

HOPSWORKS-2604 add option to specify additional insecure docker registry

HOPSWORKS-2597 Add hops Python helper methods for model and dataset modules

HOPSWORKS-2584 Increase Schema Registry database schema column size

HOPSWORKS-2578 Use a separate Data directory to store Hopsworks state

HOPSWORKS-2575 Make Online FS Kafka topic partition number configurable

HOPSWORKS-2560 Support Memory, Cores and GPU as a resource for model serving

HOPSWORKS-2559 Add input paths option for Docker jobs and multiple commands

HOPSWORKS-2552 Send monitoring URLs as part of execution dto

HOPSWORKS-2551 Bump hops dependencies to 3.2.0.4-SNAPSHOT

HOPSWORKS-2529 Package dockerfile:experimental in the base.tar for

HOPSWORKS-2528 Add support to provide PIP extra-index-url in the cluster definition

HOPSWORKS-2523 Use own java cookbook instead of upstream

HOPSWORKS-2506 Feature Validation Improvements

HOPSWORKS-2502 Add JOB audience to delete endpoint for training datasets

HOPSWORKS-2492 Improve readiness check of KFServing inference services

HOPSWORKS-2356 Alerting for hopsworks services

HOPSWORKS-2167 Add inference logging sidecar to KFServing deployments

New Feature

HOPSWORKS-2381 Project level configuration for Jupyter notebooks and Jobs

HOPSWORKS-2334 Support custom transformers in KFServing deployments

HOPSWORKS-2319 Add API to interact with the secret store

HOPSWORKS-2104 Model Serving with KFServing

Task

HOPSWORKS-2637 Bump hops dependencies to 3.2.0.4-RC0

HOPSWORKS-2622 pin six to 1.15.0

HOPSWORKS-2621 Bump pip in hops-system to last version that supports python 2.7

HOPSWORKS-2601 Update swagger code to be compatible with the latest swagger hub and enable for demo cluster

HOPSWORKS-2498 Remove filebeat beam recipes

HOPSWORKS-2424 Attach transformation function to training dataset

HOPSWORKS-2420 Bump hops dependencies to 3.2.0.3-SNAPSHOT

HOPSWORKS-1942 Upgrade to Spark 3.x