3.2.2 release cherry-pick cp: Reduce the size of tarball to continue …

…ASF release (#16042) * add version of 3.2.2 * Fix rpc channel leak due to concurrent operation (#16021) * Fix rpc channel leak due to concurrent operation * Throw channel create failed exception --------- Co-authored-by: Rick Cheng <[email protected]> * Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027) * cp: Reduce the size of tarball to continue ASF release (#15004) (#15540) * Reduce the size of tarball to continue ASF release for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt # Conflicts: # tools/dependencies/known-dependencies.txt * Increase block-until-connected in ZookeeperRegistryTestCase (#16041) * [Fix][CI] fix the ci error of Values.datasource.profile (#16031) Co-authored-by: Eric Gao <[email protected]> * Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033) Co-authored-by: Rick Cheng <[email protected]> * [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967) * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * improvement project code * remove useless imports * remove method onClearSearchTaskType * add parameter doc * optimisation logic * code conflict resolution * code conflict resolution * [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978) * update * test * add monitor enhance ui * update * update * update doc * fix spotless * update * update * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <[email protected]> * update * fix spotless * update --------- Co-authored-by: Wenjun Ruan <[email protected]> * [Improvement][Monitor] Add UT for montor (#15998) * formatting Code * add pom * Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038) * Removed unused StateEventHandleFailure (#16052) * Remove OmitStackTraceInFastThrow in start.sh (#16054) * Update Chart.yaml Rollback APPversion * modify log msg (#16062) Co-authored-by: xiangzihao <[email protected]> * [DS-16046][fix] Set PreparedStatement parameter type (#16050) Fix the PreparedStatement parameter is TIME, set it to java.sql.Time Co-authored-by: Aaron Wang <[email protected]> Co-authored-by: Rick Cheng <[email protected]> * Update pom.xml * [DSIP-42] Add dolphinscheduler-aws-authentication module (#16043) --------- Co-authored-by: wangxj <wangxj31> Co-authored-by: Wenjun Ruan <[email protected]> Co-authored-by: Rick Cheng <[email protected]> Co-authored-by: Eric Gao <[email protected]> Co-authored-by: TianXinCoord <[email protected]> Co-authored-by: 小可耐 <[email protected]> Co-authored-by: sleo <[email protected]> Co-authored-by: xiangzihao <[email protected]> Co-authored-by: yinxiaolog <[email protected]> Co-authored-by: Aaron Wang <[email protected]>
apache · May 25, 2024 · 646bbf4 · 646bbf4
1 parent 7b7a333
commit 646bbf4
Show file tree

Hide file tree

Showing 151 changed files with 2,554 additions and 1,357 deletions.
diff --git a/deploy/docker/.env b/deploy/docker/.env
@@ -15,8 +15,8 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-HUB=ghcr.io/apache/dolphinscheduler
-TAG=latest
+HUB=apache
+TAG=3.2.2
 
 TZ=Asia/Shanghai
 DATABASE=postgresql

diff --git a/deploy/kubernetes/dolphinscheduler/Chart.yaml b/deploy/kubernetes/dolphinscheduler/Chart.yaml
@@ -35,7 +35,7 @@ type: application
 
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
-version: 3.1.0
+version: 3.2.2
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application.

diff --git a/deploy/kubernetes/dolphinscheduler/README.md b/deploy/kubernetes/dolphinscheduler/README.md
@@ -120,6 +120,12 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
 | conf.auto | bool | `false` | auto restart, if true, all components will be restarted automatically after the common configuration is updated. if false, you need to restart the components manually. default is false |
 | conf.common."alert.rpc.port" | int | `50052` | rpc port |
 | conf.common."appId.collect" | string | `"log"` | way to collect applicationId: log, aop |
+| conf.common."aws.credentials.provider.type" | string | `"AWSStaticCredentialsProvider"` | |
+| conf.common."aws.s3.access.key.id" | string | `"minioadmin"` | The AWS access key. if resource.storage.type=S3, and credentials.provider.type is AWSStaticCredentialsProvider. This configuration is required |
+| conf.common."aws.s3.access.key.secret" | string | `"minioadmin"` | The AWS secret access key. if resource.storage.type=S3, and credentials.provider.type is AWSStaticCredentialsProvider. This configuration is required |
+| conf.common."aws.s3.bucket.name" | string | `"dolphinscheduler"` | The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name. |
+| conf.common."aws.s3.endpoint" | string | `"http://minio:9000"` | You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn |
+| conf.common."aws.s3.region" | string | `"ca-central-1"` | The AWS Region to use. if resource.storage.type=S3, This configuration is required |
 | conf.common."conda.path" | string | `"/opt/anaconda3/etc/profile.d/conda.sh"` | set path of conda.sh |
 | conf.common."data-quality.jar.dir" | string | `nil` | data quality option |
 | conf.common."data.basedir.path" | string | `"/tmp/dolphinscheduler"` | user data local directory path, please make sure the directory exists and have read write permissions |
@@ -138,11 +144,6 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
 | conf.common."resource.alibaba.cloud.oss.bucket.name" | string | `"dolphinscheduler"` | oss bucket name, required if you set resource.storage.type=OSS |
 | conf.common."resource.alibaba.cloud.oss.endpoint" | string | `"https://oss-cn-hangzhou.aliyuncs.com"` | oss bucket endpoint, required if you set resource.storage.type=OSS |
 | conf.common."resource.alibaba.cloud.region" | string | `"cn-hangzhou"` | alibaba cloud region, required if you set resource.storage.type=OSS |
-| conf.common."resource.aws.access.key.id" | string | `"minioadmin"` | The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required |
-| conf.common."resource.aws.region" | string | `"ca-central-1"` | The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required |
-| conf.common."resource.aws.s3.bucket.name" | string | `"dolphinscheduler"` | The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name. |
-| conf.common."resource.aws.s3.endpoint" | string | `"http://minio:9000"` | You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn |
-| conf.common."resource.aws.secret.access.key" | string | `"minioadmin"` | The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required |
 | conf.common."resource.azure.client.id" | string | `"minioadmin"` | azure storage account name, required if you set resource.storage.type=ABS |
 | conf.common."resource.azure.client.secret" | string | `"minioadmin"` | azure storage account key, required if you set resource.storage.type=ABS |
 | conf.common."resource.azure.subId" | string | `"minioadmin"` | azure storage subId, required if you set resource.storage.type=ABS |
@@ -158,6 +159,7 @@ Please refer to the [Quick Start in Kubernetes](../../../docs/docs/en/guide/inst
 | conf.common."yarn.application.status.address" | string | `"http://ds1:%s/ws/v1/cluster/apps/%s"` | if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname |
 | conf.common."yarn.job.history.status.address" | string | `"http://ds1:19888/ws/v1/history/mapreduce/jobs/%s"` | job history status url when application number threshold is reached(default 10000, maybe it was set to 1000) |
 | conf.common."yarn.resourcemanager.ha.rm.ids" | string | `"192.168.xx.xx,192.168.xx.xx"` | if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty |
+| datasource.profile | string | `"postgresql"` | The profile of datasource |
 | externalDatabase.database | string | `"dolphinscheduler"` | The database of external database |
 | externalDatabase.driverClassName | string | `"org.postgresql.Driver"` | The driverClassName of external database |
 | externalDatabase.enabled | bool | `false` | If exists external database, and set postgresql.enable value to false. external database will be used, otherwise Dolphinscheduler's internal database will be used. |

diff --git a/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl b/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl
@@ -51,7 +51,6 @@ Create a default common labels.
 {{- define "dolphinscheduler.common.labels" -}}
 app.kubernetes.io/instance: {{ .Release.Name }}
 app.kubernetes.io/managed-by: {{ .Release.Service }}
-app.kubernetes.io/version: {{ .Chart.AppVersion }}
 {{- end -}}
 
 {{/*

diff --git a/deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-alert.yaml b/deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-alert.yaml
@@ -33,15 +33,13 @@ data:
  banner:
  charset: UTF-8
  datasource:
- profile: postgresql
- config:
- driver-class-name: org.postgresql.Driver
- url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
- username: root
- password: root
- hikari:
- connection-test-query: select 1
- pool-name: DolphinScheduler
+ driver-class-name: org.postgresql.Driver
+ url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
+ username: root
+ password: root
+ hikari:
+ connection-test-query: select 1
+ pool-name: DolphinScheduler
  
  # Mybatis-plus configuration, you don't need to change it
  mybatis-plus:
@@ -102,4 +100,16 @@ data:
  
  metrics:
  enabled: true
+ 
+ # Override by profile
+ ---
+ spring:
+ config:
+ activate:
+ on-profile: mysql
+ datasource:
+ driver-class-name: com.mysql.cj.jdbc.Driver
+ url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
+ username: root
+ password: root
 {{- end }}
diff --git a/deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-api.yaml b/deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-api.yaml
@@ -54,15 +54,13 @@ data:
  messages:
  basename: i18n/messages
  datasource:
- profile: postgresql
- config:
- driver-class-name: org.postgresql.Driver
- url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
- username: root
- password: root
- hikari:
- connection-test-query: select 1
- pool-name: DolphinScheduler
+ driver-class-name: org.postgresql.Driver
+ url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
+ username: root
+ password: root
+ hikari:
+ connection-test-query: select 1
+ pool-name: DolphinScheduler
  quartz:
  auto-startup: false
  job-store-type: jdbc
@@ -239,6 +237,21 @@ data:
  application-name: ""
  # Doplhinscheduler login url
  redirect-url: ""
+ 
+ # Override by profile
+ ---
+ spring:
+ config:
+ activate:
+ on-profile: mysql
+ datasource:
+ driver-class-name: com.mysql.cj.jdbc.Driver
+ url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
+ username: root
+ password: root
+ quartz:
+ properties:
+ org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
 {{- end }}
 
 

diff --git a/deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml b/deploy/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml
@@ -33,15 +33,13 @@ data:
  time-zone: UTC
  date-format: "yyyy-MM-dd HH:mm:ss"
  datasource:
- profile: postgresql
- config:
- driver-class-name: org.postgresql.Driver
- url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
- username: root
- password: root
- hikari:
- connection-test-query: select 1
- pool-name: DolphinScheduler
+ driver-class-name: org.postgresql.Driver
+ url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
+ username: root
+ password: root
+ hikari:
+ connection-test-query: select 1
+ pool-name: DolphinScheduler
  quartz:
  job-store-type: jdbc
  jdbc:
@@ -157,4 +155,19 @@ data:
  
  metrics:
  enabled: true
+ 
+ # Override by profile
+ ---
+ spring:
+ config:
+ activate:
+ on-profile: mysql
+ datasource:
+ driver-class-name: com.mysql.cj.jdbc.Driver
+ url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
+ username: root
+ password: root
+ quartz:
+ properties:
+ org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
 {{- end }}
diff --git a/deploy/kubernetes/dolphinscheduler/templates/configmap.yaml b/deploy/kubernetes/dolphinscheduler/templates/configmap.yaml
@@ -37,7 +37,7 @@ data:
  {{- range $key, $value := index .Values.conf "common" }}
  {{- if and $.Values.minio.enabled }}
  {{- if eq $key "resource.storage.type" }}{{ $value = "S3" }}{{- end }}
- {{- if eq $key "resource.aws.s3.endpoint" }}{{ $value = print "http://" (include "dolphinscheduler.minio.fullname" $) ":9000" }}{{- end }}
+ {{- if eq $key "aws.s3.endpoint" }}{{ $value = print "http://" (include "dolphinscheduler.minio.fullname" $) ":9000" }}{{- end }}
  {{- end }}
  {{ $key }}={{ $value }}
  {{- end }}

diff --git a/deploy/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml b/deploy/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml
@@ -131,5 +131,5 @@ spec:
  name: {{ include "dolphinscheduler.fullname" . }}-configs
  - name: alert-config-volume
  configMap:
- name: { { include "dolphinscheduler.fullname" . } }-alert
+ name: {{ include "dolphinscheduler.fullname" . }}-alert
 {{- end }}
diff --git a/deploy/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml b/deploy/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml
@@ -141,7 +141,7 @@ spec:
  name: {{ include "dolphinscheduler.fullname" . }}-configs
  - name: api-config-volume
  configMap:
- name: { { include "dolphinscheduler.fullname" . } }-api
+ name: {{ include "dolphinscheduler.fullname" . }}-api
  {{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }}
  {{- include "dolphinscheduler.fsFileResource.volume" . | nindent 8 }}
  {{- include "dolphinscheduler.ldap.ssl.volume" . | nindent 8 }}

diff --git a/deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml b/deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml
@@ -127,7 +127,7 @@ spec:
  {{- end }}
  - name: master-config-volume
  configMap:
- name: { { include "dolphinscheduler.fullname" . } }-master
+ name: {{ include "dolphinscheduler.fullname" . }}-master
  {{- include "dolphinscheduler.sharedStorage.volume" . | nindent 8 }}
  - name: config-volume
  configMap:

diff --git a/deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml b/deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml
@@ -144,7 +144,7 @@ spec:
  {{- end }}
  - name: worker-config-volume
  configMap:
- name: { { include "dolphinscheduler.fullname" . } }-worker
+ name: {{ include "dolphinscheduler.fullname" . }}-worker
  - name: config-volume
  configMap:
  name: {{ include "dolphinscheduler.fullname" . }}-configs

diff --git a/deploy/kubernetes/dolphinscheduler/values.yaml b/deploy/kubernetes/dolphinscheduler/values.yaml
@@ -33,7 +33,7 @@ image:
  # -- Docker image repository for the DolphinScheduler
  registry: apache
  # -- Docker image version for the DolphinScheduler
- tag: latest
+ tag: 3.2.2
  # -- Image pull policy. Options: Always, Never, IfNotPresent
  pullPolicy: "IfNotPresent"
  # -- Specify a imagePullSecrets
@@ -49,6 +49,10 @@ image:
  # -- tools image
  tools: dolphinscheduler-tools
 
+datasource:
+ # -- The profile of datasource
+ profile: postgresql
+
 postgresql:
  # -- If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL
  enabled: true
@@ -246,20 +250,25 @@ conf:
  # -- resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
  resource.storage.upload.base.path: /dolphinscheduler
 
- # -- The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
- resource.aws.access.key.id: minioadmin
+ # The AWS credentials provider type. support: AWSStaticCredentialsProvider, InstanceProfileCredentialsProvider
+ # AWSStaticCredentialsProvider: use the access key and secret key to authenticate
+ # InstanceProfileCredentialsProvider: use the IAM role to authenticate
+ aws.credentials.provider.type: AWSStaticCredentialsProvider
+
+ # -- The AWS access key. if resource.storage.type=S3, and credentials.provider.type is AWSStaticCredentialsProvider. This configuration is required
+ aws.s3.access.key.id: minioadmin
 
- # -- The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
- resource.aws.secret.access.key: minioadmin
+ # -- The AWS secret access key. if resource.storage.type=S3, and credentials.provider.type is AWSStaticCredentialsProvider. This configuration is required
+ aws.s3.access.key.secret: minioadmin
 
- # -- The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required
- resource.aws.region: ca-central-1
+ # -- The AWS Region to use. if resource.storage.type=S3, This configuration is required
+ aws.s3.region: ca-central-1
 
  # -- The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name.
- resource.aws.s3.bucket.name: dolphinscheduler
+ aws.s3.bucket.name: dolphinscheduler
 
  # -- You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn
- resource.aws.s3.endpoint: http://minio:9000
+ aws.s3.endpoint: http://minio:9000
 
  # -- alibaba cloud access key id, required if you set resource.storage.type=OSS
  resource.alibaba.cloud.access.key.id: <your-access-key-id>