Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support seatunnel 2.3.2 #607

Merged
merged 5 commits into from
Aug 22, 2023
Merged

Support seatunnel 2.3.2 #607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
070b5f1
[Feature][Seatunnel] StarRocks add base-url property
LiuBodong Aug 21, 2023
65fbcd1
Merge remote-tracking branch 'origin/dev' into support-seatunnel-2.3.2
LiuBodong Aug 22, 2023
969cc49
[Feature][setunnel-plugin-seatunnel-connector] Update starrocks plugi…
LiuBodong Aug 22, 2023
1646c0a
[Feature][setunnel-plugin-seatunnel-connector] Change file system con…
LiuBodong Aug 22, 2023
162a437
[Bugfix][Seatunnel] Seatunnel pulsar connector property type error.
LiuBodong Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions scaleph-datasource/src/main/java/cn/sliew/scaleph/ds/modal/olap/StarRocksDataSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ public class StarRocksDataSource extends AbstractDataSource {
@Schema(description = "Node Urls")
private String nodeUrls;

@NotBlank
@Schema(description = "Base url")
private String baseUrl;

@NotBlank
@Schema(description = "username")
private String username;
Expand Down
4 changes: 2 additions & 2 deletions ...main/java/cn/sliew/scaleph/plugin/seatunnel/flink/connectors/file/FileSinkProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ public enum FileSinkProperties {
.addValidator(Validators.NON_BLANK_VALIDATOR)
.validateAndBuild();

public static final PropertyDescriptor<String> FILE_FORMAT = new PropertyDescriptor.Builder<String>()
.name("file_format")
public static final PropertyDescriptor<String> FILE_FORMAT_TYPE = new PropertyDescriptor.Builder<String>()
.name("file_format_type")
.description("We supported as the following file types:text, csv, excel, parquet, orc, json")
.type(PropertyType.STRING)
.defaultValue("text")
Expand Down
2 changes: 1 addition & 1 deletion ...a/cn/sliew/scaleph/plugin/seatunnel/flink/connectors/file/ftp/sink/FtpFileSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public FtpFileSinkPlugin() {

final List<PropertyDescriptor> props = new ArrayList<>();
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
2 changes: 1 addition & 1 deletion ...cn/sliew/scaleph/plugin/seatunnel/flink/connectors/file/hdfs/sink/HDFSFileSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ public HDFSFileSinkPlugin() {

final List<PropertyDescriptor> props = new ArrayList<>();
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
2 changes: 1 addition & 1 deletion .../sliew/scaleph/plugin/seatunnel/flink/connectors/file/local/sink/LocalFileSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ public LocalFileSinkPlugin() {

final List<PropertyDescriptor> props = new ArrayList<>();
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
2 changes: 1 addition & 1 deletion .../java/cn/sliew/scaleph/plugin/seatunnel/flink/connectors/file/oss/sink/OSSSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public OSSSinkPlugin() {

final List<PropertyDescriptor> props = new ArrayList<>();
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
2 changes: 1 addition & 1 deletion ...liew/scaleph/plugin/seatunnel/flink/connectors/file/ossjindo/sink/OSSJindoSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public OSSJindoSinkPlugin() {

final List<PropertyDescriptor> props = new ArrayList<>();
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
2 changes: 1 addition & 1 deletion ...in/java/cn/sliew/scaleph/plugin/seatunnel/flink/connectors/file/s3/sink/S3SinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public S3SinkPlugin() {
final List<PropertyDescriptor> props = new ArrayList<>();
props.add(HADOOP_S3_PROPERTIES);
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
2 changes: 1 addition & 1 deletion ...cn/sliew/scaleph/plugin/seatunnel/flink/connectors/file/sftp/sink/SftpFileSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public SftpFileSinkPlugin() {

final List<PropertyDescriptor> props = new ArrayList<>();
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(CUSTOM_FILENAME);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
Expand Down
4 changes: 2 additions & 2 deletions ...sliew/scaleph/plugin/seatunnel/flink/connectors/pulsar/source/PulsarSourceProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,8 @@ public enum PulsarSourceProperties {
public static final PropertyDescriptor<String> CURSOR_STOP_TIMESTAMP = new PropertyDescriptor.Builder()
.name("cursor.stop.timestamp")
.description("Stop from the specified epoch timestamp (in milliseconds).")
.type(PropertyType.STRING)
.parser(Parsers.STRING_PARSER)
.type(PropertyType.LONG)
.parser(Parsers.LONG_PARSER)
.addValidator(Validators.NON_BLANK_VALIDATOR)
.validateAndBuild();

Expand Down
2 changes: 1 addition & 1 deletion ...sliew/scaleph/plugin/seatunnel/flink/connectors/s3redshift/sink/S3RedshiftSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ public S3RedshiftSinkPlugin() {

props.add(HADOOP_S3_PROPERTIES);
props.add(PATH);
props.add(FILE_FORMAT);
props.add(FILE_FORMAT_TYPE);
props.add(FILE_NAME_EXPRESSION);
props.add(FILENAME_TIME_FORMAT);
props.add(FIELD_DELIMITER);
Expand Down
4 changes: 4 additions & 0 deletions ...n/sliew/scaleph/plugin/seatunnel/flink/connectors/starrocks/sink/StarRocksSinkPlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ public StarRocksSinkPlugin() {
"Used to send data to StarRocks. Both support streaming and batch mode. The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.",
StarRocksSinkPlugin.class.getName());
final List<PropertyDescriptor> props = new ArrayList<>();
props.add(BASE_URL);
props.add(DATABASE);
props.add(TABLE);
props.add(LABEL_PREFIX);
Expand Down Expand Up @@ -81,6 +82,9 @@ public ObjectNode createConf() {
if (StringUtils.hasText(dataSource.getPassword())) {
conf.putPOJO(PASSWORD.getName(), dataSource.getPassword());
}
if (StringUtils.hasText(dataSource.getBaseUrl())) {
conf.putPOJO(BASE_URL.getName(), dataSource.getBaseUrl());
}
return conf;
}

Expand Down
14 changes: 10 additions & 4 deletions ...iew/scaleph/plugin/seatunnel/flink/connectors/starrocks/sink/StarRocksSinkProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,21 @@

package cn.sliew.scaleph.plugin.seatunnel.flink.connectors.starrocks.sink;

import cn.sliew.scaleph.plugin.framework.property.Parsers;
import cn.sliew.scaleph.plugin.framework.property.PropertyDescriptor;
import cn.sliew.scaleph.plugin.framework.property.PropertyType;
import cn.sliew.scaleph.plugin.framework.property.Validators;
import cn.sliew.scaleph.plugin.framework.property.*;
import com.fasterxml.jackson.databind.JsonNode;

public enum StarRocksSinkProperties {
;

public static final PropertyDescriptor<String> BASE_URL = new PropertyDescriptor.Builder<>()
.name("base-url")
.description("The JDBC URL like jdbc:mysql://localhost:9030/ or jdbc:mysql://localhost:9030 or jdbc:mysql://localhost:9030/db")
.properties(Property.Required)
.type(PropertyType.STRING)
.parser(Parsers.STRING_PARSER)
.addValidator(Validators.NON_BLANK_VALIDATOR)
.validateAndBuild();

public static final PropertyDescriptor<String> LABEL_PREFIX = new PropertyDescriptor.Builder()
.name("labelPrefix")
.description("The prefix of StarRocks stream load label")
Expand Down
3 changes: 2 additions & 1 deletion scaleph-ui-react/src/locales/zh-CN/pages/project.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -570,6 +570,7 @@ export default {


// starrocks
'pages.project.di.step.starrocks.base-url': '基础连接地址',
'pages.project.di.step.starrocks.database': '数据库',
'pages.project.di.step.starrocks.table': '数据表',
'pages.project.di.step.starrocks.labelPrefix': '标签前缀',
Expand Down Expand Up @@ -616,7 +617,7 @@ export default {

// s3redshift
'pages.project.di.step.s3redshift.jdbcUrl': 'Jdbc URL',
'pages.project.di.step.s3redshift.jdbcUser': 'Jdbc哟你见过户',
'pages.project.di.step.s3redshift.jdbcUser': 'Jdbc用户名',
'pages.project.di.step.s3redshift.jdbcPassword': 'Jdbc密码',
'pages.project.di.step.s3redshift.executeSql': 'SQL查询语句',
'pages.project.di.step.s3redshift.executeSql.placeholoder':
Expand Down
2 changes: 1 addition & 1 deletion scaleph-ui-react/src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/constant.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,6 @@ export const BaseFileParams = {
dateFormat: 'date_format',
timeFormat: 'time_format',
datetimeFormat: 'datetime_format',
fileFormat: 'file_format',
customFilename: 'custom_filename',
fileNameExpression: 'file_name_expression',
filenameTimeFormat: 'filename_time_format',
Expand Down Expand Up @@ -519,6 +518,7 @@ export const DorisParams = {
};

export const StarRocksParams = {
baseUrl: 'base-url',
database: 'database',
table: 'table',
labelPrefix: 'labelPrefix',
Expand Down
10 changes: 5 additions & 5 deletions ...t/src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-ftp-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,14 @@ const SinkFtpFileStepForm: React.FC<
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -109,7 +109,7 @@ const SinkFtpFileStepForm: React.FC<
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
10 changes: 5 additions & 5 deletions .../src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-hdfs-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,13 +78,13 @@ const SinkHdfsFileStepForm: React.FC<
rules={[{required: true}]}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -106,7 +106,7 @@ const SinkHdfsFileStepForm: React.FC<
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
10 changes: 5 additions & 5 deletions ...src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-local-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,14 +78,14 @@ const SinkLocalFileStepForm: React.FC<
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -107,7 +107,7 @@ const SinkLocalFileStepForm: React.FC<
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
10 changes: 5 additions & 5 deletions ...t/src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-oss-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,14 @@ const SinkOSSFileStepForm: React.FC<
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -109,7 +109,7 @@ const SinkOSSFileStepForm: React.FC<
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
10 changes: 5 additions & 5 deletions .../pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-ossjindo-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,14 @@ const SinkOSSJindoFileStepForm: React.FC<
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -109,7 +109,7 @@ const SinkOSSJindoFileStepForm: React.FC<
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
10 changes: 5 additions & 5 deletions ...ct/src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-s3-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,14 +127,14 @@ const SinkS3FileStepForm: React.FC<
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -156,7 +156,7 @@ const SinkS3FileStepForm: React.FC<
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
8 changes: 4 additions & 4 deletions ...ages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-s3redshift-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ const SinkS3RedshiftStepForm: React.FC<ModalFormProps<{
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
valueEnum={{
Expand All @@ -145,9 +145,9 @@ const SinkS3RedshiftStepForm: React.FC<ModalFormProps<{
csv: 'csv',
}}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand Down
10 changes: 5 additions & 5 deletions .../src/pages/Project/Workspace/Artifact/DI/DiJobFlow/Dag/steps/sink/sink-sftp-file-step.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,14 +78,14 @@ const SinkSftpFileStepForm: React.FC<ModalFormProps<{
colProps={{span: 24}}
/>
<ProFormSelect
name={'file_format'}
name={BaseFileParams.fileFormatType}
label={intl.formatMessage({id: 'pages.project.di.step.baseFile.fileFormat'})}
colProps={{span: 24}}
options={['json', 'parquet', 'orc', 'text', 'csv', 'excel']}
/>
<ProFormDependency name={['file_format']}>
{({file_format}) => {
if (file_format == 'text' || file_format == 'csv') {
<ProFormDependency name={[BaseFileParams.fileFormatType]}>
{({file_format_type}) => {
if (file_format_type == 'text' || file_format_type == 'csv') {
return (
<ProFormGroup>
<ProFormText
Expand All @@ -107,7 +107,7 @@ const SinkSftpFileStepForm: React.FC<ModalFormProps<{
</ProFormGroup>
);
}
if (file_format == 'excel') {
if (file_format_type == 'excel') {
return (
<ProFormGroup>
<ProFormText
Expand Down
Loading