Skip to content

Commit

Permalink
Removed Issue tag Plugin-1669 in Runtime scenarios and test runner file,
Browse files Browse the repository at this point in the history
updated correct conn string,
removed space in yml,
removed verify goal in pom.
changed sql connector java version to run in github actions and
bqmt secret updated
  • Loading branch information
priyabhatnagar25 authored and rahuldash171 committed Dec 21, 2023
1 parent ab3c7fc commit 0d56779
Show file tree
Hide file tree
Showing 6 changed files with 10 additions and 15 deletions.
8 changes: 3 additions & 5 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ jobs:
MYSQL_USERNAME:cdapio-github-builds/MYSQL_USERNAME
MYSQL_PASSWORD:cdapio-github-builds/MYSQL_PASSWORD
MYSQL_PORT:cdapio-github-builds/MYSQL_PORT
CONNECTION_STRING:cdapio-github-builds/CONNECTION_STRING
BQMT_CONNECTION_STRING:cdapio-github-builds/BQMT_CONNECTION_STRING
- name: Run required e2e tests
if: github.event_name != 'workflow_dispatch' && github.event_name != 'push' && steps.filter.outputs.e2e-test == 'false'
Expand All @@ -89,7 +89,7 @@ jobs:
MYSQL_USERNAME: ${{ steps.secrets.outputs.MYSQL_USERNAME }}
MYSQL_PASSWORD: ${{ steps.secrets.outputs.MYSQL_PASSWORD }}
MYSQL_PORT: ${{ steps.secrets.outputs.MYSQL_PORT }}
CONNECTION_STRING: ${{ steps.secrets.outputs.CONNECTION_STRING }}
BQMT_CONNECTION_STRING: ${{ steps.secrets.outputs.BQMT_CONNECTION_STRING }}

- name: Run all e2e tests
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push' || steps.filter.outputs.e2e-test == 'true'
Expand All @@ -99,22 +99,20 @@ jobs:
MYSQL_USERNAME: ${{ steps.secrets.outputs.MYSQL_USERNAME }}
MYSQL_PASSWORD: ${{ steps.secrets.outputs.MYSQL_PASSWORD }}
MYSQL_PORT: ${{ steps.secrets.outputs.MYSQL_PORT }}
CONNECTION_STRING: ${{ steps.secrets.outputs.CONNECTION_STRING }}
BQMT_CONNECTION_STRING: ${{ steps.secrets.outputs.BQMT_CONNECTION_STRING }}

- name: Upload report
uses: actions/upload-artifact@v3
if: always()
with:
name: Cucumber report - ${{ matrix.tests }}
path: ./plugin/target/cucumber-reports

- name: Upload debug files
uses: actions/upload-artifact@v3
if: always()
with:
name: Debug files - ${{ matrix.tests }}
path: ./**/target/e2e-debug

- name: Upload files to GCS
uses: google-github-actions/upload-cloud-storage@v0
if: always()
Expand Down
4 changes: 1 addition & 3 deletions pom.xml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?>

<!--
~ Copyright © 2016-2019 Cask Data, Inc.
~
Expand Down Expand Up @@ -1195,7 +1194,6 @@
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
Expand Down Expand Up @@ -1246,7 +1244,7 @@
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.19</version>
<version>8.0.25</version>
<scope>test</scope>
</dependency>
<dependency>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
@BQMT_SINK
Feature: BigQueryMultiTable sink -Verification of Multiple Database Tables to BigQueryMultiTable successful data transfer using macros

@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required @PLUGIN-1669
@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required
Scenario:Verify data is getting transferred from Multiple Database Tables to BQMT sink with all datatypes using macros
Given Open Datafusion Project to configure pipeline
When Expand Plugin group in the LHS plugins list: "Source"
Expand Down Expand Up @@ -71,7 +71,7 @@ Feature: BigQueryMultiTable sink -Verification of Multiple Database Tables to Bi
Then Verify the pipeline status is "Succeeded"
Then Validate the values of records transferred to BQMT sink is equal to the value from source MultiDatabase table

@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required @PLUGIN-1669
@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required
Scenario:Verify data is getting transferred from Multiple Database Tables to BQMT sink with split field using macros
Given Open Datafusion Project to configure pipeline
When Expand Plugin group in the LHS plugins list: "Source"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
@BQMT_SINK
Feature: BigQueryMultiTable sink -Verification of MultipleDatabaseTable to BigQueryMultiTable successful data transfer

@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required @PLUGIN-1669
@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required
Scenario:Verify data is getting transferred from Multiple Database Tables to BQMT sink with all datatypes
Given Open Datafusion Project to configure pipeline
When Expand Plugin group in the LHS plugins list: "Source"
Expand Down Expand Up @@ -58,7 +58,7 @@ Feature: BigQueryMultiTable sink -Verification of MultipleDatabaseTable to BigQu
Then Verify the pipeline status is "Succeeded"
Then Validate the values of records transferred to BQMT sink is equal to the value from source MultiDatabase table

@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required @PLUGIN-1669
@MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required
Scenario:Verify data is getting transferred from Multiple Database Tables to BQMT sink with split field
Given Open Datafusion Project to configure pipeline
When Expand Plugin group in the LHS plugins list: "Source"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@
features = {"src/e2e-test/features"},
glue = {"io.cdap.plugin.bigquerymultitable.stepsdesign", "io.cdap.plugin.gcs.stepsdesign", "stepsdesign",
"io.cdap.plugin.common.stepsdesign"},
tags = {"@BQMT_SINK and not @PLUGIN-1669"},
//TODO: Enable test once issue is fixed https://cdap.atlassian.net/browse/PLUGIN-1669
tags = {"@BQMT_SINK"},
monochrome = true,
plugin = {"pretty", "html:target/cucumber-html-report/bigquerymultitable",
"json:target/cucumber-reports/cucumber-bigquerymultitable.json",
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/resources/pluginParameters.properties
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ host=MYSQL_HOST
user=MYSQL_USERNAME
pass=MYSQL_PASSWORD
port=MYSQL_PORT
connectionString=CONNECTION_STRING
connectionString=BQMT_CONNECTION_STRING
mySQLSourceTable=dummy
sourceTable=dummy
tableNameField=tableName
Expand Down

0 comments on commit 0d56779

Please sign in to comment.