Skip to content

e2e tests CDAP-19055 - Override service account details #1002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
itsankit-google merged 1 commit into from
May 11, 2022
Merged

e2e tests CDAP-19055 - Override service account details #1002

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1005,6 +1005,20 @@
<forkCount>2</forkCount> <!--Number of JVM processes -->
<reuseForks>true</reuseForks>
<!--End configuration to run TestRunners in parallel-->
<environmentVariables>
<GOOGLE_APPLICATION_CREDENTIALS>
${GOOGLE_APPLICATION_CREDENTIALS}
</GOOGLE_APPLICATION_CREDENTIALS>
<SERVICE_ACCOUNT_TYPE>
${SERVICE_ACCOUNT_TYPE}
</SERVICE_ACCOUNT_TYPE>
<SERVICE_ACCOUNT_FILE_PATH>
${SERVICE_ACCOUNT_FILE_PATH}
</SERVICE_ACCOUNT_FILE_PATH>
<SERVICE_ACCOUNT_JSON>
${SERVICE_ACCOUNT_JSON}
</SERVICE_ACCOUNT_JSON>
</environmentVariables>
</configuration>
<executions>
<execution>
Expand Down
4 changes: 4 additions & 0 deletions src/e2e-test/features/bigquery/sink/BigQuerySinkError.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Enter BigQuery sink property GCS upload request chunk size "bqInvalidChunkSize"
Expand All @@ -32,6 +33,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
When Sink is BigQuery
Then Open BigQuery sink properties
Then Enter BigQuery property reference name
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "bqInvalidSinkDataset"
Then Enter BigQuery sink property table name
Then Verify the BigQuery validation error message for invalid property "dataset"
Expand All @@ -42,6 +44,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
When Sink is BigQuery
Then Open BigQuery sink properties
Then Enter BigQuery property reference name
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery property table "bqInvalidSinkTable"
Then Verify the BigQuery validation error message for invalid property "table"
Expand All @@ -54,6 +57,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"
Expand Down
8 changes: 8 additions & 0 deletions src/e2e-test/features/bigquery/sink/GCSToBigQuery.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,15 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
When Source is GCS
When Sink is BigQuery
Then Open GCS source properties
Then Override Service account details if set in environment variables
Then Enter the GCS source mandatory properties
Then Validate "GCS" plugin properties
Then Close the GCS properties
Then Open BigQuery sink properties
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Enter BiqQuery property encryption key name "cmekBQ" if cmek is enabled
Expand Down Expand Up @@ -41,6 +43,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
When Sink is BigQuery
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Override Service account details if set in environment variables
Then Enter GCS source property path "gcsCsvRangeFile"
Then Toggle GCS source property skip header to true
Then Select GCS property format "csv"
Expand All @@ -51,6 +54,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Toggle BigQuery sink property truncateTable to true
Expand Down Expand Up @@ -82,6 +86,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
When Sink is BigQuery
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Override Service account details if set in environment variables
Then Enter GCS source property path "gcsCsvRangeFile"
Then Toggle GCS source property skip header to true
Then Select GCS property format "csv"
Expand All @@ -92,6 +97,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Toggle BigQuery sink property truncateTable to true
Expand All @@ -118,6 +124,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
When Sink is BigQuery
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Override Service account details if set in environment variables
Then Enter GCS source property path "gcsCsvRangeFile"
Then Toggle GCS source property skip header to true
Then Select GCS property format "csv"
Expand All @@ -128,6 +135,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Enter BigQuery sink property GCS upload request chunk size "bqChunkSize"
Expand Down
16 changes: 10 additions & 6 deletions src/e2e-test/features/bigquery/sink/GCSToBigQuery_WithMacro.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Open GCS source properties
Then Enter GCS property reference name
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
Then Enter GCS property "serviceAccountFilePath" as macro argument "gcsServiceAccount"
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter GCS property "path" as macro argument "gcsSourcePath"
Then Enter GCS source property "skipHeader" as macro argument "gcsSkipHeader"
Then Enter GCS property "format" as macro argument "gcsFormat"
Expand All @@ -20,7 +22,9 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Enter BigQuery property reference name
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "bqServiceAccount"
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
Then Enter BigQuery property "table" as macro argument "bqTargetTable"
Then Enter BigQuery cmek property "encryptionKeyName" as macro argument "cmekBQ" if cmek is enabled
Expand All @@ -32,14 +36,14 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Save the pipeline
Then Preview and run the pipeline
Then Enter runtime argument value "projectId" for key "gcsProjectId"
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
Then Enter runtime argument value "projectId" for key "bqProjectId"
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
Then Enter runtime argument value "dataset" for key "bqDataset"
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
Then Enter runtime argument value "cmekBQ" for BigQuery cmek property key "cmekBQ" if BQ cmek is enabled
Expand All @@ -51,14 +55,14 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "projectId" for key "gcsProjectId"
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
Then Enter runtime argument value "projectId" for key "bqProjectId"
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
Then Enter runtime argument value "dataset" for key "bqDataset"
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
Then Enter runtime argument value "cmekBQ" for BigQuery cmek property key "cmekBQ" if BQ cmek is enabled
Expand Down
2 changes: 2 additions & 0 deletions src/e2e-test/features/bigquery/source/BigQuerySourceError.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
When Source is BigQuery
Then Open BigQuery source properties
Then Enter BigQuery property reference name
Then Override Service account details if set in environment variables
Then Enter the BigQuery source properties with incorrect property "<property>" value "<value>"
Then Validate BigQuery source incorrect property error for table "<property>" value "<value>"
Examples:
Expand Down Expand Up @@ -50,6 +51,7 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"
Expand Down
12 changes: 12 additions & 0 deletions src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Enter BigQuery source property filter "bqFilter"
Then Validate output schema with expectedSchema "bqSourceSchema"
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Open BigQuery sink properties
Then Override Service account details if set in environment variables
Then Enter the BigQuery sink mandatory properties
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Expand Down Expand Up @@ -44,13 +46,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Enter BigQuery source property filter "bqInvalidFilter"
Then Validate output schema with expectedSchema "bqSourceSchema"
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Open BigQuery sink properties
Then Override Service account details if set in environment variables
Then Enter the BigQuery sink mandatory properties
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Expand All @@ -68,13 +72,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Enter BigQuery source properties partitionStartDate and partitionEndDate
Then Validate output schema with expectedSchema "bqPartitionSourceSchema"
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Open BigQuery sink properties
Then Override Service account details if set in environment variables
Then Enter the BigQuery sink mandatory properties
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Expand Down Expand Up @@ -102,13 +108,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Enter BigQuery source properties partitionStartDate "bqFuturePartitionStartDate" and partitionEndDate "bqFuturePartitionEndDate"
Then Validate output schema with expectedSchema "bqPartitionSourceSchema"
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Open BigQuery sink properties
Then Override Service account details if set in environment variables
Then Enter the BigQuery sink mandatory properties
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Expand All @@ -130,6 +138,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
Expand All @@ -139,6 +148,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Toggle BigQuery sink property truncateTable to true
Expand Down Expand Up @@ -167,6 +177,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery source property table name
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
Expand All @@ -176,6 +187,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
Then Enter BigQuery property reference name
Then Enter BigQuery property projectId "projectId"
Then Enter BigQuery property datasetProjectId "projectId"
Then Override Service account details if set in environment variables
Then Enter BigQuery property dataset "dataset"
Then Enter BigQuery sink property table name
Then Toggle BigQuery sink property truncateTable to true
Expand Down
Loading