Skip to content

Commit e92e702

Browse files
itsankit-googleitsankit-google
authored
Merge pull request #1002 from sawantpritam/CDAP-19055
e2e tests CDAP-19055 - Override service account details
2 parents 8e44352 + ef616a3 commit e92e702

29 files changed

+206
-31
lines changed

‎pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1010,6 +1010,20 @@
10101010
<forkCount>2</forkCount> <!--Number of JVM processes -->
10111011
<reuseForks>true</reuseForks>
10121012
<!--End configuration to run TestRunners in parallel-->
1013+
<environmentVariables>
1014+
<GOOGLE_APPLICATION_CREDENTIALS>
1015+
${GOOGLE_APPLICATION_CREDENTIALS}
1016+
</GOOGLE_APPLICATION_CREDENTIALS>
1017+
<SERVICE_ACCOUNT_TYPE>
1018+
${SERVICE_ACCOUNT_TYPE}
1019+
</SERVICE_ACCOUNT_TYPE>
1020+
<SERVICE_ACCOUNT_FILE_PATH>
1021+
${SERVICE_ACCOUNT_FILE_PATH}
1022+
</SERVICE_ACCOUNT_FILE_PATH>
1023+
<SERVICE_ACCOUNT_JSON>
1024+
${SERVICE_ACCOUNT_JSON}
1025+
</SERVICE_ACCOUNT_JSON>
1026+
</environmentVariables>
10131027
</configuration>
10141028
<executions>
10151029
<execution>

‎src/e2e-test/features/bigquery/sink/BigQuerySinkError.feature‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
2121
Then Enter BigQuery property reference name
2222
Then Enter BigQuery property projectId "projectId"
2323
Then Enter BigQuery property datasetProjectId "projectId"
24+
Then Override Service account details if set in environment variables
2425
Then Enter BigQuery property dataset "dataset"
2526
Then Enter BigQuery sink property table name
2627
Then Enter BigQuery sink property GCS upload request chunk size "bqInvalidChunkSize"
@@ -32,6 +33,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
3233
When Sink is BigQuery
3334
Then Open BigQuery sink properties
3435
Then Enter BigQuery property reference name
36+
Then Override Service account details if set in environment variables
3537
Then Enter BigQuery property dataset "bqInvalidSinkDataset"
3638
Then Enter BigQuery sink property table name
3739
Then Verify the BigQuery validation error message for invalid property "dataset"
@@ -42,6 +44,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
4244
When Sink is BigQuery
4345
Then Open BigQuery sink properties
4446
Then Enter BigQuery property reference name
47+
Then Override Service account details if set in environment variables
4548
Then Enter BigQuery property dataset "dataset"
4649
Then Enter BigQuery property table "bqInvalidSinkTable"
4750
Then Verify the BigQuery validation error message for invalid property "table"
@@ -54,6 +57,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
5457
Then Enter BigQuery property reference name
5558
Then Enter BigQuery property projectId "projectId"
5659
Then Enter BigQuery property datasetProjectId "projectId"
60+
Then Override Service account details if set in environment variables
5761
Then Enter BigQuery property dataset "dataset"
5862
Then Enter BigQuery sink property table name
5963
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"

‎src/e2e-test/features/bigquery/sink/GCSToBigQuery.feature‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,15 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
77
When Source is GCS
88
When Sink is BigQuery
99
Then Open GCS source properties
10+
Then Override Service account details if set in environment variables
1011
Then Enter the GCS source mandatory properties
1112
Then Validate "GCS" plugin properties
1213
Then Close the GCS properties
1314
Then Open BigQuery sink properties
1415
Then Enter BigQuery property reference name
1516
Then Enter BigQuery property projectId "projectId"
1617
Then Enter BigQuery property datasetProjectId "projectId"
18+
Then Override Service account details if set in environment variables
1719
Then Enter BigQuery property dataset "dataset"
1820
Then Enter BigQuery sink property table name
1921
Then Enter BiqQuery property encryption key name "cmekBQ" if cmek is enabled
@@ -41,6 +43,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
4143
When Sink is BigQuery
4244
Then Open GCS source properties
4345
Then Enter GCS property projectId and reference name
46+
Then Override Service account details if set in environment variables
4447
Then Enter GCS source property path "gcsCsvRangeFile"
4548
Then Toggle GCS source property skip header to true
4649
Then Select GCS property format "csv"
@@ -51,6 +54,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
5154
Then Enter BigQuery property reference name
5255
Then Enter BigQuery property projectId "projectId"
5356
Then Enter BigQuery property datasetProjectId "projectId"
57+
Then Override Service account details if set in environment variables
5458
Then Enter BigQuery property dataset "dataset"
5559
Then Enter BigQuery sink property table name
5660
Then Toggle BigQuery sink property truncateTable to true
@@ -82,6 +86,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
8286
When Sink is BigQuery
8387
Then Open GCS source properties
8488
Then Enter GCS property projectId and reference name
89+
Then Override Service account details if set in environment variables
8590
Then Enter GCS source property path "gcsCsvRangeFile"
8691
Then Toggle GCS source property skip header to true
8792
Then Select GCS property format "csv"
@@ -92,6 +97,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
9297
Then Enter BigQuery property reference name
9398
Then Enter BigQuery property projectId "projectId"
9499
Then Enter BigQuery property datasetProjectId "projectId"
100+
Then Override Service account details if set in environment variables
95101
Then Enter BigQuery property dataset "dataset"
96102
Then Enter BigQuery sink property table name
97103
Then Toggle BigQuery sink property truncateTable to true
@@ -118,6 +124,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
118124
When Sink is BigQuery
119125
Then Open GCS source properties
120126
Then Enter GCS property projectId and reference name
127+
Then Override Service account details if set in environment variables
121128
Then Enter GCS source property path "gcsCsvRangeFile"
122129
Then Toggle GCS source property skip header to true
123130
Then Select GCS property format "csv"
@@ -128,6 +135,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
128135
Then Enter BigQuery property reference name
129136
Then Enter BigQuery property projectId "projectId"
130137
Then Enter BigQuery property datasetProjectId "projectId"
138+
Then Override Service account details if set in environment variables
131139
Then Enter BigQuery property dataset "dataset"
132140
Then Enter BigQuery sink property table name
133141
Then Enter BigQuery sink property GCS upload request chunk size "bqChunkSize"

‎src/e2e-test/features/bigquery/sink/GCSToBigQuery_WithMacro.feature‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
99
Then Open GCS source properties
1010
Then Enter GCS property reference name
1111
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
12-
Then Enter GCS property "serviceAccountFilePath" as macro argument "gcsServiceAccount"
12+
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
13+
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
14+
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
1315
Then Enter GCS property "path" as macro argument "gcsSourcePath"
1416
Then Enter GCS source property "skipHeader" as macro argument "gcsSkipHeader"
1517
Then Enter GCS property "format" as macro argument "gcsFormat"
@@ -20,7 +22,9 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
2022
Then Enter BigQuery property reference name
2123
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
2224
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
23-
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "bqServiceAccount"
25+
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
26+
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
27+
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
2428
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
2529
Then Enter BigQuery property "table" as macro argument "bqTargetTable"
2630
Then Enter BigQuery cmek property "encryptionKeyName" as macro argument "cmekBQ" if cmek is enabled
@@ -32,14 +36,14 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
3236
Then Save the pipeline
3337
Then Preview and run the pipeline
3438
Then Enter runtime argument value "projectId" for key "gcsProjectId"
35-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
39+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
40+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
3641
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
3742
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
3843
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
3944
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
4045
Then Enter runtime argument value "projectId" for key "bqProjectId"
4146
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
42-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
4347
Then Enter runtime argument value "dataset" for key "bqDataset"
4448
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
4549
Then Enter runtime argument value "cmekBQ" for BigQuery cmek property key "cmekBQ" if BQ cmek is enabled
@@ -51,14 +55,14 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
5155
Then Deploy the pipeline
5256
Then Run the Pipeline in Runtime
5357
Then Enter runtime argument value "projectId" for key "gcsProjectId"
54-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
58+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
59+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
5560
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
5661
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
5762
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
5863
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
5964
Then Enter runtime argument value "projectId" for key "bqProjectId"
6065
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
61-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
6266
Then Enter runtime argument value "dataset" for key "bqDataset"
6367
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
6468
Then Enter runtime argument value "cmekBQ" for BigQuery cmek property key "cmekBQ" if BQ cmek is enabled

‎src/e2e-test/features/bigquery/source/BigQuerySourceError.feature‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
1919
When Source is BigQuery
2020
Then Open BigQuery source properties
2121
Then Enter BigQuery property reference name
22+
Then Override Service account details if set in environment variables
2223
Then Enter the BigQuery source properties with incorrect property "<property>" value "<value>"
2324
Then Validate BigQuery source incorrect property error for table "<property>" value "<value>"
2425
Examples:
@@ -50,6 +51,7 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
5051
Then Enter BigQuery property reference name
5152
Then Enter BigQuery property projectId "projectId"
5253
Then Enter BigQuery property datasetProjectId "projectId"
54+
Then Override Service account details if set in environment variables
5355
Then Enter BigQuery property dataset "dataset"
5456
Then Enter BigQuery source property table name
5557
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"

‎src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
1010
Then Enter BigQuery property reference name
1111
Then Enter BigQuery property projectId "projectId"
1212
Then Enter BigQuery property datasetProjectId "projectId"
13+
Then Override Service account details if set in environment variables
1314
Then Enter BigQuery property dataset "dataset"
1415
Then Enter BigQuery source property table name
1516
Then Enter BigQuery source property filter "bqFilter"
1617
Then Validate output schema with expectedSchema "bqSourceSchema"
1718
Then Validate "BigQuery" plugin properties
1819
Then Close the BigQuery properties
1920
Then Open BigQuery sink properties
21+
Then Override Service account details if set in environment variables
2022
Then Enter the BigQuery sink mandatory properties
2123
Then Validate "BigQuery" plugin properties
2224
Then Close the BigQuery properties
@@ -44,13 +46,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
4446
Then Enter BigQuery property reference name
4547
Then Enter BigQuery property projectId "projectId"
4648
Then Enter BigQuery property datasetProjectId "projectId"
49+
Then Override Service account details if set in environment variables
4750
Then Enter BigQuery property dataset "dataset"
4851
Then Enter BigQuery source property table name
4952
Then Enter BigQuery source property filter "bqInvalidFilter"
5053
Then Validate output schema with expectedSchema "bqSourceSchema"
5154
Then Validate "BigQuery" plugin properties
5255
Then Close the BigQuery properties
5356
Then Open BigQuery sink properties
57+
Then Override Service account details if set in environment variables
5458
Then Enter the BigQuery sink mandatory properties
5559
Then Validate "BigQuery" plugin properties
5660
Then Close the BigQuery properties
@@ -68,13 +72,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
6872
Then Enter BigQuery property reference name
6973
Then Enter BigQuery property projectId "projectId"
7074
Then Enter BigQuery property datasetProjectId "projectId"
75+
Then Override Service account details if set in environment variables
7176
Then Enter BigQuery property dataset "dataset"
7277
Then Enter BigQuery source property table name
7378
Then Enter BigQuery source properties partitionStartDate and partitionEndDate
7479
Then Validate output schema with expectedSchema "bqPartitionSourceSchema"
7580
Then Validate "BigQuery" plugin properties
7681
Then Close the BigQuery properties
7782
Then Open BigQuery sink properties
83+
Then Override Service account details if set in environment variables
7884
Then Enter the BigQuery sink mandatory properties
7985
Then Validate "BigQuery" plugin properties
8086
Then Close the BigQuery properties
@@ -102,13 +108,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
102108
Then Enter BigQuery property reference name
103109
Then Enter BigQuery property projectId "projectId"
104110
Then Enter BigQuery property datasetProjectId "projectId"
111+
Then Override Service account details if set in environment variables
105112
Then Enter BigQuery property dataset "dataset"
106113
Then Enter BigQuery source property table name
107114
Then Enter BigQuery source properties partitionStartDate "bqFuturePartitionStartDate" and partitionEndDate "bqFuturePartitionEndDate"
108115
Then Validate output schema with expectedSchema "bqPartitionSourceSchema"
109116
Then Validate "BigQuery" plugin properties
110117
Then Close the BigQuery properties
111118
Then Open BigQuery sink properties
119+
Then Override Service account details if set in environment variables
112120
Then Enter the BigQuery sink mandatory properties
113121
Then Validate "BigQuery" plugin properties
114122
Then Close the BigQuery properties
@@ -130,6 +138,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
130138
Then Enter BigQuery property reference name
131139
Then Enter BigQuery property projectId "projectId"
132140
Then Enter BigQuery property datasetProjectId "projectId"
141+
Then Override Service account details if set in environment variables
133142
Then Enter BigQuery property dataset "dataset"
134143
Then Enter BigQuery source property table name
135144
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
@@ -139,6 +148,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
139148
Then Enter BigQuery property reference name
140149
Then Enter BigQuery property projectId "projectId"
141150
Then Enter BigQuery property datasetProjectId "projectId"
151+
Then Override Service account details if set in environment variables
142152
Then Enter BigQuery property dataset "dataset"
143153
Then Enter BigQuery sink property table name
144154
Then Toggle BigQuery sink property truncateTable to true
@@ -167,6 +177,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
167177
Then Enter BigQuery property reference name
168178
Then Enter BigQuery property projectId "projectId"
169179
Then Enter BigQuery property datasetProjectId "projectId"
180+
Then Override Service account details if set in environment variables
170181
Then Enter BigQuery property dataset "dataset"
171182
Then Enter BigQuery source property table name
172183
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
@@ -176,6 +187,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
176187
Then Enter BigQuery property reference name
177188
Then Enter BigQuery property projectId "projectId"
178189
Then Enter BigQuery property datasetProjectId "projectId"
190+
Then Override Service account details if set in environment variables
179191
Then Enter BigQuery property dataset "dataset"
180192
Then Enter BigQuery sink property table name
181193
Then Toggle BigQuery sink property truncateTable to true

0 commit comments

Comments
 (0)