Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/assets/source-datastores/catalog/aborted.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/source-datastores/catalog/catalog.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/source-datastores/catalog/daily.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/source-datastores/catalog/hourly.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/source-datastores/catalog/run.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/source-datastores/catalog/weekly.png
Binary file added docs/assets/source-datastores/profile/daily.png
Binary file added docs/assets/source-datastores/profile/hourly.png
Binary file added docs/assets/source-datastores/profile/next.png
Binary file added docs/assets/source-datastores/profile/run.png
Binary file added docs/assets/source-datastores/profile/search.png
Binary file added docs/assets/source-datastores/profile/weekly.png
Binary file added docs/assets/source-datastores/scan/activity.png
Binary file added docs/assets/source-datastores/scan/advanced.png
Binary file added docs/assets/source-datastores/scan/anomalyy.png
Binary file added docs/assets/source-datastores/scan/click-next.png
Binary file added docs/assets/source-datastores/scan/computed.png
Binary file added docs/assets/source-datastores/scan/daily.png
Binary file added docs/assets/source-datastores/scan/drop-down.png
Binary file added docs/assets/source-datastores/scan/hourly.png
Binary file added docs/assets/source-datastores/scan/monthly.png
Binary file added docs/assets/source-datastores/scan/next.png
Binary file added docs/assets/source-datastores/scan/nextt.png
Binary file added docs/assets/source-datastores/scan/result.png
Binary file added docs/assets/source-datastores/scan/run-now.png
Binary file added docs/assets/source-datastores/scan/run.png
Binary file added docs/assets/source-datastores/scan/running.png
Binary file added docs/assets/source-datastores/scan/sacn.png
Binary file added docs/assets/source-datastores/scan/schedule.png
Binary file added docs/assets/source-datastores/scan/side-menu.png
Binary file added docs/assets/source-datastores/scan/specific.png
Binary file added docs/assets/source-datastores/scan/success-2.png
Binary file added docs/assets/source-datastores/scan/sucess.png
Binary file added docs/assets/source-datastores/scan/tag.png
Binary file added docs/assets/source-datastores/scan/warning.png
Binary file added docs/assets/source-datastores/scan/weekly.png
4 changes: 2 additions & 2 deletions docs/changelog-2023.md
Original file line number Diff line number Diff line change
Expand Up @@ -806,8 +806,8 @@

- ![Screenshot](assets/changelog/absolute-relative-change-limit.png#only-light){: style="height:100px"}
![Screenshot](assets/changelog/absolute-relative-change-limit.png#only-dark){: style="height:100px"}
- The [`Absolute Change Limit`](/checks/metric-check.md#comparison-options/) check is designed to monitor changes in a field's value by a fixed amount. If the field's value changes by more than the specified limit since the last applicable scan, an anomaly is generated.
- The [`Relative Change Limit`](/checks/metric-check.md#comparison-options/) check works similarly but tracks changes in terms of percentages. If the change in a field's value exceeds the defined percentage limit since the last applicable scan, an anomaly is generated.
- The [`Absolute Change Limit`](/data-quality-checks/metric-check.md#comparison-options/) check is designed to monitor changes in a field's value by a fixed amount. If the field's value changes by more than the specified limit since the last applicable scan, an anomaly is generated.
- The [`Relative Change Limit`](/data-quality-checks/metric-check.md#comparison-options/) check works similarly but tracks changes in terms of percentages. If the change in a field's value exceeds the defined percentage limit since the last applicable scan, an anomaly is generated.

#### General Fixes

Expand Down
138 changes: 87 additions & 51 deletions docs/fields/field-status/concepts/field-status-faq.md

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions docs/settings/security/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ In Qualytics, every user is assigned a role: ```Admin```,```Manager``` or ```Mem
| | Team (Manage) | Manage teams by adding or removing members, and setting team-specific permissions. |
| | Health | Monitor the system’s health status and performance metrics. |
| | Restart Analytical Engine | Restart the analytics engine to refresh data processing or resolve issues. |
| Field Status | Manage | Full access to all field status operations (mask, unmask, exclude, restore, delete, merge). |
| | Masking Audit Log | View the masking audit log to review all masked value reveal actions. |
| Datastore Grouping | Manage | Create, edit, and delete datastore groups. |
| | Assign | Assign or remove datastores from groups (requires Editor team permission on the datastore). |

- **Manager**: Manager role has limited administrative access over global assets but remains subject to team permissions when interacting with datastores. Managers cannot manage user roles or teams. They can list all datastores (but cannot view their content without explicit team permission) and create datastores for teams where they have Editor permission. Additionally, Managers on a team with Editor permission can manage datastore teams. They can manage global assets such as Tags, Templates, and Notifications but do not have the ability to manage user accounts or team permissions like Admins.

Expand Down Expand Up @@ -65,6 +69,9 @@ In Qualytics, every user is assigned a role: ```Admin```,```Manager``` or ```Mem
| | Sync | Sync external data with the system to ensure the most up-to-date information.|
| Health | View | View the health status of the system to monitor performance and stability. |
| API only (ATM) | Transaction History | View the history of transactions made via the API for auditing and tracking.|
| Field Status | Manage | Manage field status operations (mask, unmask, exclude, restore, delete, merge) on datastores where the user has Editor team permission. |
| Datastore Grouping | Manage | Create, edit, and delete datastore groups. |
| | Assign | Assign or remove datastores from groups (requires Editor team permission on the datastore). |

- **Member**: Members are normal users with access explicitly granted to them, usually inherited from the teams they are assigned to.

Expand All @@ -81,6 +88,8 @@ In Qualytics, every user is assigned a role: ```Admin```,```Manager``` or ```Mem
| | Restore | Reactivate previously revoked tokens. |
| | Delete | Permanently remove tokens. |
| | View | Access and review all token details. |
| Field Status | Manage | Manage field status operations (mask, unmask, exclude, restore, delete, merge) on datastores where the user has Editor team permission. |
| Datastore Grouping | Assign | Assign or remove datastores from groups (requires Editor team permission on the datastore). Members cannot create, edit, or delete groups. |

## Manage Users

Expand Down
19 changes: 19 additions & 0 deletions docs/settings/security/team-permissions.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,15 @@ Legend:
| Manage Tags (create / update / delete) | ❌ | ❌ | ✅ | ✅ | ✅ |
| Assign Tags to Assets (Datastore, Container, Field, Check, Anomaly) | ❌ | ✅ | ✅ | ✅ |✅ |
| Assign Tags in Flows | ❌ | ❌ | ✅ | ✅ | ✅ |
| View Datastore Groups | ✅ | ✅ | ✅ | ✅ | ✅ |
| Assign Datastore to Group | ❌ | ❌ | ❌ | ❌ | ✅ |
| View Field Status | ✅ | ✅ | ✅ | ✅ | ✅ |
| Mask / Unmask Field | ❌ | ❌ | ❌ | ❌ | ✅ |
| Exclude Field | ❌ | ❌ | ❌ | ❌ | ✅ |
| Restore Field | ❌ | ❌ | ❌ | ❌ | ✅ |
| Delete Field | ❌ | ❌ | ❌ | ❌ | ✅ |
| Merge Fields | ❌ | ❌ | ❌ | ❌ | ✅ |
| View Masked Field Values | ❌ | ❌ | ❌ | ❌ | ✅ |

## Add Team

Expand Down Expand Up @@ -91,6 +100,12 @@ Editor role allows users to manage datastore functions comprehensively. They can
| | Delete Computed | ❌ | ✅ |
| **Field Context** | Edit Field Context | ❌ | ✅ |
| | Delete Field Context | ❌ | ✅ |
| **Field Status** | Mask / Unmask Field | ❌ | ✅ |
| | Exclude Field | ❌ | ✅ |
| | Restore Field | ❌ | ✅ |
| | Delete Field | ❌ | ✅ |
| | Merge Fields | ❌ | ✅ |
| | View Masked Field Values | ✅ | ✅ |

#### Author

Expand All @@ -107,6 +122,7 @@ Author role focuses on managing checks within the system. Users can activate, va
| | Edit Metadata | ❌ | ✅ |
| | Anomalies | ❌ | ✅ |
| **Anomalies** | Change Status of Anomalies | ❌ | ✅ |
| **Field Status** | View Field Status | ✅ | ❌ |

#### Drafter

Expand All @@ -121,6 +137,7 @@ Drafter role is designed specifically for adding and saving data within the syst
| **Profiles** | Add Check | ❌ | ✅ |
| **Checks** | Create as Draft | ❌ | ✅ |
| **Field Context** | Add Check | ❌ | ✅ |
| **Field Status** | View Field Status | ✅ | ❌ |

#### Viewer

Expand All @@ -137,6 +154,7 @@ Viewer role is focused on viewing anomalies within the system and creating notes
| | Preview | ✅ | ❌ |
| **Explore** | Anomalies | ✅ | ❌ |
| | Source Records | ✅ | ❌ |
| **Field Status** | View Field Status | ✅ | ❌ |

#### Reporter

Expand All @@ -155,6 +173,7 @@ Reporter role provides access to all report-related information, including dashb
| | Checks | ✅ | ❌ |
| | Anomalies | ✅ | ❌ |
| | Fields (Containers) | ✅ | ❌ |
| **Field Status** | View Field Status | ✅ | ❌ |
| **Enrichment Datastores**| List | ✅ | ❌ |
| **Explore** | Insights | ✅ | ❌ |
| | Activity | ✅ | ❌ |
Expand Down
6 changes: 3 additions & 3 deletions docs/source-datastore/delete-datastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@

**Step 1**: Click on the **Delete** option in the settings icon.

![delete](../assets/datastore-settings/delete-datastore/delete-light-22.png)
![delete](../assets/source-datastores/datastore-settings/delete-datastore/delete-22.png)

**Step 2**: A modal window titled **Delete Datastore** will appear.

![delete-datastore](../assets/datastore-settings/delete-datastore/delete-datastore-light-23.png)
![delete-datastore](../assets/source-datastores/datastore-settings/delete-datastore/delete-datastore-23.png)

**Step 3**: Enter the name of the datastore in the given field (confirmation check) and then click on the **I’M SURE, DELETE THIS DATASTORE** button to delete the datastore.

![confirm-delete](../assets/datastore-settings/delete-datastore/confirm-delete-light-24.png)
![confirm-delete](../assets/source-datastores/datastore-settings/delete-datastore/confirm-delete-24.png)

After clicking the **I’M SURE, DELETE THIS DATASTORE** button, a success notification appears confirming the deletion.
8 changes: 4 additions & 4 deletions docs/source-datastore/edit-datastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,20 @@

**Step 1:** Click on the **Edit** option.

![edit-datastore](../assets/datastore-settings/edit-datastore/edit-datastore-light-3.png)
![edit-datastore](../assets/source-datastores/datastore-settings/edit-datastore/edit-datastore-3.png)

**Step 2:** After selecting the **Edit** option, a modal window will appear, displaying the connection details. This window allows you to modify any specific connection details.

![connection-details](../assets/datastore-settings/edit-datastore/connection-details-light-4.png)
![connection-details](../assets/source-datastores/datastore-settings/edit-datastore/connection-details-4.png)

**Step 3:** After editing the connection details, click on the **Test Connection** button to check and verify its connection.

![test](../assets/datastore-settings/edit-datastore/test-light.png)
![test](../assets/source-datastores/datastore-settings/edit-datastore/test.png)

If the credentials and provided connection details are verified, a success message will be displayed indicating that the connection has been verified.

**Step 4:** Click on the **Save** button.

![save-datastore](../assets/datastore-settings/edit-datastore/save-datastore-light-5.png)
![save-datastore](../assets/source-datastores/datastore-settings/edit-datastore/save-datastore-5.png)

After clicking on the **Save** button, a success notification appears on the screen showing the action was completed successfully.
12 changes: 6 additions & 6 deletions docs/source-datastore/external-scan.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,35 +8,35 @@ Let’s get started 🚀

**Step 1:** Select a source datastore from the side menu to perform the external scan operation.

![datastore](../assets//external-scan/datastore-light-1.png)
![datastore](../assets/source-datastores/external-scan/datastore-1.png)

**Step 2:** After selecting your preferred source datastore, you will be taken to the details page. From there, click on **"Tables"** and select the table you want to perform the external scan operation on.

!!! note
This example is based on a JDBC table, but the same steps apply to a DFS as well. For DFS source datastores, you will need to click on **"File Patterns"** and select a File Pattern to run the external scan.

![tables](../assets//external-scan/tables-light-2.png)
![tables](../assets/source-datastores/external-scan/tables-2.png)

For demonstration purposes, we have selected the **“CUSTOMER”** table.

![container](../assets//external-scan/container-light-3.png)
![container](../assets/source-datastores/external-scan/container-3.png)

## External Scan Configuration

**Step 1:** Click on the **“Run”** button and select the **“External Scan”** option.

![external-scan](../assets//external-scan/external-scan-light-4.png)
![external-scan](../assets/source-datastores/external-scan/external-scan-4.png)

**Step 2:** After selecting the **"External Scan"** option, a modal window will appear with an input for uploading your external file. After uploading the file, click the **“Run”** button to start the operation.

![external-file](../assets//external-scan/external-file-light-5.png)
![external-file](../assets/source-datastores/external-scan/external-file-5.png)

!!! note
An External Scan operation supports the following file formats: CSV, XLSX, and XLS.

**Step 3:** After clicking the **"Run"** button, the external scan operation will begin, and you will receive a confirmation message if the operation is successfully triggered.

![success](../assets//external-scan/success-light-6.png)
![success](../assets/source-datastores/external-scan/success-6.png)

## Supported File Formats

Expand Down
20 changes: 10 additions & 10 deletions docs/source-datastore/link-enrichment.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

**Step 1:** Click on the **Enrichment** from the dropdown list.

![enrichment](../assets/datastore-settings/link-enrichment/enrichment-light-6.png)
![enrichment](../assets/source-datastores/datastore-settings/link-enrichment/enrichment-6.png)

A modal window **Link Enrichment Datastore** will appear, providing you with two options to link an **enrichment datastore**.

![link-enrichment](../assets/datastore-settings/link-enrichment/link-enrichment-light-7.png)
![link-enrichment](../assets/source-datastores/datastore-settings/link-enrichment/link-enrichment-7.png)

| REF. | FIELDS | ACTIONS |
| :---- | :---- | :---- |
Expand All @@ -23,11 +23,11 @@ If the toggle for **Add new connection** is turned on, then this will prompt you

**Step 1:** Click on the caret button and select **Add Enrichment Datastore**.

![caret](../assets/datastore-settings/link-enrichment/caret-light-8.png)
![caret](../assets/source-datastores/datastore-settings/link-enrichment/caret-8.png)

A modal window **Link Enrichment Datastore** will appear. Enter the following details to create an enrichment datastore with a new connection.

![window](../assets/datastore-settings/link-enrichment/window-light.png)
![window](../assets/source-datastores/datastore-settings/link-enrichment/window.png)

| REF. | FIELDS | ACTIONS |
| :---- | :---- | :---- |
Expand All @@ -44,17 +44,17 @@ A modal window **Link Enrichment Datastore** will appear. Enter the following de
!!! note
Connection details can vary from datastore to datastore. For illustration, we have demonstrated linking BigQuery as a new enrichment datastore.

![select-new-enrichment](../assets/datastore-settings/link-enrichment/select-new-enrichment-light-9.png)
![select-new-enrichment](../assets/source-datastores/datastore-settings/link-enrichment/select-new-enrichment-9.png)

**Step 3:** After adding the source datastore details, click on the **Test Connection** button to check and verify its connection.

![connector-creds](../assets/datastore-settings/link-enrichment/connector-creds-light-10.png)
![connector-creds](../assets/source-datastores/datastore-settings/link-enrichment/connector-creds-10.png)

If the credentials and provided details are verified, a success message will be displayed indicating that the connection has been verified.

**Step 4:** Click on the **Save** button.

![save-enrichment](../assets/datastore-settings/link-enrichment/save-enrichment-light-11.png)
![save-enrichment](../assets/source-datastores/datastore-settings/link-enrichment/save-enrichment-11.png)

After clicking on the **Save** button, a success notification appears on the screen showing the action was completed successfully.

Expand All @@ -64,11 +64,11 @@ If the **Use an existing enrichment datastore** option is selected from the drop

**Step 1:** Click on the caret button and select **Use Enrichment Datastore**.

![caret](../assets/datastore-settings/link-enrichment/caret2-light.png)
![caret](../assets/source-datastores/datastore-settings/link-enrichment/caret2.png)

**Step 2:** A modal window **Link Enrichment Datastore** will appear. Add a prefix name and select an existing enrichment datastore from the dropdown list.

![modal](../assets/datastore-settings/link-enrichment/modal-light.png)
![modal](../assets/source-datastores/datastore-settings/link-enrichment/modal.png)

| REF. | FIELDS | ACTIONS |
| :---- | :---- | :---- |
Expand All @@ -80,7 +80,7 @@ If the **Use an existing enrichment datastore** option is selected from the drop

**Step 3:** View and check the connection details of the enrichment datastore and click on the **Save** button.

![save](../assets/datastore-settings/link-enrichment/save-light-15.png)
![save](../assets/source-datastores/datastore-settings/link-enrichment/save-15.png)

After clicking on the **Save** button, a success notification appears on the screen showing the action was completed successfully.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,28 @@ In this documentation, we will explore the steps to assign a tag to the datastor

**Step 1**: Login in to your Qualytics account and select the **datastore** from the left menu on which you want to assign a tag.

![add-datastore](../assets/assign-tags/add-datastore-light-1.png)
![add-datastore](../../assets/source-datastores/navigation-and-organization/assign-tags/add-datastore-1.png)

**Step 2**: Click on **Assign Tag to this Datastore** located at the bottom-left corner of the interface.

![assign-tag](../assets/assign-tags/assign-tag-light-2.png)
![assign-tag](../../assets/source-datastores/navigation-and-organization/assign-tags/assign-tag-2.png)

**Step 3**: A drop-up menu will appear, providing you with a list of tags. Assign an appropriate **tag** to your datastore to simplify sorting, accessing, and managing data.

![select-tag](../assets/assign-tags/select-tag-light-3.png)
![select-tag](../../assets/source-datastores/navigation-and-organization/assign-tags/select-tag-3.png)

You can also create a new tag by clicking on the call to action (➕) button.

![call-to-action](../assets/assign-tags/action-light-4.png)
![call-to-action](../../assets/source-datastores/navigation-and-organization/assign-tags/action-4.png)

A modal window will appear, providing the options to create the tag. Enter the required values to get started.

![modal-window](../assets/assign-tags/modal-window-light-5.png)
![modal-window](../../assets/source-datastores/navigation-and-organization/assign-tags/modal-window-5.png)

For more information on creating tags, refer to the [Add Tag section](../tags/add-tag.md).
For more information on creating tags, refer to the [Add Tag section](../../tags/add-tag.md).

**Step 4**: Once you have assigned a tag, the tag will be instantly labeled on your source Datastore, and all related records will be updated.

For demonstration, we have assigned the **High** tag for the Snowflake source datastore **Covid-19 Data**, so it will automatically be applied to all related tables and checks within the datastore.

![tag-snowflake](../assets/assign-tags/tag-snowflake-light-6.png)
![tag-snowflake](../../assets/source-datastores/navigation-and-organization/assign-tags/tag-snowflake-6.png)
Loading
Loading