Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 54 additions & 6 deletions packages/microsoft_sqlserver/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@

The Microsoft SQL Server integration package allows you to search, observe, and visualize the SQL Server audit logs, as well as performance and transaction log metrics, through Elasticsearch.

This integration supports **SQL Server 2019, 2022, and 2025**. Where newer versions expose additional DMVs or columns, the integration uses version-safe dynamic SQL (`EXEC sp_executesql` guarded by `OBJECT_ID` or `SERVERPROPERTY('ProductMajorVersion')` checks) so the same configuration runs on all supported versions without errors. Fields that are only available on newer versions are documented individually.

## Data streams

The Microsoft SQL Server integration collects two types of data streams: logs and metrics.

**Log** data streams provide records of events happening in Microsoft SQL Server:

* `audit`: Events from the configured Windows event log channel, providing detailed auditing information. See [SQL Server Audit](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine?view=sql-server-ver15).
* `audit`: Events from the configured Windows event log channel, providing detailed auditing information. See [SQL Server Audit](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine?view=sql-server-ver16).
* `logs`: Error logs created by the Microsoft SQL server for troubleshooting and system events.

Other log sources, such as files, are not supported.
Expand All @@ -32,19 +34,22 @@

Before you can start sending data to Elastic, make sure you have the necessary Microsoft SQL Server permissions.

If you browse Microsoft Developer Network (MSDN) for the following tables, you will find a "Permissions" section that defines the permission needed for each table (for example, [the "Permissions" section on the `sys.dm_db_log_space_usage`](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver15#permissions) page).
If you browse Microsoft Developer Network (MSDN) for the following tables, you will find a "Permissions" section that defines the permission needed for each table (for example, [the "Permissions" section on the `sys.dm_db_log_space_usage`](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver16#permissions) page).

1. `transaction_log`:
- [sys.databases](https://learn.microsoft.com/en-us/sql/relational-databases/system-compatibility-views/sys-sysdatabases-transact-sql?view=sql-server-ver16)
- [sys.dm_db_log_space_usage](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-space-usage-transact-sql?view=sql-server-ver16)
- [sys.dm_db_log_stats (DB_ID)](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-db-log-stats-transact-sql?view=sql-server-ver16) (Available on SQL Server (MSSQL) 2016 (13.x) SP 2 and later)
- [sys.dm_tran_persistent_version_store_stats](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-tran-persistent-version-store-stats?view=sql-server-ver16) (**SQL Server 2022+ only** — Accelerated Database Recovery PVS health metrics; returns no data on earlier versions)
2. `performance`:
- [sys.dm_os_performance_counters](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver16)
3. `availability_groups`:
- [sys.availability_groups](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-availability-groups-transact-sql?view=sql-server-ver16)
- [sys.dm_hadr_availability_group_states](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-hadr-availability-group-states-transact-sql)
- [sys.availability_replicas](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-availability-replicas-transact-sql?view=sql-server-ver16)
- [sys.dm_hadr_availability_replica_states](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-hadr-availability-replica-states-transact-sql)

Please make sure the user has the permissions to system as well as user-defined databases. For the particular user used in the integration, the following requirements are met:

Check warning on line 52 in packages/microsoft_sqlserver/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint Documentation

Elastic.DontUse: Don't use 'Please'.

User setup options:

Expand Down Expand Up @@ -95,11 +100,33 @@

There are several levels of auditing for SQL Server, depending on government or standards requirements for your installation. The SQL Server Audit feature enables you to audit server-level and database-level groups of events and individual events.

For more information on the different audit levels, refer to [SQL Server Audit Action Groups and Actions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-action-groups-and-actions?view=sql-server-ver15).
Then to enable auditing for SQL Server, refer to these [instructions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/create-a-server-audit-and-server-audit-specification?view=sql-server-ver15).
For more information on the different audit levels, refer to [SQL Server Audit Action Groups and Actions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-action-groups-and-actions?view=sql-server-ver16).
Then to enable auditing for SQL Server, refer to these [instructions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/create-a-server-audit-and-server-audit-specification?view=sql-server-ver16).

>Note: For the integration package to be able to read and send audit events the event target must be configured to be Windows event log.

**SQL Server 2022+ audit fields** — automatically extracted from the audit payload when present:

| Field | Notes |
|---|---|
| `sqlserver.audit.obo_middle_tier_app_id` | OAuth On-Behalf-Of middle-tier application ID. NULL when not an OAuth delegated flow. |
| `sqlserver.audit.obo_middle_tier_app_name` | OAuth On-Behalf-Of middle-tier application display name. |
| `sqlserver.audit.external_policy_permissions_checked` | Azure Purview external authorization policy decision. NULL when not configured. |

**SQL Server 2022+ audit action groups** recognized by the ingest pipeline:

| Action ID | Action Group | Notes |
|---|---|---|
| `LDGR` | `LEDGER_OPERATION_GROUP` | CREATE/ALTER/DROP ledger tables and VERIFY LEDGER operations. |
| `EXPO` | `EXTERNAL_POLICY_OPERATION_GROUP` | Azure Purview external policy check operations. |

**SQL Server 2025+ audit action groups**:

| Action ID | Action Group | Notes |
|---|---|---|
| `VECT` | `VECTOR_OPERATION_GROUP` | Vector index CRUD operations. |
| `ERRE` | `EXTERNAL_REST_ENDPOINT_GROUP` | Calls to `sp_invoke_external_rest_endpoint` and native AI functions. |

#### Audit events

Collects SQL Server audit events from the specified windows event log channel.
Expand All @@ -121,13 +148,22 @@

The feature `merge_results` has been introduced in 8.4 beats which creates a single event by combining the metrics in a single event. For more details, refer to [SQL module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-sql.html#_example_merge_multiple_queries_into_a_single_event).

Read more in [instructions about each performance counter metrics](https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver15).
Read more in [instructions about each performance counter metrics](https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver16).


**SQL Server 2025+ performance counters**:

| Field | Counter Name | Object | Notes |
|---|---|---|---|
| `mssql.metrics.external_rest_endpoint_total_executions` | `Total executions` | `SQLServer:External REST Endpoint` | Cumulative count of `sp_invoke_external_rest_endpoint` calls since last restart. Use `rate()` in Kibana to derive calls/sec. Returns no data on SQL Server 2022 and earlier. |

Check notice on line 158 in packages/microsoft_sqlserver/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint Documentation

Elastic.Wordiness: Consider using 'because' instead of 'since'.

#### Transaction log metrics

The system-level database `transaction_log` metrics for SQL Server instances are collected by default. Metrics for user-level databases can be collected by specifying a list of user databases or by enabling the `Fetch from all databases` toggle to collect metrics from all databases on the server.

Read more in [instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver15).
On **SQL Server 2022 and later**, additional Accelerated Database Recovery (ADR) Persistent Version Store (PVS) metrics are collected from `sys.dm_tran_persistent_version_store_stats`. These fields (`mssql.metrics.pvs_*`) are absent on SQL Server 2019 and earlier — no errors occur, they simply do not appear in events.

Read more in [instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver16).

#### Fetch from all databases

Expand All @@ -145,6 +181,18 @@

- `sys.availability_groups`
- `sys.dm_hadr_availability_group_states`
- `sys.availability_replicas`
- `sys.dm_hadr_availability_replica_states`

**Version-conditional fields** — each column is guarded by a `sys.columns` existence check in dynamic SQL; returns `NULL` when the column is absent on the running SQL Server version:

| Field | Available On | Notes |
|---|---|---|
| `mssql.metrics.basic_availability_group` | SQL Server 2016–2022 | 1 = Basic AG (Standard Edition, single DB, no readable secondary). **Removed in SQL Server 2025** — returns `NULL` on SQL Server 2025 and later. |
| `mssql.metrics.is_distributed` | SQL Server 2016+ | 1 = Distributed Availability Group spanning two WSFC clusters. |
| `mssql.metrics.is_contained` | SQL Server 2022+ | 1 = Contained Availability Group (own system databases/logins). `NULL` on SQL Server 2019 and earlier. |

Per-replica state fields (`role`, `connected_state`, `replica_synchronization_health`, `last_connect_error_*`) are collected from `sys.dm_hadr_availability_replica_states` on all supported versions.

**Note:** Always On Availability Groups must be enabled on your SQL Server instance for this dataset to collect metrics. This feature is available in SQL Server Enterprise and Standard editions (with limitations in Standard).

Expand Down
4 changes: 2 additions & 2 deletions packages/microsoft_sqlserver/_dev/deploy/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
ARG IMAGE=${IMAGE:-mcr.microsoft.com/mssql/server:2019-latest}
FROM ${IMAGE}
ARG MSSQL_VERSION=${MSSQL_VERSION:-2025-latest}
FROM mcr.microsoft.com/mssql/server:${MSSQL_VERSION}

ENV ACCEPT_EULA='Y'
ENV SA_PASSWORD='1234_asdf'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
version: '2.3'
services:
microsoft_sqlserver:
image: mcr.microsoft.com/mssql/server:2019-latest
user: root
# SQL Server 2025 has a strict CPU-topology assertion in sosnumap.cpp:
# NUMA_nodes × cores_per_node == total_logical_processors
# On Intel hybrid CPUs (P-cores with HT + E-cores without HT) the full
# host topology is non-uniform and triggers a fatal crash in SQL Server 2025.
# Pinning to 4 logical CPUs (first 2 P-cores, both HT threads each) presents
# a uniform 1 NUMA × 2 cores × 2 threads = 4 topology that satisfies the assertion.
# SQL Server 2019 has no such assertion and works without this restriction.
cpuset: "0-3"
build:
context: .
args:
MSSQL_VERSION: ${MSSQL_VERSION:-2019-latest}
MSSQL_VERSION: ${MSSQL_VERSION:-2025-latest}
ports:
- 1433
volumes:
Expand Down
6 changes: 4 additions & 2 deletions packages/microsoft_sqlserver/_dev/deploy/variants.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
variants:
MSSQL_2019-latest:
IMAGE: 2019-latest
default: 2019-latest
MSSQL_VERSION: 2019-latest
MSSQL_2025-latest:
MSSQL_VERSION: 2025-latest
default: MSSQL_2025-latest
16 changes: 16 additions & 0 deletions packages/microsoft_sqlserver/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,20 @@
# newer versions go on top
- version: "2.17.0"
changes:
- description: >
Add SQL Server 2022 and 2025 support.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this readable?

availability_groups: collect per-replica state from sys.dm_hadr_availability_replica_states;
add is_contained (SQL 2022+), basic_availability_group (SQL 2016–2022, removed in SQL Server 2025),
and is_distributed (SQL 2016+) fields via sys.columns-guarded dynamic SQL so all three columns
return NULL instead of an error when absent on the running SQL Server version.
audit: add obo_middle_tier_app_id, obo_middle_tier_app_name, external_policy_permissions_checked fields (SQL 2022+);
add LEDGER_OPERATION_GROUP, EXTERNAL_POLICY_OPERATION_GROUP (SQL 2022), VECTOR_OPERATION_GROUP, and
EXTERNAL_REST_ENDPOINT_GROUP (SQL 2025) audit action group mappings.
performance: add external_rest_endpoint_total_executions counter via SQLServer:External REST Endpoint (SQL 2025+).
transaction_log: add ADR Persistent Version Store (PVS) metrics from
sys.dm_tran_persistent_version_store_stats (SQL 2022+) via version-safe dynamic SQL.
type: enhancement
link: https://github.com/elastic/integrations/pull/18262
- version: "2.16.0"
changes:
- description: Add health metrics for Always On Availability Groups.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1127,6 +1127,34 @@ processors:
type:
- access
action: unsafe-assembly
# SQL Server 2022+ action groups
"LDGR":
# https://learn.microsoft.com/en-us/sql/relational-databases/security/ledger/ledger-overview
# Fires when a CREATE/ALTER/DROP LEDGER TABLE or VERIFY LEDGER operation occurs.
value: "LEDGER_OPERATION_GROUP"
type:
- change
action: ledger-operation
"EXPO":
# Azure Purview external authorization policy operations (SQL Server 2022+)
# Fires when an external policy is checked or applied.
value: "EXTERNAL_POLICY_OPERATION_GROUP"
type:
- info
action: external-policy-operation
# SQL Server 2025+ action groups
"VECT":
# Vector index CRUD operations (SQL Server 2025+)
value: "VECTOR_OPERATION_GROUP"
type:
- change
action: vector-operation
"ERRE":
# Calls to sp_invoke_external_rest_endpoint and native AI functions (SQL Server 2025+)
value: "EXTERNAL_REST_ENDPOINT_GROUP"
type:
- info
action: external-rest-endpoint-invoked
source: |-
def actionIdKey = ctx.sqlserver.audit.action_id;
def actions = params.get('actions');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,22 @@
- name: user_defined_information
type: text
description: User defined information
# SQL Server 2022+ fields
- name: obo_middle_tier_app_id
type: keyword
description: >
Application ID of the middle-tier application acting on behalf of a principal (OAuth delegated access).
NULL when the action is not performed via OAuth On-Behalf-Of flow.
Available on SQL Server 2022 (16.x) and later.
- name: obo_middle_tier_app_name
type: keyword
description: >
Display name of the middle-tier application acting on behalf of a principal (OAuth delegated access).
NULL when the action is not performed via OAuth On-Behalf-Of flow.
Available on SQL Server 2022 (16.x) and later.
- name: external_policy_permissions_checked
type: keyword
description: >
Information related to the external authorization policy permission check performed during the audit event,
such as Azure Purview external policy decisions. NULL when external policy is not configured.
Available on SQL Server 2022 (16.x) and later.
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,49 @@ raw_data.enabled: true
merge_results: false
driver: "mssql"
sql_queries:
# Availability groups metrics
# Availability groups metrics (compatible with SQL Server 2019+)
# basic_availability_group (added SQL Server 2016, removed SQL Server 2025),
# is_distributed (SQL Server 2016+), and is_contained (SQL Server 2022+) are
# guarded by checking sys.columns so the query is safe across all versions.
- query: >-
DECLARE @hasBasicAG BIT = CASE WHEN EXISTS (SELECT 1 FROM sys.columns WHERE object_id = OBJECT_ID('sys.availability_groups') AND name = 'basic_availability_group') THEN 1 ELSE 0 END;
DECLARE @hasIsDistributed BIT = CASE WHEN EXISTS (SELECT 1 FROM sys.columns WHERE object_id = OBJECT_ID('sys.availability_groups') AND name = 'is_distributed') THEN 1 ELSE 0 END;
DECLARE @hasIsContained BIT = CASE WHEN EXISTS (SELECT 1 FROM sys.columns WHERE object_id = OBJECT_ID('sys.availability_groups') AND name = 'is_contained') THEN 1 ELSE 0 END;
DECLARE @sql NVARCHAR(MAX) = N'SELECT
@@servername AS server_name,
ag.name,
CONVERT(NVARCHAR(36), ag.group_id) AS group_id,
ags.primary_replica,
ags.synchronization_health,
ags.synchronization_health_desc,
ags.primary_recovery_health,
ags.secondary_recovery_health'
+ CASE WHEN @hasBasicAG = 1 THEN N', CAST(ag.basic_availability_group AS INT) AS basic_availability_group' ELSE N', NULL AS basic_availability_group' END
+ CASE WHEN @hasIsDistributed = 1 THEN N', CAST(ag.is_distributed AS INT) AS is_distributed' ELSE N', NULL AS is_distributed' END
+ CASE WHEN @hasIsContained = 1 THEN N', CAST(ag.is_contained AS INT) AS is_contained' ELSE N', NULL AS is_contained' END
+ N' FROM sys.dm_hadr_availability_group_states ags
JOIN sys.availability_groups ag ON ags.group_id = ag.group_id;';
EXEC sp_executesql @sql;
response_format: table
# Per-replica state metrics — available on all SQL Server versions that support AG
- query: "SELECT
@@servername AS server_name,
ag.name,
CONVERT(NVARCHAR(36), ag.group_id) AS group_id,
ags.primary_replica,
ags.synchronization_health,
ags.synchronization_health_desc,
ags.primary_recovery_health,
ags.secondary_recovery_health
FROM sys.dm_hadr_availability_group_states ags
JOIN sys.availability_groups ag
ON ags.group_id = ag.group_id;"
@@servername AS server_name,
CONVERT(NVARCHAR(36), rs.group_id) AS group_id,
ar.replica_server_name,
rs.role,
rs.role_desc,
rs.operational_state,
rs.operational_state_desc,
rs.connected_state,
rs.connected_state_desc,
rs.synchronization_health AS replica_synchronization_health,
rs.synchronization_health_desc AS replica_synchronization_health_desc,
rs.last_connect_error_number,
rs.last_connect_error_description,
rs.last_connect_error_timestamp
FROM sys.dm_hadr_availability_replica_states rs
JOIN sys.availability_replicas ar
ON rs.replica_id = ar.replica_id;"
response_format: table
{{#if processors}}
processors:
Expand Down
Loading
Loading