Skip to content

Fix trace-file accumulation and Data Retention job failure (#972) #973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
erikdarlingdata merged 1 commit into from
May 21, 2026
Merged

Fix trace-file accumulation and Data Retention job failure (#972) #973

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Fixed

- **Data Retention job no longer fails with `xp_delete_file` error 22049** ([#972]) — the trace-file cleanup added in v2.11.0 passed a wildcard path to `xp_delete_file`, raising an uncatchable `Msg 22049` that failed the entire `PerformanceMonitor - Data Retention` Agent job on every run once any `Monitor_LongQueries_*.trc` files existed. `xp_delete_file` also cannot delete `.trc` files at all — it only accepts SQL Server backup files and Maintenance Plan report files — so that cleanup step has been removed from `config.data_retention`

### Changed

- **Trace files are now bounded at the source** ([#972]) — `collect.trace_management_collector` creates the long-query trace with a rollover file-count cap (`@filecount`, via the new `@max_files` parameter, default 5), so SQL Server itself deletes the oldest `.trc` file as the trace rolls. The scheduled collector also now issues `START` instead of `RESTART`: it keeps one trace running rather than tearing it down and spawning a fresh timestamped trace — and a fresh batch of orphaned files — every cycle

### Added

- **`tools/Remove-OrphanedTraceFiles.ps1`** ([#972]) — one-time cleanup script for `Monitor_LongQueries_*.trc` files left on disk by versions through 2.11.0. Run it on the SQL Server host; it skips files belonging to a running trace and files that are in use

[#972]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/972

## [2.11.0] - 2026-05-19

### Important
Expand Down
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,15 @@ WHERE collection_status = 'ERROR'
ORDER BY collection_time DESC;
```

**Orphaned `Monitor_LongQueries_*.trc` files (issue #972)** — versions through 2.11.0 accumulated stale SQL Trace files in the SQL Server error log directory. Newer versions bound the long-query trace with a rollover file-count cap, so SQL Server prunes its own files going forward — but trace files already on disk are not removed automatically (`xp_delete_file` cannot delete `.trc` files). Sweep them once with `tools/Remove-OrphanedTraceFiles.ps1`, run **on the SQL Server host** as a local Administrator or the SQL Server service account:

```powershell
.\Remove-OrphanedTraceFiles.ps1 -WhatIf # preview what would be deleted
.\Remove-OrphanedTraceFiles.ps1 # delete
```

It skips files belonging to a running trace and files that are in use.

### Lite Edition

Application logs are written to the `logs/` folder. Collection success/failure is also logged to the `collection_log` table in DuckDB.
Expand Down
38 changes: 34 additions & 4 deletions install/30_collect_trace_management.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ ALTER PROCEDURE
@duration_threshold_ms bigint = 2000000, /*minimum duration in microseconds (2 seconds)*/
@cpu_threshold_ms integer = 1000, /*minimum CPU time in milliseconds (1 second)*/
@max_file_size_mb bigint = 200, /*maximum trace file size in MB*/
@max_files integer = 5, /*max rollover files to keep; SQL Server deletes the oldest beyond this (issue #972)*/
@debug bit = 0 /*prints additional diagnostic information*/
)
WITH RECOMPILE
Expand All @@ -50,6 +51,7 @@ BEGIN
*/
DECLARE
@trace_id integer = NULL,
@trace_stoptime datetime = NULL, /*sp_trace_create requires @stoptime once @filecount is passed; NULL = run until stopped*/
@file_path nvarchar(4000) = N'',
@trace_search_pattern nvarchar(4000) = N'',
@error_log_path nvarchar(4000) = N'',
Expand Down Expand Up @@ -85,6 +87,16 @@ BEGIN
RETURN;
END;

/*
SQL Trace requires a rollover file count greater than 1 (issue #972).
*/
IF @max_files < 2 OR @max_files > 1000
BEGIN
SET @error_message = N'@max_files must be between 2 and 1000';
RAISERROR(@error_message, 16, 1);
RETURN;
END;

/*
Get SQL Server error log path dynamically
*/
Expand Down Expand Up @@ -225,9 +237,22 @@ BEGIN
ELSE IF @action IN (N'START', N'RESTART')
BEGIN
/*
Stop ALL existing traces first if RESTART (regardless of timestamp)
Stop existing traces before creating a fresh one. Runs for RESTART,
and for START when the running trace has no rollover file-count cap
(one created by versions <= 2.11.0), so the issue #972 fix
self-heals without waiting for a SQL Server restart.
*/
IF @action = N'RESTART'
OR EXISTS
(
SELECT
1/0
FROM sys.traces AS t
WHERE t.path LIKE @trace_search_pattern
AND t.is_default = 0
AND t.status = 1
AND ISNULL(t.max_files, 0) < 2
)
BEGIN
DECLARE @restart_cursor CURSOR;

Expand Down Expand Up @@ -273,8 +298,10 @@ BEGIN
END;

/*
Check for existing running traces (any with this trace name)
If already running, return success (idempotent behavior for scheduled operation)
START is idempotent: if a bounded trace (one created with a
rollover file-count cap) is already running, leave it alone. An
unbounded trace from an older version was stopped just above, so it
no longer matches here and a fresh capped trace is created (#972).
*/
IF @action = N'START'
AND EXISTS
Expand All @@ -285,6 +312,7 @@ BEGIN
WHERE t.path LIKE @trace_search_pattern
AND t.is_default = 0
AND t.status = 1
AND ISNULL(t.max_files, 0) > 1
)
BEGIN
/*
Expand Down Expand Up @@ -346,7 +374,9 @@ BEGIN
@traceid = @trace_id OUTPUT,
@options = 2, /*file rollover enabled*/
@tracefile = @file_path,
@maxfilesize = @max_file_size_mb;
@maxfilesize = @max_file_size_mb,
@stoptime = @trace_stoptime,
@filecount = @max_files; /*issue #972: bound rollover so SQL Server deletes the oldest file itself*/

IF @debug = 1
BEGIN
Expand Down
8 changes: 7 additions & 1 deletion install/42_scheduled_master_collector.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,13 @@ BEGIN
END;
ELSE IF @collector_name = N'trace_management_collector'
BEGIN
EXECUTE collect.trace_management_collector @action = N'RESTART', @debug = @debug;
/*
Issue #972: START, not RESTART. START is idempotent - it
leaves an already-running trace alone. RESTART tore the
trace down and built a fresh timestamped one every cycle,
orphaning the previous trace's .trc files on disk.
*/
EXECUTE collect.trace_management_collector @action = N'START', @debug = @debug;
END;
ELSE IF @collector_name = N'trace_analysis_collector'
BEGIN
Expand Down
194 changes: 12 additions & 182 deletions install/43_data_retention.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -557,189 +557,19 @@ WHERE ' + QUOTENAME(@time_column_name) + N' < @retention_date_param;';
/*Cursor variables don't require DEALLOCATE*/

/*
Issue #951: clean up Monitor_LongQueries_*.trc files left in the SQL Server
error log directory by collect.trace_management_collector. xp_delete_file
skips files that are currently open, so the running trace is safe.

Retention source mirrors the table-cleanup logic above:
- @apply_schedule_override = 1 (caller passed @retention_days = NULL):
read retention_days from config.collection_schedule for
trace_management_collector, falling back to 30 days.
- @apply_schedule_override = 0 (caller passed a flat @retention_days):
use @effective_retention_days as the cutoff.
Issue #972: stale Monitor_LongQueries_*.trc cleanup has been removed
from this procedure. xp_delete_file cannot delete trace files at all -
it only accepts SQL Server backup files or Maintenance Plan report
files and validates the header - so the #951 implementation here could
never have worked, and the malformed wildcard path it passed raised an
uncatchable Msg 22049 that failed the Data Retention Agent job.

Trace files are now bounded at the source: collect.trace_management_collector
creates the trace with a rollover file-count cap (@filecount), so SQL
Server itself deletes the oldest file as the trace rolls. Trace files
left on disk by versions <= 2.11.0 are cleaned up once by
tools/Remove-OrphanedTraceFiles.ps1.
*/
BEGIN TRY
DECLARE
@trace_retention_days integer,
@trace_cutoff_string varchar(23),
@trace_error_log_path nvarchar(4000),
@trace_file_pattern nvarchar(4000);

IF @apply_schedule_override = 1
BEGIN
SELECT
@trace_retention_days = cs.retention_days
FROM config.collection_schedule AS cs
WHERE cs.collector_name = N'trace_management_collector';

SET @trace_retention_days = ISNULL(@trace_retention_days, 30);
END
ELSE
BEGIN
SET @trace_retention_days = @effective_retention_days;
END;

/*
xp_delete_file takes the cutoff as an ISO 8601 string ('YYYY-MM-DDTHH:MM:SS.SSS').
*/
SET @trace_cutoff_string =
CONVERT(varchar(23), DATEADD(DAY, -@trace_retention_days, SYSDATETIME()), 126);

/*
Derive the SQL Server error log directory the same way
collect.trace_management_collector does.
*/
SELECT @trace_error_log_path =
LEFT
(
CONVERT(nvarchar(4000), SERVERPROPERTY('ErrorLogFileName')),
LEN(CONVERT(nvarchar(4000), SERVERPROPERTY('ErrorLogFileName'))) -
CHARINDEX('\', REVERSE(CONVERT(nvarchar(4000), SERVERPROPERTY('ErrorLogFileName')))) + 1
);

SET @trace_file_pattern = @trace_error_log_path + N'Monitor_LongQueries_*.trc';

/*
Pre-check via xp_dirtree so we only call xp_delete_file when matching
files actually exist. Without this, xp_delete_file prints a noisy
"Msg 22049, 'system cannot find the file specified'" to client/Agent
job output on every run that finds nothing to delete.
*/
DECLARE
@trace_dir_listing TABLE
(
subdirectory nvarchar(512) NOT NULL,
depth integer NOT NULL,
is_file bit NOT NULL
);

INSERT INTO
@trace_dir_listing
(
subdirectory,
depth,
is_file
)
EXECUTE master.sys.xp_dirtree
@trace_error_log_path,
1,
1;

DECLARE
@trace_files_present integer =
(
SELECT
COUNT_BIG(*)
FROM @trace_dir_listing AS d
WHERE d.is_file = 1
AND d.subdirectory LIKE N'Monitor_LongQueries[_]%.trc'
);

IF @debug = 1
BEGIN
SET @message =
N'Found ' + CONVERT(nvarchar(10), @trace_files_present)
+ N' Monitor_LongQueries_*.trc file(s) in ' + @trace_error_log_path
+ N'; cutoff=' + @trace_cutoff_string;
RAISERROR(@message, 0, 1) WITH NOWAIT;
END;

IF @trace_files_present > 0
BEGIN
/*
xp_delete_file argument 1 = 1 means "delete files matching the pattern
older than the cutoff date". It returns no row count.
*/
EXECUTE master.dbo.xp_delete_file
1,
@trace_file_pattern,
N'trc',
@trace_cutoff_string;
END;

INSERT INTO
config.collection_log
(
collector_name,
collection_status,
error_message
)
VALUES
(
N'data_retention',
N'SUCCESS',
N'Trace file cleanup: pattern=' + @trace_file_pattern
+ N', cutoff=' + @trace_cutoff_string
+ N', files_present=' + CONVERT(nvarchar(10), @trace_files_present)
);
END TRY
BEGIN CATCH
/*
Error 22049 from xp_delete_file means "no files matched the pattern"
(the Windows ERROR_FILE_NOT_FOUND surfaced as SQL Msg 22049). Older
SQL versions raise this as a catchable error; SQL 2022 only prints
it as an info message. Treat as benign — there was simply nothing
old enough to delete.
*/
IF ERROR_NUMBER() = 22049
BEGIN
IF @debug = 1
BEGIN
SET @message =
N'No trace files older than ' + @trace_cutoff_string
+ N' matched ' + @trace_file_pattern;
RAISERROR(@message, 0, 1) WITH NOWAIT;
END;

INSERT INTO
config.collection_log
(
collector_name,
collection_status,
error_message
)
VALUES
(
N'data_retention',
N'SUCCESS',
N'Trace file cleanup: no files older than ' + @trace_cutoff_string
+ N' matched ' + @trace_file_pattern
);
END
ELSE
BEGIN
SET @message = N'Error cleaning trace files: ' + ERROR_MESSAGE();

IF @debug = 1
BEGIN
RAISERROR(@message, 0, 1) WITH NOWAIT;
END;

INSERT INTO
config.collection_log
(
collector_name,
collection_status,
error_message
)
VALUES
(
N'data_retention',
N'ERROR',
@message
);
END;
END CATCH;

/*
Log retention operation
Expand Down
Loading
Loading