Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
271 changes: 270 additions & 1 deletion MaiChartManager/Controllers/Tools/VideoConvertToolController.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
using System.Text.Json;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Batch conversion no longer refreshes MovieDataMap after file changes, which can leave metadata out of sync with disk state until another scan runs.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At MaiChartManager/Controllers/Tools/VideoConvertToolController.cs, line 11:

<comment>Batch conversion no longer refreshes `MovieDataMap` after file changes, which can leave metadata out of sync with disk state until another scan runs.</comment>

<file context>
@@ -8,7 +8,7 @@ namespace MaiChartManager.Controllers.Tools;
 [ApiController]
 [Route("MaiChartManagerServlet/[action]Api")]
-public class VideoConvertToolController(ILogger<VideoConvertToolController> logger, StaticSettings settings) : ControllerBase
+public class VideoConvertToolController(ILogger<VideoConvertToolController> logger) : ControllerBase
 {
     public enum VideoConvertEventType
</file context>

using System.Threading.Channels;
using MaiChartManager.Utils;
using Microsoft.AspNetCore.Mvc;
using Microsoft.VisualBasic.FileIO;

namespace MaiChartManager.Controllers.Tools;

Expand Down Expand Up @@ -83,4 +86,270 @@ await VideoConvert.ConvertVideoToUsm(
await Response.Body.FlushAsync();
}
}
}

public enum BatchConvertPvDirection
{
/// <summary>USM/DAT → MP4</summary>
UsmToMp4,
/// <summary>MP4 → USM/DAT</summary>
Mp4ToUsm
}

public enum BatchConvertPvEventType
{
/// <summary>整体 + 当前文件进度,data 为 JSON</summary>
Progress,
/// <summary>单文件失败,仍然继续处理后续文件</summary>
FileError,
/// <summary>全部完成,data 为 "processed/total|failedCount"</summary>
Success,
/// <summary>致命错误,停止</summary>
Error,
/// <summary>被取消,data 为 "processed/total"</summary>
Cancelled
}

private static readonly JsonSerializerOptions BatchJsonOptions = new()
{
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
};

private record BatchProgressPayload(int Processed, int Total, int FileProgress, string FileName, int Failed);

/// <summary>
/// 批量转换用户选择的文件夹内所有 PV:USM/DAT ↔ MP4。
/// 使用 SSE 实时推送整体进度(已处理/总数)+ 当前文件进度。
/// 客户端断开连接时通过 RequestAborted 触发取消,循环在下一个文件之间退出。
/// 所有 SSE 写入通过单写者 Channel 串行化,避免 Xabe 同步进度事件触发的 async-void 写入交错。
/// </summary>
[HttpPost]
public async Task BatchConvertPvTool([FromQuery] string folderPath, [FromQuery] BatchConvertPvDirection direction)
{
Response.Headers.Append("Content-Type", "text/event-stream");

// PV 转换属于赞助功能
if (IapManager.License != IapManager.LicenseStatus.Active)
{
await Response.WriteAsync($"event: {BatchConvertPvEventType.Error}\ndata: {SanitizeSseLine(Locale.BatchConvertPvNeedLicense)}\n\n");
await Response.Body.FlushAsync();
return;
}

if (string.IsNullOrWhiteSpace(folderPath) || !Directory.Exists(folderPath))
{
await Response.WriteAsync($"event: {BatchConvertPvEventType.Error}\ndata: {SanitizeSseLine(Locale.BatchConvertPvFolderNotFound)}\n\n");
await Response.Body.FlushAsync();
return;
}

// 直接枚举用户选择的文件夹(不递归),按方向筛选源扩展名
var sourceExtensions = direction == BatchConvertPvDirection.UsmToMp4
? new HashSet<string>(StringComparer.OrdinalIgnoreCase) { ".dat", ".usm" }
: new HashSet<string>(StringComparer.OrdinalIgnoreCase) { ".mp4" };

var files = Directory.EnumerateFiles(folderPath)
.Where(f => sourceExtensions.Contains(Path.GetExtension(f)))
.OrderBy(f => f, StringComparer.OrdinalIgnoreCase)
.ToList();

if (files.Count == 0)
{
await Response.WriteAsync($"event: {BatchConvertPvEventType.Error}\ndata: {SanitizeSseLine(Locale.BatchConvertPvNoFiles)}\n\n");
await Response.Body.FlushAsync();
return;
}

var total = files.Count;
var processed = 0;
var failedCount = 0;
var cancellationToken = HttpContext.RequestAborted;

// 单写者 Channel:所有 SSE 帧(不论来自循环还是 OnProgress)都进入这条队列
var sseChannel = Channel.CreateUnbounded<string>(new UnboundedChannelOptions
{
SingleReader = true,
SingleWriter = false,
});

var writer = WriteSseFrames(sseChannel.Reader, cancellationToken);
Comment on lines +168 to +174
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (bug_risk): 建议对 SSE 进度通道进行限界或节流,以避免在客户端较慢或进度事件非常频繁的情况下导致内存无限增长。

由于这里使用的是无界的 Channel<string>,并结合 EnqueueProgressFireAndForget,当客户端处理速度较慢或被阻塞、且进度回调很频繁时,尤其是在长时间运行的批量任务中,队列可能会无限增长。建议考虑使用有界 Channel,并对进度事件采用丢弃/覆盖策略,或者对进度更新进行节流/合并(例如只在进度变化 ≥1% 或按固定时间间隔入队),从而控制内存使用。

Suggested implementation:

        // 单写者 Channel:所有 SSE 帧(不论来自循环还是 OnProgress)都进入这条队列
        // 使用有界 Channel,并在队列满时丢弃最旧的进度帧,防止在慢客户端/高频进度回调下无限占用内存
        var sseChannel = Channel.CreateBounded<string>(new BoundedChannelOptions(100)
        {
            SingleReader = true,
            SingleWriter = false,
            FullMode = BoundedChannelFullMode.DropOldest,
        });
  1. 如果 EnqueueProgress/EnqueueProgressFireAndForget 当前使用的是 WriteAsync 这类可能阻塞的写入方式,则需要改为使用 TryWrite,或对 WriteAsync 的失败/取消做容错处理,以适配 BoundedChannelFullMode.DropOldest 的策略。
  2. 如果希望进一步节流/合并进度事件(例如仅在进度提升 ≥1% 或按固定时间间隔发送),可以在 EnqueueProgress 内部记录上一次发送的进度百分比/时间戳,在未满足阈值时直接返回而不写入 Channel。
Original comment in English

suggestion (bug_risk): Consider bounding or throttling the SSE progress channel to avoid unbounded memory growth under slow clients or very frequent progress events.

Because this is an unbounded Channel<string> combined with EnqueueProgressFireAndForget, a slow or blocked client plus frequent progress callbacks can cause unbounded queue growth, especially for long-running batch jobs. Consider either using a bounded channel with a drop/overwrite policy for progress events, or throttling/coalescing updates (e.g., only enqueue on ≥1% change or at a fixed interval) to keep memory usage controlled.

Suggested implementation:

        // 单写者 Channel:所有 SSE 帧(不论来自循环还是 OnProgress)都进入这条队列
        // 使用有界 Channel,并在队列满时丢弃最旧的进度帧,防止在慢客户端/高频进度回调下无限占用内存
        var sseChannel = Channel.CreateBounded<string>(new BoundedChannelOptions(100)
        {
            SingleReader = true,
            SingleWriter = false,
            FullMode = BoundedChannelFullMode.DropOldest,
        });
  1. EnqueueProgress/EnqueueProgressFireAndForget 目前如果使用 WriteAsync 之类的阻塞写入,需要调整为使用 TryWrite 或对 WriteAsync 的失败/取消做容错处理,以配合 BoundedChannelFullMode.DropOldest 的策略。
  2. 如果希望进一步节流/合并进度事件(例如仅在进度提升 ≥1% 或按固定时间间隔发送),可以在 EnqueueProgress 内部记录上一次发送的进度百分比/时间戳,在未满足阈值时直接返回而不写入 Channel。


try
{
foreach (var inputPath in files)
{
if (cancellationToken.IsCancellationRequested) break;

var fileName = Path.GetFileName(inputPath);
await EnqueueProgress(sseChannel.Writer, processed, total, 0, fileName, failedCount);

try
{
var directory = Path.GetDirectoryName(inputPath)!;
var nameWithoutExt = Path.GetFileNameWithoutExtension(inputPath);

if (direction == BatchConvertPvDirection.UsmToMp4)
{
var outputPath = Path.Combine(directory, nameWithoutExt + ".mp4");
var snapshot = (Processed: processed, Failed: failedCount);
await VideoConvert.ConvertUsmToMp4(
inputPath,
outputPath,
percent => EnqueueProgressFireAndForget(sseChannel.Writer, snapshot.Processed, total, percent, fileName, snapshot.Failed));
}
else
{
// MP4 → USM(VP9):先输出到临时文件,验证后再覆盖目标
var finalPath = Path.Combine(directory, nameWithoutExt + ".dat");
var tempPath = finalPath + ".tmp";
var snapshot = (Processed: processed, Failed: failedCount);
try
{
await VideoConvert.ConvertVideo(new VideoConvert.VideoConvertOptions
{
InputPath = inputPath,
OutputPath = tempPath,
NoScale = StaticSettings.Config.NoScale,
UseH264 = false,
UseYuv420p = StaticSettings.Config.Yuv420p,
Padding = 0,
TaskbarProgress = false,
OnProgress = percent => EnqueueProgressFireAndForget(sseChannel.Writer, snapshot.Processed, total, percent, fileName, snapshot.Failed)
});

if (!System.IO.File.Exists(tempPath) || new FileInfo(tempPath).Length == 0)
{
throw new Exception("Converted DAT is missing or empty");
}

// 取消检查必须在覆盖/删除前,避免取消时仍然损毁源文件
cancellationToken.ThrowIfCancellationRequested();

if (System.IO.File.Exists(finalPath))
{
System.IO.File.Delete(finalPath);
}
System.IO.File.Move(tempPath, finalPath);

// 源 MP4 送进回收站,而非永久删除,最大程度避免用户数据丢失
try
{
FileSystem.DeleteFile(inputPath, UIOption.OnlyErrorDialogs, RecycleOption.SendToRecycleBin);
}
catch (Exception delEx)
{
logger.LogWarning(delEx, "Failed to move source MP4 to recycle bin after batch convert: {Path}", inputPath);
await EnqueueEvent(sseChannel.Writer, BatchConvertPvEventType.FileError, $"{fileName}: moved to .dat but failed to remove source MP4 ({delEx.Message})");
}
}
catch
{
try { if (System.IO.File.Exists(tempPath)) System.IO.File.Delete(tempPath); }
catch { /* ignored */ }
throw;
}
}

processed++;
await EnqueueProgress(sseChannel.Writer, processed, total, 100, fileName, failedCount);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception fileEx)
{
logger.LogError(fileEx, "Failed to convert PV file {File}", inputPath);
failedCount++;
processed++;
await EnqueueEvent(sseChannel.Writer, BatchConvertPvEventType.FileError, $"{fileName}: {fileEx.Message}");
await EnqueueProgress(sseChannel.Writer, processed, total, 100, fileName, failedCount);
}
}

if (cancellationToken.IsCancellationRequested)
{
await EnqueueEvent(sseChannel.Writer, BatchConvertPvEventType.Cancelled, $"{processed}/{total}");
}
else
{
await EnqueueEvent(sseChannel.Writer, BatchConvertPvEventType.Success, $"{processed}/{total}|{failedCount}");
}
}
catch (OperationCanceledException)
{
await EnqueueEvent(sseChannel.Writer, BatchConvertPvEventType.Cancelled, $"{processed}/{total}");
}
catch (Exception ex)
{
logger.LogError(ex, "Batch PV conversion failed");
SentrySdk.CaptureException(ex);
try
{
await EnqueueEvent(sseChannel.Writer, BatchConvertPvEventType.Error, string.Format(Locale.ConvertFailed, ex.Message));
}
catch
{
// 客户端可能已断开
}
}
finally
{
sseChannel.Writer.TryComplete();
try
{
await writer;
}
catch
{
// writer 自己负责吞掉客户端断开异常
}
}
}

private static string SanitizeSseLine(string data) =>
data.Replace("\r", " ").Replace("\n", " ");

private static ValueTask EnqueueEvent(ChannelWriter<string> writer, BatchConvertPvEventType eventType, string data) =>
writer.WriteAsync($"event: {eventType}\ndata: {SanitizeSseLine(data)}\n\n");

private static ValueTask EnqueueProgress(ChannelWriter<string> writer, int processed, int total, int fileProgress, string fileName, int failed)
{
var payload = JsonSerializer.Serialize(new BatchProgressPayload(processed, total, fileProgress, fileName, failed), BatchJsonOptions);
return writer.WriteAsync($"event: {BatchConvertPvEventType.Progress}\ndata: {payload}\n\n");
}

private static void EnqueueProgressFireAndForget(ChannelWriter<string> writer, int processed, int total, int fileProgress, string fileName, int failed)
{
var payload = JsonSerializer.Serialize(new BatchProgressPayload(processed, total, fileProgress, fileName, failed), BatchJsonOptions);
// Channel 是无界的,TryWrite 同步入队,避免在 Xabe 的同步进度事件里 await
writer.TryWrite($"event: {BatchConvertPvEventType.Progress}\ndata: {payload}\n\n");
}

private async Task WriteSseFrames(ChannelReader<string> reader, CancellationToken cancellationToken)
{
try
{
await foreach (var frame in reader.ReadAllAsync(cancellationToken))
{
try
{
await Response.WriteAsync(frame, cancellationToken);
await Response.Body.FlushAsync(cancellationToken);
}
catch (OperationCanceledException)
{
return;
}
catch (Exception ex)
{
logger.LogDebug(ex, "SSE frame write failed (client disconnected?)");
return;
}
}
}
catch (OperationCanceledException)
{
// ignore
}
}
}
26 changes: 26 additions & 0 deletions MaiChartManager/Front/src/locales/en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,32 @@ tools:
imageToAbError: Image to AB conversion error
videoOptions:
processing: Still processing, please wait...
batchPv:
label: Batch Convert PVs
title: Batch Convert PVs
selectFolder: Pick folder
changeFolder: Change folder
selectedFolder: Selected folder
folderNotFound: Selected folder does not exist
direction: Conversion direction
directionUsmToMp4: USM / DAT → MP4
directionMp4ToUsm: MP4 → USM / DAT
directionUsmToMp4Hint: Writes a sibling .mp4 next to each DAT/USM in the chosen folder; originals are kept
directionMp4ToUsmHint: Writes a sibling .dat next to each MP4 in the chosen folder; source .mp4 files are moved to the recycle bin
start: Start
cancel: Cancel
cancelling: Cancelling...
cancelHint: Cancel takes effect after the current file finishes
close: Close
overall: Overall
currentFile: Current file
currentFileProgress: Current file progress
completedSummary: 'Completed: {success} / {total} succeeded ({failed} failed)'
cancelledSummary: 'Cancelled: {completed} / {total} done'
noFiles: No PV files in that folder match the selected direction
needLicense: This feature requires sponsor activation
error: Batch PV conversion error
fileErrors: 'File errors encountered:'
error:
title: Error
unknown: Unknown error occurred
Expand Down
26 changes: 26 additions & 0 deletions MaiChartManager/Front/src/locales/zh-TW.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,32 @@ tools:
imageToAbError: 圖片轉 AB 出錯
videoOptions:
processing: 還在處理,別急…
batchPv:
label: 批次轉換 PV
title: 批次轉換 PV
selectFolder: 選擇資料夾
changeFolder: 重新選擇
selectedFolder: 已選擇的資料夾
folderNotFound: 選擇的資料夾不存在
direction: 轉換方向
directionUsmToMp4: USM / DAT → MP4
directionMp4ToUsm: MP4 → USM / DAT
directionUsmToMp4Hint: 在該資料夾內為每個 DAT/USM 產生對應的 MP4,原檔案保留
directionMp4ToUsmHint: 在該資料夾內為每個 MP4 產生對應的 DAT,原 MP4 會被移入資源回收筒
start: 開始轉換
cancel: 取消
cancelling: 正在取消…
cancelHint: 取消會在目前檔案轉換完成後生效
close: 關閉
overall: 總進度
currentFile: 目前檔案
currentFileProgress: 目前檔案進度
completedSummary: 轉換完成:成功 {success} / {total}(失敗 {failed})
cancelledSummary: 已取消:已完成 {completed} / {total}
noFiles: 該資料夾下找不到符合方向的 PV 檔案
needLicense: 此功能需要贊助啟用
error: 批次轉換 PV 出錯
fileErrors: 處理過程中的檔案錯誤:
error:
title: 錯誤
unknown: 發生未知錯誤
Expand Down
26 changes: 26 additions & 0 deletions MaiChartManager/Front/src/locales/zh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,32 @@ tools:
imageToAbError: 图片转 AB 出错
videoOptions:
processing: 还在处理,别急…
batchPv:
label: 批量转换 PV
title: 批量转换 PV
selectFolder: 选择文件夹
changeFolder: 重新选择
selectedFolder: 已选择的文件夹
folderNotFound: 选择的文件夹不存在
direction: 转换方向
directionUsmToMp4: USM / DAT → MP4
directionMp4ToUsm: MP4 → USM / DAT
directionUsmToMp4Hint: 在该文件夹内为每个 DAT/USM 生成对应的 MP4,原文件保留
directionMp4ToUsmHint: 在该文件夹内为每个 MP4 生成对应的 DAT,原 MP4 会被移入回收站
start: 开始转换
cancel: 取消
cancelling: 正在取消…
cancelHint: 取消会在当前文件转换完成后生效
close: 关闭
overall: 总进度
currentFile: 当前文件
currentFileProgress: 当前文件进度
completedSummary: 转换完成:成功 {success} / {total}(失败 {failed})
cancelledSummary: 已取消:已完成 {completed} / {total}
noFiles: 该文件夹下没有找到匹配方向的 PV 文件
needLicense: 此功能需要赞助激活
error: 批量转换 PV 出错
fileErrors: 处理过程中的文件错误:
error:
title: 错误
unknown: 发生未知错误
Expand Down
Loading