Skip to content

Fix/latency issues #150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
chrishagglund-ship-it wants to merge 2 commits into
base: gated-metrics-standardization
Choose a base branch
from
Draft

Fix/latency issues #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b9e5399
fix a couple things on the hot path for the harness worker but this f…
chrishagglund-ship-it May 13, 2026
652e0e8
experimental wip trying to replace fake async methods with real ones
chrishagglund-ship-it May 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,22 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [Unreleased — async executor / thread-starvation fix]

### Changed

- `WorkflowTaskExecutor`: converted `async void` methods (`WorkOnce`, `ProcessTasks`, `ProcessTask`) to `async Task` so the poll loop properly awaits each batch before re-entering. Previously, `async void` caused untracked continuations — the `RunningWorkerDone()` monitor count drifted, and any exception after the first `await` was unobserved on the thread pool.
- `WorkflowTaskExecutor`: replaced all `Thread.Sleep` calls (poll interval, error backoff, retry backoff) with `await Task.Delay`, releasing thread-pool threads during waits instead of blocking them.
- `ApiClient`: added `CallApiAsync` overload that accepts `Configuration` for async token-refresh retry (mirrors the sync `CallApi` + `RetryRestClientCallApi` path but uses `RestClient.ExecuteAsync`).
- `TaskResourceApi.BatchPollAsync` / `UpdateTaskAsync(TaskResult)`: now truly async — previously wrapped the synchronous `*WithHttpInfo` call in `Task.FromResult(...)`, providing zero async benefit.
- `IWorkflowTaskClient`: added `PollTaskAsync` and `UpdateTaskAsync` to the interface; `WorkflowTaskHttpClient` implements them via the now-truly-async `TaskResourceApi` methods.

### Fixed

- `WorkflowTaskExecutor`: `task_update_time_seconds` metric now records per-attempt HTTP latency. Previously a single `Stopwatch` spanned the entire retry loop including `Thread.Sleep` backoff (2–8s per retry), inflating the metric 6–15× beyond actual network time.
- `WorkflowTaskExecutor`: cancellation check in `ProcessTask`'s `finally` block was inverted (`== CancellationToken.None` instead of `!=`), so it never fired when a real token was provided.

## [Unreleased — metrics]

### Added

Expand Down
941 changes: 587 additions & 354 deletions Conductor/Api/ApplicationResourceApi.cs
View file
Open in desktop

Large diffs are not rendered by default.

196 changes: 125 additions & 71 deletions Conductor/Api/AuthorizationResourceApi.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,41 @@ public Object GetPermissions(string type, string id)
/// <returns>Object</returns>
public async ThreadTask.Task<object> GetPermissionsAsync(string type, string id)
{
ApiResponse<object> localVarResponse = await ThreadTask.Task.FromResult(GetPermissionsWithHttpInfo(type, id));
return localVarResponse.Data;
// verify the required parameter 'type' is set
if (type == null)
throw new ApiException(400, "Missing required parameter 'type' when calling AuthorizationResourceApi->GetPermissions");
// verify the required parameter 'id' is set
if (id == null)
throw new ApiException(400, "Missing required parameter 'id' when calling AuthorizationResourceApi->GetPermissions");

var localVarPath = "/auth/authorization/{type}/{id}";
var localVarPathParams = new Dictionary<String, String>();
var localVarQueryParams = new List<KeyValuePair<String, String>>();
var localVarHeaderParams = new Dictionary<String, String>(this.Configuration.DefaultHeader);
var localVarFormParams = new Dictionary<String, String>();
var localVarFileParams = new Dictionary<String, FileParameter>();
Object localVarPostBody = null;

// to determine the Content-Type header
String[] localVarHttpContentTypes = new String[] {
};
String localVarHttpContentType = this.Configuration.ApiClient.SelectHeaderContentType(localVarHttpContentTypes);

// to determine the Accept header
String[] localVarHttpHeaderAccepts = new String[] {
"application/json"
};
String localVarHttpHeaderAccept = this.Configuration.ApiClient.SelectHeaderAccept(localVarHttpHeaderAccepts);
if (localVarHttpHeaderAccept != null)
localVarHeaderParams.Add("Accept", localVarHttpHeaderAccept);

if (type != null) localVarPathParams.Add("type", this.Configuration.ApiClient.ParameterToString(type)); // path parameter
if (id != null) localVarPathParams.Add("id", this.Configuration.ApiClient.ParameterToString(id)); // path parameter

return (await this.Configuration.ApiClient.ExecuteAsync<Object>(localVarPath,
Method.Get, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams,
localVarFileParams, localVarPathParams, localVarHttpContentType, this.Configuration,
ExceptionFactory, "GetPermissions")).Data;
}

/// <summary>
Expand Down Expand Up @@ -165,28 +198,11 @@ public ApiResponse<object> GetPermissionsWithHttpInfo(string type, string id)

if (type != null) localVarPathParams.Add("type", this.Configuration.ApiClient.ParameterToString(type)); // path parameter
if (id != null) localVarPathParams.Add("id", this.Configuration.ApiClient.ParameterToString(id)); // path parameter
// authentication (api_key) required
if (!String.IsNullOrEmpty(this.Configuration.AccessToken))
{
localVarHeaderParams["X-Authorization"] = this.Configuration.AccessToken;
}

// make the HTTP request
RestResponse localVarResponse = (RestResponse)this.Configuration.ApiClient.CallApi(localVarPath,
Method.Get, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams, localVarFileParams,
localVarPathParams, localVarHttpContentType, this.Configuration);

int localVarStatusCode = (int)localVarResponse.StatusCode;

if (ExceptionFactory != null)
{
Exception exception = ExceptionFactory("GetPermissions", localVarResponse);
if (exception != null) throw exception;
}

return new ApiResponse<Object>(localVarStatusCode,
localVarResponse.Headers.ToDictionary(x => x.Name, x => string.Join(",", x.Value)),
(Object)this.Configuration.ApiClient.Deserialize(localVarResponse, typeof(Object)));
return this.Configuration.ApiClient.Execute<Object>(localVarPath,
Method.Get, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams,
localVarFileParams, localVarPathParams, localVarHttpContentType, this.Configuration,
ExceptionFactory, "GetPermissions");
}

/// <summary>
Expand All @@ -209,8 +225,45 @@ public Response GrantPermissions(AuthorizationRequest body)
/// <returns>Response</returns>
public async ThreadTask.Task<Response> GrantPermissionsAsync(AuthorizationRequest body)
{
ApiResponse<Response> localVarResponse = await ThreadTask.Task.FromResult(GrantPermissionsWithHttpInfo(body));
return localVarResponse.Data;
// verify the required parameter 'body' is set
if (body == null)
throw new ApiException(400, "Missing required parameter 'body' when calling AuthorizationResourceApi->GrantPermissions");

var localVarPath = "/auth/authorization";
var localVarPathParams = new Dictionary<String, String>();
var localVarQueryParams = new List<KeyValuePair<String, String>>();
var localVarHeaderParams = new Dictionary<String, String>(this.Configuration.DefaultHeader);
var localVarFormParams = new Dictionary<String, String>();
var localVarFileParams = new Dictionary<String, FileParameter>();
Object localVarPostBody = null;

// to determine the Content-Type header
String[] localVarHttpContentTypes = new String[] {
"application/json"
};
String localVarHttpContentType = this.Configuration.ApiClient.SelectHeaderContentType(localVarHttpContentTypes);

// to determine the Accept header
String[] localVarHttpHeaderAccepts = new String[] {
"application/json"
};
String localVarHttpHeaderAccept = this.Configuration.ApiClient.SelectHeaderAccept(localVarHttpHeaderAccepts);
if (localVarHttpHeaderAccept != null)
localVarHeaderParams.Add("Accept", localVarHttpHeaderAccept);

if (body != null && body.GetType() != typeof(byte[]))
{
localVarPostBody = this.Configuration.ApiClient.Serialize(body); // http body (model) parameter
}
else
{
localVarPostBody = body; // byte array
}

return (await this.Configuration.ApiClient.ExecuteAsync<Response>(localVarPath,
Method.Post, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams,
localVarFileParams, localVarPathParams, localVarHttpContentType, this.Configuration,
ExceptionFactory, "GrantPermissions")).Data;
}
/// <summary>
/// Grant access to a user over the target
Expand Down Expand Up @@ -254,28 +307,10 @@ public ApiResponse<Response> GrantPermissionsWithHttpInfo(AuthorizationRequest b
{
localVarPostBody = body; // byte array
}
// authentication (api_key) required
if (!String.IsNullOrEmpty(this.Configuration.AccessToken))
{
localVarHeaderParams["X-Authorization"] = this.Configuration.AccessToken;
}

// make the HTTP request
RestResponse localVarResponse = (RestResponse)this.Configuration.ApiClient.CallApi(localVarPath,
Method.Post, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams, localVarFileParams,
localVarPathParams, localVarHttpContentType, this.Configuration);

int localVarStatusCode = (int)localVarResponse.StatusCode;

if (ExceptionFactory != null)
{
Exception exception = ExceptionFactory("GrantPermissions", localVarResponse);
if (exception != null) throw exception;
}

return new ApiResponse<Response>(localVarStatusCode,
localVarResponse.Headers.ToDictionary(x => x.Name, x => string.Join(",", x.Value)),
(Response)this.Configuration.ApiClient.Deserialize(localVarResponse, typeof(Response)));
return this.Configuration.ApiClient.Execute<Response>(localVarPath,
Method.Post, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams,
localVarFileParams, localVarPathParams, localVarHttpContentType, this.Configuration,
ExceptionFactory, "GrantPermissions");
}

/// <summary>
Expand All @@ -298,8 +333,45 @@ public Response RemovePermissions(AuthorizationRequest body)
/// <returns>Response</returns>
public async ThreadTask.Task<Response> RemovePermissionsAsync(AuthorizationRequest body)
{
ApiResponse<Response> localVarResponse = await ThreadTask.Task.FromResult(RemovePermissionsWithHttpInfo(body));
return localVarResponse.Data;
// verify the required parameter 'body' is set
if (body == null)
throw new ApiException(400, "Missing required parameter 'body' when calling AuthorizationResourceApi->RemovePermissions");

var localVarPath = "/auth/authorization";
var localVarPathParams = new Dictionary<String, String>();
var localVarQueryParams = new List<KeyValuePair<String, String>>();
var localVarHeaderParams = new Dictionary<String, String>(this.Configuration.DefaultHeader);
var localVarFormParams = new Dictionary<String, String>();
var localVarFileParams = new Dictionary<String, FileParameter>();
Object localVarPostBody = null;

// to determine the Content-Type header
String[] localVarHttpContentTypes = new String[] {
"application/json"
};
String localVarHttpContentType = this.Configuration.ApiClient.SelectHeaderContentType(localVarHttpContentTypes);

// to determine the Accept header
String[] localVarHttpHeaderAccepts = new String[] {
"application/json"
};
String localVarHttpHeaderAccept = this.Configuration.ApiClient.SelectHeaderAccept(localVarHttpHeaderAccepts);
if (localVarHttpHeaderAccept != null)
localVarHeaderParams.Add("Accept", localVarHttpHeaderAccept);

if (body != null && body.GetType() != typeof(byte[]))
{
localVarPostBody = this.Configuration.ApiClient.Serialize(body); // http body (model) parameter
}
else
{
localVarPostBody = body; // byte array
}

return (await this.Configuration.ApiClient.ExecuteAsync<Response>(localVarPath,
Method.Delete, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams,
localVarFileParams, localVarPathParams, localVarHttpContentType, this.Configuration,
ExceptionFactory, "RemovePermissions")).Data;
}

/// <summary>
Expand Down Expand Up @@ -344,28 +416,10 @@ public ApiResponse<Response> RemovePermissionsWithHttpInfo(AuthorizationRequest
{
localVarPostBody = body; // byte array
}
// authentication (api_key) required
if (!String.IsNullOrEmpty(this.Configuration.AccessToken))
{
localVarHeaderParams["X-Authorization"] = this.Configuration.AccessToken;
}

// make the HTTP request
RestResponse localVarResponse = (RestResponse)this.Configuration.ApiClient.CallApi(localVarPath,
Method.Delete, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams, localVarFileParams,
localVarPathParams, localVarHttpContentType, this.Configuration);

int localVarStatusCode = (int)localVarResponse.StatusCode;

if (ExceptionFactory != null)
{
Exception exception = ExceptionFactory("RemovePermissions", localVarResponse);
if (exception != null) throw exception;
}

return new ApiResponse<Response>(localVarStatusCode,
localVarResponse.Headers.ToDictionary(x => x.Name, x => string.Join(",", x.Value)),
(Response)this.Configuration.ApiClient.Deserialize(localVarResponse, typeof(Response)));
return this.Configuration.ApiClient.Execute<Response>(localVarPath,
Method.Delete, localVarQueryParams, localVarPostBody, localVarHeaderParams, localVarFormParams,
localVarFileParams, localVarPathParams, localVarHttpContentType, this.Configuration,
ExceptionFactory, "RemovePermissions");
}
}
}
Loading