Add Usage Tracking feature with OpenAI support by rahulsprajapati · Pull Request #1060 · 10up/classifai

rahulsprajapati · 2026-02-25T15:37:28Z

Description of the Change

This PR adds OpenAI Usage Tracking to ClassifAI: the ability to monitor usage/costs from the OpenAI API and enforce soft (alert-only) and hard (disable features) spending limits.

Design overview:

New service & feature: A new UsageTracking service and OpenAIUsage feature are introduced, with provider OpenAI\UsageTracking that calls the OpenAI organization costs API.
Usage data: Usage (MTD, YTD, all time) is fetched and cached with a configurable refresh interval (default 15 minutes) via Action Scheduler. A "Force refresh" action is available via REST and in the UI.
Thresholds:
Soft threshold — when usage exceeds the configured amount for the chosen scope, a dismissible admin notice is shown and an optional email can be sent (once per period).
Hard threshold — when exceeded, an option is set, all OpenAI API requests are blocked in APIRequest::get/post/post_form (returning a WP_Error), and a non-dismissible error notice is shown with a link to re-enable from the pricing/usage page. Email can be sent once per period.
Scopes: Both thresholds support scope: current month, year to date, or all time.
UI:
- Settings under Tools → ClassifAI → Usage tracking: Admin API key, optional project ID, refresh interval, force refresh button, and soft/hard threshold toggles with amount, scope, and comma-separated emails.
- A dashboard widget shows MTD/YTD/all-time usage and links to configure alerts and force refresh.
Integration:
- Plugin.php registers the Usage Tracking service and OpenAI Usage feature and adds openai_usage_tracking support to load the Action Scheduler.
- APIRequest checks is_request_allowed() (which respects the hard-limit option and the classifai_can_make_request filter) before any get/post/post_form call for OpenAI providers.
- Notifications.php gains render_openai_threshold_notice() so admins see soft/hard threshold warnings in the admin.

Verification:

Enable the feature, set an OpenAI org Admin API key (and optional project ID), save.
Confirm usage appears in the dashboard widget and in the usage-tracking settings.
Set a soft threshold (e.g. $1), trigger a refresh so it's exceeded: dismissible notice and (if emails set) one email per period.
Set a hard threshold and exceed it: OpenAI features are blocked, non-dismissible notice appears, and (if emails set) one email per period.
Re-enable from the linked pricing/usage page and confirm requests work again.
Confirm "Force refresh" in the UI and on the dashboard triggers a refresh and that the REST endpoint /classifai/v1/openai-usage/force-refresh is only allowed when the feature is enabled.

Benefits: Visibility into OpenAI spend (MTD/YTD/all time), alerts and optional automatic disabling to avoid cost overruns, and one email per period per threshold to avoid spam.

Possible drawback / note: Uses an organization-level Admin API key; docs should clarify this is required and separate from project keys.

Closes #122 (comment)

How to test the Change

Install/activate ClassifAI and ensure the new settings UI is used (legacy settings off).
Go to Tools → ClassifAI and open the Usage tracking section for the OpenAI Usage feature.
Verify you see "OpenAI usage tracking" section. Click on toggle button for this feature to enable it.
Click on "Settings" button in "OpenAI usage tracking" section itself, and verify you see "OpenAI usage tracking Settings" section with following fields:
- Enable Feature
- "Select a provider" with "OpenAI Usage Tracking" provider
- Admin API Key
- OpenAI Project ID
- Refresh interval (minutes)
- Force refresh data
- "Soft threshold (alert)" and "Hard threshold (alert)" section with toggle to enable alert and scope, email fields.
Enter an OpenAI organization Admin API key (sk-admin-…) to "Admin API Key". Optionally set OpenAI Project ID to restrict to one project.
Save Settings
Wait for 15min cron to run (or whatever Refresh interval you set) to update the data. OR click on "Force refresh data" button and wait for it to finish refreshing data. (You will see "Force refresh data" button disabled and processing UI until it complete the data updates).
Verify Usage display:: Open the Dashboard and confirm the "OpenAI usage (ClassifAI)" widget shows This month / Year to date / All time (and "Last updated" or "Updating…").
Soft threshold:
- Enable Soft threshold (alert) and set a low amount (e.g. $1), scope (e.g. Current month), and at least one email. Save.
- Force refresh so current usage exceeds the amount.
- Confirm a dismissible warning notice appears and that the configured email(s) receive one alert for that period.
- Dismiss the notice and confirm it does not reappear until the next period (or after clearing the dismissed meta).
Hard threshold:
- Enable Hard threshold (disable features) and set an amount already exceeded (or very low), with scope and emails. Save
- Force refresh so the hard limit is exceeded.
- Confirm a red, non-dismissible error notice appears and that OpenAI-powered features (e.g. ChatGPT, image generation) are blocked (e.g. requests return an error about the hard limit).
- Use the link in the notice to open the pricing/usage page and re-enable; confirm the notice can be cleared and OpenAI features work again.
- Confirm one hard-limit email per period when emails are set.
Regression: With usage tracking disabled, confirm existing OpenAI features still work and no usage notices appear. Confirm the new REST route and dashboard script only load when the feature is enabled.

Changelog Entry

Added - OpenAI usage tracking: monitor usage/costs via OpenAI org API, optional soft (notice and email alert) and hard (notice, email and disable OpenAI features) spending thresholds with configurable scope (current month, YTD, all time), dashboard widget, and force refresh.

Credits

Props @rahulsprajapati

Checklist:

I agree to follow this project's Code of Conduct.
I have updated the documentation accordingly.
I have added Critical Flows, Test Cases, and/or End-to-End Tests to cover my change.
All new and existing tests pass.

github-actions · 2026-02-25T15:41:56Z

✅ WordPress Plugin Check Report

✅ Status: Passed

📊 Report

All checks passed! No errors or warnings found.

🤖 Generated by WordPress Plugin Check Action • Learn more about Plugin Check

dkotter

Left a decent number of comments here that need addressed.

That said, I still don't know if this PR matches my expectations in terms of extensibility (same comment I had on the first PR).

If I wanted to add usage tracking for another Provider, I'd have to duplicate basically all the code here. I still think there's a better approach where we can extract out all of the shared functionality, making it much easier and cleaner to add support in the future.

I'm also not sure on the use of our Feature and Provider classes here. I know that's how the rest of ClassifAI is set up but I don't think these things really fit neatly into that paradigm. Doesn't necessarily need changed but something to consider.

Edit: at the very least, I think if we want to follow this structure we need to make the Usage Tracking Feature generic, it should not be tied to OpenAI at all (this is how all other Features work). Any OpenAI specific things can then be in the Usage Provider. I'd also recommend we create a general Usage Provider that contains all shared functionality. Then the OpenAI specific Usage Provider can extend that (or similar)

Second edit: Thinking about this some more, the above wouldn't quite work as we only support a single Feature -> Provider setup, where here ideally we can turn on usage tracking for multiple Providers. So I go back to my original thought that not sure our standard Feature / Provider setup works for this, at least not without some changes to how we render the settings and allow things to be turned on and configured

dkotter · 2026-02-25T20:31:31Z

includes/Classifai/Providers/OpenAI/APIRequest.php

 	 * @return array|WP_Error
 	 */
 	public function get( string $url, array $options = [] ) {
+		if ( ! $this->is_request_allowed() ) {


See my comment from the other PR: #1054 (comment). I know we have a new check using OpenAIUsage::get_openai_provider_ids but I'd still suggest this sort of disabling shouldn't happen here as this class won't exist much longer

Let's discuss this over sync. Not sure what other possible solution for this. Disabling feature all together seems confusing for end users.

includes/Classifai/Services/UsageTracking.php

includes/Classifai/Services/ServicesManager.php

dkotter · 2026-02-25T21:02:57Z

includes/Classifai/Features/OpenAIUsage.php

+	const CRON_HOOK = 'classifai_openai_usage_refresh';
+
+	/**
+	 * Cron hook for the usage refresh.


This is the same docblock as the one above. Should update these to be unique

dkotter · 2026-02-25T21:03:32Z

includes/Classifai/Providers/OpenAI/UsageTracking.php

+		switch ( $this->feature_instance::ID ) {
+			case FeatureOpenAIUsage::ID:
+				return array_merge(
+					$common_settings,
+					[
+						'api_key' => '',
+					]
+				);
+		}


This is redundant right now so I'd remove it

dkotter · 2026-02-25T21:05:31Z

includes/Classifai/Providers/OpenAI/APIRequest.php

+		 *
+		 * @return bool
+		 */
+		$allowed = (bool) apply_filters( 'classifai_can_make_request', true, $this->feature, $this->provider );


Suggested change

$allowed = (bool) apply_filters( 'classifai_can_make_request', true, $this->feature, $this->provider );

$allowed = (bool) apply_filters( 'classifai_openai_can_make_request', true, $this->feature, $this->provider );

dkotter · 2026-02-25T21:07:36Z

includes/Classifai/Features/OpenAIUsage.php

+		if (
+			strpos( $route, '/classifai/v1/openai-usage/force-refresh' ) !== 0
+			|| ! function_exists( 'as_enqueue_async_action' )
+			|| (
+				function_exists( 'as_has_scheduled_action' )
+				&& \as_has_scheduled_action( self::FORCE_CRON_HOOK, [], 'classifai' )
+			)
+		) {
+			return parent::rest_endpoint_callback( $request );
+		}


Not sure I understand what this logic is trying to do? Particularly in the case where the action is already scheduled, why do we need to call the parent method?

includes/Classifai/Providers/OpenAI/UsageTracking.php

rahulsprajapati · 2026-02-26T05:52:56Z

That said, I still don't know if this PR matches my expectations in terms of extensibility (#1054 (review) I had on the first PR).

I tried using that approach, but with that it will be whole different structure compare to current ClassifAI structure. Because of that we don't get to reuse same UI/UX and need lots of changes to make UI/UX match current one. So I tried to use existing Provider + Feature classes, which help reuse existing UI/UX. Maybe we need to separate out UI/UX code from current structure and allow any new/extend feature with same UI/UX without having to reuse Provider+Feature structure.

If I wanted to add usage tracking for another Provider, I'd have to duplicate basically all the code here. I still think there's a better approach where we can extract out all of the shared functionality, making it much easier and cleaner to add support in the future.

I think since I'm not sure what other provide feature looks like (how we get usage data, do we need to calculate similar way or get it directly from one endpoint, or require diff admin api key only) because of that I'm unsure to keep which feature/code shared. Like we can have fetch_period method shared but again not sure if we will have same type of api calls. So we can have interface without defined method but not sure what else we can do here.
Maybe once we have another provider as well, we can get more idea on which code/method can be shared.

I'm also not sure on the use of our Feature and Provider classes here. I know that's how the rest of ClassifAI is set up but I don't think these things really fit neatly into that paradigm. Doesn't necessarily need changed but something to consider.

For this, looks like we might need full refactor on plugin itself. Since without using this Feature and Provider we don't get to reuse existing UI/UX and require lots of changes in style and scripts.

About the blocking direct api calls:

I didn't wanted to disable feature itself since that looks confusing when usage is exceeded. Since editor might see as an bug/issue that feature disabled automatically without any notice and try request admin to enable again. We do have notice but that doesn't show up on post edit screen or media uploads screen.
When we block it from api itself, we are keeping all feature as it is but show usage exceed error on each feature usage, which is clear message for why we can't use feature. So to me, displaying api call error when using feature looks better.
Even though this will move out somewhere else, this api call will happen from some or the other file. That can still use same code we already have. This will be moved somewhere else but won't be removed.

But let me know what you think. If this doesn't make sense I can move this block to disable feature itself or whichever other place we can do this.

I'll update other changes and setup a sync to discuss more on this and get this feature ready asap.

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

dkotter · 2026-02-26T14:37:10Z

includes/Classifai/Services/UsageTracking.php

+	 */
+	public static function get_service_providers(): array {
+		/**
+		 * Filter the service providers for Recommendation service.


This isn't accurate, referencing the Recommendation service

rahulsprajapati added 6 commits February 25, 2026 19:01

Add usage tracking feature for OpenAI.

9de09e7

Add openai_usage_tracking credentials masking.

b1bcb83

Update refresh interval filter

8e0c53c

Block api request for usage hard limit reached.

9fdca45

Update usage limit notice to be dismissible for soft alerts only.

8c8d39b

Remove filter from setting description.

c36dd16

github-actions bot assigned rahulsprajapati Feb 25, 2026

github-actions bot added this to the 3.8.0 milestone Feb 25, 2026

Fix phpstan errors

382ce1d

Fix js linting errors

5cc47ed

rahulsprajapati marked this pull request as ready for review February 25, 2026 16:18

rahulsprajapati requested review from a team, dkotter and jeffpaul as code owners February 25, 2026 16:18

github-actions bot added the needs:code-review This requires code review. label Feb 25, 2026

rahulsprajapati mentioned this pull request Feb 25, 2026

Add OpenAI pricing/costs usage feature. #1054

Closed

4 tasks

Fix vip 5s timeout issue and return 0 for float return function.

dc855d5

dkotter requested changes Feb 25, 2026

View reviewed changes

rahulsprajapati and others added 10 commits February 26, 2026 12:39

Update includes/Classifai/Services/ServicesManager.php

c5828a4

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Update includes/Classifai/Services/ServicesManager.php

6fd3d94

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Update includes/Classifai/Features/OpenAIUsage.php

61e9d98

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Update includes/Classifai/Features/OpenAIUsage.php

899cb1f

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Update includes/Classifai/Features/OpenAIUsage.php

7781634

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Update includes/Classifai/Features/OpenAIUsage.php

8e17890

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Apply typo suggestions from code review

fc46262

Co-authored-by: Darin Kotter <darin.kotter@gmail.com>

Fix PR feedbacks.

32a8832

Update dashboard widget markup.

53a9873

Fix lint error for translators comment.

26cc07f

dkotter reviewed Feb 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Usage Tracking feature with OpenAI support#1060

Add Usage Tracking feature with OpenAI support#1060
rahulsprajapati wants to merge 19 commits intodevelopfrom
feature/api-usage-122

rahulsprajapati commented Feb 25, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 25, 2026 •

edited

Loading

Uh oh!

dkotter left a comment •

edited

Loading

Uh oh!

dkotter Feb 25, 2026

Uh oh!

rahulsprajapati Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkotter Feb 25, 2026

Uh oh!

dkotter Feb 25, 2026

Uh oh!

dkotter Feb 25, 2026

Uh oh!

dkotter Feb 25, 2026

Uh oh!

Uh oh!

rahulsprajapati commented Feb 26, 2026 •

edited

Loading

Uh oh!

dkotter Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	$allowed = (bool) apply_filters( 'classifai_can_make_request', true, $this->feature, $this->provider );
	$allowed = (bool) apply_filters( 'classifai_openai_can_make_request', true, $this->feature, $this->provider );

Conversation

rahulsprajapati commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the Change

How to test the Change

Changelog Entry

Credits

Checklist:

Uh oh!

github-actions bot commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ WordPress Plugin Check Report

📊 Report

Uh oh!

dkotter left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkotter Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

rahulsprajapati Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkotter Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rahulsprajapati commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkotter Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rahulsprajapati commented Feb 25, 2026 •

edited

Loading

github-actions bot commented Feb 25, 2026 •

edited

Loading

dkotter left a comment •

edited

Loading

rahulsprajapati commented Feb 26, 2026 •

edited

Loading