Add OpenAI pricing/costs usage feature.#1054
Add OpenAI pricing/costs usage feature.#1054rahulsprajapati wants to merge 15 commits intodevelopfrom
Conversation
✅ WordPress Plugin Check Report
📊 ReportAll checks passed! No errors or warnings found. 🤖 Generated by WordPress Plugin Check Action • Learn more about Plugin Check |
This reverts commit 969f576.
rahulsprajapati
requested review from
a team,
dkotter and
jeffpaul
as code owners
space: no space:
Recommendation Service: Recommendation settings: Can probably leverage some of the existing copy:
Example: Potential copy for Usage Tracking... Heading: OpenAI usage tracking settings Alert: checkbox Enable Deactivate features: checkbox Enable left-aligned secondary action button: Back to dashboard Let's move the "Force refresh data" from the settings directly to the Dashboard widget. Updating there to make the "Configure usage tracking settings" the primary action button and "Force refresh data" a secondary action button only displayed for admins. Within the
I think that captures my initial feedback, but without me sharing or making visual edits it might be best to jump on a call to tweak these together or accept we may need another round of minor tweaks. |
dkotter
left a comment
dkotter
left a comment
There was a problem hiding this comment.
I've left a few comments on things I'm seeing, still need to do a more thorough review.
That said, one thing I'm noticing here is how specific to OpenAI everything is. I know the end goal is to only support OpenAI for this initial release but there is potential to extend this to other Providers in the future.
At the moment, we'd need to either do a lot of refactoring at that point or have a lot of code duplication. I don't necessarily want to slow down this PR but thinking we may want to refactor here to have more generic classes and functions that we can then extend with OpenAI specific needs. For instance, I'd imagine any Provider we want to support is going to use the same functionality in regards to showing admin notices or rendering the dashboard widget. Or even getting usage information. The actual API requests will need to be unique but most everything else feels like it could be shared.
Let me know if you have any thoughts on that or if I can explain things better.
| * @return array|WP_Error | ||
| */ | ||
| public function get( string $url, array $options = [] ) { | ||
| if ( ! $this->openai_request_allowed() ) { |
There was a problem hiding this comment.
We'll need to update how we turn off Features as other Providers use this same APIRequest class (lots of Providers follow the OpenAI API spec). We're also looking to remove this class in the near future and replace with a central APIRequest class.
My suggestion for how to handle this is to use the Feature enable filters ('classifai_' . static::ID . '_is_feature_enabled' or 'classifai_' . static::ID . '_is_enabled'). If neither of those filters will work, we can look to add a new filter to handle this use case
| } | ||
|
|
||
| $key = 'openai_threshold_reached'; | ||
| if ( get_user_meta( get_current_user_id(), "classifai_dismissed_{$key}", true ) ) { |
There was a problem hiding this comment.
I'm wondering if we want this dismissible? I typically err on the side of making things dismissible to avoid user's being annoyed but if threshold limits are reached, seems like this is an admin notice we would want to always show. Maybe dismissible on the soft threshold, non-dismissible on the hard?
| $classes[] = 'notice-error'; | ||
| $message = sprintf( | ||
| /* translators: 1: amount with currency, 2: link to settings */ | ||
| __( 'OpenAI features are currently disabled due to exceeded your hard limit of %1$s for this period. <a href="%2$s">Re-enable it from the pricing page</a>.', 'classifai' ), |
There was a problem hiding this comment.
| __( 'OpenAI features are currently disabled due to exceeded your hard limit of %1$s for this period. <a href="%2$s">Re-enable it from the pricing page</a>.', 'classifai' ), | |
| __( 'OpenAI Features are currently disabled due to you exceeding your hard limit of %1$s for this period. <a href="%2$s">Re-enable it from the pricing page</a>.', 'classifai' ), |
| ], | ||
| ]; | ||
|
|
||
| $response = safe_wp_remote_get( $url, $options ); |
There was a problem hiding this comment.
Could we use the APIRequest class to handle this request? If so, would reduce the amount of code here a bit
| 'soft_threshold_amount' => isset( $pricing['soft_threshold_amount'] ) ? abs( (float) $pricing['soft_threshold_amount'] ) : 0, | ||
| 'soft_threshold_scope' => $soft_scope, | ||
| 'soft_threshold_emails' => isset( $pricing['soft_threshold_emails'] ) ? sanitize_textarea_field( $pricing['soft_threshold_emails'] ) : ( $current['soft_threshold_emails'] ?? '' ), | ||
| 'hard_threshold_amount' => isset( $pricing['hard_threshold_amount'] ) ? abs( (float) $pricing['hard_threshold_amount'] ) : 0, |
There was a problem hiding this comment.
I'd suggest we have better defaults than $0 for the thresholds. Right now if I turn things on but don't modify the value, I get alerts immediately because I've exceeded $0. Maybe something like $5 for the soft, $10 for the hard?
| description={ sprintf( | ||
| /* translators: %d: default minutes */ | ||
| __( | ||
| 'How often to fetch usage (default %d). Filter: classifai_openai_usage_refresh_interval_minutes', |
There was a problem hiding this comment.
| 'How often to fetch usage (default %d). Filter: classifai_openai_usage_refresh_interval_minutes', | |
| 'How often to fetch usage (default %d)', |
|
|
||
| $interval = $this->get_refresh_interval_minutes(); | ||
| if ( ! \as_has_scheduled_action( self::CRON_HOOK ) ) { | ||
| \as_schedule_recurring_action( time(), $interval, self::CRON_HOOK, [], 'classifai' ); |
There was a problem hiding this comment.
So I don't think this is actually working right now, as Action Scheduler is only available if certain Features are turned on. So if those Features are on, this works. But if none of those are, these functions won't exist. We'll need to modify our Action Scheduler loading logic to account for this
| 'enabled' => false, | ||
| 'admin_api_key' => '', | ||
| 'project_id' => '', | ||
| 'refresh_interval_minutes' => ( self::DEFAULT_REFRESH_INTERVAL / MINUTE_IN_SECONDS ), | ||
| 'soft_threshold_enabled' => false, | ||
| 'soft_threshold_amount' => 0, | ||
| 'soft_threshold_scope' => 'current_month', | ||
| 'soft_threshold_emails' => '', | ||
| 'hard_threshold_amount' => 0, | ||
| 'hard_threshold_scope' => 'current_month', | ||
| 'hard_threshold_emails' => '', | ||
| 'hard_threshold_enabled' => false, | ||
| 'api_start_year' => 2020, | ||
| 'all_year_pricing' => [], | ||
| 'this_month_total' => 0, | ||
| 'ytd_total' => 0, | ||
| 'all_time_total' => 0, | ||
| 'usage_currency' => 'USD', | ||
| 'usage_last_updated' => 0, |
There was a problem hiding this comment.
All of these fields are repeated multiple places throughout the code, both on the PHP side and JS side. I don't know exact approach but seems like there should be a better way to manage these that doesn't require us to repeat all of these over and over
To this point, did a quick refactor using Claude Code that I think is closer to what I envision (noting I have not fully tested this, it was a quick iteration with AI so definitely needs more work). But was hoping this would better explain how this could work: https://github.com/10up/classifai/compare/refactor/usage-tracking-extensibility?expand=1 |
Yeah this make sense and now that we do have some time(few more days), we can refactor this. I tried to add this feature asap given the time. I tried to reuse as much as code/structure I could, but majority of time I ended up spending time on fixing issue, so created some new file/template. I'll review this https://github.com/10up/classifai/compare/refactor/usage-tracking-extensibility?expand=1 and update this PR. |
I'd prefer to refactor before shipping, especially since known extensions on this can leverage this branch for now (or an iteration of the branch) while we get things "better" for broad usage in ClassifAI core. |
|
@dkotter , @jeffpaul I created new PR #1060 to keep clean PR with refactoring. I used existing code structure to get proper UI for this. This fixes all the spacing issue and keeps the consistence UI across all features. Also, used provider and features classes so it can be reused to add new provider for this feature. We can close this ticket and start review on new one. |
|
Closing in favor of #1060 |
3 participants
Description of the Change
Features
This PR adds OpenAI usage monitoring and spending controls to ClassifAI:
OpenAI Pricing Controller (
OpenAIPricingController.php): Central admin controller that:UsageCostsclass.classifai_openai_pricingoption and refreshes it on a configurable interval (default 15 minutes) using Action Scheduler.classifai_openai_hard_limit_reachedoption, which gates all OpenAI API requests inAPIRequest(get/post/post_form) and shows an error notice; optionally sends an email. Users can re-enable by increasing the limit or disabling the hard threshold on the Pricing page.UsageCosts (
UsageCosts.php): New provider class that calls the OpenAI organization Costs API (/v1/organization/costs), with optional project ID filter, supports pagination, and providesfetch_costs(start_ts, end_ts)andfetch_all_time_costs()(from 2020).Admin notices (
Notifications.php): Newrender_openai_threshold_notice()— shows a dismissible notice when soft (warning) or hard (error) threshold is exceeded, with link to Tools → ClassifAI → Pricing.REST API (
Settings.php): New routes:GET classifai/v1/openai-usage— returns cached usage for the dashboard/settings UI.GET classifai/v1/openai-pricing-settings— returns pricing settings with API key obfuscated andhard_limit_reachedflag.POST classifai/v1/openai-pricing-settings— updates pricing settings; supportsforce_refresh: trueto trigger an immediate usage refresh.Request gating (
APIRequest.php): Before eachget,post, andpost_form, callsopenai_request_allowed(), which respects the filterclassifai_openai_can_make_requestand the hard-limit option. When not allowed, returns aWP_Errorwith a message directing users to re-enable via ClassifAI → Pricing.Plugin registration (
Plugin.php): RegistersOpenAIPricingControllerinadmin_helpersso it runs in admin (and only when not using legacy settings).Settings UI: New “Pricing” tab in the ClassifAI settings (React) with:
Documentation (
useful-snippets.md): Documents filters and actions:classifai_openai_usage_refresh_interval_minutes,classifai_openai_cached_usage,classifai_openai_can_make_request,classifai_openai_admin_api_key,classifai_openai_usage_updated,classifai_openai_soft_threshold_exceeded,classifai_openai_hard_threshold_exceeded.Closes #122 (comment)
How to test the Change
Changelog Entry
Credits
Props @rahulsprajapati
Checklist: