Update queuing logic to avoid lock notify contention

[AMOOOMA]

Make sure we wake the next thread in line instead of notifying all and race to hope that the first item in queue wins.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @AMOOOMA, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the queuing and synchronization logic within the ModelManager to mitigate lock contention, particularly when many threads are waiting to acquire a model. By transitioning from a broadcast-based notification system (notify_all) to a more precise, ticket-based approach that wakes only the next eligible thread, the changes aim to improve the fairness and efficiency of resource acquisition, especially under high load.

Highlights

• Targeted Thread Notification: Replaced threading.Condition.notify_all() with a mechanism to wake only the next thread in the queue, reducing contention and improving fairness.
• QueueTicket Introduction: Introduced a QueueTicket class to encapsulate priority, ticket number, tag, and a dedicated threading.Event for individual thread waking, enabling more precise control over thread notifications.
• Efficient Cancellation Handling: Added a _cancelled_tickets set and logic to efficiently remove cancelled requests from the head of the wait queue without requiring a full heap re-heapify operation.
• Refactored Waiting Logic: Implemented _wake_next_in_queue and _wait_in_queue helper methods to streamline the queuing and notification process within the acquire_model method.
• Enhanced Timeout Test: Updated the test_model_manager_timeout to simulate higher concurrency (1000 threads) and added a post-timeout check to verify the ModelManager remains functional after a timeout scenario.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• sdks/python/apache_beam/ml/inference/model_manager.py
  • Introduced QueueTicket class with priority, ticket_num, tag, and wake_event attributes, and a custom __lt__ method for heap ordering.
  • Added _cancelled_tickets set to ModelManager to track tickets that have been cancelled.
  • Implemented _wake_next_in_queue method to notify the next valid ticket in the priority queue.
  • Implemented _wait_in_queue method to allow a specific ticket to wait on its wake_event.
  • Modified acquire_model to create and use QueueTicket objects instead of raw tuples.
  • Replaced self._cv.wait() with self._wait_in_queue(my_ticket) for waiting threads.
  • Replaced self._cv.notify_all() with self._wake_next_in_queue() in relevant sections, including model release and priority re-evaluation.
  • Updated _evict_to_make_space to access the tag attribute directly from QueueTicket objects.
  • Improved cleanup logic in acquire_model's finally block to mark cancelled tickets instead of searching and re-heapifying.
• sdks/python/apache_beam/ml/inference/model_manager_test.py
  • Expanded test_model_manager_timeout to submit 1000 acquire_model calls concurrently.
  • Added a post-timeout check to ensure the ModelManager can still acquire and release models, verifying its continued functionality.

Activity

• The pull request includes a standard contribution checklist.
• CI build statuses for Python, Java, and Go tests are displayed.
• No human comments or reviews have been recorded yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 0% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.87%. Comparing base (4e218f0) to head (aee2900).
⚠️ Report is 9 commits behind head on master.

Files with missing lines	Patch %	Lines
...s/python/apache_beam/ml/inference/model_manager.py	0.00%	35 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #37528      +/-   ##
============================================
- Coverage     35.87%   35.87%   -0.01%     
  Complexity     1676     1676              
============================================
  Files          1063     1063              
  Lines        166481   166580      +99     
  Branches       1208     1208              
============================================
+ Hits          59722    59756      +34     
- Misses       104578   104643      +65     
  Partials       2181     2181              

Flag	Coverage Δ
python	39.68% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[AMOOOMA]

R: @damccorm

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[damccorm]

/gemini review

[gemini-code-assist]

Code Review

The pull request refactors the model acquisition queuing logic within the ModelManager to improve concurrency and reduce contention. It introduces a QueueTicket class and a specific threading.Event based notification mechanism (_wake_next_in_queue, _wait_in_queue) to wake up only the next eligible thread, replacing the less efficient notify_all(). Additionally, a _cancelled_tickets set is introduced to efficiently handle tickets that are no longer valid but might still be in the priority queue, avoiding costly heap re-organizations. The test suite has also been enhanced to better validate the timeout and concurrency behavior.

[damccorm]

I see a few flakes related to #37528 but https://github.com/apache/beam/actions/runs/21734367880/job/62696049665?pr=37528 maybe looks concerning

self = <apache_beam.ml.inference.model_manager_test.TestModelManager testMethod=test_single_model_convergence_with_fluctuations>

    def test_single_model_convergence_with_fluctuations(self):
      """
      Tests that the estimator converges to the true usage with fluctuations.
      """
      model_name = "fluctuating_model"
      model_cost = 3000.0
      load_cost = 2500.0
      # Fix random seed for reproducibility
      random.seed(42)
    
      def loader():
        self.mock_monitor.allocate(load_cost)
        return model_name
    
      model = self.manager.acquire_model(model_name, loader)
      self.manager.release_model(model_name, model)
      initial_est = self.manager._estimator.get_estimate(model_name)
      self.assertEqual(initial_est, load_cost)
    
      def run_inference():
        model = self.manager.acquire_model(model_name, loader)
        noise = model_cost - load_cost + random.uniform(-300.0, 300.0)
        self.mock_monitor.allocate(noise)
        time.sleep(0.1)
        self.mock_monitor.free(noise)
        self.manager.release_model(model_name, model)
        return
    
      with ThreadPoolExecutor(max_workers=8) as executor:
        futures = [executor.submit(run_inference) for _ in range(100)]
    
      for f in futures:
        f.result()
    
      est_cost = self.manager._estimator.get_estimate(model_name)
>     self.assertAlmostEqual(est_cost, model_cost, delta=100.0)
E     AssertionError: 2768.996662756712 != 3000.0 within 100.0 delta (231.00333724328812 difference)

I don't think this is caused by this PR, but any ideas on what might be causing this?

[damccorm]

I see a few flakes related to #37528 but https://github.com/apache/beam/actions/runs/21734367880/job/62696049665?pr=37528 maybe looks concerning

self = <apache_beam.ml.inference.model_manager_test.TestModelManager testMethod=test_single_model_convergence_with_fluctuations>

    def test_single_model_convergence_with_fluctuations(self):
      """
      Tests that the estimator converges to the true usage with fluctuations.
      """
      model_name = "fluctuating_model"
      model_cost = 3000.0
      load_cost = 2500.0
      # Fix random seed for reproducibility
      random.seed(42)
    
      def loader():
        self.mock_monitor.allocate(load_cost)
        return model_name
    
      model = self.manager.acquire_model(model_name, loader)
      self.manager.release_model(model_name, model)
      initial_est = self.manager._estimator.get_estimate(model_name)
      self.assertEqual(initial_est, load_cost)
    
      def run_inference():
        model = self.manager.acquire_model(model_name, loader)
        noise = model_cost - load_cost + random.uniform(-300.0, 300.0)
        self.mock_monitor.allocate(noise)
        time.sleep(0.1)
        self.mock_monitor.free(noise)
        self.manager.release_model(model_name, model)
        return
    
      with ThreadPoolExecutor(max_workers=8) as executor:
        futures = [executor.submit(run_inference) for _ in range(100)]
    
      for f in futures:
        f.result()
    
      est_cost = self.manager._estimator.get_estimate(model_name)
>     self.assertAlmostEqual(est_cost, model_cost, delta=100.0)
E     AssertionError: 2768.996662756712 != 3000.0 within 100.0 delta (231.00333724328812 difference)

I don't think this is caused by this PR, but any ideas on what might be causing this?

Actually, I see it here as well #37506

So it is probably unrelated, but may need follow up as well. Rerunning checks now

[damccorm]

LGTM assuming checks are green (or at least unrelated red given our current flake situation :( ). I'll merge once we get that signal