Skip to content

Conversation

@knst
Copy link
Collaborator

@knst knst commented Dec 4, 2025

Issue being fixed or feature implemented

Fixes intermittent failure in feature_governance_cl.py

This error happens ~10-20% on my localhost in case of multi-parallel runs of this specific test.

    raise AssertionError(self._node_msg(msg))
AssertionError: [node 0] Expected messages "['UpdateCachesAndClean -- Governance Objects: 0']" does not partially match log:

 - 2025-12-04T13:01:49.091536Z (mocktime: 1417720362) [      http] [httpserver.cpp:250] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:47922
 - 2025-12-04T13:01:49.091576Z (mocktime: 1417720362) [httpworker.3] [rpc/request.cpp:180] [parse] [rpc] ThreadRPCServer method=setmocktime user=__cookie__
 - 2025-12-04T13:01:49.091601Z (mocktime: 1417720962) [httpworker.3] [httprpc.cpp:92] [~RpcHttpRequest] [bench] HTTP RPC request handled: user=__cookie__ command=setmocktime external=false status=200 elapsed_time_ms=0
 - 2025-12-04T13:01:49.091855Z (mocktime: 1417720962) [      http] [httpserver.cpp:250] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:47922
 - 2025-12-04T13:01:49.091900Z (mocktime: 1417720962) [httpworker.0] [rpc/request.cpp:180] [parse] [rpc] ThreadRPCServer method=mockscheduler user=__cookie__
 - 2025-12-04T13:01:49.091945Z (mocktime: 1417720962) [httpworker.0] [httprpc.cpp:92] [~RpcHttpRequest] [bench] HTTP RPC request handled: user=__cookie__ command=mockscheduler external=false status=200 elapsed_time_ms=0
 - 2025-12-04T13:01:49.091960Z (mocktime: 1417720962) [ scheduler] [wallet/bdb.cpp:604] [PeriodicFlush] [walletdb] Flushing wallet.dat
 - 2025-12-04T13:01:49.092156Z (mocktime: 1417720962) [ scheduler] [wallet/bdb.cpp:612] [PeriodicFlush] [walletdb] Flushed wallet.dat 0ms
 - 2025-12-04T13:01:49.092194Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1149] [RequestGovernanceObjectVotes] [gobject] CGovernanceManager::RequestGovernanceObjectVotes -- start: vTriggerObjHashes 1 vOtherObjHashes 0 mapAskedRecently 3
 - 2025-12-04T13:01:49.092203Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1194] [RequestGovernanceObjectVotes] [gobject] CGovernanceManager::RequestGovernanceObjectVotes -- end: vTriggerObjHashes 0 vOtherObjHashes 0 mapAskedRecently 3
 - 2025-12-04T13:01:49.092268Z (mocktime: 1417720962) [ scheduler] [net_processing.cpp:5761] [CheckForStaleTipAndEvictPeers] Potential stale tip detected, will try using extra outbound peer (last tip update: 600 seconds ago)
 - 2025-12-04T13:01:49.092279Z (mocktime: 1417720962) [ scheduler] [net.cpp:2787] [SetTryNewOutboundPeer] [net] setting try another outbound peer=true
 - 2025-12-04T13:01:49.103502Z (mocktime: 1417720962) [ scheduler] [net.cpp:2754] [DumpAddresses] [net] Flushed 0 addresses to peers.dat  0ms
 - 2025-12-04T13:01:49.103538Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1393] [RequestOrphanObjects] [gobject] CGovernanceObject::RequestOrphanObjects -- number objects = 0
 - 2025-12-04T13:01:49.103543Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:470] [CheckAndRemove] [gobject] CGovernanceManager::UpdateCachesAndClean
 - 2025-12-04T13:01:49.103556Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1519] [CleanAndRemoveTriggers] [gobject] CGovernanceManager::CleanAndRemoveTriggers -- mapTrigger.size() = 1
 - 2025-12-04T13:01:49.103560Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1535] [CleanAndRemoveTriggers] [gobject] CGovernanceManager::CleanAndRemoveTriggers -- superblock status = 2
 - 2025-12-04T13:01:49.103564Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1545] [CleanAndRemoveTriggers] [gobject] CGovernanceManager::CleanAndRemoveTriggers -- Valid trigger found
 - 2025-12-04T13:01:49.103569Z (mocktime: 1417720962) [ scheduler] [governance/classes.cpp:354] [IsExpired] [gobject] CSuperblock::IsExpired -- nBlockHeight = 260, nExpirationBlock = 280
 - 2025-12-04T13:01:49.103572Z (mocktime: 1417720962) [ scheduler] [governance/classes.cpp:357] [IsExpired] [gobject] CSuperblock::IsExpired -- Outdated trigger found
 - 2025-12-04T13:01:49.103577Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1557] [CleanAndRemoveTriggers] [gobject] CGovernanceManager::CleanAndRemoveTriggers -- marked for removal
 - 2025-12-04T13:01:49.103583Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:1566] [CleanAndRemoveTriggers] [gobject] CGovernanceManager::CleanAndRemoveTriggers -- Removing trigger object {"event_block_height": 260, "payment_addresses": "yZV6odagTGg4KNNTLUSH5Zbtmes4niNGsz|ySG7HrWALVPAdaQ7n7ua6gZ1JpD3rkRkJH", "payment_amounts": "1.10000000|3.30000000", "proposal_hashes": "fb196adc9f048cef43a5f138163a8925e8d7f992da14f9e23a1384e9587261e5|f75c0c0a77977191178b22c93bc09539d7e8e4445df837da30d00c662c3623b2", "type":2}
 - 2025-12-04T13:01:49.103591Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:510] [CheckAndRemove] [gobject] CGovernanceManager::UpdateCachesAndClean -- Checking object for deletion: 38c677b5b9803d0be5a3e4159280d8d6dce6dc731068e897044eea39ad628f23, deletion time = 1417720962, time since deletion = 0, delete flag = 1, expired flag = 1
 - 2025-12-04T13:01:49.103598Z (mocktime: 1417720962) [ scheduler] [governance/governance.cpp:575] [CheckAndRemove] [gobject] CGovernanceManager::UpdateCachesAndClean -- Governance Objects: 1 (Proposals: 0, Triggers: 1, Other: 0; Erased: 2), Votes: 4, m_requested_hash_time size=0

What was done?

Generated one more block to trigger scheduler.
I am not sure why it helps, but it drops failure rate to almost 0% (unrelated failures) for my local environment.
See this PR #6926 which improved reliability of feature_governance_cl.py significantly in the past.

How Has This Been Tested?

Run multiple times with and without changes from this PR:

test/functional/test_runner.py -j20 feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py feature_governance_cl.py 

Breaking Changes

N/A

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone

@github-actions
Copy link

github-actions bot commented Dec 4, 2025

✅ No Merge Conflicts Detected

This PR currently has no conflicts with other open PRs.

@knst knst force-pushed the fix-governance-cl branch from 6437fa0 to 55136b6 Compare December 4, 2025 13:13
@coderabbitai
Copy link

coderabbitai bot commented Dec 4, 2025

Walkthrough

The changes to test/functional/feature_governance_cl.py adjust the timing and sequencing of block generation operations within a governance cleanup test. An additional block-generation step is reintroduced before the main cleanup loop begins, while an earlier generate call after bump_mocktime(156) is removed. This restructuring ensures block progression is synchronized with governance cleanup event evaluation, with explicit generate calls inserted at key points to align state transitions with trigger cleanup timing expectations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

  • Verify the logic of moving/adding generate calls affects test correctness as intended
  • Confirm the removal of the early generate call does not inadvertently skip necessary state progression
  • Ensure the block advancement timing aligns with the governance cleanup trigger evaluation expectations

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title refers to making scheduler calls more reliable in feature_governance_cl.py, which aligns with the main change of adding block-generation steps to improve test reliability.
Description check ✅ Passed The description clearly explains the intermittent test failure, the fix applied (adding an extra block generation), testing methodology, and references a prior reliability improvement PR.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
test/functional/feature_governance_cl.py (1)

145-155: Per-iteration block generation before scheduler looks sound; consider documenting intent

The added generate(..., sync_fun=lambda: self.sync_blocks(self.nodes[0:5])) inside the (delta, expected) loop cleanly advances the chain on non-isolated nodes before each governance-cleanup scheduler run, which matches the goal of making the expected log sequence deterministic. The ordering with self.mocktime += delta and subsequent mockscheduler(delta) calls is internally consistent and doesn’t introduce obvious race or state issues.

Given that this extra block-per-phase is a subtle test stabilization detail, consider adding a brief comment above the loop explaining that the additional generation is required to reliably age and delete old governance triggers across nodes. This would help future readers understand why the extra block is needed here.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 09d533a and 55136b6.

📒 Files selected for processing (1)
  • test/functional/feature_governance_cl.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
test/functional/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Functional tests in test/functional/ must be written in Python (minimum version specified in .python-version) and depend on dashd and dash-node

Files:

  • test/functional/feature_governance_cl.py
🧠 Learnings (2)
📓 Common learnings
Learnt from: kwvg
Repo: dashpay/dash PR: 6718
File: test/functional/test_framework/test_framework.py:2102-2102
Timestamp: 2025-06-09T16:43:20.996Z
Learning: In the test framework consolidation PR (#6718), user kwvg prefers to limit functional changes to those directly related to MasternodeInfo, avoiding scope creep even for minor improvements like error handling consistency.
Learnt from: kwvg
Repo: dashpay/dash PR: 6543
File: src/wallet/receive.cpp:240-251
Timestamp: 2025-02-06T14:34:30.466Z
Learning: Pull request #6543 is focused on move-only changes and refactoring, specifically backporting from Bitcoin. Behavior changes should be proposed in separate PRs.
Learnt from: UdjinM6
Repo: dashpay/dash PR: 6926
File: test/functional/feature_governance_cl.py:0-0
Timestamp: 2025-10-28T08:54:00.392Z
Learning: In Dash tests, the scheduler (mockscheduler) operates independently of network state. Isolated nodes should still run scheduler-based cleanup processes like governance cleanup, even if they have different state due to network isolation.
📚 Learning: 2025-11-24T16:41:22.457Z
Learnt from: CR
Repo: dashpay/dash PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-11-24T16:41:22.457Z
Learning: Applies to src/governance/**/*.{cpp,h} : Governance implementation must support governance objects (proposals, triggers, superblock management) and on-chain voting with tallying

Applied to files:

  • test/functional/feature_governance_cl.py
🧬 Code graph analysis (1)
test/functional/feature_governance_cl.py (1)
test/functional/test_framework/test_framework.py (1)
  • sync_blocks (812-832)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
  • GitHub Check: linux64_multiprocess-build / Build source
  • GitHub Check: linux64_nowallet-build / Build source
  • GitHub Check: win64-build / Build source
  • GitHub Check: mac-build / Build source
  • GitHub Check: linux64_tsan-build / Build source
  • GitHub Check: linux64-build / Build source
  • GitHub Check: linux64_fuzz-build / Build source
  • GitHub Check: linux64_ubsan-build / Build source
  • GitHub Check: linux64_sqlite-build / Build source
  • GitHub Check: arm-linux-build / Build source

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant