The Grafana chart & alert for Balances (milliFIL) should take into account pending transactions

[bajtos]

spark-evaluate is not able to commit new transactions:

2024-08-09T10:20:01Z app[e2867541be3e68] cdg [info]    message: 'failed to check balance: not enough funds including pending messages (required: 192.747690270205419759 FIL, balance: 192.688139350355372554 FIL): validation failure'

Yet, our dashboard monitoring the balances shows all is fine, and no alert was triggered.

[juliangruber]

I don't understand this, which transaction would require 192 FIL to complete?

[bajtos]

message: 'failed to check balance: not enough funds including pending messages (required: 192.747690270205419759 FIL, balance: 192.688139350355372554 FIL)

I don't understand this, which transaction would require 192 FIL to complete?

My understanding of the problem:

• To pay for all pending transactions and the new transaction we are creating, we need 192.74 FIL.
• The fees for the pending transactions are not deducted from the wallet balance
• Our wallet has only 192.68 FIL, which was enough to cover all previous pending transactions, but it's not enough to also cover the new TX we are creating

Let's say there are 900 pending transactions and each transaction requires 0.214 FIL on average.

• 900 * 0.214 = 192.6 - we have enough funds to pay for all of these TXs
• 901 * 0.214 = 192.814 - we don't have enough funds to pay for the 901th TX

[juliangruber]

I understand the pending txs situation now 👍

[juliangruber]

Do you think this is urgent, especially given the recent cancellation of stuck txs? Otherwise I propose to do this a bit later

[bajtos]

The problem is that we don't have visibility into the actual amount of remaining funds in service wallets. The Grafana charts give us a false sense that all is good.

I'd say the priority depends on how easy/difficult this is to implement. If it's just adding another telegraf section to make an API call to get the amount reserved by the pending transactions, then I'd say let's do it soon.

I'll leave the decision for you to make, you know more about this part of our stack and how bad it is that we don't see amount reserved for pending transactions.

[juliangruber]

Asked about this in https://filecoinproject.slack.com/archives/CRK2LKYHW/p1725016742675359

[bajtos]

We encountered this problem again this morning.

Sep 20 08:20:19 9b22 vector: spark-evaluate 
CANNOT SUBMIT SCORES FOR ROUND 15534n (CALL 1/10): 
Error: could not coalesce error (error={ 
  "code": 1, 
  "message": "failed to check balance: not enough funds including pending messages (required: 85.823851039775149633 FIL, balance: 85.810686571479888285 FIL): validation failure" 
}