Alternative approach to retry failed queries

[philippose]

Hi,
After the discussion regarding retrying of failed kraken requests, I did a lot of reading up into the concept of making HTTP(S) requests more robust. This was also partly because I was getting very frustrated with the load related issues kraken was/has been facing over the last two months or so.

After scouring through different concepts, I found the one described on the following website to be quite elegant:
https://www.peterbe.com/plog/best-practice-with-retries-with-requests

Here is my implementation of the retry logic... Again, there is no "mandatory" change required to the currently existing API... and by default everything is disabled to preserve the present functionality as it is.

I have not been able to test this implementation on kraken because they seemed to have improved their service :-)

Could you kindly have a look at it and see if it is something you want to include into your wrapper?

Thanks, and wish you a wonderful new year :-)

Regards,
Philippose

[veox]

What is the benefit of doing it like this, compared to PR #65 or the retry-on-failure branch linked therein?..

The one small up-side I see is the use of backoff with an ever-increasing delay.

The article you linked sounds like @peterbe's code is intended to interface with arbitrary network services, that could have various misconfigurations or failure modes. We're just connecting to one here, api.kraken.com.

[philippose]

Hello Noel, A Good Evening to you. Thank you for the feedback regarding the code for retrying requests. The reason why I felt the solution from the blog I referenced was better was because it uses the already in-built functionality within the requests and urllib libraries to handle exactly such cases of latency and unavailability of services when connecting with the server. You can specify the list of return codes to handle automatically, and it also handles the number of retries and wait time between retries without any additional code on the application side. Through the configuration method in the class, you can modify the number of retries, the backoff factor and the return codes which triggers a retry, making it possible to modify it on a per call basis (for example, differentiation between private and public calls, etc...) without overloading the "query_public" and "query_private" methods with too many user arguments. My attempt was to reduce the amount of "specialist" code in the krakenex API, be using available functionality. However, krakenex is your creation :-) And if you think that the technique used in the earlier implementation is more in line with your targets and concepts, I am perfectly ok with using that. Maybe we can think about merging some form of retry logic into the master branch soon, because it looks like using kraken in real situations does require some form of ability to retry requests to the kraken servers. wish you a good evening ahead. Regards, Philippose

…

On 2 Jan 2018 16:47, "Noel Maersk" ***@***.***> wrote: What is the benefit of doing it like this, compared to PR #65 <#65> or the retry-on-failure branch linked therein?.. The one small up-side I see is the use of backoff with an ever-increasing delay. The article you linked sounds like @peterbe <https://github.com/peterbe>'s code is intended to interface with arbitrary network services, that could have various misconfigurations or failure modes. We're just connecting to one here, api.kraken.com. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACcov5nlMz6eIeLgJx-h-s_pPCWsu1clks5tGk94gaJpZM4RPQXd> .

[veox]

Much time has passed, and seems Kraken is experiencing high loads again - it's been brought up in issue #98.

After pondering a lot, I think you're right:

My attempt was to reduce the amount of "specialist" code in the krakenex API, be using available functionality.

PR #65 does take a somewhat NIH approach, which tends to become a rake eventually. :)

The configurability of delay periods on failure is also a plus - it might be sensible to make them non-linear in some cases.

I'll try to get to this at the closest opportunity.

[philippose]

Hi,
I just went through my side of the code, merged your current master into the master of the fork, and merged the whole lot into my branch ENH_requests-retry-2.
So this should bring this branch, and consequently the pull request up to date with the current state of your code.

[veox]

I've pulled it in locally, did some clean-up, and am hoping to get to testing it on a live set-up.

There's still some clean-up that I need to do, hopefully will find time this week...

[veox]

Closing this - incorporated into PR #99.