[DEV-12952] Add Configurable Retry Behavior

[mawelborn]

Summary

This PR implements configurable, best-practices retry behavior for both IndicoClient and AsyncIndicoClient with consistent socket connect and read timeouts. This replaces the existing retry behavior that's only present on the async client and isn't configurable. IndicoConfig has new options to control retry behavior, with reasonable defaults:

IndicoConfig(
    retry_count: int = 4,  # Retry API calls this many times.
    retry_wait: float = 1,  # Wait this many seconds after the first error before retrying.
    retry_backoff: float = 4,  # Multiply the wait time by this amount for each additional error.
    retry_jitter: float = 1,  # Add a random amount of time (up to this percent as a decimal) to the wait time to prevent simultaneous retries.
)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• Jira story/task is put into PR title
• Code has been self-reviewed
• Hard-to-understand areas have been commented
• Documentation has been updated
• My changes generate no new warnings
• Unit tests have been added for base functionality and known edge cases
• Any dependent changes have been merged and published in downstream modules

[mawelborn]

Hi @IndicoDataSolutions/pr-be-indicodata-ai folks! I rebased this PR onto main and have all the checks passing. Would appreciate a review when you have a chance. This functionality will be super useful for ProServe and customers.

[thearchitector]

if this is an urgent thing, this can probably wait, but on the platform side we've been converging on this library, which might be worth considering here instead of shipping our own? it has fancy timing, logging, and backoff stuff we could probably expose if we wanted for better script integrations

[mawelborn]

Hello again! Sorry, just now getting back to this.

I don't have strong opinions regarding the retry implementation--whether we ship our own or use a library like Tenacity. I used the retry implementation from the toolkit since it's a known quantity, and shipping our own was already being done for AsyncIndicoClient.

Do I need to swap out the implementation or make any other changes? Or is the current implementation good enough for now?

Thanks!

[thearchitector]

we shouldn't need both inner and outer return type here i don't think? both the wrapper and the fn should be typed to return the same thing (since they do). that would probably alleviate the any typing problem too

also | isn't valid for <3.10. has to be Union[]

[mawelborn]

Ah good catch! Those two return types are the same in this version.

It's valid on 3.9 when string escaped, like the other types that are only imported when type checking. I've run it and type checked it with mypy --strict on 3.9 without issue.

[mawelborn]

@IndicoDataSolutions/pr-be-indicodata-ai it looks like one of you will have to approve and/or merge?