[Feature Request] Track exception occurrences accros test runs

[zied-klouz-lb]

Hi,

Appreciate if there would be a dedicated metrics for Exceptions.

Motivation: keep track of occurred exceptions, even when already handled by TRY/EXCEPT.

Suggestions:

• Table for all occurred exceptions
• Bar graph for exceptions (similar to messages graph) for all displayed runs
• Same bar graph for latest run

[timdegroot1996]

Thanks for creating an issue. I will look into the possibilities with the output.xml file and update you when I know more!

[timdegroot1996]

@zied-klouz-lb I'm looking into this right now and so far I tried the following scenario:

*** Test Cases ***
Exceptional Test
    TRY
        Some Keyword With An Exception
    EXCEPT    I'm a failure of a keyword
        Log    message=Caught Exception
    END


*** Keywords ***
Some Keyword With An Exception
    Fail    msg=I'm a failure of a keyword

I'm assuming in above usecase you would be interested in the "I'm a failure of a keyword" message as that is the "exception" that is occuring. Am I right in assuming so? And I'm also wondering if generally failing keywords like below should be handled in the solution you are aiming for. Think about stuff like this:

${status} Run Keyword And Return Status Fail msg=I failed

Should "I failed" in this scenario also be an "exception" that should be in the list of data that you are requesting?

Apart from that I found the following when looking at the result visitor api on a keyword level:

print(keyword): Fail    msg=I'm a failure of a keyword
print(dir(keyword)): ['BREAK', 'CONTINUE', 'ELSE', 'ELSE_IF', 'ERROR', 'EXCEPT', 'FAIL', 'FINALLY', 'FOR', 'GROUP', 'IF', 'IF_ELSE_ROOT', 'ITERATION', 'KEYWORD', 'KEYWORD_TYPES', 'MESSAGE', 'NOT_RUN', 'NOT_SET', 'PASS', 'RETURN', 'SETUP', 'SKIP', 'SUITE', 'TASK', 'TEARDOWN', 'TEST', 'TRY', 'TRY_EXCEPT_ROOT', 'VAR', 'WHILE', '__annotations__', '__bool__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_datetime_to_timestr', '_elapsed_time', '_elapsed_time_from_children', '_end_time', '_get_id', '_include_in_repr', '_repr_format', '_setter__body', '_setter__tags', '_setup', '_start_time', '_teardown', '_timestr_to_datetime', 'args', 'assign', 
'body', 'body_class', 'config', 'copy', 'deepcopy', 'doc', 'elapsed_time', 'elapsedtime', 'end_time', 'endtime', 'failed', 'from_dict', 'from_json', 'full_name', 'has_setup', 'has_teardown', 'id', 'kwname', 'libname', 'message', 'messages', 'name', 'not_run', 'owner', 'parent', 'passed', 'repr_args', 'setup', 'skipped', 'source_name', 'sourcename', 'start_time', 'starttime', 'status', 'tags', 'teardown', 'timeout', 'to_dict', 'to_json', 'type', 'visit']       
print(keyword.failed): True
print(keyword.body): [robot.result.Message(message="I'm a failure of a keyword", level='FAIL')]
print(keyword.message): I'm a failure of a keyword

print(keyword): Some Keyword With An Exception
print(dir(keyword)): ** same huge list as above ** 
print(keyword.failed): True
print(keyword.body): [robot.result.Keyword(name='Fail', args=("msg=I'm a failure of a keyword",), assign=())]
print(keyword.message): I'm a failure of a keyword

print(keyword): Log    message=Caught Exception
print(dir(keyword)): ** same huge list as above **
print(keyword.failed): (empty)
print(keyword.body): [robot.result.Message(message='Caught Exception', level='INFO')]
print(keyword.message): (empty)

As it appears there are no statistics for "caught in exception" but I can check that on a more detailed level in #robotframework in slack, if that is necessary.

So my actual questions for you are the following:

1. Do you want to track "all" failing keywords or only the ones that are in try/except clauses?
2. What part of the error would you want to see visualized? Keyword name + message?

Let me know your thoughts!
Tim

[timdegroot1996]

@zied-klouz-lb Do you have an update on my comment above?

[timdegroot1996]

@zied-klouz-lb, I tried to implement a first version of this exception handling. I tried this test suite below as data for run 1 & 3. And for run 2 I tweaked it a bit to give different exceptions.

I added a new graph in the keyword section and I added a table in the tables tab. Let me know what you think about this implementation!

*** Settings ***
Library    Collections
Library    String

*** Keywords ***
Keyword That Fails With Timeout
    Fail    Timeout exceeded: operation did not complete within 5 seconds

Keyword That Fails With Connection
    Fail    ConnectionError: Unable to connect to remote server

Keyword That Fails With Value
    Fail    ValueError: invalid literal for int() with base 10: 'abc'

Keyword That Fails With Assertion
    Fail    AssertionError: Expected 'OK' but got 'ERROR'

Keyword That Fails With Permission
    Fail    PermissionError: Access denied to resource /admin

Nested Inner Keyword
    Keyword That Fails With Timeout

*** Test Cases ***
Test With Timeout Exception
    [Documentation]    Single TRY with a timeout exception
    TRY
        Keyword That Fails With Timeout
    EXCEPT
        Log    Caught timeout exception
    END
    Log    Test completed successfully

Test With Connection Exception
    [Documentation]    Single TRY with a connection exception
    TRY
        Keyword That Fails With Connection
    EXCEPT
        Log    Caught connection exception
    END

Test With Nested Exceptions
    [Documentation]    Nested TRY/EXCEPT blocks with different exceptions
    TRY
        TRY
            Keyword That Fails With Value
        EXCEPT
            Log    Caught inner value error
        END
        Keyword That Fails With Connection
    EXCEPT
        Log    Caught outer connection error
    END

Test With Multiple Exceptions In Sequence
    [Documentation]    Multiple TRY/EXCEPT blocks one after another
    TRY
        Keyword That Fails With Timeout
    EXCEPT
        Log    Caught first timeout
    END
    TRY
        Keyword That Fails With Assertion
    EXCEPT
        Log    Caught assertion error
    END
    TRY
        Keyword That Fails With Timeout
    EXCEPT
        Log    Caught second timeout
    END

Test With Deeply Nested Exceptions
    [Documentation]    Three levels of nested TRY/EXCEPT
    TRY
        TRY
            TRY
                Keyword That Fails With Permission
            EXCEPT
                Log    Caught at level 3
            END
            Keyword That Fails With Value
        EXCEPT
            Log    Caught at level 2
        END
        Log    Outer TRY continues
    EXCEPT
        Log    Should not reach here
    END

Test With Exception And Failure
    [Documentation]    Has a caught exception but then fails outside TRY
    TRY
        Keyword That Fails With Connection
    EXCEPT
        Log    Caught connection error
    END
    Fail    This test intentionally fails after handling exception

Test With Recurring Assertion Error
    [Documentation]    Same assertion error caught multiple times
    TRY
        Keyword That Fails With Assertion
    EXCEPT
        Log    First assertion caught
    END
    TRY
        Keyword That Fails With Assertion
    EXCEPT
        Log    Second assertion caught
    END

Test With Nested And Recurring Timeout
    [Documentation]    Nested TRY with recurring timeout exceptions
    TRY
        TRY
            Keyword That Fails With Timeout
        EXCEPT
            Log    Inner timeout caught
        END
        Nested Inner Keyword
    EXCEPT
        Log    Outer timeout caught via nested keyword
    END

Test With Exception In Loop
    [Documentation]    Exception occurring inside a FOR loop within TRY
    TRY
        FOR    ${i}    IN RANGE    3
            TRY
                Keyword That Fails With Value
            EXCEPT
                Log    Caught value error iteration ${i}
            END
        END
    EXCEPT
        Log    Should not reach here
    END

Test Without Any Exception
    [Documentation]    A normal passing test with no TRY/EXCEPT
    Log    This is a simple test
    ${result}=    Set Variable    42
    Should Be Equal As Numbers    ${result}    42

log-20260416-004700.html

robot_dashboard.html