'encoding' cannot be specified in python 3

[jonapich]

raven-python/raven/utils/json.py

Line 50 in 1a0d697

kwargs['encoding'] = 'safe-utf-8'

The 'json.dumps' command no longer have an encoding argument in python 3 and causes this stacktrace:

File "/usr/local/lib/python3.6/site-packages/raven/handlers/logging.py", line 89, in emit
 return self.emit(record)
 File "/usr/local/lib/python3.6/site-packages/raven/handlers/logging.py", line 182, in _emit
 *kwargs)
 File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 645, in capture
 self.send(*data)
 File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 740, in send
 message = self.encode(data)
 File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 782, in encode
 return zlib.compress(json.dumps(data).encode('utf8'))
 File "/usr/local/lib/python3.6/site-packages/raven/utils/json.py", line 51, in dumps
 return json.dumps(value, cls=BetterJSONEncoder, **kwargs)
 File "/usr/local/lib/python3.6/json/init.py", line 238, in dumps
 **kw).encode(obj)
TypeError: __init_() got an unexpected keyword argument 'encoding'

[ashwoods]

@jonapich I'm having problems trying to create a test case for this issue. Have an example of sentry data where this exception is raised?

[jonapich]

No I don't. I'm not sure a sentry was even raised for this, it probably ended up in our logs instead. I guess there was some invalid UTF data in the exception/locals somewhere?

[cenkalti]

@ashwoods we experience this too. The code puts encoding kwarg into json.dumps call but it is not valid for Python 3. If this code path is executed, it will always fail on Python 3. We see this frequently on Sentry. Here is an example traceback: https://cl.ly/okPZ

Let me know if you need anything else for debugging.

[cenkalti]

@jonapich In our case, there was a type with custom __repr__ mehod that raises an exception. I have opened a PR that fixes this case. If you still have access to logs, can you confirm that there is a __repr__ call in the exception chain?

[cenkalti]

@ashwoods can you review my PR please? #1204

[jonapich]

@cenkalti I only have the stacktrace I posted above, sorry.

[miracle2k]

I don't think this should be closed.

I am seeing this still, and it's a bit confusing, but here is what is happening. We have this code in Sentry:

raven-python/raven/utils/json.py

Lines 46 to 51 in 1a0d697

	def dumps(value, **kwargs):
	try:
	return json.dumps(value, cls=BetterJSONEncoder, **kwargs)
	except Exception:
	kwargs['encoding'] = 'safe-utf-8'
	return json.dumps(value, cls=BetterJSONEncoder, **kwargs)

The logic here seems to be: Try to serialize everything to JSON, and if that fails for any reason, try it again, but this time use a special codec that ignores any errors. (Question: Why use a catch-all here, rather than just catching encoding-related exceptions?).

The problem, as stated above, is that on Python 3, the encoding argument is no longer supported. So if for any reason this fallback-branch triggers, we get an exception like this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/raven/handlers/logbook.py", line 49, in emit
    return self._emit(record)
  File "/usr/local/lib/python3.6/site-packages/raven/handlers/logbook.py", line 100, in _emit
    **handler_kwargs
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 651, in capture
    self.send(**data)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 746, in send
    message = self.encode(data)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 788, in encode
    return zlib.compress(json.dumps(data).encode('utf8'))
  File "/usr/local/lib/python3.6/site-packages/raven/utils/json.py", line 63, in dumps
    return json.dumps(value, cls=BetterJSONEncoder, **kwargs)
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
TypeError: __init__() got an unexpected keyword argument 'encoding'

So the task here would probably to a) find a way to support the fallback-branch on Python 3, or disable the fallback branch on Py3 so the exception is less confusing.

[lbolla]

This is a simple script to reproduce the issue:

In [2]: import raven.utils.json
In [3]: raven.utils.json.dumps({(): 'foo'})
...long stacktrace...
TypeError: __init__() got an unexpected keyword argument 'encoding'

I agree that the catchall is misleading and the utf-8 encoding hack should not be tried in Py3.

In particular, in my application the un-jsonifiable object is in Raven's breadcrumbs and it looks like this:

{'type': 'default',
 'timestamp': 1552913527.0162847,
 'level': '\x1b[0;37mdebug\x1b[1;',
 'message': "some message",
 'category': 'some category',
 'data': {('cn', 1002): 3}}

Note the tuple as key in the 'data' dictionary. Simply using json.dumps(skipkeys=True) would fix the issue, by ignoring invalid keys.