UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 3860: invalid start byte

[fti-lit]

With the new version of the pack, the actions sometimes fail: our powershell output has latin-1 encoding, so the action fails with

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py", line 395, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/python_runner/python_action_wrapper.py", line 214, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/activedirectory/actions/run_cmdlet.py", line 9, in run
    return self.run_ad_cmdlet(cmdlet, **kwargs)
  File "/opt/stackstorm/packs/activedirectory/actions/lib/action.py", line 299, in run_ad_cmdlet
    stdout = stdout.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 3860: invalid start byte

With the previous 1.0.0 version, stdout was of byte type, and stdout_dict contained the decoding error (it "failed" silently). We had to do the decodings ourselves (res_ad = json.loads(ad_result["stdout"].decode("latin1"))) , but at least, we had something to work with.

  "result": "{'stdout': b'[\\r\\n
				   {\\r\\n
				   \"GivenName\":  ...  }\\r\\n
				]\\r\\n',
	'stderr': b'',
	'exit_status': 0,
	'stdout_dict': {'parse_error': 'Traceback (most recent call last):\\n
		  File \"/opt/stackstorm/packs/activedirectory/actions/lib/action.py\", line 241, in parse_output\\n
			parsed_output = json.loads(output_str)\\n
		  File \"/usr/lib/python3.8/json/__init__.py\", line 343, in loads\\n
			s = s.decode(detect_encoding(s), \\'surrogatepass\\')\\n
		UnicodeDecodeError: \\'utf-8\\' codec can\\'t decode byte 0x82 in position 5896: invalid start byte\\n
	'},
 'stderr_dict': {}}"

I don’t know what’s a better/easier fix (and with ones are feasible):  

• try to guess the encoding (I guess not)
• set an encoding as a parameter to the action
• set a default encoding in the .yaml config file and use it in run_ad_cmdlet
• somehow force run_ps to have an utf-8 encoding
• only put a try/except around the decoding and if it fails do nothing (ie still return bytes objects)

For the moment we have reverted to v1.0.0 of the pack.