[Discussion] UnicodeDecodeError with vdf.load

[DavidoTek]

We encountered an error where vdf.load(open("config.vdf")) would crash when the vdf file contains an invalid UTF-8 character. This was reported a few times now. See DavidoTek/ProtonUp-Qt#424 for details.

The error occurs in this line of vdf/__init__.py:

  File "/app/lib/python3.11/site-packages/pupgui2/steamutil.py", line 221, in get_steam_global_ctool_name
    d = get_steam_vdf_compat_tool_mapping(vdf.load(open(config_vdf_file)))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/vdf/__init__.py", line 211, in load
    return parse(fp, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/vdf/__init__.py", line 97, in parse
    for lineno, line in enumerate(fp, 1):
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2170: invalid start byte

We are not sure how often the error occurs and were not able to reproduce it. It is likely caused by a misconfigured system locale on Linux.
A similar issue was reported and fixed for vdf.binary_loads in ValvePython#20.

The fix we implemented looks like this (see ProtonUp-Qt PR #455). Invalid UTF-8 characters are simply replaced by "�".

with open("config.vdf", "r", encoding="utf-8", errors="replace") as f:
    x = vdf.loads(f.read())

# Something like this would probably also work - untested:
vdf.load( open("config.vdf", "r", encoding="utf-8", errors="replace") )

I don't think this is something that can be fixed from your side as vdf.load only accepts the file object and the Unicode-decoding and error handling is part of open() which is called by the user. I just thought it might be interesting to share in case others also have a similar issue (this could go into "GitHub Discussions").

[WinterPhoenix]

Yeah, nothing we can do about it. Has to be handled by the open() call or whatever is used to create the file object