Update: I have found a workaround but not a solution. This is definitely a bug as far as I cant tell.
One option, if you do not require Greek alphabet, is to simply use HTML code, simply using italics, superscripts or subscripts.
If one wants to use Greek letters, HTML entities don’t work, but a few decimal and hexadecimal codes do work.
This issue should be addressed, especially if Plotly is to be used for publishing purposes. Otherwise, Matplotlib is the alternative that does work.