Oh snap. I figured it out. This is caused by setting custom margin with fig.update_layout. If margin is set to 0 for left and right, the y-axis title(s) is pushed within the plot.
E.g.
Will leave this here for others.