I have now found that the correct terminology for this behaviour is zooming and panning.
So, to rephrase my question:
Can I disable only the zoom functionality but keep the pan functionality? Setting fixedrange=True will disable both.
Now this seems to be a duplicate question, already asked here back in 2018. Sadly, this is appearantly still not possible.