Great question @cforelle! It seems like these should be the default or at least an option in the Graph object. Something like Graph(..., freeze=True) or Graph(..., preserve_interactions=True) or something. This comes up if the user zooms into a region and the updates a callback: the graph scales back out (zoomed region is not preserved).
For now, you can add the figure as a state=dash.dependencies.State() property which will provide the value of the figure but will not trigger the callback when that value changes (which would cause an infinite loop). So:
@app.callback(
dash.dependencies.Output('test_graph', 'figure'),
[dash.dependencies.Input('test_slider', 'value')],
state=[dash.dependencies.State('test_graph', 'figure')])
def update_figure(value, previous_figure):