I am creating a figure with go.Figure. After I create the figure, I can look at the x and y data in the first trace with figure[‘data’][0][‘x’‘] and figure[‘data’][0][‘y’]. x and y are numpy.ndarrays. When I get the figure back in the Status of a callback, x and y are changed to a dict with keys dict_keys([‘dtype’, ‘bdata’, ‘_inputArray’]). It looks like I could get to the data for x and y by figure[‘data’][0][‘x’][’_inputArray’].values() and figure[‘data’][0][y’][‘_inputArray’].values() Is there a better or “proper” way to retrieve the x and y data for a trace in a figure?
@Brent thanks for the question.
Can you please share the code you wrote for this figure and the minimal Dash app examples?
#!/usr/bin/env python3
"""Minimal dash program."""
from dash import callback, Dash, dcc, Input, Output, State
import dash_bootstrap_components as dbc
import pandas as pd
import plotly.express as px
app = Dash(__name__, external_stylesheets=[dbc.themes.BOOTSTRAP], suppress_callback_exceptions=True)
dataframe = pd.DataFrame({'x': [1, 3, 5], 'y': [2, 4, 6]})
figure = px.line(dataframe, x='x', y='y')
print(f"x after creating figure: {figure['data'][0]['x']}")
app.layout = dbc.Container([dcc.Graph(id='graph', figure=figure),
dcc.RangeSlider(1, 5, value=[1, 5], id='range-slider')])
@callback(Output('graph', 'figure'),
Input('range-slider', 'value'),
State('graph', 'figure'))
def fill_graph(fig_range, fig):
print(f"x in callback: {fig['data'][0]['x']}")
print(f"x values in callback: {list(fig['data'][0]['x']['_inputArray'].values())[:-3]}")
fig['layout']['xaxis']['range'] = fig_range
return fig
if __name__ == '__main__':
app.run_server(debug=True)
@Brent, are you trying to return a new figure with the updated ranges?
If so, I think you can use Patch:
from dash import callback, Dash, dcc, Input, Output, Patch
import dash_bootstrap_components as dbc
import pandas as pd
import plotly.express as px
app = Dash(__name__, external_stylesheets=[dbc.themes.BOOTSTRAP])
dataframe = pd.DataFrame({'x': [1, 3, 5], 'y': [2, 4, 6]})
figure = px.line(dataframe, x='x', y='y')
app.layout = dbc.Container([
dcc.Graph(id='graph', figure=figure),
dcc.RangeSlider(1, 5, value=[1, 5], id='range-slider')
])
@callback(
Output('graph', 'figure'),
Input('range-slider', 'value')
)
def fill_graph(fig_range):
# Modify the layout range
patched_fig = Patch()
patched_fig['layout']['xaxis']['range'] = fig_range
return patched_fig
if __name__ == '__main__':
app.run(debug=True)
Or are you trying to retrieve all the x and y data of the figure after using the rangeSlider?
@adamschroeder, my actual use case is more complex of course. Patch does look very useful. However, in this case I am doing more than updating the figure. The data in the graph is a histogram. I actually have 2 range sliders. One for the x values and another one for the percentile of the x values. The sliders work in tandem. If the user moves/changes the x value slider, I also have to show the corresponding movement on the percentile slider and vice versa. In order to compute the percentile slider value, I need the x and y values for the graph. I am trying to prevent having to rebuild the histogram every time the range sliders are moved. I could use a Store to store the x and y values but they can be large.
hi @Brent ,
What is the main issue at stake? Is retrieving the x and y values too slow, too complex? Are you looking for a more efficient way?
It works fine the way I do it but it is not very fast. I also have to use _inputArray and usually _ variables are intended to be hidden. I was just wondering if there was a more efficient or official way.
Hi @Brent
Your workaround is good, but relying on _inputArray might be fragile.
For some background, starting in Plotly 6 NumPy arryay are encoded as base64 to improve performance. You can find more in the Plotly docs here: High performance visualization in Python
NumPy and NumPy Convertible Arrays for Improved Performance
New in Plotly.py version 6
You can improve the performance of generating Plotly figures that use a large number of data points by passing data as NumPy arrays, or in a format that Plotly can convert easily to NumPy arrays, such as pandas and Polars Series or DataFrames. These formats will usually show better performance than passing data as a Python list.
Plotly.py uses Plotly.js for rendering, which supports typed arrays. In Plotly.py, NumPy arrays and NumPy-convertible arrays are base64 encoded before being passed to Plotly.js for rendering.
As you saw in your app, this change does make it more more difficult to extract data from figures in a Dash callback. I’m working on adding a utility that could be imported from plotly and used in a callback. Would you like to give this a try and see if it works for you?
import base64
import numpy as np
SHORT_TO_NUMPY = {
"i1": np.int8,
"u1": np.uint8,
"i2": np.int16,
"u2": np.uint16,
"i4": np.int32,
"u4": np.uint32,
"f4": np.float32,
"f8": np.float64,
}
def decode_typed_arrays(obj):
if isinstance(obj, dict):
if (
"dtype" in obj
and "bdata" in obj
and obj["dtype"] in SHORT_TO_NUMPY
):
try:
arr = np.frombuffer(
base64.b64decode(obj["bdata"]),
dtype=SHORT_TO_NUMPY[obj["dtype"]],
)
if "shape" in obj:
shape = tuple(
int(x.strip())
for x in str(obj["shape"]).split(",")
if x.strip()
)
arr = arr.reshape(shape)
return arr.tolist()
except Exception:
return obj
return {
k: decode_typed_arrays(v)
for k, v in obj.items()
if k != "_inputArray"
}
if isinstance(obj, list):
return [decode_typed_arrays(v) for v in obj]
return obj
Then in the callback you can use it like this:
x = decode_typed_arrays(fig['data'][0]['x'])
I’m looking forward to your feedback. It would be helpful to test this in a real use-case before doing a PR to add this to Plotly.
Hi @AnnMarieW,
Thank you. I was hoping for something like this so I wouldn’t have to depend on the “hidden”_inputArray. I was finally able to test it. It seemed to work fine. I have cases that go through the dict/bdata/dtype path returning arr.tolist() and the return obj path. The question I have is why not just return arr instead of arr.tolist(). For the two cases I have, they return 2 different types. A list and an array. It seems to me like it would be better to be consistent and for the dict case just return arr. I suppose you could make them both list also but most people would probably expect an array since they are probably doing some kind of numpy task with the result. I do not really have a use case for the other returns or even know how I would build them.
Thanks for giving it a try! I’m not sure what you mean by this - can you give me more details or an example?
This returns a list.
That returns a numpy array.
Oh, I see what you mean. However, this is intended to decode the figure prop in a callback which has been serialized to json. There shouldn’t be any numpy arrays there.
In my app I build the figure and then I need to retrieve x and y from that figure to update the x axis before I send the figure back in the callback. When I call your function at that point x and y are numpy arrays. When I call your function for x and y, your function simply returns obj which is a numpy array.
In another location in my app I need to retrieve x and y from the figure to also update the x axis. In this case figure is an input to the callback so x and y have already been converted and your function returns a list from return arr.tolist().
I have a generic function to update the x axis that handles both cases so I just call your function.
Before your function, I was handling the 2 cases like this:
try:
y = np.array(list(figure['data'][0]['y']['_inputArray'].values())[:-3])
x = np.array(list(figure['data'][0]['x']['_inputArray'].values())[:-3])
except IndexError:
y = figure['data'][0]['y']
x = figure['data'][0]['x']
It is really the same generic use case for me(i.e. retrieve x and y from a figure). The figure just happens to be in 2 different states. 1. it has not been sent back in a callback and 2. it has been sent back in a callback.
It is not a big deal to me to get a list one time and an array another since python, numpy, pandas generally handle both.
OK, I was trying to match what you would have gotten back in a callback with the figure prop in Plotly 5 where the numpy (or equivalent) arrays were converted to lists. But you make a good case for returning them as numpy array.
However would that be confusing for people using Polars or Pandas? They might think they are getting the same thing back, but Plotly converts those to numpy before encoding them.
Hi @AnnMarieW,
I do think returning a numpy array would be better than a list.
I wrote a minimal test program to look at this. Things that I found:
- When a plot is created with a pandas dataframe, x and y are numpy arrays initially.
- When the figure is an input in a callback, by that time, x and y are converted as you describe above.
- When a trace is added to that plot using lists for x and y, x and y are tuples initially.
- When the figure is an input in a callback, x and y become lists for the trace created with the lists.
- It would seem like x and y should also be converted for the trace as you describe above but they are not.
I wrote a function with expanded functionality that may be useful. It can be called in the following ways: - One trace at a time for x or y like this: x = decode_x_y(figure[‘data’][0][‘x’])
- One trace at a time for x and y: x, y = decode_x_y(figure[‘data’][0])
- All traces for x and y: data = decode_x_y(figure[‘data’]) or data = decode_x_y(figure)
data[0] is the first trace
data[1] is the 2nd trace
…
data[0][0] is x in the first trace
data[0][1] is y in the first trace
data[1][0] is x in the 2nd trace
data[1][1] is y in the 2nd trace
…
Here is the test program. Move the range slider to trigger the callback.:
#!/usr/bin/env python3
"""Minimal dash program."""
import base64
from dash import callback, Dash, dcc, Input, Output, State
import dash_bootstrap_components as dbc
import numpy as np
import pandas as pd
import plotly.express as px
from plotly.basedatatypes import BaseTraceType, BaseFigure
SHORT_TO_NUMPY = {
"i1": np.int8,
"u1": np.uint8,
"i2": np.int16,
"u2": np.uint16,
"i4": np.int32,
"u4": np.uint32,
"f4": np.float32,
"f8": np.float64,
}
def decode_typed_arrays(obj):
if isinstance(obj, dict):
if (
"dtype" in obj
and "bdata" in obj
and obj["dtype"] in SHORT_TO_NUMPY
):
try:
arr = np.frombuffer(
base64.b64decode(obj["bdata"]),
dtype=SHORT_TO_NUMPY[obj["dtype"]],
)
if "shape" in obj:
shape = tuple(
int(x.strip())
for x in str(obj["shape"]).split(",")
if x.strip()
)
arr = arr.reshape(shape)
return arr.tolist()
except Exception:
return obj
return {
k: decode_typed_arrays(v)
for k, v in obj.items()
if k != "_inputArray"
}
if isinstance(obj, list):
return [decode_typed_arrays(v) for v in obj]
return obj
def decode_x_y(obj):
"""Get x/y from figure."""
ret_val = obj
if isinstance(obj, BaseFigure):
ret_val = decode_x_y(obj['data'])
elif isinstance(obj, BaseTraceType):
ret_val = decode_x_y(obj['x']), decode_x_y(obj['y'])
elif isinstance(obj, (list, tuple)):
try:
if 'x' in obj[0]:
ret_val = [decode_x_y(ii) for ii in obj]
except TypeError:
pass
elif isinstance(obj, dict):
if 'dtype' in obj and 'bdata' in obj and obj['dtype'] in SHORT_TO_NUMPY:
try:
arr = np.frombuffer(base64.b64decode(obj["bdata"]), dtype=SHORT_TO_NUMPY[obj['dtype']])
if 'shape' in obj:
arr = arr.reshape(tuple(int(x.strip()) for x in str(obj['shape']).split(",") if x.strip()))
ret_val = arr
except Exception:
pass
elif 'x' in obj:
ret_val = decode_x_y(obj['x']), decode_x_y(obj['y'])
# ret_val = {kk: decode_x_y(vv) for kk, vv in obj.items() if kk != '_inputArray'}
elif 'data' in obj:
ret_val = decode_x_y(obj['data'])
return ret_val
def print_all_call_variants(fig, msg):
"""Demonstrate and print all of the decode_x_y call variants."""
xx = decode_x_y(fig['data'][0]['x'])
yy = decode_x_y(fig['data'][0]['y'])
print("\ndecode_x_y(fig['data'][0 or 1]['x' or 'y']")
print_x_y(xx, yy, 'pandas', msg)
xx = decode_x_y(fig['data'][1]['x'])
yy = decode_x_y(fig['data'][1]['y'])
print_x_y(xx, yy, 'list', msg)
xx, yy = decode_x_y(fig['data'][0])
print("\ndecode_x_y(fig['data'][0 or 1]")
print_x_y(xx, yy, 'pandas', msg)
xx, yy = decode_x_y(fig['data'][1])
print_x_y(xx, yy, 'list', msg)
data = decode_x_y(fig['data'])
print("\ndecode_x_y(fig['data'])")
print_x_y(data[0][0], data[0][1], 'pandas', msg)
print_x_y(data[1][0], data[1][1], 'list', msg)
data = decode_x_y(fig)
print("\ndecode_x_y(fig)")
print_x_y(data[0][0], data[0][1], 'pandas', msg)
print_x_y(data[1][0], data[1][1], 'list', msg)
def print_x_y(xx, yy, what, msg):
"""Print x and y values and their types."""
print_value_type('x', xx, what, msg)
print_value_type('y', yy, what, msg)
def print_value_type(x_y_str, x_y, what, msg):
"""Print x or y, values and their types."""
print(f"{what} {x_y_str} {msg}: {x_y}")
print(f"type {what} {x_y_str} {msg}: {type(x_y)}")
app = Dash(__name__, external_stylesheets=[dbc.themes.BOOTSTRAP], suppress_callback_exceptions=True)
x = [1, 3, 5]
y = [2, 4, 6]
dataframe = pd.DataFrame({'x': x, 'y': y})
figure = px.line(dataframe, x='x', y='y')
figure.add_scatter(x=y, y=x, mode='lines')
print("figure['data'][0] was created with a pandas dataframe")
print("figure['data'][1] was created with lists for x and y\n")
print_x_y(figure['data'][0]['x'], figure['data'][0]['y'], 'pandas', 'after_creating figure')
print_x_y(figure['data'][1]['x'], figure['data'][1]['y'], 'list', 'after_creating figure')
print('\nCall decode_type_arrays for both\n')
x = decode_typed_arrays(figure['data'][0]['x'])
y = decode_typed_arrays(figure['data'][0]['y'])
print_x_y(x, y, 'pandas', 'after_creating figure and decode_type_arrays')
x = decode_typed_arrays(figure['data'][1]['x'])
y = decode_typed_arrays(figure['data'][1]['y'])
print_x_y(x, y, 'list', 'after_creating figure and decode_type_arrays')
print('\nCall decode_x_y with all variants for both\n')
print_all_call_variants(figure, 'after creating figure and decode_x_y')
app.layout = dbc.Container([dcc.Graph(id='graph', figure=figure),
dcc.RangeSlider(1, 5, value=[1, 5], id='range-slider')])
@callback(Output('graph', 'figure'),
Input('range-slider', 'value'),
State('graph', 'figure'),
prevent_initial_call=True)
def fill_graph(fig_range, fig):
xx = fig['data'][0]['x']
yy = fig['data'][0]['y']
print('\n')
print_x_y(xx, yy, 'pandas', 'in callback')
xx = fig['data'][1]['x']
yy = fig['data'][1]['y']
print('\n')
print_x_y(xx, yy, 'list', 'in callback')
xx = decode_typed_arrays(fig['data'][0]['x'])
yy = decode_typed_arrays(fig['data'][0]['y'])
print('\n')
print_x_y(xx, yy, 'pandas', 'in callback and decode_typed_arrays')
xx = decode_typed_arrays(fig['data'][1]['x'])
yy = decode_typed_arrays(fig['data'][1]['y'])
print('\n')
print_x_y(xx, yy, 'list', 'in callback and decode_typed_arrays')
print_all_call_variants(fig, 'in callback and decode_x_y')
fig['layout']['xaxis']['range'] = fig_range
return fig
if __name__ == '__main__':
app.run(debug=True, threaded=False, use_reloader=False)
Here is a version where decode_x_y converts x and y to a numpy array if they are not already.
#!/usr/bin/env python3
"""Minimal dash program."""
import base64
from dash import callback, Dash, dcc, Input, Output, State
import dash_bootstrap_components as dbc
import numpy as np
import pandas as pd
import plotly.express as px
from plotly.basedatatypes import BaseTraceType, BaseFigure
SHORT_TO_NUMPY = {
"i1": np.int8,
"u1": np.uint8,
"i2": np.int16,
"u2": np.uint16,
"i4": np.int32,
"u4": np.uint32,
"f4": np.float32,
"f8": np.float64,
}
def decode_typed_arrays(obj):
if isinstance(obj, dict):
if (
"dtype" in obj
and "bdata" in obj
and obj["dtype"] in SHORT_TO_NUMPY
):
try:
arr = np.frombuffer(
base64.b64decode(obj["bdata"]),
dtype=SHORT_TO_NUMPY[obj["dtype"]],
)
if "shape" in obj:
shape = tuple(
int(x.strip())
for x in str(obj["shape"]).split(",")
if x.strip()
)
arr = arr.reshape(shape)
return arr.tolist()
except Exception:
return obj
return {
k: decode_typed_arrays(v)
for k, v in obj.items()
if k != "_inputArray"
}
if isinstance(obj, list):
return [decode_typed_arrays(v) for v in obj]
return obj
def decode_x_y(obj):
"""Get x/y from figure."""
ret_val = obj
convert_to_numpy_array = True
if isinstance(obj, BaseFigure):
convert_to_numpy_array = False
ret_val = decode_x_y(obj['data'])
elif isinstance(obj, BaseTraceType):
convert_to_numpy_array = False
ret_val = decode_x_y(obj['x']), decode_x_y(obj['y'])
elif isinstance(obj, (list, tuple)):
try:
if 'x' in obj[0]:
convert_to_numpy_array = False
ret_val = [decode_x_y(ii) for ii in obj]
except TypeError:
pass
elif isinstance(obj, dict):
if 'dtype' in obj and 'bdata' in obj and obj['dtype'] in SHORT_TO_NUMPY:
try:
arr = np.frombuffer(base64.b64decode(obj["bdata"]), dtype=SHORT_TO_NUMPY[obj['dtype']])
if 'shape' in obj:
arr = arr.reshape(tuple(int(x.strip()) for x in str(obj['shape']).split(",") if x.strip()))
ret_val = arr
except Exception:
pass
elif 'x' in obj:
convert_to_numpy_array = False
ret_val = decode_x_y(obj['x']), decode_x_y(obj['y'])
# ret_val = {kk: decode_x_y(vv) for kk, vv in obj.items() if kk != '_inputArray'}
elif 'data' in obj:
convert_to_numpy_array = False
ret_val = decode_x_y(obj['data'])
try:
if convert_to_numpy_array and not isinstance(ret_val, np.ndarray):
ret_val = np.asarray(ret_val)
except Exception:
pass
return ret_val
def print_all_call_variants(fig, msg):
"""Demonstrate and print all of the decode_x_y call variants."""
xx = decode_x_y(fig['data'][0]['x'])
yy = decode_x_y(fig['data'][0]['y'])
print("\ndecode_x_y(fig['data'][0 or 1]['x' or 'y']")
print_x_y(xx, yy, 'pandas', msg)
xx = decode_x_y(fig['data'][1]['x'])
yy = decode_x_y(fig['data'][1]['y'])
print_x_y(xx, yy, 'list', msg)
xx, yy = decode_x_y(fig['data'][0])
print("\ndecode_x_y(fig['data'][0 or 1]")
print_x_y(xx, yy, 'pandas', msg)
xx, yy = decode_x_y(fig['data'][1])
print_x_y(xx, yy, 'list', msg)
data = decode_x_y(fig['data'])
print("\ndecode_x_y(fig['data'])")
print_x_y(data[0][0], data[0][1], 'pandas', msg)
print_x_y(data[1][0], data[1][1], 'list', msg)
data = decode_x_y(fig)
print("\ndecode_x_y(fig)")
print_x_y(data[0][0], data[0][1], 'pandas', msg)
print_x_y(data[1][0], data[1][1], 'list', msg)
def print_x_y(xx, yy, what, msg):
"""Print x and y values and their types."""
print_value_type('x', xx, what, msg)
print_value_type('y', yy, what, msg)
def print_value_type(x_y_str, x_y, what, msg):
"""Print x or y, values and their types."""
print(f"{what} {x_y_str} {msg}: {x_y}")
print(f"type {what} {x_y_str} {msg}: {type(x_y)}")
app = Dash(__name__, external_stylesheets=[dbc.themes.BOOTSTRAP], suppress_callback_exceptions=True)
x = [1, 3, 5]
y = [2, 4, 6]
dataframe = pd.DataFrame({'x': x, 'y': y})
figure = px.line(dataframe, x='x', y='y')
figure.add_scatter(x=y, y=x, mode='lines')
print("figure['data'][0] was created with a pandas dataframe")
print("figure['data'][1] was created with lists for x and y\n")
print_x_y(figure['data'][0]['x'], figure['data'][0]['y'], 'pandas', 'after_creating figure')
print_x_y(figure['data'][1]['x'], figure['data'][1]['y'], 'list', 'after_creating figure')
print('\nCall decode_type_arrays for both\n')
x = decode_typed_arrays(figure['data'][0]['x'])
y = decode_typed_arrays(figure['data'][0]['y'])
print_x_y(x, y, 'pandas', 'after_creating figure and decode_type_arrays')
x = decode_typed_arrays(figure['data'][1]['x'])
y = decode_typed_arrays(figure['data'][1]['y'])
print_x_y(x, y, 'list', 'after_creating figure and decode_type_arrays')
print('\nCall decode_x_y with all variants for both\n')
print_all_call_variants(figure, 'after creating figure and decode_x_y')
app.layout = dbc.Container([dcc.Graph(id='graph', figure=figure),
dcc.RangeSlider(1, 5, value=[1, 5], id='range-slider')])
@callback(Output('graph', 'figure'),
Input('range-slider', 'value'),
State('graph', 'figure'),
prevent_initial_call=True)
def fill_graph(fig_range, fig):
xx = fig['data'][0]['x']
yy = fig['data'][0]['y']
print('\n')
print_x_y(xx, yy, 'pandas', 'in callback')
xx = fig['data'][1]['x']
yy = fig['data'][1]['y']
print('\n')
print_x_y(xx, yy, 'list', 'in callback')
xx = decode_typed_arrays(fig['data'][0]['x'])
yy = decode_typed_arrays(fig['data'][0]['y'])
print('\n')
print_x_y(xx, yy, 'pandas', 'in callback and decode_typed_arrays')
xx = decode_typed_arrays(fig['data'][1]['x'])
yy = decode_typed_arrays(fig['data'][1]['y'])
print('\n')
print_x_y(xx, yy, 'list', 'in callback and decode_typed_arrays')
print_all_call_variants(fig, 'in callback and decode_x_y')
fig['layout']['xaxis']['range'] = fig_range
return fig
if __name__ == '__main__':
app.run(debug=True, threaded=False, use_reloader=False)
This true for the entire figure as well. For the items you need in the callback, if you use a list instead of a numpy array or dataframe series etc, it does not get encoded which means you won’t need a function to decode.
Unless you are dealing with millions of datapoints, it should not affect performance.