Dash DataTable: df.to_dict('records') fails when using datatable instead of pandas

Hi there, I am using datatable (not Dash DataTable, but the equivalent to pandas) for my dataframes. When using pandas I simply can use those datafremes to create the Dash DataTables. But when the dataframe was created with datatable I can’t use it create the Dash DataTable. Sorry for the confusing wording.
Here is the in App error message:

I guess my questions is the following: what sources can be used to create Dash DataTables besides pandas. If I want to use datatables what changes to my code are necessary?

Please let me know if I have provide to details or further clarification.

dash_table.DataTable accepts a list of dictionaries where each dictionary key corresponds to the column is and each dictionary value corresponds to the cell value. All that to_dict(‘records’) does is convert a data frame object into this list. So you’ll need to just convert your object into a list of dictionaries.

Hi @chriddyp, yes the simple solution would be to convert the datatable to a pandas df. But I was wondering if there might be some native support to directly use the datatable as ‘input’ for Dash DataTable.

I am using datatable especially because it is so much faster than pandas. Converting it to a pandas dataframe seems a bit like taking a step back.

Thank you very much for your help.

You don’t need to convert it to a pandas data frame, all you need to do is convert it to a list of dictionaries.

Hi @chriddyp,

Suppose df_sub is my datatable. This is my code:

c_names = df_sub.names #column names

df_sub = df_sub.to_list() #convert datatable to list (as shown in datatable docs)

df_sub = [{c_names[i]: df_sub[i]} for i in range(len(c_names))] #lastly create list of dict

Sadly it produces the following error:

What changes to my code are necessary?

Thank you :slight_smile:

Just do data = df_sub. .to_dict('records') is a pandas method that converts a dataframe to a list of dictionaries. Since you are converting this yourself, you don’t need to call .to_dict

Hmm, this does not work for me.

I guess I just convert it to a pandas dataframe:

df_sub = df_sub.to_pandas()

This at least makes the App work, although it seems a bit messy.

I really appreciate your help with this. Have a nice day

What doesn’t work? Is there any error message?

Yes I do get the following error:

Huh, that seems like an unrelated issue to the data= property. What is the value of print(dervied_viewport_row_ids[0]?

Where would I place the:

print(dervied_viewport_row_ids[0]

codeline?

I do not use the property dervied_viewport_row_ids anywhere in my code if that is of importance.

Ah OK, that’s interesting. You aren’t listening to that property in a callback either? i.e. you don’t have Input('table_month', 'derived_viewport_row_ids')?

This is the complete callback (and only callback; based on a dropdown a df is filtered and some data transformation occurs, the result is supposed to be displayed inside the App, more is currently not happening in the App):

@app.callback(
    Output('table_container', 'children'),
    [Input('dd_symbol', 'value')])
def update_table(symbol):
    #took some code out to focus on the relevant
    # below is the final datatable that should be displayed as a Dash Datatable
    df_sub = df_sub[0, {'value_date': f.datetime, 'value_min': f.bid}, by('id'), sort('bid')]

    c_names = df_sub.names  # column names

    df_sub = df_sub.to_list()  # convert datatable to list (as shown in datatable docs)

    df_sub = [{c_names[i]: df_sub[i]} for i in range(len(c_names))]  # lastly create list of dict

    div = html.Div([
        dash_table.DataTable(
            id='table_month',
            columns=[{"name": i, "id": i} for i in c_names],
            data=df_sub,
            style_cell={'textAlign': 'left',
                        'overflow': 'hidden',
                        'textOverflow': 'ellipsis',
                        'maxWidth': 0},
            page_size=100, #usually equal to nrow, hence virtualization = True, 
                           #but since I am switching between datatable and pandas I set it to 100 for simplicity
            fixed_rows={'headers': True},
            style_table={'height': 400},
            filter_action="native",
            sort_action="native",
            page_action='none',
            virtualization=True
        )
    ])

    return div

Edit: also the App behaves unusually slow, e.g. when clicking on the error messages to expand them.

Order function is data table is much faster than base function order(). Reason being, order in data table uses radix order sort which impart additional boost.