Large file display in dash aggrid

Dash Python
1

Hi community friends,

I have a requirement of displaying 10 million rows and 100 columns using Dash-ag-grid in the browser with 8 dcc.Dropdowns to filter in addition to filter in the dash-ag-grid.

I have tried pagination, preset pag# but its very slow and takes a very longtime.

Dash-ag-grid 0.34 has pagination issues too. Right?

Can someone help me out with this?

Thank you very much,
BKS

2

Here is an example of big data, you can refer to it. https://plotly.com/blog/polars-to-build-fast-dash-apps-for-large-datasets/

3

Hello @Bijoy,

You can try pagination with rowModelType infinite.

4

@Bijoy

You can find a lot more information on handling large files with the Infinite Row Model in the docs:

https://dash.plotly.com/dash-ag-grid/infinite-row-model

5

@ Liripro,

Thank you very much. Your reference really helped me put together the Dash App in Polars format.

Best,
Bijoy

6

Hello @jinnyzor,

Awesome! That was perfect!! The only change I made is I adapted it to Polars since it is blazing fast. Please see below:

@app.callback(
Output(“grid”, “getRowsResponse”),
Input(“grid”, “getRowsRequest”),
State(“filters”, “value”),
)
def update_grid(request, filters_data):
if request:
filtered_df = df.filter(df[‘colname’] == filters_data)

    partial = filtered_df.slice(request["startRow"], request["endRow"] - request["startRow"])
    return {"rowData": partial.to_pandas().to_dict("records"), "rowCount": len(filtered_df)}

Thank you very much! Appreciate it a ton!

Best,
BIjoy

7

Hi @AnnMarieW ,

I would like to express my sincere gratitude for all the help and support you have given to the community members who are ardent fans of Dash & Plotly!

Thank you very much for the references. It helped me figure the solution quick.

Best,
Bijoy

1 Like