How to create views of dataframe without pumping entire thing through to_dict?

Dash Python
1

I am looking for an example that does not send an entire dataframe to the front end but instead does aggregation in backend.

All the examples of datatables that I see are calling some to_dict and do not appear to have dynamic query generation from the front end.

This is what I am expecting this library to implement:

a) start with dataframe abstraction (either in memory or virtual like spark). 10 million rows is pretty standard in memory.
b) meta-data comes from dataframes, this + widget specs populates the front end UI widget stuff.
c) widget selections generate a query for data which goes to the backend and hits the dataframes, this data is max-rowed etc and sent to the front end and “visualized”.
d) charts and tables have optional click through behaviour that populates the widgety filters and groupings etc. This comes from the “chart” spec.

Is there some way to do this? Can someone point me to an example if so?

2

I think it might be here:

https://dash.plotly.com/datatable/callbacks

If this is true (I need to read it), this should be example number one. Make it nearly impossible to do things another way. One way to do things.

If there are client side manipulations, those exist AFTER the backend ops regardless of whether you use the backend ops.

3

So my understanding is the you need to manually wrap the whole dataframe columns in a loop, extract the types, generate the custom groupers, slideres etc then turn that into a div in layout, then somehow extract all those choices into the callback.

Are there no patterns to do this somewhere?