Black Lives Matter. Please consider donating to Black Girls Code today.

How to work with large DataFrames

My goal is to display groups in a dash data table, allowing user to select each group + metric and then I will display the plot of that groups 100 days or that metric.

What strategies should I pursue if I have data for 100 days * 1000s of groups. This is already an expensive query to the DB, and it is too big to put all into a hidden div. I saw the documentation for paging / sorting datatables on the backend, but I can’t figure out where I’d store the DF between callbacks as global variables are not allowed, which confuses me because the examples all show data that is usually just loaded into what is essentially a global top level variable “df”.

Would greatly appreciate some suggestions for good patterns or ways to implement this!