Efficient Way of Loading Data from Flat File

Hi there!

I am quite new to dash and currently I am trying to build a dashboard that displays different information depending on the pathname.

For example, www.mywebsite.com/dash/case1 will display a set of data, and /case2 will display another set of data.

I am currently using dcc.Location to get the pathname in a call back and hence load all the relevant csv files (that differ based on pathname /case1) I have with pandas, and then jsonify all the pandas objects and store in a hidden html div.

Subsequently, depending on the tabs that users click, I use pandas read json to reconstruct the data frames from the hidden html and generate various graphs.

At the moment it takes around 4-6 sec to load the various files on load. I am wondering if there is a more efficient way of organising data for this use case.

All comments are highly appreciated. Thanks in advance.

It’s likely the hidden DIV component that’s making it take a while as you have to send all your data back up rather than just down. I see why this situation is a little tricky though, as normally you’d just return a new layout to inject into Location.url, which would have callbacks associated with the containing IDs, each of which manages its own data. but in you case, it sounds like you want to share the same layout and callbacks across the page, but have them be parameterised by the current pathname, and accessing the pathname within callbacks other than the router becomes tricky due to issues of shared state.

Assuming I’ve understood your situation, here’s one way I just thought of; there’s probably others…

  1. Start by defining a dictionary mapping pathnames to DataFrames needed by each page, let’s call it DATAMAP.

  2. Create a function let’s call it make_pages that takes as input a list of pathname identifiers, and then returns a corresponding dictionary that maps pathname identifiers to layouts. Assign the value returned by this function to a variable, say PAGEMAP

  3. make_pages is going to need to do two things: generate the layouts to put in PAGEMAP, and create callbacks the layouts needs.

  4. To make the layouts, for each pathanme, it needs to create one layout, and it sounds like they’ll be be the same layout, but inside each, all your Graph and Inputs etc are given a different ID prefixed with the pathname. so maybe define another function that generates auto-prefixed versions of your layout template).

  5. make_pages also needs to create all possible callbacks that all of these layouts are going to need. so you’ll need some kind of factory function that creates the functions that will become callbacks you need for each graph. To to this it needs to get the right data from DATAMAP and return a function that can then be given a callback decorator, which is automatically prefixed by pathname to match the IDs you made for the layouts.

  6. Then you just gotta create your router callback watching Location.url and outputing to your container DIV that is being swapped in and out. It just needs to return PAGEMAP[pathname].

The general outcome is that it still feels like you’ve got the one logical layout and set of callbacks, because in each case, they’re defined precisely once, however you’re actually dynamically generating different ones for each page, where everything is automatically prefixed by the pathname, so everything is nicely namespaced.

Hopefully that both makes sense and works :smiley:

1 Like