Hi there, I am facing some issues with fetching data directly from the database. I have billions of rows of data. When I am trying to pull all that data it is not working. I decided to do something like this first I will put all values of a particular column in filter options. Based on the selected value I want to fetch data from the database in Dash using SQL command. Is this approach work and If there are any other solutions please share those details. I want some code templates to try. I successfully made connections between database and dash. I am able to read data into a data frame using SQL queries. My concern is that based on the selected option from the dashboard I need to automatically fetch data from the database using SQL query and charts should be shown accordingly.
Evening!
Large datasets have a few issues simply because of their inherent size. There’s a few things you can try:
1.) Check out the various ‘gl’ chart types: Webgl Vs Svg
2.) Do as much of your filtering as you can on the database/server side so you aren’t locking the client up trying to stuff too much data into it.
3.) Try datashading: Datashader
Few other places to read:
Proper way to plot large datasets - #8 by lamourj
https://www.reddit.com/r/learnpython/comments/q65kl3/plotlydash_with_large_datasets/
I want to know that based on the user input on the filter, I want to fetch data from the database how it can be solved. Is there any solution for this kind of approach?
Hmm well the code for that would depend on what database you are using and which python (assuming you are using python) connector you’d like to use for that database to connect your app to it.
I used Connecting a Data Source to Dash - YouTube this video a lot when i first started in dash, might be a good starting point for you.
I already made snowflake connection with python and I am able to fetch data from snowflake. I want to automate the process such as based on the user input from dropdown list how to fetch data from snowflake. I am able to write the SQL query for fetching the data. My doubt is where to keep that query inside callbacks or outside callbacks. If the user input particular value that can used in def function getting from input. Where to put query so that I can use that data for all other charts.
Ah, snowflake is a good choice, love that platform. I would probably load all of your data into a store Like here but more specifically for you I’d try this example:
You can put all your data into a dcc.store components and then use that data in any number of charts, reports, etc. I’d probably make 1 callback to get your data from snowflake and put it into the store and then use the store as the input for your callbacks the build your charts