Load the data using databricks sql connector in a optimized way

Dash Python
1

Hi All,

Recently, I started exploring the Dash. For my analysis, I am fetching the data from databricks sql warehouse. The problem with that is it takes around 97 seconds to load the data to Dash.

The table which i am fetching has around 4,00,000 rows

Is there any other to achieve it in lesser runtime?

2

Hello @MBharathi,

Welcome to the community!

If possible, could you run this task on a schedule and store the data in a paraquet file locally?

This would make it a lot easier to display in Dash.

3

Hi @MBharathi

You will find a lot of good information here: