Best approach to using Dash and S3 database

Hi all,

We’re looking into using Dash for an IoT project, where multiple devices in the field are uploading data to an S3 bucket.

We intend to process this incoming data and convert it to some database format (e.g. MySQL, CSV or other format). The resulting database will most likely be 100-200 MB and should serve as the basis for displaying charts in Dash so that the end users can quickly visualize the data.

We struggled to find guides/examples of how to approach this - in particular regarding below questions:

1) What would be the best database format to select for this?

2) If the database is stored on S3, how do we best connect it to Dash (without having to download e.g. 100 MB every time a user polls data)? We saw another example suggesting the use of boto3 get_object and io - but I’d assume this to be problematic if the database is 100+ MB?

Any thoughts on the above would be greatly appreciated,
Mads