Good morning everyone,
For the past few weeks, I’ve been developing a multi-page application for my company. This app allows users to select clients and then choose from various analytical cycles, such as revenue analysis or logistic analysis, with each cycle presented as a dashboard. I’ve implemented dash-auth for authentication and successfully set up an authorization system to control access based on the client or cycle.
However, I’ve encountered a scalability issue with the data management aspect. Storing data in CSV format, especially as files grow to 30GB and beyond, has led to significant performance bottlenecks, with data querying operations taking up to three minutes. To address this, I’m considering migrating our data storage to a database system, aiming for improved performance and manageability. Is this the best practice for such a situation? Would a database significantly enhance the application’s speed and efficiency?
Additionally, given the sensitive nature of our data, I’m exploring the deployment of our application and the database on an intranet using a Windows server. What are the best practices for secure deployment in such an environment? I would greatly appreciate any advice or resources that could help deepen my understanding in these areas.
Although I recently graduated with a master’s degree in Data Sciences, specializing in Deep Learning, my current role encompasses responsibilities in software engineering and data analysis—fields I’m less experienced in. I’m eager to learn and adapt, so any guidance or insights you can offer would be incredibly valuable. Even books would be great as I am not afraid to deep dive into various areas.
Thank you very much for your help.