I’m trying to create a tool for displaying large, interactive line chart plots for some massive data log files. I really like the fact that Plotly is able to support zooming into segments of the data to plot it with the interactive plots, which really helps for rapid data analysis. Sharing these easily through HTML to other teammates is important to me, so I’ve been using Chrome, Jupyter notebooks, and cufflinks. However, the browser seems to crash when using iplot with large CSV files, e.g. 50-150 MB, 150,000-370,000+ rows. Sometimes I see “Aw snap!” Or “Out of memory”. It happens with a crash in FireFox as well. After searching around, I realize there may be a limitation to the file/datasize, and there may be some better big data methods to use to visualize this, and I’m wondering what the best method would be to do this?
For reference, I did see this: https://plot.ly/python/big-data-analytics-with-pandas-and-sqlite/ but I didn’t think I needed it since it seems the dataframe gets loaded in with no problem usually, and it even plots the chart most of the time (albeit it takes a while). However, after doing nothing for a few seconds, or when zooming, it crashes. Task manager’s memory usage also doesnt appear to be exceeding my system’s RAM amount either…
EDIT: the code I use to plot with cufflinks is:
Any guidance is much appreciated.