I have a lot of data that is updating very frequently. Right now, I store the data in pickle files and I read from the pickle files. The problem is that the pickle files often get corrupted because there are a bunch of processes reading and writing at the same time.
I thought I would try redis but it looks like that is not available on windows. I’m wondering if disk cache might be sufficient and, if not, then if there are other in memory databases that are better for plotly on windows because redis appears to be unavailable
I tried to use sqlite but it was very slow for reading data. It took multiple seconds whereas I can read all the data from pickle in a fairly small number of milliseconds
Is your issue you have a large amount of transactions occurring at once or do you have a large amount of data? or both?
If you have to use windows, could something like postgre work? I don’t know your data but MongoDB might also be a solution. Neither of these are necessarily in-memory but for my uses (few million rows) they seem to work fine.
It is a large amount of transactions occurring at once. Right now, there are five pickle files that have different small datasets (about one hundred thousand rows each across ten columns). The dashboard reads from each file every six hundred milliseconds or so and a different process writes to each file at about the same frequency.
I would want a database that can mimic the performance in terms of reading and writing speed that I get from pickle but is more reliable and stable. The pickle files often get corrupted because they’re not made for simultaneous read / writes.