Show and Tell - dash-uploader (Upload large files)

Dash Python
21

Hey Niko, thanks for the help again. So while programming I have come across another issue. Is there a way to upload the files without upload folder but instead in the “memory”? I have a folder of measurement data that I want to upload and then display in the dashboard but I don’t want to create another folder with he exact same files next to the one that already exists. Is there a way to do that? I appreciate your help.

23

Hi @sidneywilke . Did I understand correctly that you are using a Dash app locally on a PC? If that is so, and the files are already on the disk, why would you need an upload component? For selecting the files? Or is there some special reason for it?

1 Like
24

So I run the code in the IDE and then I get a local IP in the terminal from which I can open the Dashboard in my browser. The upload component is a way for me to “insert” files into the dashboard so it can display graphs etc.

25

@sidneywilke Would it be okay just to use normal file selector dialog (tkinter?) to select files from your disk? I would add a a button to dash that has a callback, which opens a file selector dialog. There is probably no need to “upload” files if you will only run your app locally.

1 Like
26

That sounds like the ideal solution, how is this button called to open file selector dialog, tkinter?

27

For example, from this SO question:

import tkinter as tk
from tkinter import filedialog

root = tk.Tk()
root.withdraw()

file_path = filedialog.askopenfilename()

Just put the call to the askopenfilename() inside a normal dash Button component callback.

1 Like
28

Hi @fohrloop,

I like your idea to use Tkinter to upload local files into Dash. Follow your advice, I put “file_path = filedialog.askopenfilename()” inside a callback of html.Button().
However, I ran into this error: ‘RuntimeError: main thread is not in main loop’.

Would you please elaborate more?
Thanks

29

@nqv I tested this and it seems that dash is running all the callbacks in a separate process/thread. Tkinter works only in the main thread so I guess tkinter is out of the game (unless you find a way to run the callbacks in the main process). Using dash-uploader to select and upload the files might be then a good workaround. Just remove the uploaded file in the callback. Note that the callback should not return “large amount of data” as this is sent to the browser (process your data before returning).

30

@fohrloop,

Thanks for quick reply.

I was hoping Tkinter would work, so it would be faster to upload the local files into Dash.
My issue is that I need to upload 3 files into Dash, and one of them is pretty big (~100MB). Thus, I could not load that file using Dash’s dcc.upload().

I will install your dash-uploader and try your suggested approach above to see if it works out for me.
Will report back to you if I run into any issues.

Thanks

31

Is there a possibility to post files using HTTP requests to dash-uploader, like:

import requests
myurl = ‘http://127.0.0.1:8050/api-upload’
files = {‘file’: open(‘some_file.csv’, ‘rb’)}
getdata = requests.post(myurl, files=files)
print(getdata.text)

Thanks

32

As a quick answer, yes. At least it is possible to use the same endpoint for POST requests coming from any kind of source. The endpoint is configured in Flask when you configure the dash uploader. You can setup your custom HTTP request handler if you wish. See the http_request_handler argument of the du.configure_upload for the details: dash-uploader/dash-uploader.md at dev · np-8/dash-uploader · GitHub

Note that if you want to do something like that, you need to add to the HTTP POST request some special fields. See the source code of the HttpRequestHandler for examples.

Another way would be to setup your own Flask routes in similar manner that dash-uploader does.

Hope it helps!

33

Thank you for your fast reply. I will see if I can get it working. If so, I will post the details here.

34

I am still having trouble to get the post files by using HTTP requests to dash-uploader working.
Do you have some sample code available for this?

35

Yeah it will not be easy task but surely manageable. I don’t unfortunately have any example code at hand. By the way, is the some specific reason you would not just directly create Flask routes for your Post requests? You could then take some example from the chunk handling of dash-uploader for your app.

36

We managed to get it working using the code below for posting files with a workaround.

We could not get the parameters (resumableTotalChunks, resumableChunkNumber, resumableFilename, resumableIdentifier, upload_id) working. Furthermore the unique id’s are now generated in the “post” script instead of in the dash script.

What do we miss here?

===== code= ====
import requests
import uuid
import os

unique_id = uuid.uuid1()
to_upload_files_path = “sample_csv_files/”
for filename in os.listdir(to_upload_files_path):
filename_with_path = f"{to_upload_files_path}{filename}"

upload_url = 'http://127.0.0.1:8050/API/dash-uploader'
files = {'file': open(filename_with_path, 'rb')}
payload = {
    'resumableFilename': filename,
    'upload_id': "test",
    'resumableTotalChunks': 1,
}
getdata = requests.post(upload_url, files=files, data=payload)
print(getdata.text)
37

What do you mean by this?

38

The HTTP requesthandler uses this is input, we assumed this should be also implemented in the “post” function.
Now we do not use these as inputs.

39

In the source code of the BaseHttpRequestHandler._post() you can see how the POST requests are handled. In short, if you want to send the files in small chunks, you need to include the fields it is expecting (resumableTotalChunks etc.). This is option A.

However, I have not ever tried to hack around like this and this part is considered to be internal to dash-uploader, and these fields might be changing over time. This is because the current fields are chosen by the JS library resumable.js which is used under the hood.

What you could do is to subclass HttpRequestHandler (or BaseHttpRequestHandler) and edit the post method as needed. This is option B.

Or, you could also do option A + B (=option C), of course.

As said earlier, you could also just open Flask route yourself. You will not need dash-uploader for that!

It is slightly difficult to guide you further as I don’t really understand what is the problem. Are you getting some sort of error messages?

40

I do not get any error messages, but it would be nice if I could exactly replicate the functionality of the dash uploader using http post requests. But this is also working for what I was planning to do. Thank you for your support!

41

Is it possible to integrate this with an upload button and not necessarily an uploading area as it is possible with the dcc.Upload component?