Hey @chriddyp!
Generally speaking, there is no native networking in WASM-land, and so the solution is to expose the JavaScript networking APIs and use them for data retrieval. These include WebSocket and XMLHttpRequest/Fetch.
Specifically, in Pyodide, which is CPython compiled to WASM, we have Python bindings to those JavaScript APIs. So we could have something like that:
# Taken from https://github.com/pyodide/pyodide/blob/b19e2762a851a679cf809ac582c753d5c9c0bb0d/src/py/pyodide/_base.py#L15
def open_url(url: str) -> StringIO:
"""
Fetches a given URL
Parameters
----------
url : str
URL to fetch
Returns
-------
io.StringIO
the contents of the URL.
"""
from js import XMLHttpRequest
req = XMLHttpRequest.new()
req.open("GET", url, False)
req.send(None)
return StringIO(req.response)
This allows us to write something like that when retrieving data for use in pd.read_csv
:
data = pyodide.open_url('https://plotly.github.io/datasets/country_indicators.csv')
df = pd.read_csv(data)
In fact, that’s exactly what we’re doing in the adapted Dash demo app you wrote!
pd.read_csv
and many other libraries rely on urllib
. In the current configuration both urllib
and requests
calls will result in no-ops. With that said, it should be possible to monkey-patch urllib
so that it’s using the js
API introduced above for any HTTP requests (this should also apply to requests.get
). If we manage to get that to work then we will be able to eliminate the friction of adding pyodide.open()
and be able to use pd.read_csv
with an address directly.
For database queries, native socket connections will not work out of the box unfortunately (for the same reasons mentioned above). But the good news is you can still use some databases which support WebSocket connections. And for those which do not, it is possible to use a gateway server which receives WebSocket connections and relays them as regular socket connections. There’s an overhead there which is not ideal, but it’s a solution.
As for the last point about mixing and matching server/client on the WASM stack - it’s such an exciting prospect! Because Pyodide is essentially CPython running in WASM, everything (including the server) is essentially pushed on to the WASM stack to begin with - packages like Pandas and Numpy are compiled to WASM and run inside the browser. This allows minimally modified Dash apps to “just work” in the browser.
We can bundle multiple WASM modules, so anything in addition to the Python kernel, - although admittedly I did not experiment with that! Perhaps there’s scope to add something to Dash Renderer? I would love to hear any feedback or ideas you may have ,