Help/Advice debugging background callbacks on Mac with Python3.11

Dash Python
,
1

Hi all,

I have a bit of a tricky bug with background callbacks on Mac with Python3.11 and would greatly appreciate any input regarding the issue itself or how to draft a good bug report for Issues · plotly/dash · GitHub

Problem Description

Running the example from Background Callback Caching | Dash for Python Documentation | Plotly on a MacBook Pro with Python 3.11 starts up fine but when clicking the “Run Job!” button yields the following error:

Process Process-2:
Traceback (most recent call last):
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/dash/long_callback/managers/diskcache_manager.py", line 197, in job_fn
    ctx.run(run)
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/dash/long_callback/managers/diskcache_manager.py", line 195, in run
    cache.set(result_key, user_callback_output)
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/diskcache/core.py", line 800, in set
    with self._transact(retry, filename) as (sql, cleanup):
  File "/Users/ndr/.pyenv/versions/3.11.8/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/diskcache/core.py", line 714, in _transact
    sql = self._sql
          ^^^^^^^^^
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/diskcache/core.py", line 652, in _sql
    return self._con.execute
           ^^^^^^^^^
  File "/Users/ndr/code/test_caching/.venv/lib/python3.11/site-packages/diskcache/core.py", line 623, in _con
    con = self._local.con = sqlite3.connect(
                            ^^^^^^^^^^^^^^^^
sqlite3.OperationalError: disk I/O error

When running the same code under Python3.10 on the same machine it works without problems.
Running the same code with Python3.11 under either Windows or Ubuntu also works fine.

Mitigation attempts

Going back to the Mac with Python3.11 and disabling debug mode of the app e.g.

    app.run(debug=False)

Makes the error only come up sometimes i.e. sometimes, the click leads to a page update and sometimes to the sqlite3 disk I/O error.

Additionally disabling threading and using only a single process

    app.run(debug=False, processes=1, threaded=False)

makes the error go away and the page working “properly” but severely limiting the usage of debug mode and threading.

To me, it seems to be a multiprocessing file access issue specific to Mac (maybe related to forking?) but I am not familiar with debugging these kind of issues, specifically with a Mac.

I tried additionally allowing “Full Disk Access” to the Terminal and my IDE and setting the environment variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES (as I googled this might be related somewhere else) but that did not appear to change anything.

2

Hello @ndr,

This sounds more like an issue with sqlite, sqlite is designed to only have one write operation with it. If you have multiple processes trying to write to it, then you will run into issues.

Another thing, is that the connection string is designed to be handled in the main thread, to allow for async processes, you need to adjust the connection string.

sqlite3.connect("db.sqlite", check_same_thread=False)
3

Hi @jinnyzor

thanks for the suggestion and sorry for the late reply.

I completely agree. Note, however, that the connection is initiated by diskcache and they check they “check the process ID to support forking” by closing the connection upon process id change and updating the process ID: python-diskcache/diskcache/core.py at master · grantjenks/python-diskcache · GitHub

I tried modifying the diskcache source locally to include the check_same_thread=False as you suggested

image
image1034×356 37.7 KB

But that still gave the same result.

I am wondering where to post a bug issue to. Diskcache, Dash or SQLite? Or debug it further myself. Unfortunately, when debugging the issue with e.g. the PyCharm Debugger, the error message is not coming through anymore upon breaking at the connection line but the page also does not work properly, so the issue seems to remain.