Dynamic setting of port number in dash_duo testing

For our in-house Plotly Dash component library, we have the following use case:

  • Unit tests with dash[testing], i.e. pytest, selenium and dash_duo.
  • CI pipeline on GitHub using GitHub Actions and Linux runners.
  • GitHub Actions matrix strategy, i.e. testing for Python 3.8 and 3.9.

These are the first lines of the GitHub Actions YAML file:

name: QA
on:
  pull_request:
  push:
    branches:
      - master
      - main

jobs:
  test:
    name: Tests
    runs-on: [ self-hosted, GFBBPT ]
    strategy:
      matrix:
        python-version: [ '3.8', '3.9' ]
    steps:
    - uses: actions/checkout@v2
    - name: Install Python
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

Problem:
GitHub Actions runs the tests for 3.8 and 3.9 in parallel. Each test calls dash_duo.start_server() and tries to bind port 8050 by default. Usually, either a test for 3.8 or 3.9 already occupies port 8050. So it is not available to the other one. Most unfortunately, the application then fails silently inside the thread. At the end, it says the test fails due to assertion errors because a unit test in 3.8 connected to the server in 3.9 and vice versa.

Discussion:
Running unit tests in parallel, or my use case of CI in GitHub Actions with a matrix strategy, are not too exotic, and therefore in my opinion warrant a feature request in Plotly Dash.
From the top of my head, I am able to think of three implementations:

  1. Ask for permission: Check if the port is available, e.g. using ThreadRunner.available(), and automatically increase port number by 2, starting at port 8050.
  2. Ask for forgiveness: Run server. Check if server actually runs – at the thread, Dash or Flask level. Have not fully thought this through. Pinging the port is definitely not enough for my use case because another port would ping back!
  3. Defer responsibility. At least, raise an error message if the port is occupied, and delegate the task to the caller of dash_duo.start_server().

The first one is easy to implement. As a quick fix, I wrote the following function:

def start_server(
    dash_duo,
    app: dash.Dash,
):
    port = 8050

    for _ in range(100):
        if dash_duo.server.accessible(f"http://127.0.0.1:{port}/"):
            port += 2
        else:
            break

    dash_duo.start_server(app, port=port)

Nevertheless, this is not thread-safe. At the moment, I do not know ThreadRunner, Dash or Flask well enough to say if there is a way to check with the application server inside the thread is running. Any ideas? :slight_smile: