Dash_clientside.set_props 50 times slower than document.getElementById(id).innerText

Continue this topic How to detect changes by javascript in dash

Is it normal, that dash_clientside.set_props 50 times slower than element.innerText = value;?

I tested for 1000 elements, dash_clientside.set_props takes about 900-1000 ms to update, while with document.getElementById(id).innerText= it takes about 20 ms.

Js code for both cases:

function update_dom_with_id_to_raw_data_as_dash(updates) {
    console.log('update_dom_with_id_to_raw_data_as_dash got data', Object.keys(updates).length, updates); // Adjusted for object
    Object.keys(updates).forEach(function (id) {
        let id2 = id.startsWith('{')? JSON.parse(id):id; // for dicts
        const now = get_human_time_ms();
        value = `${updates[id]}     /   set at ${now}`;
        dash_clientside.set_props(id2, {children:value})
    });
}

function update_dom_with_id_to_raw_data(updates) {
    console.log('update_dom_with_id_to_raw_data got data', Object.keys(updates).length, updates); // Adjusted for object
    Object.keys(updates).forEach(function (id) {
        const element = document.getElementById(id);
        // console.log(`id=${id}`, element, updates[id]);
        if (element) {
            const now = get_human_time_ms();
            value = `${updates[id]}     /   set at ${now}`;
            element.innerText = value;
        } else {
            console.log(`Element with ID ${id} not found.`);
        }
    });
}

Look at the speed with getElementById - wonderful!
Screen Recording 2024-03-28 at 19.47.05

And 1 second on 1000 elements with set_props :frowning:
Screen Recording 2024-03-28 at 19.56.57

Is that normal?
Is there any way to make such bulk actions quicker?

Hello @kuplo,

I think that this is working differently because dash is piling them up in order to save rendering power. set_props should be about the same speed as if you were updating it in a clientside callback.

As far as setting the innerText, yeah, that will be immediate because you are rendering it immediately, whereas for set_props you are sending it to the dash_renderer to the react component which then renders the updates.

Have you tried it compared to one vs one?

Comparing set_props vs innerText, in ms:

For 100 elements: 20 milliseconds vs 1.5
For 200: 55 vs 2.5
For 500: 250 vs 6
For 1000: 1000 vs 20

I need bulk updates to display, so one by one is not my scenario.

Whatā€™s your real use-case scenario for this?

Why do you need the components synced with the ecosystem?


Also, my original design was to group all the updates together. But it resulted in inconsistent updates.

1 Like

Great question! Was waiting for it :slight_smile:

Solving that topic https://community.plotly.com/t/subject-how-to-process-calculations-with-a-queue-celery-rq-etc-and-display-results-immediately-after-completion/83280/4 I came to the solution, where I create new tasks in dash and send them with websockets to workers. With websockets workers inform me of progress and send me results, and I set that progress and results directly by element.innerText

Seems like I use all possible powers together: the power of Dash in quickly creating interfaces , the power of queues for heavy tasks, the power of websockets for quick messaging, and the power of javascript for quick manipulation with dom (set results).

All would be good, but when I add new tasks - all previous updates are cleared by the dash, it displays their starting inner states - with no results :frowning:

Demo:
Screen Recording 2024-03-28 at 21.00.58

And that is the main problem. And I started to dive into that issue.

Here is the code for creating tasks in Dash:

app.layout = dmc.NotificationsProvider(
    [
        dbc.Container(
            [
                # two stores, one to track which job was most recently started, one to
                # track which job was most recently completed. if they differ, then
                # there is a job still running.
                # dcc.Store(id="submitted_store", data={}),
                # dcc.Store(id="finished_store", data={}),
                dcc.Interval(id="interval", interval=1000),
                html.H2("MULTI Redis / RQ demo", className="display_4"),
                html.P(EXPLAINER),
                html.Hr(),
                dbc.Textarea(value="1" * CNT_ELEMENTS, id="text", className="mb_3"),
                dbc.Button(
                    "Upper case", id="button", color="primary", className="mb_3"
                ),
                dbc.Collapse(
                    dbc.Progress(id="progress", className="mb_3"), id="collapse"
                ),
                html.Hr(),
                html.Div(
                    [
                        html.H1(f"WebSocket Update Example: {CNT_ELEMENTS} elements"),
                        # html.Div(["connecting"], id="connection_status"),
                        html.Div(id="log"),
                        html.Div(["notification_wrapper"], id="notification_wrapper"),
                    ]
                ),
                html.Hr(),
                html.P(id="output_result"),
                html.P(id="log_result"),
                # adding websockets elements
            ]
        )
    ]
)


@app.callback(
    [
        Output("output_result", "children"),
        Output("log_result", "children"),
        # Output("submitted_store", "data"),
        Output("interval", "disabled"),
    ],
    [Input("button", "n_clicks")],
    [
        State("text", "value"),
        State("output_result", "children"),
        # State("submitted_store", "data"),
    ],
)
def submit_to_multiple_tasks(n_clicks, text, output_result):
    """
    Submit a job to the queue, log the id in submitted_store
    """
    submitted_store = global_submitted_store
    output_result = output_result or []
    message = "empty_log"
    if n_clicks:
        logger.debug(f"submit_to_multiple_tasks {submitted_store=}")
        ws_boss.notify_rendered(f"adding {len(text)} tasks")
        for num, symbol in enumerate(text, 1):
            id_ = f"{num}_{uuid.uuid4()}"

            # queue the task
            # queue.enqueue(slow_loop, text, job_id=id_)
            queue.enqueue(slow_dash_div, symbol * 3, job_id=id_)
            submitted_store[id_] = {"status": "queued"}
            global_dynamic_progress_id.add(id_)

            output_result.insert(
                0,
                html.Div(
                    [
                        html.Div(
                            [f"at {get_human_time_ms()} added {num}/{len(text)} {id_}"],
                            id={"index": id_, "type": "dynamic_first_message"},
                        ),
                        html.Div(
                            ["no progress"],
                            id={"index": id_, "type": "dynamic_progress"},
                        ),
                        html.Div(
                            id={"index": id_, "type": "dynamic_result"},
                        ),
                    ],
                ),
            )

            # log process id in dcc.Store
        message = f"{len(submitted_store)} {submitted_store=}"
        ws_boss.notify_rendered(f"+added tasks")
        logger.debug(message)

    interval_disabled = False
    return output_result, message, interval_disabled

Would you be open to using AG grid for the open tasks?

If so, you can use rowTransaction instead.

I canā€™t get now how AG Grid might assist in efficiently displaying various types of results, such as charts and tables.

However, my primary goal is to develop and manage a message bus system that can automatically handle tasks and generate subtasks based on the outcomes of those tasks. Let me provide a bit more context to explain what I mean.

Each task in my system is associated with a pre_div and a result_div, along with a designated func and its func_kwargs. For instance, consider a task named task1, which involves preparing financial data for analysis. The pre_div for this task is a ā€˜download dataā€™ button. When I click this button, the task is dispatched to workers, and my interface updates to indicate that the task is in progress.

Upon completion of the data preparation, the result_div is updated to display the results, such as a statistical overview of the data. However, the completion of task1 unveils several new options for further processing this data. For example, I might want to visualize the data in a chart or perform various calculations (like backtesting trading strategies). Some of these follow-up actions can be executed automatically (e.g., generating a chart if a specific checkbox is marked), while others may require manual input or actions, such as clicking a ā€˜chart dataā€™ button, entering parameters for a backtest, and then initiating the backtest by clicking a ā€˜backtestā€™ button.

The result_div is designed to present all these potential follow-up actions. In essence, the completion of task1 leads to the creation of new tasks (letā€™s call them task21, task22, task23, and so on) based on the initial taskā€™s results.

My vision is for this system to streamline the workflow by automatically managing tasks and their dependencies, thereby enhancing efficiency and user interaction. However, Iā€™m still figuring out the role AG Grid could play in this setup or if thereā€™s a more suitable tool or approach for achieving these objectives. Iā€™m open to suggestions and insights from the community.

Can you work up an app that would be able to do 10 tasks and Iā€™ll see about creating it in AG grid.

1 Like

Imagine my demo Dash_clientside.set_props 50 times slower than document.getElementById(id).innerText - #5 by kuplo , but any result is not ā€œmockuped resultā€, but result_div is one graph, one new dash grid, and one new button that on clicking can display results of tasks1.

How to solve that in AG Grid?

As far as I understand it, you only need the final result_div once everything is completed, before that you just need status updates, is this correct?


Also, this is not a full blown app that I can run. How to write a Minimal Reproducible Example (MRE)

yes, correct. If there is a status update and result_div, this should be enough.

Iā€™ll see if I can create MRE, but thanks to our conversation I have another idea now - to create that ā€œoutput_resultā€ and add new children to it the same way as I update progress and results - with direct DOM manipulations with python. This way Dash should not rewrite my updates with its initial values.

Anyway, donā€™t you have any example that use AG Grid to create new element with AG greed and chart? For some reason, I just canā€™t imagine how this is possible :slight_smile:

You can create charts as custom cell renderers, or popup modals.

My main thinking was you have the grid, then as you add tasks the grid grows. As the status updates come in, it would go to the rowTransation as an {update: [data]} which would then be applied async. To add a task to the grid it would be rowTransaction {add: [data]}, this would then stack up and you dont have to worry about the tasks disappearing.

As long as the page is active that is.

Then when you have a completed task, it would populate the other columns of the grid, if you wanted. Clicking on buttons in those columns would then open components in a modal or whatever you want. :stuck_out_tongue:


examples of custom components:

Another thing too, if you go with the Grid, you can use the api to do the transactions instead of using the props.

1 Like

Very interesting idea, thanks! I should sleep with it :slight_smile: hope to try it later.

1 Like

I have nothing to add here, just wanted to let you know that this topic is a really interesting read!

2 Likes

Hello everyone,

I want to share an update on my recent decision-making process.

Iā€™ve decided to embrace the concept of settling on a ā€œgood enoughā€ solution rather than spending an indefinite amount of time chasing the ā€œidealā€ one.

In light of my main taskā€™s specifications, I realized that there wouldnā€™t be more than 50 tasks per user at any given moment. To visualize this, I modeled a scenario with 50 tasks, each comprising 5 subtasks, being processed by 10 workers who keep the system updated on their progress.

Iā€™ve integrated Dashā€™s .set_props callback from JavaScript to leverage all the benefits Dash offers. It appears that under these circumstances, Iā€™m effectively utilizing nearly all of my desired capabilities:

  • The swift interface creation made possible by Dash
  • The robust handling of heavy tasks by queues
  • The rapid communication facilitated by websockets
  • The dynamic DOM manipulation enabled by JavaScript

Hereā€™s a glimpse of the system in action, showcasing its efficiency:

Full process, but low quality (sorry about the resolution - max 4 mb are allowed to upload, it would be good to fix that in the future)

Screen Recording 2024-03-29 at 13.42.14

Part of video in better quality to feel the speed:
Screen Recording 2024-03-29 at 13.42.14

Good enough :slight_smile:

Iā€™ve integrated tasks within Dash and established a separate function that checks worker results every 0.1 seconds. Progress and results are communicated to Dash via websockets, and if necessary, updates are made using set_props. A Dash callback, highlighted in green, signifies when the results are ready (triggered after set_props), ensuring that callbacks remain operational.

Iā€™m quite pleased with the setup :blush:.

@jinnyzor, thank you for our insightful conversation. It helped me grasp several key concepts. Iā€™m considering exploring AG Grid for this application in the future. However, at the moment, my mind is overwhelmed with a plethora of new information, so I want to give my brain a chance to ā€œfreezeā€ a little from all this new knowledge :slight_smile:

1 Like