Slow rendering of large tables

I have created a table with three columns and 6000 rows, which is large, but really not that large these days… I am returning it inside a div with with a fixed height so only the top 12 rows are visible and the user can scroll to see the rest.

My query and table creation complete in less than a half-second but it still takes 7-10 seconds for the browser to render the table on the page. After spending a few hours seeing if any suggested Bootstrap tweaks would help, I noticed that the slow loading time is the same whether I am using Dash Bootstrap Components and when I am not.

I have a similar large-table-inside-a-small-div with the table having even more columns and it displays instantaneously using Flask. Is there any cause for this delay in Dash and anything I can do about it?

Have you tried upgrading to dash 1.13.4? we made some performance improvements for this use case in the 1.13.x series

Hi Chris, I had just upgraded to 1.13.3 a few days ago for Multiple Page renders on app restart+reconnect I have gotten no response and would love a little feedback on whether that is a bug or normal behavior that I just don’t understand.

Back to this problem, I just upgraded to 1.13.4 and the results are the same. The page spends a little over one second per 1000 rows in the Loading state until it displays. Here is an example program I am testing with that is completely free of any CSS elements in the table:

import dash
import dash_html_components as html

app = dash.Dash()

rows = 10000

app.layout = html.Div([
    html.Div(html.Table([html.Tr([html.Td(f'A{i}'),
                         html.Td(f'B{i}'),
                         html.Td(f'C{i}')]) for i in range(rows)]),
             style={'height': f'400px', 'width': f'200px',
                    'border': '1px', 'overflow': 'auto'}),
])

if __name__ == '__main__':
    app.run_server(port=5555, debug=True)

Thanks for the example! It doesn’t look like you are doing anything wrong here and I see similar performance issues when I run that example.

One more thing to try: Try setting debug=False. This will turn off component validation which is a JavaScript check that we make for every single component. Turning debug=False improved the performance by a few seconds for me.

If that still isn’t fast enough, then here are your options:

  1. Use dash_table.DataTable instead with virtualization=True or pagination (frontend or backend). virtualization will handle 10k no problem and serverside pagination has no limit on the number of rows that can be rendered. Rendering an arbitrary number of rows is one of the main reasons we built this component. In general, the browser has difficulty rendering over 50K elements on the screen which is why many components resort to tricks like virtualization (render rows on-the-fly as you scroll), pagination, or alternative rendering engines like WebGL (which is what we do for scattergl plots)
  2. Implement paging with html.Table with custom buttons so that you only render 500-1k rows a a time. With 10k rows, your transfering 1MB over the wire anyway so this would improve the network performance as well as the rendering performance.
  3. Let us know which features you would need in DataTable in order to switch from html.Table to DataTable. Then, wait for us to find a way to fit these into our customer roadmap for Dash Enterprise customers (if they aren’t already planned), a community member to implement these features, or prioritize these features by funding the development directly.
  4. Fund the development of improving the performance of the rendering part of the library. The first round of rendering improvements were made in 1.13.x and took about 6-8 weeks of development work. The next round of rendering improvements would likely take a similar amount of time. As noted in 1, there are fundamental limits to the number of elements that can be rendered in the web browser (50k-100k depending on the machine) so this would only improve rendering between 1k-20k rows.
  5. Build your own table component that handles virtualization internally (using e.g. https://github.com/bvaughn/react-virtualized) that suits your needs better than dash_table.DataTable. Our team can also be contracted to build out custom components, but for table work we’d rather use funding to improve DataTable itself.

Thanks Chris, I will definitely check the performance differential without the debug setting. My environment typically has debug on all the time while developing for the lazy reloads and that just carried over.

What I am trying to accomplish is using Dash Bootstrap Components to create some really sharp looking tables with controls resembling the examples in https://mdbootstrap.com/docs/jquery/tables/sort/ I don’t see and table sorting available using Dash DBC and am sure the paging/searching are up to me. I haven’t had the time to explore DataTable yet, because I don’t need it’s input capacities but will give it a try. I didn’t like the fixed font but see that can be restyled. I will definitely implement paging too. This was just one step along the way of getting there and was really wondering why the loads were so slow. My largest tables will always be around 5000 rows or 6300 rows.

If the table data is sparse, free of CSS, rendered in a split second, and displays instantaneously using Flask, what is actually going on during the several seconds of “Loading…”?

We’re rendering everything with JavaScript rather than serving HTML directly (which would be done via Flask), so our upper bound for performance is React itself. In this case, React can load a datatable this big pretty quickly (here’s an example I made: https://codepen.io/chriddyp/pen/rNxYwNZ?editors=0110) so the bottleneck is still in our own code.

When we load a layout JSON, we crawl through the entire layout and perform some light checks on every single component before we render it. With debug=True, this includes performing verification that the properties supplied in each component (in this case, just children) are valid, which includes checking the names of the properties provided and their types. If the type is invalid (e.g. a dict was supplied to children instead of a string, component, or number) then we display an error message popup in the devtools panel. When debug mode is false, I believe if we’re still doing some checks like seeing if the component is in a loading state and needs the “is-loading” attribute. We also create a new data structure(s) that maps the IDs of the components to their location in the layout tree so that when we make a callback update, we can quickly address that component without needing to crawl the entire tree again on every callback (imagine if row 9421 and column 3 had a component with an ID that was updated via a callback. We don’t really know upfront where these components might be nested within the layout). Recursively crawling 40K items (A 10K table with 3 columns has 10K Tr elements, 30K Td elements) and performing these light checks & data structure initialization routines ends up taking a few seconds for large layouts. This used to take a lot longer but we optimized some of this code in 1.13.x by removing some of the checks. I’m a little hazy on the details here: I wrote the original version 3 years ago but it has since been reworked & improved by other engineers on the team.

As we’re crawling through the layout tree, we’re also generating the object for React to render: For each element, we call React.createElement recursively from the bottom of the layout tree up towards the root of the tree (40K function calls). It seems that constructing an object this way is slower than how React has been optimized to generate layouts with one big render, but we’d have to dig in here more to understand why. It might not be… it might be that the light checks that we perform during initialization is the main cause of slowness. The other engineers that did the latest performance improvements would know more here.

Then finally, we pass that object to React to turn that JavaScript object into actual HTML that you see on the page in the DOM. This itself is quite fast within the 10K-40K element range but will slow down above 50K-100K rows due to browser limitations with the DOM. Server-side frameworks (like pure flask) would have the same limitations here.