How to fast render of markers/tooltips on a map?

I’ve been stucked this last days cause I can’t get my app to run faster. It’s already hosted: Dash (buscafes.azurewebsites.net)
And the code is in Buscafe/app.py at main · lucaschicco/Buscafe (github.com)

The code is long, so I don’t expect you to analyze it. As you can see, the app consists of a map of coffee places. I’ve already asked some questions in the forum. The issue I’m facing has to do with the initial rendering and the rendering when the filters are applied. It takes a long time to render all the markers, tooltips, and popups, and I can’t seem to find a solution to make it faster.

I haven’t had success with clientside callbacks; I don’t know how to code them, and ChatGPT isn’t helping much with that. Clustering is not an option, and neither is loading fewer markers.

Other options include using Redis caching (which requires payment) or upgrading to a better Azure plan (I’m currently on the basic plan).


But I don’t want to unnecesary pay if the code can be improved.

Any ideas that can help?


Yeah, the apps glitchy and thats a substantial dataset you are you working with… I recently participated in Figure Friday and built out a map with ~10k lines of data and mapped it all. Dash

The cluster worked well but if I remove the cluster it practically crashes the application.

So with cluster not an option I have two other solutions you could try:

  1. Setup an async function, split the data up into like 10 dl.GeoJSON’s with [0:500], [500,1000], [1000:1500]… and so on. Then try rendering it into the map

  2. Setup a dl.GeoJSON with an outline of the neighborhoods. Instead of showing all the coffee shops you’d see the regions you’ve setup for the territory. You could add an id to the borders and when you click on a neighborhood the map zooms into that location and populates the map with the coffee shops in that location. You’d have to do a few things in order to get this to work though. You’d need to assign each row of data with a region so it knows what to show on reference to a border click . To create the regions you could use Dash Leaflet Edit Control and create a custom border via the polygon then save the output to a file which loads into dash automatically and connects to that dl.GeoJSON.

Hope you find a solution soon, curious whatcha find works.

1 Like

Thanks pip. Glad to hear I’m not the only one with the crushing problem. I’ll try both options and then answer. Btw, as you are more experienced than me, how about the payments plan? will it work for example having a better one? or the issue will go on?

I believe the issue would persist regardless of your payment plan, I think the issue is with leaflet being forced to render such a large dataset. Your going to need to get creative rather than trying a brute force solution

I haven’t done any profiling on your app, but typically the performance bottle necks for this type of use case are,

a) client side rendering performance
b) limited bandwidth between client and server

The symptom of (b) is slow initial load time, while (a) cases a laggy/unresponsive map experience. Scaling up your Azure resources is unlikely to help much with either.

What you can do is to,

  1. Use the GeoJSON component. For a large amount of markers, it provides superior rendering performance
  2. If your data is static, you can host your markers as assets and load them through an URL (helps on (b) only). Here, Azure might be able to improve performance even further, if you host the asset(s) through their CDN
  3. Use clustering. I know said you didn’t want to, but if you adjust the cluster size small enough, it would only make the UX better IMO (I don’t see that it provides any value to to render a bunch of markers on top of each other)

Adopting these points, you should be able to visualize ~ a few millions of points. If you need to go even higher, you can use tiling. It requires a bit more effort though, so I would only recommend going that route if you really need to :blush:

4 Likes

Hi Emil, sorry I didn’t answer before, I was exploring your solutions. I tried hosting the markers but the app is still really slow, because the problem are the tooltips and popups, I believe that is not “hostable”, because is just information taken from the dataset (or geojson in this case).

I briefly tried clustering and it works faster, but I’m not being able to add the tooltips and popups and personalized markers, I should investigate more.

The only way I found to get my app to run very fast is when I used plotly graph objects. I didn’t have personalized markers, but I did have tooltips and popups, and the response was automatically from the app. The con was (or is) that I cannot have customized markers, maybe using maki icons, but well, I wanted my markers.

I also tried pip’s solution of spliting the data into various geojsons but it didn’t work (maybe I did it wrong), but I believe it was because of the same I said before: tooltips and popups (if I just delete them from de app it works perfectly, even with the 2200 (personalized) markers I have). His second option is out of the table by now, it would take me time and I wanted to explore other options.

Anyway, I’ll keep you updated. Thanks!

Great to hear that you are making progress! Just so that I understand - did you switch to (or test with) the GeoJSON component? I am asking because I would expect this change to make a significant difference in itself, and it is a prerequisite for getting the most out of (2) and (3). For inspiration, you can see some examples in the docs,

https://www.dash-leaflet.com/docs/geojson_tutorial

You can retain the tooltip and the icon with cluster this is how I was able to do it, not exact but it should give a good starting point to adapt from:

point_to_layer_js = assign(
    """
function(feature, latlng){
    const iconData = feature.properties.icon_data;
    const flag = L.icon({
        iconUrl: iconData.url, 
        iconSize: [35, 35],
        tooltipAnchor: [20, 0]
    });
    const marker = L.marker(latlng, {icon: flag});
    marker.bindTooltip('<img src="' + feature.properties.image + '" style="width: 30vw; height: 20vh; max-width: 250px; max-height: 250px;">');
    marker.on('click', function(e) {
        var event = new CustomEvent('marker_click', {detail: feature.properties.pk});
        window.dispatchEvent(event);
    });
    return marker;
}
"""
)

dl.GeoJSON(
                            data=geojson_dict,
                            id="locations_layer",
                            cluster=True,
                            zoomToBoundsOnClick=True,
                            superClusterOptions=dict(radius=40),
                            pointToLayer=point_to_layer_js,
                        )

I’ve also been able to setup a modal on click of the icon but its pretty advanced and think its outside of your project at this current point but if you find yourself wanting actions with clicking the icon let me know and I have some code you can reference.

I’m also using the import geopandas as gpd instead or the typical pandas which I found works better for my project.

1 Like

Well, I tried at first but couldn’t arrive to anything, but that was because I wan’t figuring it out , neither chatgpt haha. But, BUT! I passed the " Interactivity via the hideout prop" code from the geojson link to chatgpt and started constructing my app since there, and little by little y could add the rest of the features of my app.

I haven’t tried out on production, but in Jupyter the speed difference is really huge, I’m impressed.
Basically, the changes where:

dl.GeoJSON(data=geojson_data, filter=geojson_filter, hideout=dd_defaults, id="geojson", zoomToBounds=True,
                           options=dict(pointToLayer=assign(
                                """function(feature, latlng){
                                       return L.marker(latlng, {
                                           icon: L.icon({
                                               iconUrl: feature.properties.icon_url,
                                               iconSize: [15, 23],
                                               iconAnchor: [12, 23],
                                               popupAnchor: [1, -34],
                                               shadowSize: [41, 41]
                                           })
                                       }).bindTooltip(feature.properties.tooltip, {direction: "top", offset: L.point(0, -20), opacity: 0.9, className: 'marker-tooltip'})
                                         .bindPopup(feature.properties.popup);
                                }""")))

I add that part to the “app.layout([html.Div([dl.Map(children” section, with the “geojson_data = dlx.dicts_to_geojson([{” previuos to the app.layout.

The other significant change was converting the return from “update_map” from a dataframe to a geojson

@app.callback(
    Output('geojson', 'data'),
    [Input('feature-filter', 'value'),
     Input('filtro-dias', 'value'),
     Input('filtro-barrios', 'value'),
     Input('search-input', 'value'),
     Input('rating-slider', 'value'),
     Input('map-style-dropdown', 'value')]
)
def update_map(features, days, barrios, search, rating, map_style):
    filtered_df = df.copy()
    
    # Aplicar los filtros
    if features:
        for feature in features:
            filtered_df = filtered_df[filtered_df[feature] == True]
    
    if days:
        day_filters = [f"{day}_open" for day in days]
        filtered_df = filtered_df.dropna(subset=day_filters, how='all')

    if barrios:
        filtered_df = filtered_df[filtered_df['Barrio'].isin(barrios)]
    
    if search:
        filtered_df = filtered_df[filtered_df['Nombre'].str.contains(search, case=False)]

    if rating:
        filtered_df = filtered_df[(filtered_df['Rating'] >= rating[0]) & (filtered_df['Rating'] <= rating[1])]

    # Convertir los datos filtrados a GeoJSON
    geojson_data = dlx.dicts_to_geojson([{
        "name": row["Barrio"],
        "lat": row["Latitud"],
        "lon": row["Longitud"],
        "tooltip": f"""
            <p class='nombre'>{row['Nombre']}</p>
            <p class='stars'>{generate_stars(row['Rating'])}</p>
            <p><span class='bold-text'>Reviews: </span>{row['Cantidad Reviews']}</p>
            <p><span class='bold-text'>Dirección: </span>{row['Dirección']}</p>
        """,
        "popup": f"""
            <h4 style='font-family: Montserrat; font-size: 16px; font-weight: bold;'><u>{row['Nombre']}</u></h4>
            <p style='font-family: Montserrat; font-size: 14px;'><strong>Rating: </strong>{row['Rating']}</p>
            <p style='font-family: Montserrat; font-size: 14px;'><strong>Cantidad Reviews: </strong>{row['Cantidad Reviews']}</p>
            <p style='font-family: Montserrat; font-size: 14px;'><strong>Sitio Web: </strong><a href='{row['Sitio Web']}' target='_blank'>{row['Sitio Web']}</a></p>
            <p style='font-family: Montserrat; font-size: 14px;'><strong>Dirección: </strong>{row['Dirección']}</p>
            <div style='font-family: Montserrat; font-size: 14px;'>{' '.join(format_hours(row))}</div>
        """,
        "icon_url": get_icon_url(row["Rating"])
    } for _, row in filtered_df.iterrows()])

    return geojson_data

Before it was:

@app.callback(
    Output('filtered-data', 'data'),
    [Input('rating-slider', 'value'), Input('feature-filter', 'value'), Input('filtro-dias', 'value'), Input('filtro-barrios', 'value'), Input('search-input', 'value')]
)
@cache.memoize()
def filter_data(rating_range, selected_features, selected_days, selected_barrios, search_input):
    filtered_df = df2[(df2['Rating'] >= rating_range[0]) & (df2['Rating'] <= rating_range[1])]
    if search_input and isinstance(search_input, str):
        filtered_df = filtered_df[filtered_df['Nombre'].str.contains(search_input, case=False)]
    for feature in selected_features:
        filtered_df = filtered_df[filtered_df[feature] == True]
    for day in selected_days:
        open_column = f'{day}_open'
        close_column = f'{day}_close'
        filtered_df = filtered_df[(~filtered_df[open_column].isna()) & (~filtered_df[close_column].isna())]
    if selected_barrios:
        filtered_df = filtered_df[filtered_df['Barrio'].isin(selected_barrios)]
    return filtered_df.to_dict('records')
@app.callback(
    Output('layer', 'children'),
    Input('filtered-data', 'data')
)
@cache.memoize()
def update_map(filtered_data):
    filtered_df = pd.DataFrame(filtered_data)
    def generate_stars(rating):
        full_star = '★'
        empty_star = '☆'
        return full_star * int(rating) + empty_star * (5 - int(rating))
    markers = [
        dl.Marker(
            position=[row['Latitud'], row['Longitud']],
            icon={
                "iconUrl": get_marker_icon(row['Rating']),
                "iconSize": [5, 5],
                "iconAnchor": [10, 20],
            },
            children=[
                dl.Tooltip(
                    html.Div([
                        html.P(row['Nombre'], className='nombre'),  # Clase CSS 'nombre' añadida aquí
                        html.P(generate_stars(row['Rating']), className='stars'),  # Clase CSS 'stars' añadida aquí
                        html.P([html.Span("Reviews: ", className='bold-text'), row['Cantidad Reviews']]),
                        html.P([html.Span("Dirección: ", className='bold-text'), row['Dirección']])
                    ]),
                    className='marker-tooltip'
                ),
                dl.Popup(
                    html.Div(
                        children=[
                            html.H4(html.U(row['Nombre']), style={'font-family': 'Montserrat', 'font-size': '16px', 'font-weight': 'bold'}),
                            html.P([html.Strong("Rating: "), str(row['Rating'])], style={'font-family': 'Montserrat', 'font-size': '14px'}),
                            html.P([html.Strong("Cantidad Reviews: "), str(row['Cantidad Reviews'])], style={'font-family': 'Montserrat', 'font-size': '14px'}),
                            html.P([html.Strong("Sitio Web: "), html.A(row['Sitio Web'], href=row['Sitio Web'], target="_blank")], style={'font-family': 'Montserrat', 'font-size': '14px'}),
                            html.P([html.Strong("Dirección: "), str(row['Dirección'])], style={'font-family': 'Montserrat', 'font-size': '14px'}),
                            *format_hours(row)  # Aquí se añade el resultado de format_hours
                        ],
                        className='marker-popup'
                    )
                )
            ]
        ) for _, row in filtered_df.iterrows()
    ]
    return markers

In the first case, both functions “filter” and the one with the markers are in the same one, and they will return a geojson instead of the dataframe and the markers. Well, I don’t really understand at all what happened lol, I just managed to guide chatgpt, of course with your advices.
I’ll keep you in touch when trying in production.

1 Like

I have a quick test for you to run, if you could add print statement at the start of the callback with start time and at the end with end time and record the time.

Then try this setup and see the difference between callback time:

import time
import geopandas as gpd
from shapely.geometry import Point

@app.callback(
    Output('geojson', 'data'),
    [Input('feature-filter', 'value'),
     Input('filtro-dias', 'value'),
     Input('filtro-barrios', 'value'),
     Input('search-input', 'value'),
     Input('rating-slider', 'value'),
     Input('map-style-dropdown', 'value')]
)
def update_map(features, days, barrios, search, rating, map_style):
    start_time = time.time()
    
    # Convert df to GeoDataFrame if it's not already
    if not isinstance(df, gpd.GeoDataFrame):
        gdf = gpd.GeoDataFrame(
            df, 
            geometry=gpd.points_from_xy(df.Longitud, df.Latitud),
            crs="EPSG:4326"
        )
    else:
        gdf = df.copy()
    
    # Apply filters
    if features:
        for feature in features:
            gdf = gdf[gdf[feature] == True]
    
    if days:
        day_filters = [f"{day}_open" for day in days]
        gdf = gdf.dropna(subset=day_filters, how='all')

    if barrios:
        gdf = gdf[gdf['Barrio'].isin(barrios)]
    
    if search:
        gdf = gdf[gdf['Nombre'].str.contains(search, case=False)]

    if rating:
        gdf = gdf[(gdf['Rating'] >= rating[0]) & (gdf['Rating'] <= rating[1])]

    # Convert to GeoJSON
    geojson_data = gdf.to_crs("EPSG:4326").to_json()
    
    # Add custom properties to GeoJSON
    import json
    geojson_dict = json.loads(geojson_data)
    for feature, row in zip(geojson_dict['features'], gdf.itertuples()):
        feature['properties'].update({
            "name": row.Barrio,
            "tooltip": f"""
                <p class='nombre'>{row.Nombre}</p>
                <p class='stars'>{generate_stars(row.Rating)}</p>
                <p><span class='bold-text'>Reviews: </span>{row._6}</p>
                <p><span class='bold-text'>Dirección: </span>{row.Dirección}</p>
            """,
            "popup": f"""
                <h4 style='font-family: Montserrat; font-size: 16px; font-weight: bold;'><u>{row.Nombre}</u></h4>
                <p style='font-family: Montserrat; font-size: 14px;'><strong>Rating: </strong>{row.Rating}</p>
                <p style='font-family: Montserrat; font-size: 14px;'><strong>Cantidad Reviews: </strong>{row._6}</p>
                <p style='font-family: Montserrat; font-size: 14px;'><strong>Sitio Web: </strong><a href='{row._7}' target='_blank'>{row._7}</a></p>
                <p style='font-family: Montserrat; font-size: 14px;'><strong>Dirección: </strong>{row.Dirección}</p>
                <div style='font-family: Montserrat; font-size: 14px;'>{' '.join(format_hours(row))}</div>
            """,
            "icon_url": get_icon_url(row.Rating)
        })
    
    end_time = time.time()
    print(f"Callback execution time: {end_time - start_time:.2f} seconds")
    
    return json.dumps(geojson_dict)

I’ve attempted to optimize it a fair bit, not entirely sure if it would preform better but It could shave some seconds.

sorry pip, I tried but I couldn’t make that work :frowning: Anyway, the app works much better. I’m trying to “convert” the callback functions to clientside. No success either. Maybe I’ll just stop at this point