Handling failed background_callbacks elegantly

My application is deployed on Openshift (Red Hat’s Opinionated Kubernetes).

Because of our worker node size, the maximum memory requirement we can schedule a pod with is ~8GB. However, sometimes a celery worker in a pod that is processing data and rendering figures for background callbacks exceeds this maximum and is sig-9 killed. Because the pod was scheduled w/ the 8GB memory requirement satisfied, retrying the job won’t make a difference- the same data will be used and will overrun the available memory.

I’d like to report to the user that the graph they are expecting cannot be generated (we’ll be bulking up our node size in the future but I need a short-term fix) because the callback failed. Is there a Dash-native way to do this? e.g. a default return value from a background_callback if the task in the task queue fails? Alternatively, does anyone have a non-native pattern that they’ve used?

The best-case scenario would be to be aware of which signal killed the task and report that the figure cannot be generated because of resource constraints. If Dash’s celery_manager doesn’t have access to that API, I’d have a generic return graph that reports an error in lieu of a more informative graph.

Would appreciate any help very much!