Gzip compression with AWS Lambda deployed applications

I have a dashboard that has particularly heavy loads, that exceed the payload limit set by AWS (which is 6MB for those who wonder). In order to reduce the size of the response structure, I am looking at compressing the data that Dash returns, by enabling dash.Dash(compress=True) for my Dash applications. However, I have not been able to get my dashboard working correctly.

Note: if I set compress=False it does work. So, the dashboard (code) itself is in working order.

In the lambda_handler that takes care of the incoming request on AWS, and passes that event to my Dash server instance, I use the aws-wsgi package to process the output of Dash in a format that AWS can return to my browser where I am accessing my Dash app:

def lambda_handler(event, context):
    # app points to my dash.Dash(compress=True, ...) instance
    # base64_content_types is a list of all MimeTypes that should be considered as base64 encoded
    base64_content_types = [
        'text/html', 'application/xhtml+xml', 'application/xml', 'image/avif',
        'image/webp', 'image/apng', '*/*', 'application/signed-exchange'
    ]
    result = awsgi.response(app.server, event, context, base64_content_types)

Since by enabling compression in Dash, all the Dash responses are binary type and therefore I have to tag them all as base64 encoded types. When I look in my AWS logs, I can see that awsgi successfully generates a base64 string (by checking the contents of result).

However when loading my dash app in a browser I am met with the beautiful message internal server error.

Since I am very unfamiliar with the compressing mechanism in Dash (and in servers in general) I am wondering how I can fix this issue. What do I need to do to decompress the base64 string in my browser? I was under the assumption that everything should be handled automatically by whatever Dash code is loaded in my browser.

Anyone has an idea how to proceed?

UPDATE: After looking around some more, I found a way to decompress the base64 string that awsgi.response() generates:

import gzip
import zlib
import base64

base64_message = 'Fill in the base64 string here'

# Method 1 - gzip
base64_bytes = base64_message.encode('ascii')
message_bytes = base64.b64decode(base64_bytes)
message = gzip.decompress(message_bytes).decode('utf-8')
print(message)

# Method 2 - zlib
message = zlib.decompress(base64.b64decode(base64_message), 16 + zlib.MAX_WBITS).decode('utf-8')
print(message)

In this way I can see that the contents of the base64 string is indeed what I expect, so this seems to confirm that the compression in Dash is working correctly. The follow-up question here is: why isn’t my browser able to decompress this message? What do I need to enable (or set in the http request header) to tell the browser that it should decompress this data in the correct way?

Ok, problem solved! As usual, the problem I encountered was due to my own oversights rather than something wrong happening in Dash. (I forgot to add a return statement to lambda_handler, therefore AWS was not receiving any response which resulted in the internal server error). Though I have learned about how compression works in Dash application. I will summarize the lessons I learned here for future reference in case anyone want to use compression in there Dash apps that run on AWS:

  1. Enable compression in your Dash instance: app = dash.Dash(compress=True, ...)
  2. If you are using an API resource for your AWS application, in your tools’ template.yaml set "*/*" under Properties -> BinaryMediaTypes to tell your API that everything will be passed as binary data and it should not block that data.
  3. Wherever you call awsgi.response(), create a list of all Content-Type that will be communicated to and from the server. And pass that list to the base64_content_types parameter of awsgi.response().
    a. You have to be explicit in your Content-Type list as awsgi does not expand wildcard characters, i.e. you cannot just have "*/*" in the list because awsgi will do a literal comparison between the Content-Type of the communication and what is in the base64_content_types list.
    b. If you see UnicodeDecodeErrors in your CloudWatch logs coming from awsgi.__init__ then this most likely indicates that you encountered a Content-Type that you didn’t add to the list yet. If you can’t figure out what the correct type is (the Content-Type that awsgi sees is not always visible in the incoming event data), then do the following. Build your tool locally, then in the build folder, go to awsgi/__init__.py and add a print statement in the use_binary_response() function that prints the Content-Type to the AWS logs. Then deploy this adjusted version and test it out.

That’s it, the Dash code will take care of the compression/decompression by itself, just as you would expect.

1 Like