Does ds.read_file() download the entire file into memory before returning, or does it return a handle that supports lazy row-group-level access

Dash Python
1

Hi,

I’m experiencing Out of Memory (OOM) errors in my Dash Enterprise app when reading large Parquet files from an S3 data source. My current code looks like this:

**from dash_enterprise_libraries import data_sources as ds
import pyarrow.parquet as pq

file_handle = ds.read_file(path, DATA_SOURCE)
table = pq.read_table(file_handle, columns=columns, filters=filter_expr)**

I noticed that the code does a file_handle.seek(0) before the second read, which suggests ds.read_file() returns an in-memory buffer (like BytesIO) rather than a lazy/streaming file handle.

My question: Does ds.read_file() download the entire file into memory before returning, or does it return a handle that supports lazy row-group-level access — so that pyarrow’s filters= can push down predicates and avoid downloading unnecessary data from S3?