Auto adjust xshift on boxplot annotation

Looking to add point count labels to a boxplot with boxmode=‘group’ in go.figure or using the color category in plotly express.

Have figured out how to loop through and add the annotations, but can’t figure out how to tap into the offset information dynamically. I have explicitly set the xshift, which works OK until a user resizes anything.

Any ideas how to set an xshift as a percentage or otherwise figure out how the plotly rendering decides on the pixel value for the offset group?

Example code using the iris dataset.

import plotly.express as px
df = px.data.tips()
fig = px.box(df, x="time", y="total_bill", color ='sex')

# Add annotations of count
for i, sex in enumerate(df['sex'].unique()):
    xshift = 75
    if i == 0:
        xshift = -xshift 
    else:
        xshift = xshift   
        
    for time in df["time"].unique():
        subdf = df[(df['sex']==sex) & (df['time']==time)]
        text = str(len(subdf))
    
        fig.add_annotation(
            x=time, 
            y=subdf['total_bill'].median(),
            text=text,
            showarrow=False,
            xshift = xshift,
            yshift=10)
fig.show()

Original Figure:

Zoomed in on Dinner: