How to plot bar chart with grouped categorical x and uneven number of columns

I want to plot this data in a grouped bar plot. The trouble is that categories donโ€™t have the same number of columns, so using a categorical x axis leaves huge gaps in the plot.

import pandas as pd
import plotly.express as px

df = pd.DataFrame([[4, 5, 8, 6, 7], ["a", "a", "a", "b", "b"]]).T
df.columns = ["value", "group"]

df["category"] = [f"x{i+1}" for i in range(len(df))]

fig = px.bar(df, x="category", y="value", color="group", barmode="group")

fig.show()

Is kinda workaround, but the x axis is not categorical anymore and I want the columns to be clustered within groups. Is there a proper way how to this?