How do I specify exact color values for qualitative data?

Hello,
I have a shiny app with a simple line plot. I would like to use a categorical variable in the data.frame to plot different colored lines, like so:

output$plot <- renderPlotly({
  d <- get.plot.data()
  p <- d %>%
    plot_ly(x = ~Date) %>%
    add_trace(y = ~ArealDensity, type = 'scatter', mode = 'lines+markers', color = ~Name)
  p
})

This works great, but I would like to set specific colors for the names that I expect to receive from get.plot.data(). For example, ‘Bob’ = rgb(200,0,0), ‘Fred’ = rgb(100,100,40), etc.
Depending on the data, get.plot.data() might return 1-4 unique names, and I want the same name to be the same color, irrespective of how many traces I need.
I appreciate your help on this; I’ve searched for answers without success.