✊🏿 Black Lives Matter. Please consider donating to Black Girls Code today.
⚡️ Concerned about the grid? Kyle Baranko teaches how to predicting peak loads using XGBoost. Register for the August webinar!

Plotly visualization not working in Pyspark kernel on EMR Jupyterhub Notebook

I’m trying to plot graphs using plotly on EMR Jupyterhub Notebook however the graphs are not being rendered in Pyspark kernel. (Note: Python kernel renders the graph just fine)

Sample code I am trying:

data_canada = px.data.gapminder().query(“country == ‘Canada’”)
fig = px.bar(data_canada, x=‘year’, y=‘pop’)

I am able to plot a graph with %%display sparkmagic however I am not able to figure out if we can get plotly working with %%display sparkmagic -

import random
data = [(‘Person:%s’ % i, i, random.randint(1, 5)) for i in range(1, 50)]
columns = [‘Name’, ‘Age’, ‘Random’]
spark_df = spark.createDataFrame(data, columns)


Has anyone tried this successfully? Please advise.

Much appreciated!