serverSide rowModelType with Storing Changes

Thank you. I can even use filter with the help of this example ,

However, I have problem with cell editing. After I edit a cell in the table and sort/filter/scroll, the edited cell is back to what it was originally, since the fetched data has only the non-edited rows. Is there anyway to keep the edited cell data?

The request in extractRowsFromData(request, df): function in the example looks like this:
{‘startRow’: 0, ‘endRow’: 100, ‘rowGroupCols’: , ‘valueCols’: , ‘pivotCols’: , ‘pivotMode’: False, ‘groupKeys’: , ‘filterModel’: {}, ‘sortModel’: }

So there is no info regarding edited cells. Is there a way to add all edited cells to request or some other neat trick?

Hmm possibly.

If you have a unique identifier, you could store your changes in a log, and when you respond from the server you also include that as a state and merge the two together.

How would one do that exactly? I dont quite follow what is log in this context or state in that matter. In my case I do have an unique identifier for my rows.

Upon cellValueChanged as a trigger, then store the value inside a dictionary with the unique identifier as a key and then there is another dictionary inside where you would place the column as key and the value.

cellValueChanged → {uid: {col: v}}

Then when pulling info back from the server, you loop through the new data and check to see if the key exists in the stored data, if found then update that row like this:

newData={…row, …matched[uid]}
1 Like

Ahh I kind of now understand how to do it from your hint, but my understanding to the solution is not so efficient.
I make a callback with cellValueChanged, which alters the df in python. So when the serverside gets called via scrolling/filtering/sorting, the new value is in the df already. Or something like that.

But it would be smart and more efficient, if you could some how store all the altered rows in a dictionary in clientside and when the table gets scrolled/filtered/sorted this dictionary is sent to serverside aswel to be used to alter the df. You probably meant something like this, but how to do this exactly?

Yes, I meant to do it on the clientside. You just store the changes like I did above into a local store and then query it during your update request from the server.

I can explain more on Monday if I need.

I got this to work in clientside, but its not the most beautiful or efficient. I made an example on top of your example, if you want to comment or improve it and for the community. Id gladly hear some advice.

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import requests, json
import flask
import pandas as pd

app = Dash(__name__)

server = app.server

df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/ag-grid/olympic-winners.csv"
)
df.reset_index(inplace=True)
df.rename(columns={"index": "InputRowId"}, inplace=True)

rowData = df.to_dict('records')

columnDefs = [
    # Row group by country and by year is enabled.
    {"field": "country", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "year", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "athlete", "sortable": True, "filter": True},
    {"field": "age", "sortable": True, "filter": True},
    {"field": "date", "sortable": True, "filter": True},
    {"field": "sport", "sortable": True, "filter": True, 'editable': True},
    {"field": "total", "sortable": True, "filter": True, "aggFunc": "sum"},
    {"field": "InputRowId", "sortable": True, "filter": True, 'hide': True, 'suppressColumnsToolPanel': True},
]

def extractRowsFromData(request, df):
    response = []
    print(request)
    
    groupBy = []
    
    for i in request['newCells']:
        
        for j in request['newCells'][i]:
            
            cellvalue = request['newCells'][i][j]
            i = int(i)
            df.at[i, j] = cellvalue 
    dff = df.copy()
    
    if request['rowGroupCols']:
        groupBy = [i['id'] for i in request['rowGroupCols']]
    agg = {}
    if request['valueCols']:
        agg = {i['id']: i['aggFunc'] for i in request['valueCols']}
    if not request['groupKeys']:
        if groupBy:
            if agg:
                dff = dff.groupby(groupBy[0]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[0]).agg('count').reset_index()
    else:
        for i in range(len(request['groupKeys'])):
            dff = dff[dff[request['rowGroupCols'][i]['id']] == request['groupKeys'][i]]
        if len(request['groupKeys']) != len(groupBy):
            if agg:
                dff = dff.groupby(groupBy[:len(request['groupKeys'])+1]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg('count').reset_index()
    dff = dff.sort_values(by=[i['colId'] for i in request['sortModel']], ascending=[i['sort'] == 'asc' for i in request['sortModel']])
    
    return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff)}

@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json, df)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs=columnDefs,
            dashGridOptions={"rowSelection": "multiple", "sideBar": True},
            defaultColDef=dict(
                resizable=True,
                enableRowGroup=True,
                enableValue=True,
                enablePivot=True
            ),
            enableEnterpriseModules=True,
            rowModelType="serverSide",
            #getRowId="params.node.id",
            style={'overflow': 'auto', 'resize': 'both'}
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
        html.Div(id='hidden-div', style={'display':'hidden'}),
    ]
)

app.clientside_callback(
    """async function (id) {
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
        grid = await window.dash_ag_grid.getApiAsync(id)
        if (grid) {
            updateData(grid)
        }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)

app.clientside_callback(
   """function (wuu) {
        
        cellchange(wuu)
        return cellchange(wuu)
    }""",
    Output('hidden-div', 'children'),
    Input('grid', 'cellValueChanged')
)


if __name__ == "__main__":
    app.run(debug=True)

js file:


var globalData = {};

async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      console.log('ServerSideDatasource.getRows: params = ', params);
      var partsu = params.request; 
      partsu['newCells'] = globalData;
      globalData = {};
      var result = await getServerData(partsu)
      console.log('getRows: result = ', result);
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

function cellchange(params) {
  console.log('Cell change event:', params);
  
  const nestedObject = params['0'];
  console.log('Nested object:', nestedObject);
  const nesteddata = nestedObject.data;
  const inputrowid = nesteddata.InputRowId;
  const colid = nestedObject.colId;
 
  const value = nestedObject ? nestedObject.value : 'Value not found';
  
  globalData[inputrowid] = {[colid]:value};
  console.log('klobaali', globalData)
  return ''
}


I wonder how to set up the rowId for request. I now had to fetch the rowid from data key and then the index column InputRowId, which I made. If I set it up getRowId = InputRowId in grid options, the app wouldnt work. I dont quite understand why.

Sure, here is something you can do:

app.py

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import requests, json
import flask
import pandas as pd

app = Dash(__name__)

server = app.server

df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/ag-grid/olympic-winners.csv"
)
df.reset_index(inplace=True)
df.rename(columns={"index": "InputRowId"}, inplace=True)

rowData = df.to_dict('records')

columnDefs = [
    # Row group by country and by year is enabled.
    {"field": "country", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "year", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "athlete", "sortable": True, "filter": True},
    {"field": "age", "sortable": True, "filter": True},
    {"field": "date", "sortable": True, "filter": True},
    {"field": "sport", "sortable": True, "filter": True, 'editable': True},
    {"field": "total", "sortable": True, "filter": True, "aggFunc": "sum"},
    {"field": "InputRowId", "sortable": True, "filter": True, 'hide': True, 'suppressColumnsToolPanel': True},
]


def extractRowsFromData(request, df):
    response = []
    print(request)

    groupBy = []
    dff = df.copy()

    if request['rowGroupCols']:
        groupBy = [i['id'] for i in request['rowGroupCols']]
    agg = {}
    if request['valueCols']:
        agg = {i['id']: i['aggFunc'] for i in request['valueCols']}
    if not request['groupKeys']:
        if groupBy:
            if agg:
                dff = dff.groupby(groupBy[0]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[0]).agg('count').reset_index()
    else:
        for i in range(len(request['groupKeys'])):
            dff = dff[dff[request['rowGroupCols'][i]['id']] == request['groupKeys'][i]]
        if len(request['groupKeys']) != len(groupBy):
            if agg:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg('count').reset_index()
    dff = dff.sort_values(by=[i['colId'] for i in request['sortModel']],
                          ascending=[i['sort'] == 'asc' for i in request['sortModel']])

    return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff)}


@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json, df)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs=columnDefs,
            dashGridOptions={"rowSelection": "multiple", "sideBar": True},
            defaultColDef=dict(
                resizable=True,
                enableRowGroup=True,
                enableValue=True,
                enablePivot=True
            ),
            enableEnterpriseModules=True,
            rowModelType="serverSide",
            style={'overflow': 'auto', 'resize': 'both'}
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
        html.Div(id='hidden-div', style={'display': 'hidden'}),
    dcc.Store(id='stored_changes', data={}, storage_type='local')
    ]
)

app.clientside_callback(
    """async function (id) {
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
        grid = await window.dash_ag_grid.getApiAsync(id)
        if (grid) {
            updateData(grid)
        }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)

app.clientside_callback(
    """function (wuu, changes) {
         if (wuu) {
            newChanges = JSON.parse(JSON.stringify(changes))
            wuu.forEach((c) => {
                if (!newChanges[c['data']['InputRowId']]) {
                    newChanges[c['data']['InputRowId']] = {}
                }
                newChanges[c['data']['InputRowId']][c['colId']] = c['value']
            })
            return newChanges
         }
         return window.dash_clientside.no_update
     }""",
    Output('stored_changes', 'data'),
    Input('grid', 'cellValueChanged'),
    State('stored_changes', 'data'),
)

if __name__ == "__main__":
    app.run(debug=True)

assets/.js file

async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      var result = await getServerData(params.request)
      // loads stored data
      stored_changes = JSON.parse(localStorage.getItem('stored_changes'))
      newRowData = result.rowData.map((row) => {
        if (row?.InputRowId) {
            if (stored_changes[row?.InputRowId]) {
                return {...row, ...stored_changes[row?.InputRowId]}
            }
        }
        return row
      })

      // sets up response for with new data
      result['rowData'] = newRowData
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

Now, lets break it down, I adjusted your code for the extractRowsFromData since it is unnecessary for the altered data to be additionally pushed to the server, instead, I am storing the changes, locally right now, could alter to server side. Then instead of createServerSideDatasource I replace the stored changed values into the response from the server. This should keep your costs down since you wont be transferring the data as often.

You could store these as a window variable, but I like the store, because if you wanted to, you could push these changes to the server data and update a db from the change log. The window variable could also be configured to do this, it would just take a little stepping and probably creating another flask route to hand the data easier.

Let me know your thoughts. :slight_smile:

Thank you sir. It is insightful in the most delightful way.

Unfortunately the sort stops working in this method, because the changed values are not indeed sent to serverside. Also serverside filtering wouldnt work either.

In my example I dont think I transferred data anymore often? The altered row data was only sent when sorting/filtering or grouping occured. Albeit more data was sent in those occasions.

Haha, whoops.

Depends on the amount of data that you build.

Try this version, its similar to yours, except that it stores the changes on the server using your session as a key:

app.py

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State, no_update
import requests, json
import flask
import pandas as pd
import uuid

app = Dash(__name__)

server = app.server
server.secret_key = "xyz"

alterations = {}

df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/ag-grid/olympic-winners.csv"
)
df.reset_index(inplace=True)
df.rename(columns={"index": "InputRowId"}, inplace=True)

rowData = df.to_dict('records')

columnDefs = [
    # Row group by country and by year is enabled.
    {"field": "country", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "year", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "athlete", "sortable": True, "filter": True},
    {"field": "age", "sortable": True, "filter": True},
    {"field": "date", "sortable": True, "filter": True},
    {"field": "sport", "sortable": True, "filter": True, 'editable': True},
    {"field": "total", "sortable": True, "filter": True, "aggFunc": "sum"},
    {"field": "InputRowId", "sortable": True, "filter": True, 'hide': True, 'suppressColumnsToolPanel': True},
]


def extractRowsFromData(request, df):
    response = []

    ## create session uid if not already done
    if not flask.session.get('uid'):
        flask.session['uid'] = uuid.uuid4()

    ## pull session stored changes
    if not alterations.get(flask.session['uid']):
        alterations[flask.session['uid']] = {}
    myalterations = alterations.get(flask.session['uid'])

    groupBy = []
    dff = df.copy()
    for k in myalterations.keys():
        for key in myalterations[k].keys():
            dff.at[k, key] = myalterations[k][key]

    if request['rowGroupCols']:
        groupBy = [i['id'] for i in request['rowGroupCols']]
    agg = {}
    if request['valueCols']:
        agg = {i['id']: i['aggFunc'] for i in request['valueCols']}
    if not request['groupKeys']:
        if groupBy:
            if agg:
                dff = dff.groupby(groupBy[0]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[0]).agg('count').reset_index()
    else:
        for i in range(len(request['groupKeys'])):
            dff = dff[dff[request['rowGroupCols'][i]['id']] == request['groupKeys'][i]]
        if len(request['groupKeys']) != len(groupBy):
            if agg:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg('count').reset_index()
    dff = dff.sort_values(by=[i['colId'] for i in request['sortModel'] if i['colId'] in dff.columns],
                          ascending=[i['sort'] == 'asc' for i in request['sortModel'] if i['colId'] in dff.columns])

    return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff)}


@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json, df)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs=columnDefs,
            dashGridOptions={"rowSelection": "multiple", "sideBar": True},
            defaultColDef=dict(
                resizable=True,
                enableRowGroup=True,
                enableValue=True,
                enablePivot=True
            ),
            enableEnterpriseModules=True,
            rowModelType="serverSide",
            style={'overflow': 'auto', 'resize': 'both'}
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
        html.Div(id='hidden-div', style={'display': 'hidden'}),
    dcc.Store(id='stored_changes', data={}, storage_type='local')
    ]
)

app.clientside_callback(
    """async function (id) {
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
        grid = await window.dash_ag_grid.getApiAsync(id)
        if (grid) {
            updateData(grid)
        }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)

@app.callback(Output('stored_changes', 'data', allow_duplicate=True),
              Input('grid', 'cellValueChanged'),
              prevent_initial_call=True)
def syncData(changes):
    ## load and update alterations
    if not alterations.get(flask.session['uid']):
        alterations[flask.session['uid']] = {}
    myalterations = alterations.get(flask.session['uid'])
    for c in changes:
        if not myalterations.get(c['data']['InputRowId']):
            myalterations[c['data']['InputRowId']] = {}
        myalterations[c['data']['InputRowId']][c['colId']] = c['value']
    return no_update

if __name__ == "__main__":
    app.run(debug=True)

js

async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      var result = await getServerData(params.request)
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

I added a couple of fixes, your version would have only stored one column per key, this one will store all the changes on the df.

I also am making sure that the sorting columns exist in the data, as this was causing some error.

I would like to thank you again for spreading your knowledge and wisdom.

Im still learning and understanding flask. If I understood correctly, everytime a cellvalue is changed, the new value is sent to serverside in this model? I think it might be more efficient to store them clientside and then when sorting/filter/grouping happens the altered data is sent to server. Anyways I like your examples to learn more.

Thanks for noticing this!

1 Like

I have a follow up on this after many months.

The documentation in Infinite Row Model (Server-Side) | Dash for Python Documentation | Plotly says:

I find no more documentation regarding the key argument storeInfo. What possibilities does it have?

As an example:

storeinfo = 'storeInfo':{'ids':[3]} ##Doesnt work, but as an example what could be done with storeInfo? Im trying to preselect row id 3 here.
return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff), storeinfo}

Another question, which touches the subject: If I were to preselect, say row 3, everytime filter/sort/scrolling happens, how would I do it in this setup? Since the filter/sort/scrolling doesnt actually change and trigger any dash state. Perhaps it is possible to have a callback that some how listens to these events in new dash ag grid version?

Thank you!

Hey, if you are using the serverSide model, then you aren’t using the infinite model.

If you want, you could add things to the fetch request process to capture anytime that a request has been triggered to the server.

Also, with dash 2.16+, you can set props on components using the js function: dash_clientside.set_props

1 Like

Thank you sir.
The whole dash_clientside.set_props seems to be a big game changer and I didnt realize it until now that you mentioned it! I will try this out.

I just made the wrong assumption that since they are similar and seemed to have same return value that it would indeed have same return value.

Not sure how this helps without the new set props? Maybe I didnt understand.

Thank you once again!

1 Like

I tried using set props to change selectedRows state directly in the javascript. For some reason it didnt work. I could have probably got it to work by changing some dcc.store, which then triggers a callback to change the state of selectedRows, but thats not very eloquent to say the least.

In the end with some help with GPT4 I actually utilized the key argument storeInfo and it actually does work!

js.file:

function createServerSideDatasource() {
  const dataSource = {
      getRows: async (params) => {
          console.log('ServerSideDatasource.getRows: params = ', params);

          var partsu = params.request; 
          partsu['newCells'] = globalData;
          partsu['tieto'] = globaltieto;
          globalData = {};
          var result = await getServerData(partsu);
          
         
          setTimeout(function () {
              params.success(result);
              // Select the rows with the specified IDs
              if (result.storeInfo && result.storeInfo.selectedRowIds) {
                  const selectedRowIds = result.storeInfo.selectedRowIds;
                  const api = params.api;            
                  api.forEachNode(node => {
                      if (selectedRowIds.includes(node.data.InputRowId.toString())) {
                          node.setSelected(true);
                      }
                  });
              }
          }, 100);
      },
  };
  return dataSource;
}

return part in python:

{'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff), 'storeInfo': {'selectedRowIds': ['2']}}
1 Like