serverSide rowModelType with Storing Changes

Thank you. I can even use filter with the help of this example ,

However, I have problem with cell editing. After I edit a cell in the table and sort/filter/scroll, the edited cell is back to what it was originally, since the fetched data has only the non-edited rows. Is there anyway to keep the edited cell data?

The request in extractRowsFromData(request, df): function in the example looks like this:
{‘startRow’: 0, ‘endRow’: 100, ‘rowGroupCols’: , ‘valueCols’: , ‘pivotCols’: , ‘pivotMode’: False, ‘groupKeys’: , ‘filterModel’: {}, ‘sortModel’: }

So there is no info regarding edited cells. Is there a way to add all edited cells to request or some other neat trick?

Hmm possibly.

If you have a unique identifier, you could store your changes in a log, and when you respond from the server you also include that as a state and merge the two together.

How would one do that exactly? I dont quite follow what is log in this context or state in that matter. In my case I do have an unique identifier for my rows.

Upon cellValueChanged as a trigger, then store the value inside a dictionary with the unique identifier as a key and then there is another dictionary inside where you would place the column as key and the value.

cellValueChanged → {uid: {col: v}}

Then when pulling info back from the server, you loop through the new data and check to see if the key exists in the stored data, if found then update that row like this:

newData={…row, …matched[uid]}
1 Like

Ahh I kind of now understand how to do it from your hint, but my understanding to the solution is not so efficient.
I make a callback with cellValueChanged, which alters the df in python. So when the serverside gets called via scrolling/filtering/sorting, the new value is in the df already. Or something like that.

But it would be smart and more efficient, if you could some how store all the altered rows in a dictionary in clientside and when the table gets scrolled/filtered/sorted this dictionary is sent to serverside aswel to be used to alter the df. You probably meant something like this, but how to do this exactly?

Yes, I meant to do it on the clientside. You just store the changes like I did above into a local store and then query it during your update request from the server.

I can explain more on Monday if I need.

I got this to work in clientside, but its not the most beautiful or efficient. I made an example on top of your example, if you want to comment or improve it and for the community. Id gladly hear some advice.

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import requests, json
import flask
import pandas as pd

app = Dash(__name__)

server = app.server

df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/ag-grid/olympic-winners.csv"
)
df.reset_index(inplace=True)
df.rename(columns={"index": "InputRowId"}, inplace=True)

rowData = df.to_dict('records')

columnDefs = [
    # Row group by country and by year is enabled.
    {"field": "country", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "year", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "athlete", "sortable": True, "filter": True},
    {"field": "age", "sortable": True, "filter": True},
    {"field": "date", "sortable": True, "filter": True},
    {"field": "sport", "sortable": True, "filter": True, 'editable': True},
    {"field": "total", "sortable": True, "filter": True, "aggFunc": "sum"},
    {"field": "InputRowId", "sortable": True, "filter": True, 'hide': True, 'suppressColumnsToolPanel': True},
]

def extractRowsFromData(request, df):
    response = []
    print(request)
    
    groupBy = []
    
    for i in request['newCells']:
        
        for j in request['newCells'][i]:
            
            cellvalue = request['newCells'][i][j]
            i = int(i)
            df.at[i, j] = cellvalue 
    dff = df.copy()
    
    if request['rowGroupCols']:
        groupBy = [i['id'] for i in request['rowGroupCols']]
    agg = {}
    if request['valueCols']:
        agg = {i['id']: i['aggFunc'] for i in request['valueCols']}
    if not request['groupKeys']:
        if groupBy:
            if agg:
                dff = dff.groupby(groupBy[0]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[0]).agg('count').reset_index()
    else:
        for i in range(len(request['groupKeys'])):
            dff = dff[dff[request['rowGroupCols'][i]['id']] == request['groupKeys'][i]]
        if len(request['groupKeys']) != len(groupBy):
            if agg:
                dff = dff.groupby(groupBy[:len(request['groupKeys'])+1]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg('count').reset_index()
    dff = dff.sort_values(by=[i['colId'] for i in request['sortModel']], ascending=[i['sort'] == 'asc' for i in request['sortModel']])
    
    return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff)}

@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json, df)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs=columnDefs,
            dashGridOptions={"rowSelection": "multiple", "sideBar": True},
            defaultColDef=dict(
                resizable=True,
                enableRowGroup=True,
                enableValue=True,
                enablePivot=True
            ),
            enableEnterpriseModules=True,
            rowModelType="serverSide",
            #getRowId="params.node.id",
            style={'overflow': 'auto', 'resize': 'both'}
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
        html.Div(id='hidden-div', style={'display':'hidden'}),
    ]
)

app.clientside_callback(
    """async function (id) {
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
        grid = await window.dash_ag_grid.getApiAsync(id)
        if (grid) {
            updateData(grid)
        }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)

app.clientside_callback(
   """function (wuu) {
        
        cellchange(wuu)
        return cellchange(wuu)
    }""",
    Output('hidden-div', 'children'),
    Input('grid', 'cellValueChanged')
)


if __name__ == "__main__":
    app.run(debug=True)

js file:


var globalData = {};

async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      console.log('ServerSideDatasource.getRows: params = ', params);
      var partsu = params.request; 
      partsu['newCells'] = globalData;
      globalData = {};
      var result = await getServerData(partsu)
      console.log('getRows: result = ', result);
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

function cellchange(params) {
  console.log('Cell change event:', params);
  
  const nestedObject = params['0'];
  console.log('Nested object:', nestedObject);
  const nesteddata = nestedObject.data;
  const inputrowid = nesteddata.InputRowId;
  const colid = nestedObject.colId;
 
  const value = nestedObject ? nestedObject.value : 'Value not found';
  
  globalData[inputrowid] = {[colid]:value};
  console.log('klobaali', globalData)
  return ''
}


I wonder how to set up the rowId for request. I now had to fetch the rowid from data key and then the index column InputRowId, which I made. If I set it up getRowId = InputRowId in grid options, the app wouldnt work. I dont quite understand why.

Sure, here is something you can do:

app.py

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import requests, json
import flask
import pandas as pd

app = Dash(__name__)

server = app.server

df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/ag-grid/olympic-winners.csv"
)
df.reset_index(inplace=True)
df.rename(columns={"index": "InputRowId"}, inplace=True)

rowData = df.to_dict('records')

columnDefs = [
    # Row group by country and by year is enabled.
    {"field": "country", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "year", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "athlete", "sortable": True, "filter": True},
    {"field": "age", "sortable": True, "filter": True},
    {"field": "date", "sortable": True, "filter": True},
    {"field": "sport", "sortable": True, "filter": True, 'editable': True},
    {"field": "total", "sortable": True, "filter": True, "aggFunc": "sum"},
    {"field": "InputRowId", "sortable": True, "filter": True, 'hide': True, 'suppressColumnsToolPanel': True},
]


def extractRowsFromData(request, df):
    response = []
    print(request)

    groupBy = []
    dff = df.copy()

    if request['rowGroupCols']:
        groupBy = [i['id'] for i in request['rowGroupCols']]
    agg = {}
    if request['valueCols']:
        agg = {i['id']: i['aggFunc'] for i in request['valueCols']}
    if not request['groupKeys']:
        if groupBy:
            if agg:
                dff = dff.groupby(groupBy[0]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[0]).agg('count').reset_index()
    else:
        for i in range(len(request['groupKeys'])):
            dff = dff[dff[request['rowGroupCols'][i]['id']] == request['groupKeys'][i]]
        if len(request['groupKeys']) != len(groupBy):
            if agg:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg('count').reset_index()
    dff = dff.sort_values(by=[i['colId'] for i in request['sortModel']],
                          ascending=[i['sort'] == 'asc' for i in request['sortModel']])

    return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff)}


@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json, df)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs=columnDefs,
            dashGridOptions={"rowSelection": "multiple", "sideBar": True},
            defaultColDef=dict(
                resizable=True,
                enableRowGroup=True,
                enableValue=True,
                enablePivot=True
            ),
            enableEnterpriseModules=True,
            rowModelType="serverSide",
            style={'overflow': 'auto', 'resize': 'both'}
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
        html.Div(id='hidden-div', style={'display': 'hidden'}),
    dcc.Store(id='stored_changes', data={}, storage_type='local')
    ]
)

app.clientside_callback(
    """async function (id) {
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
        grid = await window.dash_ag_grid.getApiAsync(id)
        if (grid) {
            updateData(grid)
        }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)

app.clientside_callback(
    """function (wuu, changes) {
         if (wuu) {
            newChanges = JSON.parse(JSON.stringify(changes))
            wuu.forEach((c) => {
                if (!newChanges[c['data']['InputRowId']]) {
                    newChanges[c['data']['InputRowId']] = {}
                }
                newChanges[c['data']['InputRowId']][c['colId']] = c['value']
            })
            return newChanges
         }
         return window.dash_clientside.no_update
     }""",
    Output('stored_changes', 'data'),
    Input('grid', 'cellValueChanged'),
    State('stored_changes', 'data'),
)

if __name__ == "__main__":
    app.run(debug=True)

assets/.js file

async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      var result = await getServerData(params.request)
      // loads stored data
      stored_changes = JSON.parse(localStorage.getItem('stored_changes'))
      newRowData = result.rowData.map((row) => {
        if (row?.InputRowId) {
            if (stored_changes[row?.InputRowId]) {
                return {...row, ...stored_changes[row?.InputRowId]}
            }
        }
        return row
      })

      // sets up response for with new data
      result['rowData'] = newRowData
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

Now, lets break it down, I adjusted your code for the extractRowsFromData since it is unnecessary for the altered data to be additionally pushed to the server, instead, I am storing the changes, locally right now, could alter to server side. Then instead of createServerSideDatasource I replace the stored changed values into the response from the server. This should keep your costs down since you wont be transferring the data as often.

You could store these as a window variable, but I like the store, because if you wanted to, you could push these changes to the server data and update a db from the change log. The window variable could also be configured to do this, it would just take a little stepping and probably creating another flask route to hand the data easier.

Let me know your thoughts. :slight_smile:

Thank you sir. It is insightful in the most delightful way.

Unfortunately the sort stops working in this method, because the changed values are not indeed sent to serverside. Also serverside filtering wouldnt work either.

In my example I dont think I transferred data anymore often? The altered row data was only sent when sorting/filtering or grouping occured. Albeit more data was sent in those occasions.

Haha, whoops.

Depends on the amount of data that you build.

Try this version, its similar to yours, except that it stores the changes on the server using your session as a key:

app.py

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State, no_update
import requests, json
import flask
import pandas as pd
import uuid

app = Dash(__name__)

server = app.server
server.secret_key = "xyz"

alterations = {}

df = pd.read_csv(
    "https://raw.githubusercontent.com/plotly/datasets/master/ag-grid/olympic-winners.csv"
)
df.reset_index(inplace=True)
df.rename(columns={"index": "InputRowId"}, inplace=True)

rowData = df.to_dict('records')

columnDefs = [
    # Row group by country and by year is enabled.
    {"field": "country", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "year", "sortable": True, "filter": True, "rowGroup": True, "hide": True},
    {"field": "athlete", "sortable": True, "filter": True},
    {"field": "age", "sortable": True, "filter": True},
    {"field": "date", "sortable": True, "filter": True},
    {"field": "sport", "sortable": True, "filter": True, 'editable': True},
    {"field": "total", "sortable": True, "filter": True, "aggFunc": "sum"},
    {"field": "InputRowId", "sortable": True, "filter": True, 'hide': True, 'suppressColumnsToolPanel': True},
]


def extractRowsFromData(request, df):
    response = []

    ## create session uid if not already done
    if not flask.session.get('uid'):
        flask.session['uid'] = uuid.uuid4()

    ## pull session stored changes
    if not alterations.get(flask.session['uid']):
        alterations[flask.session['uid']] = {}
    myalterations = alterations.get(flask.session['uid'])

    groupBy = []
    dff = df.copy()
    for k in myalterations.keys():
        for key in myalterations[k].keys():
            dff.at[k, key] = myalterations[k][key]

    if request['rowGroupCols']:
        groupBy = [i['id'] for i in request['rowGroupCols']]
    agg = {}
    if request['valueCols']:
        agg = {i['id']: i['aggFunc'] for i in request['valueCols']}
    if not request['groupKeys']:
        if groupBy:
            if agg:
                dff = dff.groupby(groupBy[0]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[0]).agg('count').reset_index()
    else:
        for i in range(len(request['groupKeys'])):
            dff = dff[dff[request['rowGroupCols'][i]['id']] == request['groupKeys'][i]]
        if len(request['groupKeys']) != len(groupBy):
            if agg:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg(agg).reset_index()
            else:
                dff = dff.groupby(groupBy[:len(request['groupKeys']) + 1]).agg('count').reset_index()
    dff = dff.sort_values(by=[i['colId'] for i in request['sortModel'] if i['colId'] in dff.columns],
                          ascending=[i['sort'] == 'asc' for i in request['sortModel'] if i['colId'] in dff.columns])

    return {'rowData': dff.to_dict('records')[request['startRow']: request['endRow']], 'rowCount': len(dff)}


@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json, df)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs=columnDefs,
            dashGridOptions={"rowSelection": "multiple", "sideBar": True},
            defaultColDef=dict(
                resizable=True,
                enableRowGroup=True,
                enableValue=True,
                enablePivot=True
            ),
            enableEnterpriseModules=True,
            rowModelType="serverSide",
            style={'overflow': 'auto', 'resize': 'both'}
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
        html.Div(id='hidden-div', style={'display': 'hidden'}),
    dcc.Store(id='stored_changes', data={}, storage_type='local')
    ]
)

app.clientside_callback(
    """async function (id) {
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
        grid = await window.dash_ag_grid.getApiAsync(id)
        if (grid) {
            updateData(grid)
        }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)

@app.callback(Output('stored_changes', 'data', allow_duplicate=True),
              Input('grid', 'cellValueChanged'),
              prevent_initial_call=True)
def syncData(changes):
    ## load and update alterations
    if not alterations.get(flask.session['uid']):
        alterations[flask.session['uid']] = {}
    myalterations = alterations.get(flask.session['uid'])
    for c in changes:
        if not myalterations.get(c['data']['InputRowId']):
            myalterations[c['data']['InputRowId']] = {}
        myalterations[c['data']['InputRowId']][c['colId']] = c['value']
    return no_update

if __name__ == "__main__":
    app.run(debug=True)

js

async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      var result = await getServerData(params.request)
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

I added a couple of fixes, your version would have only stored one column per key, this one will store all the changes on the df.

I also am making sure that the sorting columns exist in the data, as this was causing some error.

I would like to thank you again for spreading your knowledge and wisdom.

Im still learning and understanding flask. If I understood correctly, everytime a cellvalue is changed, the new value is sent to serverside in this model? I think it might be more efficient to store them clientside and then when sorting/filter/grouping happens the altered data is sent to server. Anyways I like your examples to learn more.

Thanks for noticing this!

1 Like