Dash ag grid: load folder structure from a mounted drive using tree data dynamically

Hai,

I am trying to create a tree data structure using Dash AG Grid from a mounted drive. Instead of loading/fetching all the directories, sub-directories, and files simultaneously, I want to dynamically load the drive’s contents (using callbacks) when the user clicks on the folder or sub-folder. I found server-side tree data approach on the official Ag grid website but, I don’t know whether it is suitable for my use case or not. Moreover, it is not implemented with Python. Can somebody guide me on how to implement this?

Best regards,
Raghava Alajangi

Hmm, I don’t think that we have anything set up directly in this fashion. Master Detail can load from a callback.

If you can find an example of it in the AG grid docs, we might be able to figure out something.

As I mentioned above, I found this approach. It seems like they are using a JSON file and loading the data progressively. Isn’t it something that I need for my problem?

const columnDefs = [
  { field: 'employeeId', hide: true },
  { field: 'employeeName', hide: true },
  { field: 'jobTitle' },
  { field: 'employmentType' },
];

const gridOptions = {
  defaultColDef: {
    width: 240,
    flex: 1,
  },
  autoGroupColumnDef: {
    field: 'employeeName',
    cellRendererParams: {
      innerRenderer: (params) => {
        // display employeeName rather than group key (employeeId)
        return params.data.employeeName;
      },
    },
  },
  rowModelType: 'serverSide',
  treeData: true,
  columnDefs: columnDefs,
  animateRows: true,
  isServerSideGroupOpenByDefault: (params) => {
    // open first two levels by default
    return params.rowNode.level < 2;
  },
  isServerSideGroup: (dataItem) => {
    // indicate if node is a group
    return dataItem.group;
  },
  getServerSideGroupKey: (dataItem) => {
    // specify which group key to use
    return dataItem.employeeId;
  },
};

// setup the grid after the page has finished loading
document.addEventListener('DOMContentLoaded', function () {
  var gridDiv = document.querySelector('#myGrid');
  new agGrid.Grid(gridDiv, gridOptions);

  fetch('https://www.ag-grid.com/example-assets/small-tree-data.json')
    .then((response) => response.json())
    .then(function (data) {
      var fakeServer = createFakeServer(data);
      var datasource = createServerSideDatasource(fakeServer);
      gridOptions.api.setServerSideDatasource(datasource);
    });
});

function createFakeServer(fakeServerData) {
  const fakeServer = {
    data: fakeServerData,
    getData: function (request) {
      function extractRowsFromData(groupKeys, data) {
        if (groupKeys.length === 0) {
          return data.map(function (d) {
            return {
              group: !!d.children,
              employeeId: d.employeeId,
              employeeName: d.employeeName,
              employmentType: d.employmentType,
              jobTitle: d.jobTitle,
            };
          });
        }

        var key = groupKeys[0];
        for (var i = 0; i < data.length; i++) {
          if (data[i].employeeId === key) {
            return extractRowsFromData(
              groupKeys.slice(1),
              data[i].children.slice()
            );
          }
        }
      }

      return extractRowsFromData(request.groupKeys, this.data);
    },
  };

  return fakeServer;
}

function createServerSideDatasource(fakeServer) {
  const dataSource = {
    getRows: (params) => {
      console.log('ServerSideDatasource.getRows: params = ', params);

      var allRows = fakeServer.getData(params.request);

      var request = params.request;
      var doingInfinite = request.startRow != null && request.endRow != null;
      var result = doingInfinite
        ? {
            rowData: allRows.slice(request.startRow, request.endRow),
            rowCount: allRows.length,
          }
        : { rowData: allRows };
      console.log('getRows: result = ', result);
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };

  return dataSource;
}

If you could just give a link, that is also helpful.

I’ll take a look and see what I can come up with.

I found these links where they mentioned about lazy loading data method that loads the data dynamically.

infinite row model
server side data
server side tree data

Here is a starting point with working with their example:

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import requests, json

app = Dash(__name__)

rowData = requests.get('https://www.ag-grid.com/example-assets/small-tree-data.json').json()

grid = html.Div(
    [
        dcc.Store(id='gridData', data=rowData),
        dag.AgGrid(
            id="grid",
            columnDefs= [
                {"field": 'employeeId', "hide": True},
                {"field": 'employeeName', "hide": True},
                {"field": 'jobTitle'},
                {"field": 'employmentType'},
            ],
            defaultColDef={
                "flex": 1,
            },
            dashGridOptions={
                "autoGroupColumnDef": {
                    "field": "employeeName",
                    "cellRendererParams": {
                        "function": "groupRenderer"
                    },
                },
                "treeData": True,
                "isServerSideGroupOpenByDefault": {'function': 'params ? params.rowNode.level < 2 : null'},
                "isServerSideGroup": {'function': 'params ? params.group : null'},
                "getServerSideGroupKey": {"function": 'params ? params.employeeId : null'}
            },
            enableEnterpriseModules=True,
            rowModelType="serverSide",
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
    ]
)

app.clientside_callback(
    """async function (id, data) {
        const delay = ms => new Promise(res => setTimeout(res, ms));
        const updateData = (data, grid) => {
          var fakeServer = createFakeServer(data);
          var datasource = createServerSideDatasource(fakeServer);
          grid.setServerSideDatasource(datasource);
        };
        var grid;
            try {
                grid = dash_ag_grid.getApi(id)
            } catch {}
            count = 0
            while (!grid) {
                await delay(200)
                try {
                    grid = dash_ag_grid.getApi(id)
                } catch {}
                count++
                if (count > 20) {
                    break;
                }
            }
            if (grid) {
                updateData(data, grid)
            }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id'), State('gridData', 'data')
)


if __name__ == "__main__":
    app.run(debug=True)

And then the functions that drive it:

function createFakeServer (fakeServerData) {
  const fakeServer = {
    data: fakeServerData,
    getData: function (request) {
      function extractRowsFromData(groupKeys, data) {
        if (groupKeys.length === 0) {
          return data.map(function (d) {
            return {
              group: !!d.children,
              employeeId: d.employeeId,
              employeeName: d.employeeName,
              employmentType: d.employmentType,
              jobTitle: d.jobTitle,
            };
          });
        }
        var key = groupKeys[0];
        for (var i = 0; i < data.length; i++) {
          if (data[i].employeeId === key) {
            return extractRowsFromData(
              groupKeys.slice(1),
              data[i].children.slice()
            );
          }
        }
      }
      return extractRowsFromData(request.groupKeys, this.data);
    },
  };
  return fakeServer;
}

function createServerSideDatasource(fakeServer) {
  const dataSource = {
    getRows: (params) => {
      console.log('ServerSideDatasource.getRows: params = ', params);
      var allRows = fakeServer.getData(params.request);
      var request = params.request;
      var doingInfinite = request.startRow != null && request.endRow != null;
      var result = doingInfinite
        ? {
            rowData: allRows.slice(request.startRow, request.endRow),
            rowCount: allRows.length,
          }
        : { rowData: allRows };
      console.log('getRows: result = ', result);
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

dagfuncs.groupRenderer = function (){
        return {
            innerRenderer: (params) => {
            console.log(params)
            // display employeeName rather than group key (employeeId)
            return params.data.employeeName;
          }
          }
        }

The easiest way would probably be some way to update the getData of the createFakeServer to be responsive from a backend process like an api call to your server.

Alright, got the server side to work, this utilizing the server route, enjoy:

app.py

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import requests, json
import flask

app = Dash(__name__)

server = app.server

rowData = requests.get('https://www.ag-grid.com/example-assets/small-tree-data.json').json()

def extractRowsFromData(groupKeys, data):
    response = []
    if len(groupKeys) == 0:
        for row in data:
            response.append({
                'group': not row.get('children') is None,
                'employeeId': row['employeeId'],
                'employeeName': row['employeeName'],
                'employmentType': row['employmentType'],
                'jobTitle': row['jobTitle']
            })
        return response
    key = groupKeys[0]
    for row in data:
      if row['employeeId'] == key:
          response += extractRowsFromData(
              groupKeys[1:],
              row['children']
            )
    return response

@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json['groupKeys'], rowData)
    return json.dumps(response)


grid = html.Div(
    [
        dag.AgGrid(
            id="grid",
            columnDefs= [
                {"field": 'employeeId', "hide": True},
                {"field": 'employeeName', "hide": True},
                {"field": 'jobTitle'},
                {"field": 'employmentType'},
            ],
            defaultColDef={
                "flex": 1,
            },
            dashGridOptions={
                "autoGroupColumnDef": {
                    "field": "employeeName",
                    "cellRendererParams": {
                        "function": "groupRenderer"
                    },
                },
                "treeData": True,
                "isServerSideGroupOpenByDefault": {'function': 'params ? params.rowNode.level < 2 : null'},
                "isServerSideGroup": {'function': 'params ? params.group : null'},
                "getServerSideGroupKey": {"function": 'params ? params.employeeId : null'}
            },
            enableEnterpriseModules=True,
            rowModelType="serverSide",
        ),
    ]
)

app.layout = html.Div(
    [
        dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
        grid,
    ]
)

app.clientside_callback(
    """async function (id) {
        const delay = ms => new Promise(res => setTimeout(res, ms));
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
            try {
                grid = dash_ag_grid.getApi(id)
            } catch {}
            count = 0
            while (!grid) {
                await delay(200)
                try {
                    grid = dash_ag_grid.getApi(id)
                } catch {}
                count++
                if (count > 20) {
                    break;
                }
            }
            if (grid) {
                updateData(grid)
            }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)


if __name__ == "__main__":
    app.run(debug=True)

js file:


async function getServerData(request) {
    response = await fetch('./api/serverData', {'method': 'POST', 'body': JSON.stringify(request),
      'headers': {'content-type': 'application/json'}})
    return response.json()
}

function createServerSideDatasource() {
  const dataSource = {
    getRows: async (params) => {
      console.log('ServerSideDatasource.getRows: params = ', params);
      var allRows = await getServerData(params.request)
      var request = params.request;
      var doingInfinite = request.startRow != null && request.endRow != null;
      var result = doingInfinite
        ? {
            rowData: allRows.slice(request.startRow, request.endRow),
            rowCount: allRows.length,
          }
        : { rowData: allRows };
      console.log('getRows: result = ', result);
      setTimeout(function () {
        params.success(result);
      }, 200);
    },
  };
  return dataSource;
}

dagfuncs.groupRenderer = function (){
        return {
            innerRenderer: (params) => {
            console.log(params)
            // display employeeName rather than group key (employeeId)
            return params.data.employeeName;
          }
          }
        }
3 Likes

Do I have to save the js file with a specific name in the assets folder? because it hasn’t been recognized by the app.

How do you have it setup in the folder?

Can you send a screenshot?

How do you have the rest of the Js file setup?

And what version of dash ag grid do you have?

I upgraded the dash ag-grid. It is working now!

However, you fetched a JSON file using requests used as a rowData. Can I achieve the same for a mounted drive that is available as a docker volume? I mean to create a JSON file from directories and files (for example using os.walk or os.listdir) of the drive and feed it as a rowData.

I only used it as a rowData, you can design it to return info however you want to.

If the path is a directory, then set it as grouped.

1 Like

I made a few changes to your code. Now, its working as expected.

import dash_ag_grid as dag
from dash import Dash, Input, Output, html, dcc, State
import json
import flask
import dash_bootstrap_components as dbc
from pathlib import Path
from datetime import datetime as dt

app = Dash(__name__,
           external_stylesheets=[dbc.themes.DARKLY, dbc.icons.BOOTSTRAP],
           assets_folder="assets"
           )

server = app.server

root_dir = "path/to/a/directory"


def extractRowsFromData(groupKeys, root_dir):
    response = []
    current_path = Path(*groupKeys) if groupKeys else Path(root_dir)
    if current_path.is_dir():
        for entry in current_path.iterdir():
            if entry.is_file() and entry.suffix == ".mp3":
                modified_time = dt.fromtimestamp(
                    entry.stat().st_mtime).strftime("%d-%b-%Y %I.%M %p")
                response.append({
                    "group": entry.is_dir(),
                    "name": entry.name,
                    "path": str(entry),  # Convert Path object to a string
                    "modified_time": modified_time
                })
            elif entry.is_dir():
                modified_time = dt.fromtimestamp(
                    entry.stat().st_mtime).strftime("%d-%b-%Y %I.%M %p")
                response.append({
                    "group": entry.is_dir(),
                    "name": entry.name,
                    "path": str(entry),  # Convert Path object to a string
                    "modified_time": modified_time
                })
    return response


@server.route('/api/serverData', methods=['POST'])
def serverData():
    response = extractRowsFromData(flask.request.json['groupKeys'], root_dir)
    return json.dumps(response)


def create_grid():
    return html.Div(
        [
            dag.AgGrid(
                id="grid",
                className="ag-theme-balham-dark",
                columnDefs=[
                    {"field": 'modified_time', "hide": False},
                ],
                defaultColDef={
                    "flex": 1,
                    "sortable": True,
                    "filter": True
                },
                dashGridOptions={
                    "autoGroupColumnDef": {
                        "field": "name",
                        "cellRendererParams": {
                            # "function": "groupRenderer",
                            "suppressCount": True,
                            "checkbox": True
                        },
                    },
                    "treeData": True,
                    "rowSelection": 'multiple',
                    "groupSelectsChildren": True,
                    # "groupDefaultExpanded": 0,
                    "suppressRowClickSelection": True,
                    "isRowSelectable": {
                        "function": "params.data ? params.data.path.endsWith('.mp3') : false",
                        # "function": "params ? params.rowNode.level < 1: true"
                    },
                    "isServerSideGroupOpenByDefault": {
                        'function': 'params ? params.rowNode.level < 1: null'},
                    "isServerSideGroup": {
                        'function': 'params ? params.group : null'},
                    "getServerSideGroupKey": {
                        "function": 'params ? params.path : null'}
                },
                enableEnterpriseModules=True,
                rowModelType="serverSide",
            ),
        ]
    )


app.layout = html.Div([
    dcc.Markdown("Example: Organisational Hierarchy using Tree Data "),
    dcc.Input(id="just-filtered-input",
              placeholder="filter...",
              style={"width": "400px"}),
    create_grid(),
    html.Div(id="output"),
])

app.clientside_callback(
    """async function (id) {
        const delay = ms => new Promise(res => setTimeout(res, ms));
        const updateData = (grid) => {
          var datasource = createServerSideDatasource();
          grid.setServerSideDatasource(datasource);
        };
        var grid;
            try {
                grid = dash_ag_grid.getApi(id)
            } catch {}
            count = 0
            while (!grid) {
                await delay(200)
                try {
                    grid = dash_ag_grid.getApi(id)
                } catch {}
                count++
                if (count > 20) {
                    break;
                }
            }
            if (grid) {
                updateData(grid)
            }
        return window.dash_clientside.no_update
    }""",
    Output('grid', 'id'), Input('grid', 'id')
)


if __name__ == "__main__":
    app.run(debug=True)

However, if a directory contains quite a number of entries, it takes too much time to load. Is it possible to load the initial entries when the user clicks on a directory and progressively load the rest when it is scrolled?

I’m not 100% sure, but you can look through here React Data Grid: SSRM Tree Data

Or check on the interwebs to see if other people have done it.

You should also be able to parse out of the request where the data is.

Honestly, it might be worth having a procedure dump your folder structure to a database however many times you want to use this.