The virtual table a sounds a lot like regular-table: https://github.com/jpmorganchase/regular-table
Used in perspective: https://perspective.finos.org/ We use arrow c++ compiled with webassembly and some front end grid and chart plugins, perspective can run in a client server fashion and only sends diffs across the wire so works well for random access e.g. in pivoted views. Tim Paine tim.paine.nyc > On Jul 2, 2020, at 09:45, Matthias Vallentin <matth...@vallentin.net> wrote: > > Hi folks, > > We are reaching out to better understand the performance of ArrowJS when it > comes to viewing large amounts of data (> 1M records) in the browser’s DOM. > Our backend (https://github.com/tenzir/vast) spits out record batches, > which we are accumulating in the frontend with a RecordBatchReader. > > At first, we only want to render the data fast, line by line, with minimal > markup according to its types from the schema. We use a virtual scrolling > window to avoid overloading the DOM, that is, we lazily convert the record > batch data to DOM elements according to a scroll window defined by the > user. As the user scrolls, elements outside the window get removed and new > ones added. > > The data consists of one or more Tables that we are pulling in through the > RecordBatchReader. We use the Async Interator interface to go over the > record batches and convert them into rows. This API feels suboptimal for > our use cases, where we want random access to the data. Is there a > faster/better way to do this? > > Does anyone have any experience worth sharing with doing something similar? > The DOM is the main bottleneck, but if there are some clever things we can > do with Arrow to pull out the data in the most efficient way, that would be > nice. > > Matthias
