3
Arrow support (data representations of jagged arrays)
d
drahnreb
Working on nested arrays is a crucial task in most scientific fields. I think TileDB could perfectly leverage its strengths to support the community working in that field: https://youtu.be/jvt4v2LTGK0?t=1366
Working with Data Management in TileDB and Data Wrangling in awkward-array (https://github.com/scikit-hep/awkward-1.0) or other libraries with arrow support would be extremely beneficial workhorse.
Any updates on when Arrow will be supported?
Activity
Isaiah Norton
Hi @drahnreb, we will add Arrow support in the 2.1 release, which we are planning for the end of August. TileDB supports variable-length subarrays, so for jagged arrays of scalars (at the very least), there should be a simple mapping. Awkward looks very interesting, thanks - I will take a look.
d
drahnreb
@Isaiah Norton: I have seen your PR#1816 (https://github.com/TileDB-Inc/TileDB/pull/1816)
Great work. I understand it is an experimental C++ API implementation.
Is there plans for a python implementation or a more detailed documentation apart from the tests?
I would love to test it would be great if you could point me to further plans for arrow support.