Hi Alenka,
Great! Thank you so much for your inputs.
I have indeed tried to use schema when creating a table from a pylist and
it worked but in my use case, I wouldn't know the table schema beforehand
especially for the other columns - I need to do transformations before I
can cast it to the expected schema. Please let me know if you have any
other thoughts.
Regards,
Infant Alex
On Wed, May 3, 2023 at 9:43 AM Alenka Frim <ale...@voltrondata.com.invalid>
wrote:
> Hi Alex,
>
> passing the schema to from_pylist() method on the Table should work for
> your example (not sure if it solves your initial problem?)
>
> import pyarrow as pa
>
> table_schema = pa.schema([pa.field("id", pa.int32()),
> pa.field("names", pa.map_(pa.string(), pa.string()))])
>
> table_data = [{"id": 1,"names": {"first_name": "Tyler", "last_name":"Brady"
> }},
> {"id": 2,"names": {"first_name": "Walsh", "last_name": "Weaver"}}]
>
> pa.Table.from_pylist(table_data, schema=table_schema)
> # pyarrow.Table
> # id: int32
> # names: map<string, string>
> # child 0, entries: struct<key: string not null, value: string> not null
> # child 0, key: string not null
> # child 1, value: string
> # ----
> # id: [[1,2]]
> # names:
>
> [[keys:["first_name","last_name"]values:["Tyler","Brady"],keys:["first_name","last_name"]values:["Walsh","Weaver"]]]
>
>
> Best, Alenka
>
> On Wed, May 3, 2023 at 9:13 AM Jerald Alex <vminf...@gmail.com> wrote:
>
> > Any inputs on this please?
> >
> > On Tue, May 2, 2023 at 10:03 AM Jerald Alex <vminf...@gmail.com> wrote:
> >
> > > Hi Experts,
> > >
> > > Can anyone please highlight if it is possible to cast struct to map
> type?
> > >
> > > I tried the following but it seems to be producing an error as below.
> > >
> > > pyarrow.lib.ArrowNotImplementedError: Unsupported cast from
> > > struct<first_name: string, last_name: string> to map using function
> > cast_map
> > >
> > > Note: Snippet is just an example to show the problem.
> > >
> > > Code Snippet:
> > >
> > > table_schema = pa.schema([pa.field("id", pa.int32()), pa.field("names",
> > > pa.map_(pa.string(), pa.string()))])
> > >
> > > table_data = [{"id": 1,"names": {"first_name": "Tyler", "last_name":
> > > "Brady"}},
> > > {"id": 2,"names": {"first_name": "Walsh", "last_name": "Weaver"}}]
> > >
> > > tbl = pa.Table.from_pylist(table_data)
> > > print(tbl)
> > > print(tbl.cast(table_schema))
> > > print(tbl)
> > >
> > > Error :
> > >
> > > id: int64
> > > names: struct<first_name: string, last_name: string>
> > > child 0, first_name: string
> > > child 1, last_name: string
> > > ----
> > > id: [[1,2]]
> > > names: [
> > > -- is_valid: all not null
> > > -- child 0 type: string
> > > ["Tyler","Walsh"]
> > > -- child 1 type: string
> > > ["Brady","Weaver"]]
> > > Traceback (most recent call last):
> > > File "/Users/
> > >
> >
> infant.a...@cognitedata.com/Documents/Github/HubOcean/demo/pyarrow_types.py
> > ",
> > > line 220, in <module>
> > > print(tbl.cast(table_schema))
> > > File "pyarrow/table.pxi", line 3489, in pyarrow.lib.Table.cast
> > > File "pyarrow/table.pxi", line 523, in pyarrow.lib.ChunkedArray.cast
> > > File "/Users/
> > >
> >
> infant.a...@cognitedata.com/Library/Caches/pypoetry/virtualenvs/demo-LzMA3Hsd-py3.10/lib/python3.10/site-packages/pyarrow/compute.py
> > ",
> > > line 391, in cast
> > > return call_function("cast", [arr], options)
> > > File "pyarrow/_compute.pyx", line 560, in
> > pyarrow._compute.call_function
> > > File "pyarrow/_compute.pyx", line 355, in
> > pyarrow._compute.Function.call
> > > File "pyarrow/error.pxi", line 144, in
> > > pyarrow.lib.pyarrow_internal_check_status
> > > File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
> > > pyarrow.lib.ArrowNotImplementedError: Unsupported cast from
> > > struct<first_name: string, last_name: string> to map using function
> > cast_map
> > >
> > > Regards,
> > > Alex Vincent
> > >
> >
>
