Re: dynamically infer json data not working as expected

Also, I checked your code but it will again give the same result even if I do sampling because the schema of the "data" attribute is not fixed. Any suggestions? On Thu, Aug 8, 2024 at 12:34 PM Perez wrote: > Hi Mich, > > Thanks a lot for your answer but there is one more scenario to it. > > Th

Re: dynamically infer json data not working as expected

Hi Mich, Thanks a lot for your answer but there is one more scenario to it. The schema of the data attribute inside the steps column is not fixed. For some records, I see it as a struct and for others, I see it as an Array of objects. So at last it treats it as string only since it gets confused

Re: dynamically infer json data not working as expected

I gave an answer in SO HTH Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD Imperial College London London, United Kingdom view my Linkedin profile

Re: dynamically infer json data not working as expected

https://stackoverflow.com/questions/78835509/dynamically-infer-schema-of-json-data-using-pyspark Any help would be appreciated. Thanks, On Mon, Aug 5, 2024 at 10:35 PM Perez wrote: > Hello everyone, > > I have described my problem on the SO blog : > >

dynamically infer json data not working as expected

Hello everyone, I have described my problem on the SO blog :