I have .txt files with JSON inside it. It is generated by some API calls by
the Client.
On Wed, Apr 27, 2022 at 12:39 AM Bjørn Jørgensen <bjornjorgen...@gmail.com>
wrote:
> What is that you have? Is it txt files or json files?
> Or do you have txt files with JSON inside?
>
>
>
> tir. 26. apr. 2022 kl. 20:41 skrev Sid <flinkbyhe...@gmail.com>:
>
>> Thanks for your time, everyone :)
>>
>> Much appreciated.
>>
>> I solved it using jq utility since I was dealing with JSON. I have solved
>> it using below script:
>>
>> find . -name '*.txt' -exec cat '{}' + | jq -s '.' > output.txt
>>
>>
>> Thanks,
>>
>> Sid
>>
>>
>> On Tue, Apr 26, 2022 at 9:37 PM Bjørn Jørgensen <bjornjorgen...@gmail.com>
>> wrote:
>>
>>> and the bash script seems to read txt files not json
>>>
>>> for f in Agent/*.txt; do cat ${f} >> merged.json;done;
>>>
>>>
>>>
>>> tir. 26. apr. 2022 kl. 18:03 skrev Gourav Sengupta <
>>> gourav.sengu...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> what is the version of spark are you using? And where is the data
>>>> stored.
>>>>
>>>> I am not quite sure that just using a bash script will help because
>>>> concatenating all the files into a single file creates a valid JSON.
>>>>
>>>> Regards,
>>>> Gourav
>>>>
>>>> On Tue, Apr 26, 2022 at 3:44 PM Sid <flinkbyhe...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Can somebody help me with the below problem?
>>>>>
>>>>>
>>>>> https://stackoverflow.com/questions/72015557/dealing-with-large-number-of-small-json-files-using-pyspark
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Sid
>>>>>
>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
