Hi Bruno,
I have added a log statement in a DoFn.
logger.info(System.getProperty('file.encoding'))
and that showed ANSI as the file encoding. There isn't anything in our code
that sets ANSI file encoding. I will check with Google Support.
On Fri, Jun 16, 2023 at 7:27 AM Bruno Volpato via user <user@beam.apache.org>
wrote:
> Hi Ramana,
>
> Curious where you got ANSI_X3.4-1968 from -- I don't think there's any
> trace of this encoding anywhere in Dataflow Workers (as far as I am aware
> and looked around).
> The default encoding for JVM is UTF-8, and Dataflow doesn't appear to set
> it anywhere. I was able to check using:
>
> $ docker run -it --entrypoint '/bin/bash'
> us-central1-artifactregistry.gcr.io/google.com/dataflow-containers/worker/v1beta3/beam_java11_sdk:2.48.0
>
> # jshell
>
> > System.getProperty("file.encoding");
> $1 ==> "UTF-8"
>
>
> If you can't figure out if your job is using ANSI, I'd suggest contacting
> Google support and providing relevant job IDs so this can be looked at
> further.
> Best,
> Bruno
>
>
>
> On Thu, Jun 15, 2023 at 5:03 AM Ramana Venkata <vram...@kisi.io> wrote:
>
>> Hi,
>>
>> I accidentally discovered that the default file encoding in my Dataflow
>> runners is ANSI_X3.4-1968. We expected it to be UTF-8, and as a result,
>> some of our data has been corrupted.
>>
>> I came across this Stack Overflow answer (link:
>> https://stackoverflow.com/a/362006), but to the best of my knowledge,
>> there is no way to pass flags to the Java command in Dataflow runners.
>>
>> I would appreciate your assistance in resolving this issue.
>>
>> Let me know if you have any further questions!
>>
>> --
>>
>> Venkata Ramana
>>
>> Senior Software Engineer
>>
>> Kisi Inc, 45 Main Street, Suite 608, Brooklyn, NY 11201
>> <https://maps.google.com/?q=45+Main+Street,+Suite+723,+%C2%A0Brooklyn,+NY+11201&entry=gmail&source=g>
>>
>> www.getkisi.com
>> <http://getkisi.com/?utm_source=email&utm_medium=email&utm_campaign=email>
>>
>>
>>
>>
>> ---
>> This email is confidential/privileged. If you're not the intended
>> recipient, please delete it and notify us immediately; please do not
>> copy/use/disclose it for any purpose, to anyone. Thank you!
>>
>
--
Venkata Ramana
Senior Software Engineer
Kisi Inc, 45 Main Street, Suite 608, Brooklyn, NY 11201
<https://maps.google.com/?q=45+Main+Street,+Suite+723,+%C2%A0Brooklyn,+NY+11201&entry=gmail&source=g>
www.getkisi.com
<http://getkisi.com/?utm_source=email&utm_medium=email&utm_campaign=email>
--
---
This email is confidential/privileged. If you're not the intended
recipient, please delete it and notify us immediately; please do not
copy/use/disclose it for any purpose, to anyone. Thank you!
