from:"rahul c"

[no subject]

2021-03-10 Thread rahul c

Unsubscribe

Requesting for a support

2020-01-09 Thread rahul c

Hi team, I am very new to the community I have experience in using spark framework. I would like to help the community any way I can. Regards Timmanna

Re: Does dataframe spark API write/create a single file instead of directory as a result of write operation.

2020-02-22 Thread rahul c

Hi Kshitij, There are option to suppress the metadata files from get created. Set the below properties and try. 1) To disable the transaction logs of spark "spark.sql.sources.commitProtocolClass = org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol". This will help to disa

Re: Does dataframe spark API write/create a single file instead of directory as a result of write operation.

2020-02-22 Thread rahul c

es the CSV file for further processing. > > On Sat, 22 Feb 2020 at 14:31, rahul c wrote: > >> Hi Kshitij, >> >> There are option to suppress the metadata files from get created. >> Set the below properties and try. >> >> 1) To disable the transaction logs

Clarification on the commit protocol

2020-02-26 Thread rahul c

Hi team, Just wanted to understand. Is DBIO commit protocol available in open source spark version ?

Applying schema dynamically in dataframe

2020-05-15 Thread rahul c

Hi dev, Currently I have a scenario where I am reading the data from Kafka using spark dataframe. Multiple data sources ingest the data into kafka same topic. My dataframe now has got json string from multiple sources. I have got key column from Kafka which can be used to identify which source d

Spark-submit --files option help

2020-08-01 Thread rahul c

Hi all, I am trying to pass some configuration files via spark-submit command in cluster mode. >From logs I can see the files are transferred to each executors. But how to build the absolute path of the file in the code? Can anyone plz guide on it with some references. Appreciate your help on th