Actually, you can do partition level ingest using ForEachWriter. You just have to add each row to a list in the write method, and write to the data store in the close method
I know it’s awkward. I don’t know why Spark doesn’t provide a ForEachPartitionWriter From: Yogesh Mahajan <[email protected]> Date: Thursday, June 21, 2018 at 3:45 PM To: subramgr <[email protected]> Cc: user <[email protected]> Subject: Re: Spark 2.3.0 and Custom Sink Since ForeachWriter works at a record level so you cannot do bulk ingest into KairosDB, which supports bulk inserts. This will be slow. Instead, you can have your own Sink implementation which is a batch (DataFrame) level. Thanks, http://www.snappydata.io/blog<https://urldefense.proofpoint.com/v2/url?u=http-3A__snappydata.io&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=F2RNeGILvLdBxn7RJ4effes_QFIiEsoVM2rPi9qX1DKow5HQSjq0_WhIW109SXQ4&m=SEFEzpQgcNkmvod1F3vAjoV9R0A2scwsFxbuUwEMyJA&s=xCEVFqZ3k8jqb6oLLjPXBMcplgGBPoy5pvjGvUoA0JY&e=> On Thu, Jun 21, 2018 at 10:54 AM, subramgr <[email protected]<mailto:[email protected]>> wrote: Hi Spark Mailing list, We are looking for pushing the output of the structured streaming query output to KairosDB. (time series database) What would be the recommended way of doing this? Do we implement the *Sink* trait or do we use the *ForEachWriter* At each trigger point if I do a *dataset.collect()* the size of the data is not huge it should be in lower 10MBs Any suggestions? Thanks Girish -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=F2RNeGILvLdBxn7RJ4effes_QFIiEsoVM2rPi9qX1DKow5HQSjq0_WhIW109SXQ4&m=SEFEzpQgcNkmvod1F3vAjoV9R0A2scwsFxbuUwEMyJA&s=gudQl3hqMDKfp1XidS2tcmWwUWox0p3tZ9ewt1qXJYk&e=> --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]<mailto:[email protected]> ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
