Re: Write each group to its own file

2017-10-23 Thread Piotr Nowojski
ming (have to sort the data), 2. Can exceed the > maximum number of simultaneously opened file if number of categories is very > high. > > Piotrek > > > On 11 Oct 2017, at 17:47, rlazoti > <mailto:rodrigolaz...@gmail.com>> wrote: > > > > Hi, > &g

Re: Write each group to its own file

2017-10-23 Thread Rodrigo Lazoti
Oct 2017, at 17:47, rlazoti wrote: > > > > Hi, > > > > Is there a way to write each group to its own file using the Dataset api > > (Batch)? > > > > For example, lets use the following class: > > > > case class Product(name: String, category: St

Re: Write each group to its own file

2017-10-12 Thread Piotr Nowojski
categories is very high. Piotrek > On 11 Oct 2017, at 17:47, rlazoti wrote: > > Hi, > > Is there a way to write each group to its own file using the Dataset api > (Batch)? > > For example, lets use the following class: > > case class Product(name: String, category:

Write each group to its own file

2017-10-11 Thread rlazoti
Hi, Is there a way to write each group to its own file using the Dataset api (Batch)? For example, lets use the following class: case class Product(name: String, category: String) And the following Dataset: val products = env.fromElements(Product("i7", "cpu"), Product(&

Write each group to its own file

2017-10-11 Thread rlazoti
Hi, Is there a way to write each group to its own file using the Dataset api (Batch)? For example, lets use the following class: And the following Dataset: So in this example my output should be these 3 files: - cpu.csv i7, cpu R5, cpu - gpu.csv gtx1080, gpu vega64, gpu - ssd.csv