Re: Get attempt number in a closure

2014-10-20 Thread Yin Huai
Yes, it is for (2). I was confused because the doc of TaskContext.attemptId (release 1.1) is "the number of attempts to execute this task". Seems the per-task attempt id used to populate "attempt" field in the UI

Re: Get attempt number in a closure

2014-10-20 Thread Reynold Xin
Yes, as I understand it this is for (2). Imagine a use case in which I want to save some output. In order to make this atomic, the program uses part_[index]_[attempt].dat, and once it finishes writing, it renames this to part_[index].dat. Right now [attempt] is just the TID, which could show up l

Re: Get attempt number in a closure

2014-10-20 Thread Kay Ousterhout
Sorry to clarify, there are two issues here: (1) attemptId has different meanings in the codebase (2) we currently don't propagate the 0-based per-task attempt identifier to the executors. (1) should definitely be fixed. It sounds like Yin's original email was requesting that we add (2). On Mon

Re: Get attempt number in a closure

2014-10-20 Thread Kay Ousterhout
Are you guys sure this is a bug? In the task scheduler, we keep two identifiers for each task: the "index", which uniquely identifiers the computation+partition, and the "taskId" which is unique across all tasks for that Spark context (See https://github.com/apache/spark/blob/master/core/src/main/

Re: Get attempt number in a closure

2014-10-20 Thread Patrick Wendell
There is a deeper issue here which is AFAIK we don't even store a notion of attempt inside of Spark, we just use a new taskId with the same index. On Mon, Oct 20, 2014 at 12:38 PM, Yin Huai wrote: > Yeah, seems we need to pass the attempt id to executors through > TaskDescription. I have created

Re: Get attempt number in a closure

2014-10-20 Thread Yin Huai
Yeah, seems we need to pass the attempt id to executors through TaskDescription. I have created https://issues.apache.org/jira/browse/SPARK-4014. On Mon, Oct 20, 2014 at 1:57 PM, Reynold Xin wrote: > I also ran into this earlier. It is a bug. Do you want to file a jira? > > I think part of the p

Re: Get attempt number in a closure

2014-10-20 Thread Reynold Xin
I also ran into this earlier. It is a bug. Do you want to file a jira? I think part of the problem is that we don't actually have the attempt id on the executors. If we do, that's great. If not, we'd need to propagate that over. On Mon, Oct 20, 2014 at 7:17 AM, Yin Huai wrote: > Hello, > > Is t

Get attempt number in a closure

2014-10-20 Thread Yin Huai
Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this and this