Use Partitioner to forward messages to subtask by index

Is it possible to use a custom Partitioner to forward messages to a downstream substask by the subtask's index? I realize that it would not be safe to rely upon this across job restarts but this does not effect my particular application. I attempted to return a partition number identical to the sub

Re: Job Manager Configuration

ything special for the akka configurations. From: Joshua Griffith [mailto:jgriff...@campuslabs.com] Sent: Thursday, November 16, 2017 2:57 PM To: Chan, Regina [Tech] Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Job Manager Configuration I have an IO-dominated batch

Re: Job Manager Configuration

I have an IO-dominated batch job with 471 distinct tasks (3786 tasks with parallelism) running on 8 nodes with 12 GiB of memory and 4 CPUs each. I haven’t had any problems adding additional tasks except for 1) tasks timing out the first time the cluster is started (I suppose the JVM needs to war

Re: Delta iteration not spilling to disk

87647a9abe28be2888f4a55%7C0%7C0%7C636445440850807259&sdata=88tdKjTu8QbisJdVdQdFlJmegNpHPdUSVEOF8EBeNx0%3D&reserved=0> 2017-10-25 16:58 GMT+02:00 Joshua Griffith mailto:jgriff...@campuslabs.com>>: Hello Fabian, Thank you for your response. I tried setting the solution set to unm

Re: Case Class TypeInformation

va Tuple classes instead of a case class. You can import the Java Tuples with import org.apache.flink.api.java.tuple.{Tuple2 => JTuple2} And create a TupleTypeInfo for example with new TupleTypeInfo(new RowTypeInfo(Types.STRING, Types.LONG), Types.DOUBLE) Best, Fabian 2017-10-16 23:26 GMT+02

Re: Delta iteration not spilling to disk

he solution set to unManaged (DeltaIteration.setSolutionSetUnManaged(true)). Best, Fabian 2017-10-24 21:09 GMT+02:00 Joshua Griffith mailto:jgriff...@campuslabs.com>>: I’m currently using a delta iteration within a batch job and received the following error: java.lang.RuntimeExceptio

Delta iteration not spilling to disk

I’m currently using a delta iteration within a batch job and received the following error: java.lang.RuntimeException: Memory ran out. Compaction failed. numPartitions: 32 minPartition: 11 maxPartition: 24 number of overflow segments: 0 bucketSize: 125 Overall memory: 23232512 Partition memory:

Re: Case Class TypeInformation

Correction: I have the row’s RowTypeInfo at runtime before the job starts. I don’t have RowTypeInfo at compile time. On Oct 16, 2017, at 4:15 PM, Joshua Griffith mailto:jgriff...@campuslabs.com>> wrote: Hello, I have a case class that wraps a Flink Row and I’d like to use fields fro

Case Class TypeInformation

Hello, I have a case class that wraps a Flink Row and I’d like to use fields from that Row in a delta iteration join condition. I only have the row’s fields after the job starts. I can construct RowTypeInfo for the Row but I’m not sure how to add that to Flink’s generated type information for t

Re: Disable job graph in web UI

Upon further inspection, it appears that the web UI redraws each DOM element with every update. So I think removing the graph won’t fix the page performance issue because each task list item is being redrawn on every refresh. > On Sep 7, 2017, at 2:22 PM, Joshua Griffith wrote: > >

Disable job graph in web UI

Hello, I have an auto-generated job that creates too many tasks for web UI’s job graph to handle. The browser pinwheels while the page attempts to load. Is it possible to disable the job graph component in the web UI? For slightly smaller jobs, once the graph loads the rest of the UI is usable.

Re: Using Azure Blob Storage with Flink

GBiMV9KZW1D5EQ%3D&reserved=0> : The built jar file, named hadoop-azure.jar, also declares transitive dependencies on the additional artifacts it requires, notably the Azure Storage SDK for Java. On Tue, Aug 29, 2017 at 3:24 PM, Joshua Griffith mailto:jgriff...@campuslabs.com>> wro

Using Azure Blob Storage with Flink

I’m attempting to write to Azure Blob Storage using Flink's FileOutputFormat. I’ve included hadoop-azure within the jar I submit to Flink and configured the paths to be prefixed with wasb://{CONTAINERNAME}@{

Re: Nested Field Expressions with Rows

Indeed that worked. Thanks! > On Jul 10, 2017, at 11:57 AM, Fabian Hueske wrote: > > Hi, > > You have to add the implicit value in the main() method before you call > .map(rowFn) and not in the MapFunction. > > Best, Fabian > > > 2017-07-10 18:54 GMT+02:0

Re: Nested Field Expressions with Rows

ava/org/apache/flink/api/common/operators/Keys.java#L329-L331> > On Jul 10, 2017, at 11:54 AM, Joshua Griffith > wrote: > > Hello Fabian, > > Thank you for your response. I tried your recommendation but I’m getting the > same issue. Here’s the altered MakeRow MapFun

Re: Nested Field Expressions with Rows

w] = new RowTypeInfo( > Array[TypeInformation[_]](BasicTypeInfo.INT_TYPE_INFO, > BasicTypeInfo.INT_TYPE_INFO), > Array("id", "value") > ) > When you do this, you can also remove move the ResultTypeQueryable interface > from the MapFunction. > > Cheers, Fabian > > &g

Re: Flink Jobs disappers

Are your containers on separate nodes? Are you running in Kubernetes? Have you set hard resource limits? When I’ve run into this issue it’s been because the JobManager was restarted (I wasn’t running in HA mode). Your node could have been restarted or Docker could have OOM-killed the process if

Re: Nested Field Expressions with Rows

ying to run? > Maybe Timo or Fabian (cc'd) can help. > > > Nico > > On Friday, 7 July 2017 23:09:09 CEST Joshua Griffith wrote: >> Hello, >> >> When using nested field expressions like “Account.Id" with nested rows, I >> get the following error, “T

Nested Field Expressions with Rows

Hello, When using nested field expressions like “Account.Id" with nested rows, I get the following error, “This type (GenericType) cannot be used as key.” Is there a way to make nested field expressions work with nested rows? Thanks, Joshua