Re: low performance in running queries

2019-10-30 Thread Chris Miller
I haven't run any benchmarks with Flink or even used it enough to directly help with your question, however I suspect that the following article might be relevant: http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/ Given the computation you're performing is trivial, it's possible that

Add time attribute column to POJO stream

2019-12-03 Thread Chris Miller
I'm having trouble dealing with a DataStream of POJOs. In particular, when I perform SQL operations on it I can't figure out the syntax for referring to individual fields within the POJO. Below is an example that illustrates the problem and the various approaches I've tried. Can anyone please

Re: Accessing fields in a POJO stream

2019-12-04 Thread Chris Miller
lp, Chris -- Original Message -- From: "Jingsong Lee" To: "Chris Miller" Cc: "user" Sent: 04/12/2019 03:41:05 Subject: Re: Add time attribute column to POJO stream Hi Chris, First thing, FxRate is not POJO, a POJO should have a constructor without arguments. In

Re: Need help using AggregateFunction instead of FoldFunction

2019-12-05 Thread Chris Miller
I hit the same problem, as far as I can tell it should be fixed in Pulsar 2.4.2. The release of this has already passed voting so I hope it should be available in a day or two. https://github.com/apache/pulsar/pull/5068 -- Original Message -- From: "devinbost" To: user@flink.apache.o

Joining multiple temporal tables

2019-12-05 Thread Chris Miller
I want to decorate/enrich a stream by joining it with "lookups" to the most recent data available in other streams. For example, suppose I have a stream of product orders. For each order, I want to add price and FX rate information based on the order's product ID and order currency. Is it poss

Re: Joining multiple temporal tables

2019-12-06 Thread Chris Miller
Hi Fabian, Thanks for confirming the issue and suggesting a workaround - I'll give that a try. I've created a JIRA issue as you suggested, https://issues.apache.org/jira/browse/FLINK-15112 Many thanks, Chris -- Original Message -- From: "Fabian Hueske" To

Partitions vs. Subpartitions

2018-10-10 Thread Chris Miller
Hi, what's the difference between partitions and subpartitions? Thanks. CM

What are channels mapped to?

2018-10-11 Thread Chris Miller
Hi, in the OutputEmitter, the output channel can be selected in different manner. eg. OutputEmitter#hashPartitionDefault() What are the channels mapped to? Do they map to one IP Address or Port? Thanks. Chris

Where are the TaskManagers IPs and Ports stored?

2018-10-15 Thread Chris Miller
Hi, I'm looking for the property file where IP and Port of TaskManagers are stored in Flink. Does anyone know where it is located and when it's updated? (And for the case that there should not be such a file, where does the JobManager and TaskManagers take this information from?) Thanks.

Temporal tables not behaving as expected

2019-01-21 Thread Chris Miller
Hi all, I'm new to Flink so am probably missing something simple. I'm using Flink 1.7.1 and am trying to use temporal table functions but aren't getting the results I expect. With the example code below, I would expect 4 records to be output (one for each order), but instead I'm only seeing a

Re: Temporal tables not behaving as expected

2019-01-22 Thread Chris Miller
-- Original Message -- From: "Fabian Hueske" To: "Chris Miller" Cc: "user" Sent: 22/01/2019 11:23:23 Subject: Re: Temporal tables not behaving as expected Hi, The problem is that you are using processing time which is non-deterministic. Both inputs are consum

AssertionError: mismatched type $5 TIMESTAMP(3)

2019-01-25 Thread Chris Miller
I'm trying to group some data and then enrich it by joining with a temporal table function, however my test code (attached) is failing with the error shown below. Can someone please give me a clue as to what I'm doing wrong? Exception in thread "main" java.lang.AssertionError: mismatched type

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

2019-01-29 Thread Chris Miller
t that is required for the temporal table join. Any other ideas on how I can work around this problem? Many thanks, Chris -- Original Message -- From: "Timo Walther" To: "Chris Miller" ; "user" Sent: 29/01/2019 09:44:14 Subject: Re: AssertionError: mi

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

2019-02-05 Thread Chris Miller
Sorry to reply to my own post but I wasn't able to figure out a solution for this. Does anyone have any suggestions I could try? -- Original Message -- From: "Chris Miller" To: "Timo Walther" ; "user" Sent: 29/01/2019 10:06:47 Subject: Re: Assertion

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

2019-02-06 Thread Chris Miller
rations: https://github.com/dataArtisans/sql-training/wiki/SQL-on-streams I hope this helps. Feel free to ask further questions. Regards, Timo Am 05.02.19 um 11:30 schrieb Chris Miller: Exception in thread "main" java.lang.AssertionError: mismatched type