Hi Xingbo, Wei, Dian,
Many thanks for this plus for the high quality and prompt support overall.
Let’s close this thread here. Looking forward trying your approach.
Community, feel free to reach out with additional remarks and experiences
on structured streaming on complex/sparse objects.
Best re
Thanks Till,
Able to deduce topics by extending the KafkaSerializarion Schema class.
Prasanna.
On Wed, Dec 2, 2020 at 11:18 PM Till Rohrmann wrote:
> Hi Prasanna,
>
> I believe that what Aljoscha suggestd in the linked discussion is still
> the best way to go forward. Given your description of
What you are asking for is an HTTP(s) source. This currently does not
exist for Flink, so you would have to implement it yourself.
Additionally you would have to figure out the host on which the source
runs on yourself.
It may be easier to setup a separate HTTP(s) server that accepts data,
whi
Hello,
I am new to flink. Have few queries regarding the HA mode with zookeeper
and checkpointing. When flink is configured in HA mode with zookeeper,
where do the job checkpoints are stored? zookeeper only used for recovering
the jobmanager or even checkpoints? what is the significance of
*"**hig
Hi Narasimha,
Nothing comes to my mind immediately why it should not work. We are using the
StandaloneApplicationClusterEntryPoint to start the cluster. Can you provide
some more information about which Flink image on vvp are you trying to use and
maybe show the error message?
Best,
Fabian
Great to hear :-)
Cheers,
Till
On Thu, Dec 3, 2020 at 10:15 AM Prasanna kumar <
prasannakumarram...@gmail.com> wrote:
> Thanks Till,
>
> Able to deduce topics by extending the KafkaSerializarion Schema class.
>
> Prasanna.
>
> On Wed, Dec 2, 2020 at 11:18 PM Till Rohrmann
> wrote:
>
>> Hi Prasa
Only legacy connectors (`connector.type=kafka` instead of
`connector=kafka`) are supported in the YAML at the moment. You can use
regular DDL instead. There is a similar discussion in
https://issues.apache.org/jira/browse/FLINK-20260 these days.
Best,
Jark
On Thu, 3 Dec 2020 at 00:52, Till Rohrma
Hi,
Thanks for the hint. The infinite loop was the solution and my pipeline
works now.
Regards
Klemens
Am 24.11.20 um 16:59 schrieb Timo Walther:
For debugging you can also implement a simple non-parallel source
using
`org.apache.flink.streaming.api.functions.source.SourceFunction`. Yo
Hi Jark,
thanks for answer. I'm a bit puzzled, because in my yaml I'm using
"connector: filesystem" (not connector.type). I don't think I end up using
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connect.html#file-system-connector
- this connector as partitioning an
Hi,
Getting below error when trying to read a csv file, one of the field is
list tupe
Can someone help if fixing the issue
jobmanager_1 | Caused by: java.lang.IllegalArgumentException: The type
'java.util.List' is not supported for the CSV input format.
jobmanager_1 | at
org.apache.flink.ap
Hi all,
I have a flink job that uses FlinkKafkaConsumer to consume messages off a kafka
topic and a FlinkKafkaProducer to produce records on a Kafka topic. The
consumer works fine. However, the flink job eventually fails with the following
exception.
Caused by: org.apache.kafka.common.errors.T
I can't vouch for it personally, but perhaps the Apache Bahir Netty Source for
Flink could help you? It sounds like you want to use HTTPS, which this doesn't
support directly, but the source might be a helpful starting point to adding
the functionality you need.
On 12/3/20, 1:33 AM, "Chesnay S
Hello,
I'm running into an issue where my execution plan is creating the same
exact join operator multiple times simply because the subsequent operator
filters on a different boolean value. This is a massive duplication of
storage and work. The filtered operators which follow result in only a
smal
Hi,
When I'm looking at the Flink plan in the UI and at an operator, is there a
way to tell if an operator is in Retract mode vs Upsert mode? Ideally we
want as many of our operators in Upsert mode as possible since our data
sources are all uniquely keyed.
Thanks!
--
Rex Fenley | Software En
Our sinks are uniquely keyed as well. A couple of our joins are not until
an aggregate is performed however.
On Thu, Dec 3, 2020 at 12:46 PM Rex Fenley wrote:
> Hi,
>
> When I'm looking at the Flink plan in the UI and at an operator, is there
> a way to tell if an operator is in Retract mode vs
If a stateful operator has also a stateful operator in its input
sub-pipeline, then it may receive retract messages. Operator like group
agg, stream-stream join or rank are stateful.
We can not show if the operator are receiving retract messages in the UI.
But your request is reasonable.
Rex Fenl
Hi Kaushal,
Only the state handle pointer is stored in the ZooKeeper node. Since
ZooKeeper is built for
small data(KB level) storage. The real data will be persisted in the *"*
*high-availability.storageDir"*.
Note that it should be distributed storage(HDFS, S3, etc.).
The ZooKeeper HA service ha
Hi Rex,
Could you also attach one example for these sql / table ? And one possible
issue to confirm is that does the operators with the same names also have the
same inputs ?
Best,
Yun
--Original Mail --
Sender:Rex Fenley
Send Date:Fri Dec 4 02:55:41 2020
Yes, the same exact input operators go into both joins.
The chunk of code for the joins from the specific part of the plan I showed
is as follows. The orgUsersTable is later filtered into one table and
aggregated and another table and aggregated. The planner seems to duplicate
orgUsersTable into 2
cc Brad
On Thu, Dec 3, 2020 at 10:17 PM Rex Fenley wrote:
> Yes, the same exact input operators go into both joins.
>
> The chunk of code for the joins from the specific part of the plan I
> showed is as follows. The orgUsersTable is later filtered into one table
> and aggregated and another tab
Hi,
The CSV only supports the types listed in [1] and must use the types in
this list, thus for other types some kind of workaround is needed, like first
parsed as string and parsed again later in the program.
Best,
Yun
[1]
https://github.com/apache/flink/blob/e10e548feb2bedf54c3863bb
21 matches
Mail list logo
