I wrote a streaming job with scala, using only immutable case class. Is it
safe to enable object reuse? Will it get benefits from enabling object
reuse?
I reached to documents but they cover neither streaming cases nor immutable
data structures.
, for the first program, you have
> not registered the event time timer(though context.timerService.
> registerEventTimeTimer) and also there is also no onTimer logic defined
> to process it.
>
> 在 2019年10月28日，下午4:01，杨力 写道：
>
> It seems to be the case. But when I use timeWindow or
ermark is still not generated yet and I think that's the reason why
> it's Long.MinValue.
>
> Regards,
> Dian
>
> 在 2019年10月28日，上午11:59，杨力 写道：
>
> I'm going to sort elements in a PriorityQueue and set up timers at
> (currentWatermark + 1), following the ins
I'm going to sort elements in a PriorityQueue and set up timers at
(currentWatermark + 1), following the instructions in
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/process_function.html#timer-coalescing
.
However, it seems that context.timerService().currentWa
I'd like to provide a custom serializer for a POJO class. But that class
cannot be modified so it's not possible to add a @TypeInfo annotation to
it. Are there any other ways to register one?
The web ui keeps job ids in web browsers.
徐涛 于2018年11月18日周日 上午8:12写道：
>
> Hi Chesnay,
> Yes, it is. Later I notice before the later started cluster I have use
> ./bin/start-cluster.sh to start a local cluster, and after I
> ./bin/stop-cluster.sh to stop the cluster, the problem is gone.
> But ho
They are provided in taskmanagers.
Sayat Satybaldiyev 于 2018年9月24日周一 下午6:38写道：
> Dear all,
>
> While configuring JMX with Flink, I don't see some bean metrics that
> belongs to the job, in particular, the number in/out records per operator.
> I've checked REST API and those numbers provided ther
A java.util.Iterable is expected to provide iterators again and again.
On Fri, Sep 14, 2018 at 10:53 PM Alejandro Alcalde
wrote:
> Hello all,
>
> I am trying to replicate the code in the Docs (
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/batch/dataset_transformations.html#c
A checkpoint cannot be used for rescaling. You should take a savepoint,
stop the application, and resume from the savepoint with a new parallelism.
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html
On Fri, Sep 14, 2018 at 1:50 AM 廖嘉逸 wrote:
> Hi, all
>
>
to config at least INFO level for logger in flink, and
> currently the messages are so limited for debugging the problem.
>
> Best,
> Zhijiang
>
> ------
> 发件人：杨力
> 发送时间：2018年9月7日(星期五) 17:21
> 收件人：Zhijiang(wangz
Hi all,
I am encountering a weird problem when running flink 1.6 in yarn per-job
clusters.
The job fails in about half an hour after it starts. Related logs is
attached as an imange.
This piece of log comes from one of the taskmanagers. There are not any
other related log lines.
No ERROR-level log
equn
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/yarn_setup.html#run-a-flink-job-on-yarn
>
> On Mon, Aug 13, 2018 at 10:25 PM, 杨力 wrote:
>
>> I used to runFlink SQL in streaming mode with more than 70 sqls in
>> version 1.4. With
I used to runFlink SQL in streaming mode with more than 70 sqls in version
1.4. With so many sqls loaded, akka.framesize has to be set to 200 MB to
submit the job.
When I am trying to run the job with flink 1.6.0, the HTTP-based job
submission works perfectly but an OutOfMemoryError is thrown when
Hi,
I am running a streaming job without checkpointing enabled. A failute rate
restart strategy have been set with
StreamExecutionEvironment.setRestartStrategy.
When a task manager is lost because of memory problems, the job manager try
to restart the job without launching a new task manager, and
I am running flink SQL in streaming mode and implemented a UDAGG, which is
used in keyed HOP windows. But I found that the throughput decreases
dramatically when the function is used. Does UDAGG run in parallell? Or
does it run only in one thread?
Regards,
Bill
Maybe you can get them from yarn with rest API.
Tao Xia 于 2018年4月13日周五 上午8:09写道：
> Any good way to get access container logs from a running Flink app in YARN
> cluster in EMR?
> You can view the logs through YARN UI. But cannot programmatically access
> it and send to other services.
> The log a
You should add a column
TUMBLE_ROWTIME(t, INTERVAL '60' SECOND) AS t
to the select part of your subquery.
韩宁宁 <453673...@qq.com> 于 2018年4月3日周二 下午3:34写道：
> Thank you for your reply.
>
> I think the table registration no problem。I guess it's a question of
> subquery。
> It's no problem to execut
You can use a split operator, generating 2 streams.
Darshan Singh 于 2018年3月30日周五 上午2:53写道：
> Hi
>
> I have a dataset which has almost 99% of correct data. As of now if say
> some data is bad I just ignore it and log it and return only correct data.
> I do this inside a map function.
>
> The part
n the
> worker nodes. The size of the code depends on the number of fields in the
> schema.
> Operators and code are not shared across queries.
>
> Best, Fabian
>
> 2018-03-09 23:36 GMT+01:00 杨力 :
>
>> Thank you for your response. It occurs both in a standalone clus
tup? Logs could be
> helpful as well.
>
> Piotrek
>
> > On 9 Mar 2018, at 11:00, 杨力 wrote:
> >
> > I wrote a flink-sql app with following topography.
> >
> > KafkaJsonTableSource -> SQL -> toAppendStream -> Map ->
> JDBCAppend
To use a field in a table as timestamp, it must be declared as a rowtime
attribute for the table.
1) Call env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime).
2) Call withRowtimeAttribute on KafkaJsonTableSourceBuilder.
Reference:
1.
https://ci.apache.org/projects/flink/flink-docs-relea
I wrote a flink-sql app with following topography.
KafkaJsonTableSource -> SQL -> toAppendStream -> Map -> JDBCAppendTableSink
KafkaJsonTableSource -> SQL -> toAppendStream -> Map -> JDBCAppendTableSink
...
KafkaJsonTableSource -> SQL -> toAppendStream -> Map -> JDBCAppendTableSink
I have a dozen
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/sql.html#selecting-group-window-start-and-end-timestamps
>
>
>
> Am 3/5/18 um 11:26 AM schrieb 杨力:
>
>
>
I tried to use write a nested query with HOP window in a streaming
environment.
Table `source` consists of 3 column, a, b, and timestamp.
SELECT a FROM (SELECT a, COUNT(*) FROM source GROUP BY HOP(timestamp, ...,
...), a, b) GROUP BY HOP(timestamp, ..., ...), a HAVING ...
And flink throws an exc
24 matches
Mail list logo
