flink-conf.yaml
jobmanager.heap.size: 1024m
taskmanager.heap.size: 6144m
taskmanager.numberOfTaskSlots: 3
parallelism.default: 1
high-availability: zookeeper
high-availability.zookeeper.quorum: 10.1.1.15:2181,10.1.1.16:2181,10.1.1.17:2181
high-availability.zookeeper.path.root: /flink
high-availabil
Hi,alls:
How to use bin/pyflink.sh xxx.py in detached mode?
Thanks for answer.
Hi,
> Can you provide a pseudo-code example of how to implement this?
Processing time
If you use a TumblingProcessingTimeWindows.of(Time.seconds(1)), for each
record, you get the timestamp from System.currentTimeMillis(), say t, and
w_start = TimeWindow.getWindowStartWithOffset(t, 0, 1000), and w_
Xintong/Chesnay - Thanks for your response.
>From what I understand meter measures average throughput
>(https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#meter).
> I would like to have the absolute count in each unit of interval.
Also, I am assuming that the met
Hi,
I know that there is an improvement in Blink SQL that can deal with the top
k problem like SQL
showed below by maintaining an in-memory "heap" to store top k records.
That is not a problem
when user's score will only grow up.
> SELECT user_id, score
> FROM (
> SELECT *,
> ROW_NUMBER() O
Hi,
AFAIK Flink should remove temporary files automatically when they are not
needed anymore.
However, I'm not 100% sure that there are not corner cases when a TM
crashes.
In general it is a good idea to properly configure the directories that
Flink uses for spilling, logging, blob storage, etc.
Hi Konstantinos,
Typically the data that you are seeing is from records being spilled to disk
during groupBy/join operations, where the size of one (or multiple, for the
join case) data sets exceeds what will fit in memory.
And yes, these files can get big, e.g. as big as the sum of your input
The problem with the LATERAL JOIN (via
a LookupableTableSource+TableFunction because I need to call that function
using the userId a a parameter) is that I cannot know the window
start/end..to me it's not clear how to get that from
TimeWindow.getWindowStartWithOffset(timestamp, offset, windowSize)
Thanks.
I modify the docker-entrypoint.sh and it works.
Here's what I updated:
#exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground"$@"
$FLINK_HOME/bin/jobmanager.sh start "$@"
#exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "$@"
$FLINK_HOME/bin/t
Got it, thank you!
> On Jul 10, 2019, at 2:20 PM, Xintong Song wrote:
>
> Thanks for the kindly offer, Qi.
>
> I think this work should not take much time, so I can take care of it. It's
> just the community is currently under feature freeze for release 1.9, so we
> need to wait until the re
Hi all,
We are developing several batch processing applications using the DataSet API
of the Apache Flink.
For the time being, we are facing an issue with one of our production
environments since its disk usage increase enormously. After a quick
investigation, we concluded that the /tmp/flink-i
Hello,guys.
I use python print in flink map function to see whether i get the data,but
i didn't see anything in client or path log/***.out,just Program execution
finished.
Is there any way to solve this and see print?
Besides,I try to use raise Exception in Map() and flink still complete as
finishe
it works.
thanks very much.
Flavio Pompermaier 于2019年7月10日周三 下午5:04写道：
> Have you tried to follow the instruction to build Flink for Maven >= 3.4?
> That is:
>
> mvn clean install -DskipTestscd flink-dist
> mvn clean install
>
>
> On Wed, Jul 10, 2019 at 10:34 AM Marvin777
> wrote:
>
>> The ve
Have you tried to follow the instruction to build Flink for Maven >= 3.4?
That is:
mvn clean install -DskipTestscd flink-dist
mvn clean install
On Wed, Jul 10, 2019 at 10:34 AM Marvin777
wrote:
> The version of maven is 3.5.4.
>
> [image: image.png]
>
> Flavio Pompermaier 于2019年7月10日周三 下午4:28
The version of maven is 3.5.4.
[image: image.png]
Flavio Pompermaier 于2019年7月10日周三 下午4:28写道：
> For your specific problem try to look at
> https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html#dependency-shading
> .
> Which version of maven are you using?
>
> On Wed, Jul
For your specific problem try to look at
https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html#dependency-shading
.
Which version of maven are you using?
On Wed, Jul 10, 2019 at 10:25 AM Flavio Pompermaier
wrote:
> From some version of Flink many libraries (included jacks
>From some version of Flink many libraries (included jackson) are shaded
into org.apache.flink.shaded in order to avoid conflicts during class
loading.
See https://github.com/apache/flink-shaded
Best,
Flavio
On Wed, Jul 10, 2019 at 10:21 AM Marvin777
wrote:
> hi, all:
>
> The flink-dist package
This can't be implemented as there's no guarantee that getValue is only
called once.
Why do you want to reset the count? If you're interested in rates, why
aren't you using a meter?
On 10/07/2019 09:37, Xintong Song wrote:
Hi Singh,
Could your problem be solved by simply record the previous
hi, all:
The flink-dist package downloaded from the flink community does not have
the dependencies of Jackson and protobuf, like figure 1.
But when I pull the code from github and compile and package it using the
mvn clean package command, flink-dist has dependencies on Jackson and
protobuf, like
Hi everyone,
Can you please explain if this is expected behaviour or not?
My environment:
I have a session cluster of flink 1.8.1 and a job written in scala. I'm
submitting it to it with flink CLI, specifying the address of jobmanager.
To submit a job I'm using CLI from 2 flink docker
images flin
Hi Singh,
Could your problem be solved by simply record the previous value and
subtract it from the new value?
Thank you~
Xintong Song
On Wed, Jul 10, 2019 at 3:33 PM M Singh wrote:
> Hi:
>
> I am working on a Flink application and need to collect the number of
> events of certain type/seco
Thanks for your answer Xingcan.
Just to clarify - if the characteristic is set to IngestionTime or
ProcessingTime, the event time triggers will be ignored and not fire.
Mans
On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui
wrote:
Yes, Mans. You can use both processing-time and ev
Hi:
I am working on a Flink application and need to collect the number of events of
certain type/second (per unit of time) as a metric. I was hoping to use
Flink's metric framework with a gauge but did not find any implementation that
will reset the count after it is reported.
The example:
htt
Hi Ever,
May you share more information about your environment?
If you could not find any log in the flink dashboard, you need to check the
log4j of job manager and task manager. Make sure you could find the log
files(jobmanager.log/taskmanager.log) under /opt/flink/log in the docker
container.
I
24 matches
Mail list logo
