Hi Alexander,
From my side I still think it should be reasonable to have a jar that contains
the code that are running in the clients and also shipped to the cluster. Then
this jar could also be included in the shipping jar list.
For the second issue, similarly I think you may first build the
I did try it but this option seems to be for a third party jar. In my case
I would need to specify/ship a jar that contains the code where job is
being constracted. I'm not clear of
1. how to point to the containg jar
2. how to test such a submission from my project running in Eclipse
Alex
On Wed,
Hi Alexander,
The signature of the createRemoteEnvironment is
public static StreamExecutionEnvironment createRemoteEnvironment(
String host, int port, String... jarFiles);
Which could also ship the jars to execute to remote cluster. Could you have a
try to also pass the jar files to the r
Hi,
I am trying to build a functionality to dynamically configure a flink job
(Java) in my code based on some additional metadata and submit it to a
flink running in a session cluster.
Flink version is 1.11.2
The problem I have is how to provide a packed job to the cluster. When I am
trying the
+1 to remove the Bucketing Sink.
Thanks for the effort on ORC and `HadoopPathBasedBulkFormatBuilder`, I
think it's safe to get rid of the old Bucketing API with them.
Best,
Jingsong
On Thu, Oct 29, 2020 at 3:06 AM Kostas Kloudas wrote:
> Thanks for the discussion!
>
> From this thread I do not
issue created: https://issues.apache.org/jira/browse/FLINK-19861
On Wed, 28 Oct 2020 at 11:00, Danny Chan wrote:
> Our behavior also conflicts with the SQL standard, we should also mention
> this in the document.
>
> Till Rohrmann 于2020年10月27日周二 下午10:37写道：
>
>> Thanks for the clarification. T
Hi Ruben,
thanks for reaching out to us. Flink's native Kubernetes Application mode
[1] might be what you're looking for.
Best,
Matthias
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#flink-kubernetes-application
On Wed, Oct 28, 2020 at 11:
Hi,
from the messages you've sent on the user@ mailing list in the recent
weeks, I see that you are in the process of learning Flink. The Flink
community won't be able to provide you with full, runnable examples for
every method Flink provides.
Rather, we have a few running examples, and conceptual
Thanks for the discussion!
>From this thread I do not see any objection with moving forward with
removing the sink.
Given this I will open a voting thread tomorrow.
Cheers,
Kostas
On Wed, Oct 28, 2020 at 6:50 PM Stephan Ewen wrote:
>
> +1 to remove the Bucketing Sink.
>
> It has been very commo
Hi Yang,
Thanks again for all the help!
We are still seeing this with 1.11.2 and ZK.
Looks like others are seeing this as well and they found a solution
https://translate.google.com/translate?hl=en&sl=zh-CN&u=https://cloud.tencent.com/developer/article/1731416&prev=search
Should this solution b
+1 to remove the Bucketing Sink.
It has been very common in the past to remove code that was deprecated for
multiple releases in favor of reducing baggage.
Also in cases that had no perfect drop-in replacement, but needed users to
forward fit the code.
I am not sure I understand why this case is s
Hi Alexander,
Thanks for sharing,
I see a lot of exceptions in the logs, particularly
*Caused by: java.net.BindException: Could not start actor system on any
port in port range 6123
which means that there's probably more than one instance running and is
likely the root cause.
So it makes sense t
The values printed by the OOM killer seem indeed strange. But from the line
above the memory usage seems fine: rss=2440960.
Running the given command I see only one forked process.
Probably, this is an issue of OOM killer running in VM on Wwindows host.
Can you try with OOM killer disabled?
Regard
Then we can't remove it, because there is no way for us to ascertain
whether anyone is still using it.
Sure, the user ML is the best we got, but you can't argue that we don't
want any users to be affected and then use an imperfect mean to find users.
If you are fine with relying on the user ML,
No, I do not think that "we are fine with removing it at the cost of
friction for some users".
I believe that this can be another discussion that we should have as
soon as we establish that someone is actually using it. The point I am
trying to make is that if no user is using it, we should remove
I'm runnin the code from Eclipse, the jar exists and it contains the
classes Flink is not finding..maybe I can try to use IntelliJ in the
afternoon
On Wed, Oct 28, 2020 at 12:13 PM Chesnay Schepler
wrote:
> @Kostas: Ah, I missed that.
>
> @Flavio: the only alternative I can think your jar does n
@Kostas: Ah, I missed that.
@Flavio: the only alternative I can think your jar does not contain the
classes, or does not exist at all on the machine your application is run on.
On 10/28/2020 12:08 PM, Kostas Kloudas wrote:
Hi all,
I will have a look in the whole stack trace in a bit.
@Chesn
The alternative could also be to use a different argument than "no one
uses it", e.g., we are fine with removing it at the cost of friction for
some users because there are better alternatives.
On 10/28/2020 10:46 AM, Kostas Kloudas wrote:
I think that the mailing lists is the best we can do a
Hi all,
I will have a look in the whole stack trace in a bit.
@Chesnay Schepler I think that we are setting the correct classloader
during jobgraph creation [1]. Is that what you mean?
Cheers,
Kostas
[1]
https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/c
10x
On 2020/10/27 10:42:40, Robert Metzger wrote:
> Hi,
>
> it seems that this is what you have to do for now. However, I see that it
> would be nice if Flink would allow reading from multiple configuration
> files, so that you can have a "common configuration" and a "per cluster"
> configurati
Hi,
First time user , I'm just evaluating Flink at the moment, and I was
reading
https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html#deploy-job-cluster
and I don't fully understand if a Job Cluster will autoterminate after
the job is completed (for at batch job)
Always the same problem.
Caused by: java.lang.ClassNotFoundException: it.test.XXX
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHa
Hi Xintong,
See here:
# Top memory users
ps auxwww --sort -rss | head -10
USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND
yarn 20339 35.8 97.0 128600192 126672256 ? Sl Oct15 5975:47
/etc/alternatives/jre/bin/java -Xmx54760833024 -Xms54760833024 -XX:Max
root 524
I think that the mailing lists is the best we can do and I would say
that they seem to be working pretty well (e.g. the recent Mesos
discussion).
Of course they are not perfect but the alternative would be to never
remove anything user facing until the next major release, which I find
pretty strict
Hi Ori,
The error message suggests that there's not enough physical memory on the
machine to satisfy the allocation. This does not necessarily mean a managed
memory leak. Managed memory leak is only one of the possibilities. There
are other potential reasons, e.g., another process/container on the
hmm..it appears as if PackagedProgramUtils#createJobGraph does some
things outside the usercode classlodaer (getPipelineFromProgram()),
specifically the call to the main method.
@klou This seems like wrong behavior?
@Flavio What you could try in the meantime is wrap the call to
createJobGraph
Any help here? How can I understand why the classes inside the jar are not
found when creating the PackagedProgram?
On Tue, Oct 27, 2020 at 11:04 AM Flavio Pompermaier
wrote:
> In the logs I see that the jar is the classpath (I'm trying to debug the
> program from the IDE)..isn'it?
>
> Classpat
If the conclusion is that we shouldn't remove it if _anyone_ is using
it, then we cannot remove it because the user ML obviously does not
reach all users.
On 10/28/2020 9:28 AM, Kostas Kloudas wrote:
Hi all,
I am bringing the up again to see if there are any users actively
using the Bucketing
Hi Oleksandr,
yes you are right. The biggest problem is at the moment the lack of test
coverage and thereby confidence to make changes. We have some e2e tests
which you can find here [1]. These tests are, however, quite coarse grained
and are missing a lot of cases. One idea would be to add a Meso
Hi all,
I am bringing the up again to see if there are any users actively
using the BucketingSink.
So far, if I am not mistaken (and really sorry if I forgot anything),
it is only a discussion between devs about the potential problems of
removing it. I totally understand Chesnay's concern about no
In SQL, you can use the over window to deduplicate the messages by the id
[1], but i'm not sure if there are same semantic operators in DataStream.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication
s_penakalap...@yahoo.com 于2020年10月28日周三
下午12:34写
31 matches
Mail list logo
