Sachin Goel created FLINK-2982:
--
Summary: CLI info option throws ClassCastException for streaming
jobs
Key: FLINK-2982
URL: https://issues.apache.org/jira/browse/FLINK-2982
Project: Flink
Issue
Hi Chiwan,
Thanks for the hint, it works now! I created an issue and will update
docs/README.md.
Best,
Martin
On 06.11.2015 04:33, Chiwan Park wrote:
Hi Martin,
I had the same problem. From my investigation, current custom Jekyll plugin for
Flink is not compatible with Jekyll 3.x. If you r
Martin Junghanns created FLINK-2981:
---
Summary: Update README for building docs
Key: FLINK-2981
URL: https://issues.apache.org/jira/browse/FLINK-2981
Project: Flink
Issue Type: Improvement
Chengxiang Li created FLINK-2980:
Summary: Add CUBE/ROLLUP/GROUPING SETS operator in Table API.
Key: FLINK-2980
URL: https://issues.apache.org/jira/browse/FLINK-2980
Project: Flink
Issue Type
Hi Martin,
I had the same problem. From my investigation, current custom Jekyll plugin for
Flink is not compatible with Jekyll 3.x. If you remove Jekyll 3.x and install
Jekyll 2.x, you can build docs. I’m using Jekyll 2.5.3 to build the docs.
Regards,
Chiwan Park
On November 6, 2015 at 4:58:3
Hi Ali,
1. You can connect two streams and then use the co-map operator to consume
data from both streams. I'm not sure how much data arrives from one or the
other stream, but maybe you can store (update) the data in memory.
Read more here
https://ci.apache.org/projects/flink/flink-docs-master/api
Hi Robert,
I tried the approach you suggested and it works nicely. Thanks!
I have a few more questions if you don’t mind:
1. Is there a way to retrieve in one stream data that's stored in another
stream? I have a location stream that I can use to store the latest
subscriber location. I have anot
Hi, not sure if that's an issue or just a misconfiguration (not familiar
with Ruby).
I followed the docs/README.md and ran into:
s1ck@s1ck-T450s:~/Devel/Java/flink$ docs/build_docs.sh
Configuration file: /home/s1ck/Devel/Java/flink/docs/_config.yml
/home/s1ck/Devel/Java/flink/docs/_plugins/remo
Fixed the docs. Should be updated on the website once when the build bot
rebuilds them over night.
On Thu, Nov 5, 2015 at 6:03 PM, Stephan Ewen wrote:
> I am writing an update in the state docs now...
>
> On Thu, Nov 5, 2015 at 5:58 PM, Maximilian Michels wrote:
>
>> Hi Martin,
>>
>> You're rig
I am writing an update in the state docs now...
On Thu, Nov 5, 2015 at 5:58 PM, Maximilian Michels wrote:
> Hi Martin,
>
> You're right. The documentation needs to be updated. I have already filed a
> JIRA for that: https://issues.apache.org/jira/browse/FLINK-2938
>
> It's actually called getKey
Hi Martin,
You're right. The documentation needs to be updated. I have already filed a
JIRA for that: https://issues.apache.org/jira/browse/FLINK-2938
It's actually called getKeyValueState(...) now.
Cheers,
Max
On Thu, Nov 5, 2015 at 4:41 PM, Martin Neumann wrote:
> Hej,
>
> I'm working with
@Fabian
Is there any advantage in putting the reducer-combiner before updating the
workset vs. after (i.e. right before the join with the solution set)?
If it helps, here are the plans of these 2 alternatives:
https://drive.google.com/file/d/0BzQJrI2eGlyYcFV2RFo5dUFNXzg/view?usp=sharing
https://
Hej,
I'm working with some state full streaming operators at the moment and I
noticed that the Documentation is out of date.
The documentation says:
@Override
public void open(Configuration config) {
counter = getRuntimeContext().getOperatorState(“counter”, 0L, false);
}
getOp
You can make this one program by using a broadcast data set.
This will stream the file into the average function and the filter
together, but it will have to buffer (possibly spill) the data as well in
one of the two streams. The collect() approach is probably easier and okay,
if the file reading
Sounds good. I like the idea of presenting it as a spectrum:
Pregel -> Scatter/Gather (Spargel) -> GAS/GSA/SGA
On Tue, Nov 3, 2015 at 5:55 PM, Martin Neumann wrote:
> The problem with having many different graph model in gelly is that it
> might get quite confusing for a user.
> Maybe this can
Hi Ali,
great, the start-local-streaming.sh script sounds right.
I can explain why your first approach didn't work:
You were trying to send the CSV files from the Flink client to the cluster
using our RPC system (Akka). When you submit a job to Flink, we serialize
all the objects the user create
I did not load the CSV file using the approach you suggested. I was
loading it outside the operators (at the beginning of the main method of
my class), since the file will be needed by multiple operators for sure.
When the file was small, I saw the job registered and started, but when I
used a big
Okay.
you should be able to implement it as you described initially. I would do
the transformation in a map() operator of Flink. The RichMapFunction
provides you with an open() method which is called before the first record
arrives.
In the open() method, I would read the csv file(s) from HDFS or a
Till Rohrmann created FLINK-2979:
Summary: RollingSink does not work with Hadoop 2.7.1
Key: FLINK-2979
URL: https://issues.apache.org/jira/browse/FLINK-2979
Project: Flink
Issue Type: Bug
Hi Andre,
I'm happy you were able to solve your problem :)
Improvements to the documentation are always welcome!
To me ST != WT is straight-forward from the javadocs, but I guess it
wouldn't hurt to stress it in the docs.
Do you think you could simplify your implementation a bit to make for a
nic
Hi Robert,
The CSV file (or files as there will definitely be more than one) can be
large (let¹s say 1 GB). Memory is not an issue though. Each node has at
least 64 GB RAM mounted. The CSV files should easily fit in the memory of
each node.
Regards,
Ali
On 2015-11-05, 6:30 AM, "Robert Metzger"
Sachin Goel created FLINK-2978:
--
Summary: Integrate web submission interface into the new dashboard
Key: FLINK-2978
URL: https://issues.apache.org/jira/browse/FLINK-2978
Project: Flink
Issue Typ
Niels Basjes created FLINK-2977:
---
Summary: Cannot access HBase in a Kerberos secured Yarn cluster
Key: FLINK-2977
URL: https://issues.apache.org/jira/browse/FLINK-2977
Project: Flink
Issue Type
I plan to implement all features of the existing web client. :) I think
multiple programs will be best displayed as a list in the Entry Class
column.
-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685
On Thu, Nov 5, 2015 at 5:36 PM, Matthias J. Sax wrote:
> Looks good. How would you d
Looks good. How would you display a jar that contains multiple programs
and entry classes? Right now, WebClient can handle this:
https://ci.apache.org/projects/flink/flink-docs-master/apis/web_client.html
The Manifest entries "Main-Class" and "program-class" can have a comma
separated list of entr
Ufuk Celebi created FLINK-2976:
--
Summary: Save and load checkpoints manually
Key: FLINK-2976
URL: https://issues.apache.org/jira/browse/FLINK-2976
Project: Flink
Issue Type: Improvement
Disabling the access for secure installation: Pretty easy. Writing it right
now.
@mxm: Thanks. I haven't implemented any handlers for actually running the
job. I hope to finish it by this weekend.
One quick question: The runtime monitor will be running the job itself,
i.e., the main method of pro
Jan Thomä created FLINK-2975:
Summary: Exclude logging dependencies from shaded Hadoop jar.
Key: FLINK-2975
URL: https://issues.apache.org/jira/browse/FLINK-2975
Project: Flink
Issue Type: Improv
@Robert: I'm in favor of having a submission page which can be deactivated.
My point was simply that we don't have proper security support right now. I
agree with Stephan that we'll need to support it!
@Stephan: Good suggestion to disable the submission page for the per-job
YARN cluster.
@Sachin:
Hi Stephan
I agree. However, as and when we secure the Dashboard, everything should
get secured in one go. Or is a secure dashboard not much desired? [By
secure, I mean some form of authentication for accessing.]
-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685
On Thu, Nov 5, 2015 at
Ah. Sharing the links then:
https://drive.google.com/file/d/0B0Xlv1qTgpjDT0ZCd0pSU3VYMUU/view?usp=sharing
https://drive.google.com/file/d/0B0Xlv1qTgpjDeG51azRRc0FJYjQ/view?usp=sharing
https://drive.google.com/file/d/0B0Xlv1qTgpjDSHZXQ0h5S0FBS3c/view?usp=sharing
-- Sachin Goel
Computer Science, IIT
Security should definitely be in our minds big time.
We think we need to add secure cookies to akka very soon (makes akka
connections encrypted) and use encryption on the netty as well.
To that end, we should not introduce more loopholes now.
We can add an upload/execute section to the dashboard,
Hey Sachin, I think you can not upload images to apache mailing lists.
On Thu, Nov 5, 2015 at 11:46 AM, Sachin Goel
wrote:
> Okay. I think that does it. I've already designed the basic interface,
> although the major part for uploading jars and actually running the job are
> left. :')
> Here are
Okay. I think that does it. I've already designed the basic interface,
although the major part for uploading jars and actually running the job are
left. :')
Here are some screenshots:
[image: Inline image 2][image: Inline image 1][image: Inline image 3]
-- Sachin Goel
Computer Science, IIT Delhi
m
Robert Metzger created FLINK-2974:
-
Summary: Add periodic offset commit to Kafka Consumer if
checkpointing is disabled
Key: FLINK-2974
URL: https://issues.apache.org/jira/browse/FLINK-2974
Project: Fl
I agree with Max, but still, its extremely convenient (and much easier than
crafting akka messages) to have a web interface which accepts jar files and
executes the main method ;)
On Thu, Nov 5, 2015 at 11:42 AM, Maximilian Michels wrote:
> +1 The web client needs to go.
>
> I don't really see
+1 The web client needs to go.
I don't really see how how a submission page on the job manager would make
security worse. At the moment, anyone who sends an Akka message to the job
manager, can execute arbitrary code..
On Thu, Nov 5, 2015 at 11:16 AM, Flavio Pompermaier
wrote:
> +1 to remove th
Hi Vasia!
Many thanks for your reply! Your hints finally enabled me implementing
the problems first-choice solution using delta iteration:
https://gist.github.com/p3et/12deb7d6321b48e9efab
Do you think this could be worth to be contributed as an example within
the Flink documentation? The exa
Hi Ali,
I'm excited to hear that EMC is looking into Apache Flink. I think the
solution to this problem depends on one question: What is the size of the
data in the CSV file compared to the memory you have available in the
cluster?
Would the mapping table from the file fit into the memory of all n
+1 to remove the old web client in favor of an improved dashboard (not so
urgent however)
On Thu, Nov 5, 2015 at 11:11 AM, Robert Metzger wrote:
> Hi,
> I personally like the idea of allowing users to submit a job from the web
> interface a lot. In particular for users new to the system, its a v
Hi,
I personally like the idea of allowing users to submit a job from the web
interface a lot. In particular for users new to the system, its a very
convenient feature.
We would also get rid of this ugly, old web client :)
Regarding security: We can always add a configuration parameter which
allow
Here's the current testing document:
https://docs.google.com/document/d/1PP9ar_Astl9TZ7rX2kkGXxxuM0gPr51QFvriNVVGK3s/edit
On Thu, Nov 5, 2015 at 10:28 AM, Maximilian Michels wrote:
> Please vote on releasing the following candidate as Apache Flink version
> 0.10.0:
>
> The commit to be voted on:
Please vote on releasing the following candidate as Apache Flink version
0.10.0:
The commit to be voted on:
b2e9bf1ed1e3b60d8012b217166145b936835ea7
Branch:
release-0.10.0-rc6 (see
https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
The release artifacts to be voted on can be found at:
The current authentication mechanism is via htaccess, in the webclient. The
dashboard however isn't secured at all.
-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685
On Thu, Nov 5, 2015 at 2:33 PM, Aljoscha Krettek
wrote:
> The reason why we didn’t want to have it in the JobManager i
The reason why we didn’t want to have it in the JobManager interface by default
is that it could be a security concern. I also see, however, that security in
general is not very strong right now.
> On 05 Nov 2015, at 09:58, Matthias J. Sax wrote:
>
> I like the idea in general! Not sure if int
I like the idea in general! Not sure if integrating it into JobMangaer
Webinterface is good or not... What would be the pros/cons for
standalone vs. integrated?
-Matthias
On 11/05/2015 05:23 AM, Sachin Goel wrote:
> Hi all
> Are there any plans to upgrade the webclient interface? It could certain
46 matches
Mail list logo
