Savepoint Location from Flink REST API

2020-03-19 Thread Aaron Langford
Hey Flink Community, I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be

Re: Savepoint Location from Flink REST API

2020-03-20 Thread Aaron Langford
nk-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html > > Meanwhile, I've created an issue to update the docs: > https://issues.apache.org/jira/browse/FLINK-16696 > > Regards, > Roman > > > On Fri, Mar 20, 202

Re: How to enforce ACLs on Flink JobManager/ApplicationMaster URL on Yarn

2020-03-30 Thread Aaron Langford
I'd be curious to see how others have done this, but our setup restricts network access to machines in the YARN cluster to a jump box. Access to Flink job manager is limited to whoever can ssh to that box, and that is controlled with an Ansible playbook. Additionally, we have a list of users specif

Re: Savepoint Location from Flink REST API

2020-04-01 Thread Aaron Langford
2. Failure > > { > "status": { > "id": "completed" > }, > "savepoint": { > "failure-cause": { > "class": "string", > "stack-trace": "string", > &

Per Operator State Monitoring

2019-11-22 Thread Aaron Langford
Hey Flink Community, I'm working on a Flink application where we are implementing operators that extend the RichFlatMap and RichCoFlatMap interfaces. As a result, we are working directly with Flink's state API (ValueState, ListState, MapState). Something that appears to be extremely valuable is ha

MapState with List Type for values

2019-12-17 Thread Aaron Langford
Hello Flink Community, I have a question about using MapState with lists as values. Here is a description of the use case: I have an operator over a keyed stream where for each record that comes, it needs to look into some state to determine if a value has arrived or not. If the value has not arr

Re: MapState with List Type for values

2019-12-18 Thread Aaron Langford
ss a ๐Ÿฆž and try to search the map to know we already queued two > ๐Ÿฟ. In this time we could produce {๐Ÿฟ, ๐Ÿฆž} and set previous map state as < > ๐Ÿฟ, 1> . If you could follow this logic, the previous > serialize/deserialize of Seq could be greatly reduced. > > Best > Yun Tang &

Rich Function Thread Safety

2019-12-18 Thread Aaron Langford
Hello Flink Community, I'm hoping to verify some understanding: If I have a function with managed state, I'm wondering if a checkpoint will ever be taken while a function is mutating state. I'll try to illustrate the situation I'm hoping to be safe from: Happy Path: t0 -> processFunction invoked

Replacing a server in Zookeeper Quorum

2020-01-21 Thread Aaron Langford
Hello Flink Community, I'm working on a HA setup of Flink 1.8.1 on AWS EMR and have some questions about how Flink interacts with Zookeeper when one of the servers in the quorum specified in flink-conf.yaml goes down and is replaced by a machine with a new IP address. Currently, I configure high-

Re: Location of flink-s3-fs-hadoop plugin (Flink 1.9.0 on EMR)

2020-01-21 Thread Aaron Langford
Senthil, One of the key steps in debugging this for me was enabling debug level logs on my cluster, and then looking at the logs in the resource manager. The failure you are after happens before the exceptions you have reported here. When your Flink application is starting, it will attempt to load

Re: Replacing a server in Zookeeper Quorum

2020-01-21 Thread Aaron Langford
e quick solution is > to use a static ip for EMR > master node[1]. > > > Best, > Yang > > > [1]. > https://aws.amazon.com/premiumsupport/knowledge-center/static-private-ip-master-node-emr/?nc1=h_ls > > Aaron Langford ไบŽ2020ๅนด1ๆœˆ22ๆ—ฅๅ‘จไธ‰ ไธŠๅˆ1:48ๅ†™้“๏ผš > >> Hello F

Re: Location of flink-s3-fs-hadoop plugin (Flink 1.9.0 on EMR)

2020-01-23 Thread Aaron Langford
"Classification": "flink-log4j-yarn-session", > "Properties": { > "log4j.rootLogger": "DEBUG,stdout" > } > }] > }] > }' On Thu, Jan 23, 2020 at 11:03 AM Sen

Re: Location of flink-s3-fs-hadoop plugin (Flink 1.9.0 on EMR)

2020-01-24 Thread Aaron Langford
s > - Cannot find hdfs-site configuration-file path in > Flink config. > > > > > > *From: *Aaron Langford > *Date: *Thursday, January 23, 2020 at 12:22 PM > *To: *Senthil Kumar > *Cc: *Yang Wang , "user@flink.apache.org" < > user@fl