Great to hear you sorted things out. Looking forward to the pull request!
On Mon, Nov 9, 2015 at 4:50 PM, Stephan Ewen <se...@apache.org> wrote: > Super nice to hear :-) > > > On Mon, Nov 9, 2015 at 4:48 PM, Niels Basjes <ni...@basjes.nl> wrote: >> >> Apparently I just had to wait a bit longer for the first run. >> Now I'm able to package the project in about 7 minutes. >> >> Current status: I am now able to access HBase from within Flink on a >> Kerberos secured cluster. >> Cleaning up the patch so I can submit it in a few days. >> >> On Sat, Nov 7, 2015 at 10:01 PM, Stephan Ewen <se...@apache.org> wrote: >>> >>> The single shading step on my machine (SSD, 10 GB RAM) takes about 45 >>> seconds. HDD may be significantly longer, but should really not be more than >>> 10 minutes. >>> >>> Is your maven build always stuck in that stage (flink-dist) showing a >>> long list of dependencies (saying including org.x.y, including com.foo.bar, >>> ...) ? >>> >>> >>> On Sat, Nov 7, 2015 at 9:57 PM, Sachin Goel <sachingoel0...@gmail.com> >>> wrote: >>>> >>>> Usually, if all the dependencies are being downloaded, i.e., on the >>>> first build, it'll likely take 30-40 minutes. Subsequent builds might take >>>> 10 minutes approx. [I have the same PC configuration.] >>>> >>>> -- Sachin Goel >>>> Computer Science, IIT Delhi >>>> m. +91-9871457685 >>>> >>>> On Sun, Nov 8, 2015 at 2:05 AM, Niels Basjes <ni...@basjes.nl> wrote: >>>>> >>>>> How long should this take if you have HDD and about 8GB of RAM? >>>>> Is that 10 minutes? 20? >>>>> >>>>> Niels >>>>> >>>>> On Sat, Nov 7, 2015 at 2:51 PM, Stephan Ewen <se...@apache.org> wrote: >>>>>> >>>>>> Hi Niels! >>>>>> >>>>>> Usually, you simply build the binaries by invoking "mvn -DskipTests >>>>>> clean package" in the root flink directory. The resulting program should >>>>>> be >>>>>> in the "build-target" directory. >>>>>> >>>>>> If the program gets stuck, let us know where and what the last message >>>>>> on the command line is. >>>>>> >>>>>> Please be aware that the final step of building the "flink-dist" >>>>>> project may take a while, especially on systems with hard disks (as >>>>>> opposed >>>>>> to SSDs) and a comparatively low amount of memory. The reason is that the >>>>>> building of the final JAR file is quite expensive, because the system >>>>>> re-packages certain libraries in order to avoid conflicts between >>>>>> different >>>>>> versions. >>>>>> >>>>>> Stephan >>>>>> >>>>>> >>>>>> On Sat, Nov 7, 2015 at 2:40 PM, Niels Basjes <ni...@basj.es> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Excellent. >>>>>>> What you can help me with are the commands to build the binary >>>>>>> distribution from source. >>>>>>> I tried it last Thursday and the build seemed to get stuck at some >>>>>>> point (at the end of/just after building the dist module). >>>>>>> I haven't been able to figure out why yet. >>>>>>> >>>>>>> Niels >>>>>>> >>>>>>> On 5 Nov 2015 14:57, "Maximilian Michels" <m...@apache.org> wrote: >>>>>>>> >>>>>>>> Thank you for looking into the problem, Niels. Let us know if you >>>>>>>> need anything. We would be happy to merge a pull request once you have >>>>>>>> verified the fix. >>>>>>>> >>>>>>>> On Thu, Nov 5, 2015 at 1:38 PM, Niels Basjes <ni...@basjes.nl> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I created https://issues.apache.org/jira/browse/FLINK-2977 >>>>>>>>> >>>>>>>>> On Thu, Nov 5, 2015 at 12:25 PM, Robert Metzger >>>>>>>>> <rmetz...@apache.org> wrote: >>>>>>>>>> >>>>>>>>>> Hi Niels, >>>>>>>>>> thank you for analyzing the issue so properly. I agree with you. >>>>>>>>>> It seems that HDFS and HBase are using their own tokes which we need >>>>>>>>>> to >>>>>>>>>> transfer from the client to the YARN containers. We should be able >>>>>>>>>> to port >>>>>>>>>> the fix from Spark (which they got from Storm) into our YARN client. >>>>>>>>>> I think we would add this in >>>>>>>>>> org.apache.flink.yarn.Utils#setTokensFor(). >>>>>>>>>> >>>>>>>>>> Do you want to implement and verify the fix yourself? If you are >>>>>>>>>> to busy at the moment, we can also discuss how we share the work (I'm >>>>>>>>>> implementing it, you test the fix) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Robert >>>>>>>>>> >>>>>>>>>> On Tue, Nov 3, 2015 at 5:26 PM, Niels Basjes <ni...@basjes.nl> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Update on the status so far.... I suspect I found a problem in a >>>>>>>>>>> secure setup. >>>>>>>>>>> >>>>>>>>>>> I have created a very simple Flink topology consisting of a >>>>>>>>>>> streaming Source (the outputs the timestamp a few times per second) >>>>>>>>>>> and a >>>>>>>>>>> Sink (that puts that timestamp into a single record in HBase). >>>>>>>>>>> Running this on a non-secure Yarn cluster works fine. >>>>>>>>>>> >>>>>>>>>>> To run it on a secured Yarn cluster my main routine now looks >>>>>>>>>>> like this: >>>>>>>>>>> >>>>>>>>>>> public static void main(String[] args) throws Exception { >>>>>>>>>>> System.setProperty("java.security.krb5.conf", >>>>>>>>>>> "/etc/krb5.conf"); >>>>>>>>>>> >>>>>>>>>>> UserGroupInformation.loginUserFromKeytab("nbas...@xxxxxx.net", >>>>>>>>>>> "/home/nbasjes/.krb/nbasjes.keytab"); >>>>>>>>>>> >>>>>>>>>>> final StreamExecutionEnvironment env = >>>>>>>>>>> StreamExecutionEnvironment.getExecutionEnvironment(); >>>>>>>>>>> env.setParallelism(1); >>>>>>>>>>> >>>>>>>>>>> DataStream<String> stream = env.addSource(new >>>>>>>>>>> TimerTicksSource()); >>>>>>>>>>> stream.addSink(new SetHBaseRowSink()); >>>>>>>>>>> env.execute("Long running Flink application"); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> When I run this >>>>>>>>>>> flink run -m yarn-cluster -yn 1 -yjm 1024 -ytm 4096 >>>>>>>>>>> ./kerberos-1.0-SNAPSHOT.jar >>>>>>>>>>> >>>>>>>>>>> I see after the startup messages: >>>>>>>>>>> >>>>>>>>>>> 17:13:24,466 INFO >>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation - >>>>>>>>>>> Login >>>>>>>>>>> successful for user nbas...@xxxxxx.net using keytab file >>>>>>>>>>> /home/nbasjes/.krb/nbasjes.keytab >>>>>>>>>>> 11/03/2015 17:13:25 Job execution switched to status RUNNING. >>>>>>>>>>> 11/03/2015 17:13:25 Custom Source -> Stream Sink(1/1) switched to >>>>>>>>>>> SCHEDULED >>>>>>>>>>> 11/03/2015 17:13:25 Custom Source -> Stream Sink(1/1) switched to >>>>>>>>>>> DEPLOYING >>>>>>>>>>> 11/03/2015 17:13:25 Custom Source -> Stream Sink(1/1) switched to >>>>>>>>>>> RUNNING >>>>>>>>>>> >>>>>>>>>>> Which looks good. >>>>>>>>>>> >>>>>>>>>>> However ... no data goes into HBase. >>>>>>>>>>> After some digging I found this error in the task managers log: >>>>>>>>>>> >>>>>>>>>>> 17:13:42,677 WARN org.apache.hadoop.hbase.ipc.RpcClient >>>>>>>>>>> - Exception encountered while connecting to the server : >>>>>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by >>>>>>>>>>> GSSException: No valid credentials provided (Mechanism level: >>>>>>>>>>> Failed to find >>>>>>>>>>> any Kerberos tgt)] >>>>>>>>>>> 17:13:42,677 FATAL org.apache.hadoop.hbase.ipc.RpcClient >>>>>>>>>>> - SASL authentication failed. The most likely cause is missing or >>>>>>>>>>> invalid >>>>>>>>>>> credentials. Consider 'kinit'. >>>>>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by >>>>>>>>>>> GSSException: No valid credentials provided (Mechanism level: >>>>>>>>>>> Failed to find >>>>>>>>>>> any Kerberos tgt)] >>>>>>>>>>> at >>>>>>>>>>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> First starting a yarn-session and then loading my job gives the >>>>>>>>>>> same error. >>>>>>>>>>> >>>>>>>>>>> My best guess at this point is that Flink needs the same fix as >>>>>>>>>>> described here: >>>>>>>>>>> >>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-6918 ( >>>>>>>>>>> https://github.com/apache/spark/pull/5586 ) >>>>>>>>>>> >>>>>>>>>>> What do you guys think? >>>>>>>>>>> >>>>>>>>>>> Niels Basjes >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 27, 2015 at 6:12 PM, Maximilian Michels >>>>>>>>>>> <m...@apache.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Niels, >>>>>>>>>>>> >>>>>>>>>>>> You're welcome. Some more information on how this would be >>>>>>>>>>>> configured: >>>>>>>>>>>> >>>>>>>>>>>> In the kdc.conf, there are two variables: >>>>>>>>>>>> >>>>>>>>>>>> max_life = 2h 0m 0s >>>>>>>>>>>> max_renewable_life = 7d 0h 0m 0s >>>>>>>>>>>> >>>>>>>>>>>> max_life is the maximum life of the current ticket. However, it >>>>>>>>>>>> may be renewed up to a time span of max_renewable_life from the >>>>>>>>>>>> first ticket >>>>>>>>>>>> issue on. This means that from the first ticket issue, new tickets >>>>>>>>>>>> may be >>>>>>>>>>>> requested for one week. Each renewed ticket has a life time of >>>>>>>>>>>> max_life (2 >>>>>>>>>>>> hours in this case). >>>>>>>>>>>> >>>>>>>>>>>> Please let us know about any difficulties with long-running >>>>>>>>>>>> streaming application and Kerberos. >>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> Max >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Oct 27, 2015 at 2:46 PM, Niels Basjes <ni...@basjes.nl> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for your feedback. >>>>>>>>>>>>> So I guess I'll have to talk to the security guys about having >>>>>>>>>>>>> special >>>>>>>>>>>>> kerberos ticket expiry times for these types of jobs. >>>>>>>>>>>>> >>>>>>>>>>>>> Niels Basjes >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Oct 23, 2015 at 11:45 AM, Maximilian Michels >>>>>>>>>>>>> <m...@apache.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Niels, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for your question. Flink relies entirely on the >>>>>>>>>>>>>> Kerberos >>>>>>>>>>>>>> support of Hadoop. So your question could also be rephrased to >>>>>>>>>>>>>> "Does >>>>>>>>>>>>>> Hadoop support long-term authentication using Kerberos?". And >>>>>>>>>>>>>> the >>>>>>>>>>>>>> answer is: Yes! >>>>>>>>>>>>>> >>>>>>>>>>>>>> While Hadoop uses Kerberos tickets to authenticate users with >>>>>>>>>>>>>> services >>>>>>>>>>>>>> initially, the authentication process continues differently >>>>>>>>>>>>>> afterwards. Instead of saving the ticket to authenticate on a >>>>>>>>>>>>>> later >>>>>>>>>>>>>> access, Hadoop creates its own security tockens >>>>>>>>>>>>>> (DelegationToken) that >>>>>>>>>>>>>> it passes around. These are authenticated to Kerberos >>>>>>>>>>>>>> periodically. To >>>>>>>>>>>>>> my knowledge, the tokens have a life span identical to the >>>>>>>>>>>>>> Kerberos >>>>>>>>>>>>>> ticket maximum life span. So be sure to set the maximum life >>>>>>>>>>>>>> span very >>>>>>>>>>>>>> high for long streaming jobs. The renewal time, on the other >>>>>>>>>>>>>> hand, is >>>>>>>>>>>>>> not important because Hadoop abstracts this away using its own >>>>>>>>>>>>>> security tockens. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm afraid there is not Kerberos how-to yet. If you are on >>>>>>>>>>>>>> Yarn, then >>>>>>>>>>>>>> it is sufficient to authenticate the client with Kerberos. On >>>>>>>>>>>>>> a Flink >>>>>>>>>>>>>> standalone cluster you need to ensure that, initially, all >>>>>>>>>>>>>> nodes are >>>>>>>>>>>>>> authenticated with Kerberos using the kinit tool. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Feel free to ask if you have more questions and let us know >>>>>>>>>>>>>> about any >>>>>>>>>>>>>> difficulties. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>> Max >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Oct 22, 2015 at 2:06 PM, Niels Basjes >>>>>>>>>>>>>> <ni...@basjes.nl> wrote: >>>>>>>>>>>>>> > Hi, >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > I want to write a long running (i.e. never stop it) >>>>>>>>>>>>>> > streaming flink >>>>>>>>>>>>>> > application on a kerberos secured Hadoop/Yarn cluster. My >>>>>>>>>>>>>> > application needs >>>>>>>>>>>>>> > to do things with files on HDFS and HBase tables on that >>>>>>>>>>>>>> > cluster so having >>>>>>>>>>>>>> > the correct kerberos tickets is very important. The stream >>>>>>>>>>>>>> > is to be ingested >>>>>>>>>>>>>> > from Kafka. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > One of the things with Kerberos is that the tickets expire >>>>>>>>>>>>>> > after a >>>>>>>>>>>>>> > predetermined time. My knowledge about kerberos is very >>>>>>>>>>>>>> > limited so I hope >>>>>>>>>>>>>> > you guys can help me. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > My question is actually quite simple: Is there an howto >>>>>>>>>>>>>> > somewhere on how to >>>>>>>>>>>>>> > correctly run a long running flink application with kerberos >>>>>>>>>>>>>> > that includes a >>>>>>>>>>>>>> > solution for the kerberos ticket timeout ? >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thanks >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Niels Basjes >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Best regards / Met vriendelijke groeten, >>>>>>>>>>>>> >>>>>>>>>>>>> Niels Basjes >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best regards / Met vriendelijke groeten, >>>>>>>>>>> >>>>>>>>>>> Niels Basjes >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best regards / Met vriendelijke groeten, >>>>>>>>> >>>>>>>>> Niels Basjes >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards / Met vriendelijke groeten, >>>>> >>>>> Niels Basjes >>>> >>>> >>> >> >> >> >> -- >> Best regards / Met vriendelijke groeten, >> >> Niels Basjes > >
