Thanks for the response.
We are working quite a lot with Zeppelin and our data analysts LOVE to take
it to the edge.
We find out that once a while we manage to crash it with memory issues.
Actually, I'm using Monit to monitor the process and restart as needed.
I just bypassed the upstart scripts and use zeppelin-daemon.sh directly.
BTW
Monit monitors the process and can restart if needed, but not less
important: it gives us web API that users (such as these data scientists)
can restart Zeppelin without ssh to the machine.
On Tue, Jan 19, 2016 at 4:01 AM, Jonathan Kelly <jonathaka...@gmail.com>
wrote:
> Ophir,
>
> It's not possible to have stop/start be the last part of the line, as this
> is simply how upstart works. stop/start in this case is actually the name
> of the script. They are actually just symlinks to initctl. (That is,
> /sbin/{stop,start,etc.} are just symlinks to /sbin/initctl, and
> /sbin/initctl can either be called directly with an action as one of the
> arguments, or by using one of the symlinks to indicate the action. See "man
> initctl" for more information.)
>
> What is your use case for needing to restart the Zeppelin server on EMR so
> often?
>
> And yes, it seems that there is a bug in the stop script for Zeppelin. I
> think I have fixed it and will include the fix in a future release of EMR.
> Sorry for any inconvenience this has caused you! In the meantime you can
> kill all processes being run by the zeppelin user, and the Zeppelin server
> will automatically start back up in a few seconds (so long as you have not
> run "sudo stop zeppelin").
>
> ~ Jonathan
>
> On Sun, Jan 17, 2016 at 2:16 AM Ophir Cohen <oph...@gmail.com> wrote:
>
>> Got you.
>> OK, good to know that you guys moved to upstart.
>> Personally I think that the stop/start command should be the last in the
>> line as this is what you mostly change and it is more easy to change.
>> In test mode you need to go back to the start/stop all the time and its
>> pretty annoying.
>>
>> Anyway, can it be that the stop command does not work?
>> For some reason the start command starts my zeppelin but stop does not,
>> well, stop...
>>
>> On Fri, Jan 15, 2016 at 7:22 PM, Jonathan Kelly <jonathaka...@gmail.com>
>> wrote:
>>
>>> Thanks!
>>>
>>> I think rather than "sudo /etc/init.d/zeppelin start" the usual way
>>> would be "sudo service zeppelin start". However, on EMR 4.x, we actually
>>> use upstart (http://upstart.ubuntu.com) to manage the service processes
>>> and ensure that they are restarted automatically if they die, so the
>>> corresponding command on EMR is "sudo start zeppelin".
>>>
>>> On Jan 14, 2016, at 11:13 PM, Ophir Cohen <oph...@gmail.com> wrote:
>>>
>>> Yes, I just checked that out and find out it does not work on 0.6.
>>> Ok, good luck, you are doing great job in emr!
>>>
>>> BTW
>>> If I may say, you need to add service support to Zeppelin.
>>> I.e. to allow start of Zeppelin this way:
>>> sudo /etc/init.d/zeppelin start
>>>
>>> *that will start zeppelin under Zeppelin user.
>>> On Jan 15, 2016 9:03 AM, "Jonathan Kelly" <jonathaka...@gmail.com>
>>> wrote:
>>>
>>>> Yes, there are a lot of changes since 0.5.5, of course, but we're not
>>>> going to support an unreleased version on EMR. We did that for emr-4.1.0
>>>> (the first version to support Zeppelin), but it's much less confusing for
>>>> us to support only released versions on EMR. As I mentioned earlier, we
>>>> will upgrade to 0.5.6 once it's released, but as I also mentioned earlier,
>>>> while the Notebook API appears fixed on 0.5.6, the Job API is not. The
>>>> Job API, btw, is what allows you to run notebooks via the REST API. The
>>>> Notebook API only lets you do CRUD on notebooks/paragraphs/crons.
>>>>
>>>> FWIW, I also tried building Zeppelin from the tip of the master branch
>>>> but still hit the same error when using the Job API, but that's not too
>>>> surprising because v0.5.6 isn't too different from the tip of the master
>>>> branch, particularly in the REST API code (other than the new Shiro
>>>> Security API).
>>>>
>>>> On Thu, Jan 14, 2016 at 10:45 PM Ophir Cohen <oph...@gmail.com> wrote:
>>>>
>>>>> I havn't try the job api, in fact what is the differences between
>>>>> notebook and job api?
>>>>> Anyway, why not jumping to 0.6.0?
>>>>> Looks like there is MANY improvments and bug fixes there....
>>>>> Sounds to me that you are debugging last year snow 😉
>>>>> On Jan 15, 2016 3:07 AM, "Jonathan Kelly" <jonathaka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> But now some bad news. I was playing around with the REST API a
>>>>>> little more, and I noticed that I get the same AbstractMethodError on
>>>>>> Zeppelin 0.5.6 for a POST to /api/job/2A94M5J1Z. :( Does anybody have any
>>>>>> idea how this error could possibly have been fixed for /api/notebook in
>>>>>> one
>>>>>> release, for /api/notebook/* in the next release, and yet still fail for
>>>>>> /api/job/* in that same release?
>>>>>>
>>>>>> On Thu, Jan 14, 2016 at 4:13 PM Jonathan Kelly <
>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>
>>>>>>> Oh, hey, good news! I just tried Zeppelin 0.5.6 on EMR, and the
>>>>>>> Notebook API works fine. So now we just need Zeppelin 0.5.6 to be
>>>>>>> officially released, and then we can support it on a future version of
>>>>>>> EMR.
>>>>>>>
>>>>>>> On Thu, Jan 14, 2016 at 2:01 AM Ophir Cohen <oph...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> OK, I've made it work.
>>>>>>>> As Jonathan stated above, on emr 4.1 the notebook APIs does not
>>>>>>>> work at all. On both, emr 4.2 and emr 4.3 the /api/notebook works but
>>>>>>>> the
>>>>>>>> query for a specific notebook not.
>>>>>>>>
>>>>>>>> Deploying vanilla Zeppelin (master branch) on the same cluster
>>>>>>>> worked as a charm.
>>>>>>>>
>>>>>>>> It probably origin by Zeppelin versions (0.5.5 on emr vs. 0.6.0 the
>>>>>>>> latest).
>>>>>>>>
>>>>>>>> Now I'm encountering different error that, as far as I can see,
>>>>>>>> does not interfere Zeppelin work:
>>>>>>>> ERROR [2016-01-14 09:55:38,499] ({qtp1505370540-32}
>>>>>>>> NotebookServer.java[onMessage]:176) - Can't handle message
>>>>>>>> java.lang.NullPointerException
>>>>>>>>
>>>>>>>> Many of those.
>>>>>>>> If anyone knows what does it mean....
>>>>>>>>
>>>>>>>> On Thu, Jan 14, 2016 at 12:23 AM, Jonathan Kelly <
>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Interesting... I confirmed that removing all of the extra
>>>>>>>>> classpath entries added by EMR in zeppelin-env.sh does not fix this
>>>>>>>>> issue.
>>>>>>>>> This must mean that the issue is caused by the custom EMR builds of
>>>>>>>>> either
>>>>>>>>> Spark or Hadoop, which we use when building Zeppelin.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 13, 2016 at 2:15 PM Jonathan Kelly <
>>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Ah, never mind. I had incorrectly been excluding the jars in
>>>>>>>>>> /usr/lib/zeppelin/interpreter, and I just noticed that
>>>>>>>>>> jersey-core-1.9.jar
>>>>>>>>>> appears in two places underneath that directory (in the hive and
>>>>>>>>>> phoenix
>>>>>>>>>> interpreter libs), and those jars are included in the Zeppelin server
>>>>>>>>>> classpath. This older version of jersey-core conflicts with
>>>>>>>>>> jersey-core-1.13.jar from /usr/lib/zeppelin/lib, so this probably
>>>>>>>>>> doesn't
>>>>>>>>>> help. However, removing just these two jars did not help by
>>>>>>>>>> itself.
>>>>>>>>>>
>>>>>>>>>> Haven't tried removing classpath entries from zeppelin-env.sh yet
>>>>>>>>>> though.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 13, 2016 at 2:07 PM Jonathan Kelly <
>>>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yeah, I'm sure you're right that it must be due to something
>>>>>>>>>>> that EMR is putting in the classpath, but the thing is that I
>>>>>>>>>>> didn't see
>>>>>>>>>>> anything else in the classpath that includes the
>>>>>>>>>>> javax.ws.rs.core.Response
>>>>>>>>>>> class.
>>>>>>>>>>>
>>>>>>>>>>> And yes, I agree that trying to remove these two jars from the
>>>>>>>>>>> Zeppelin classpath was a futile attempt and would only have helped
>>>>>>>>>>> if
>>>>>>>>>>> somehow these jars were near duplicates of each other but different
>>>>>>>>>>> versions of the same thing. Besides, if removing one of the jars had
>>>>>>>>>>> helped, it would have shown that this was probably not a problem
>>>>>>>>>>> with EMR
>>>>>>>>>>> specifically but with Zeppelin itself, which of course would have
>>>>>>>>>>> been
>>>>>>>>>>> doubtful.
>>>>>>>>>>>
>>>>>>>>>>> Anyway, I'll try removing the EMR-supplied extra classpath
>>>>>>>>>>> entries from zeppelin-env.sh.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 13, 2016 at 1:48 PM Ophir Cohen <oph...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Interesting
>>>>>>>>>>>>
>>>>>>>>>>>> I'm using emr 4.1.0, I'll try 4.2.0 tomorrow.
>>>>>>>>>>>>
>>>>>>>>>>>> If I need to guess, I don't think that the mis-version
>>>>>>>>>>>> dependency comes from the zeppelin jars as Zeppelin api works when
>>>>>>>>>>>> not in
>>>>>>>>>>>> emr.
>>>>>>>>>>>> It most likely that it comes from the emr classpath.
>>>>>>>>>>>> I would have removed all the additional paths (set in
>>>>>>>>>>>> zeppelin-env.sh) and check again.
>>>>>>>>>>>>
>>>>>>>>>>>> BTW
>>>>>>>>>>>> Removing the one of the jars from Zeppelin as you did won't
>>>>>>>>>>>> work as Zeppelin needs these jars.
>>>>>>>>>>>> If you really want to test it you need to build zeppelin and
>>>>>>>>>>>> instructe maven to exclude the problematic dependency fron one of
>>>>>>>>>>>> these
>>>>>>>>>>>> jars.
>>>>>>>>>>>> In other words, maven should bring the jar, but without the
>>>>>>>>>>>> duplicate dependency.
>>>>>>>>>>>> Having said that, and as I stated above, I think that the
>>>>>>>>>>>> problematic dependency comes from the additional classpath and not
>>>>>>>>>>>> from
>>>>>>>>>>>> Zeppelin core jars.
>>>>>>>>>>>> On Jan 13, 2016 11:08 PM, "Jonathan Kelly" <
>>>>>>>>>>>> jonathaka...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, Ophir, (Jonathan from EMR here)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are you using emr-4.1.0 or emr-4.2.0? I just tried this with
>>>>>>>>>>>>> emr-4.2.0 and found that http://<my-server>:8890/api/notebook
>>>>>>>>>>>>> works for me, but http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>>>>>>>> fails. For some reason it doesn't quite fail with the same
>>>>>>>>>>>>> exception that
>>>>>>>>>>>>> you're seeing, but it definitely seems like a similar cause
>>>>>>>>>>>>> anyway. Here's
>>>>>>>>>>>>> the exception I get:
>>>>>>>>>>>>>
>>>>>>>>>>>>> WARN [2016-01-13 20:34:52,279] ({qtp508683864-37}
>>>>>>>>>>>>> ServletHandler.java[doHandle]:590) - Error for
>>>>>>>>>>>>> /api/notebook/2A94M5J1Z
>>>>>>>>>>>>> java.lang.AbstractMethodError:
>>>>>>>>>>>>> javax.ws.rs.core.Response.getStatusInfo()Ljavax/ws/rs/core/Response$StatusType;
>>>>>>>>>>>>> at
>>>>>>>>>>>>> javax.ws.rs.WebApplicationException.validate(WebApplicationException.java:186)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I found that the Zeppelin classpath includes two different
>>>>>>>>>>>>> jars that contain the javax/ws/rs/core/Response class:
>>>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar and
>>>>>>>>>>>>> /usr/lib/zeppelin/lib/jersey-core-1.13.jar
>>>>>>>>>>>>>
>>>>>>>>>>>>> What's weird though is that if I remove either of these from
>>>>>>>>>>>>> the classpath, the Zeppelin server fails to start up for
>>>>>>>>>>>>> different reasons. If
>>>>>>>>>>>>> I remove /usr/lib/zeppelin/lib/jersey-core-1.13.jar, I get a
>>>>>>>>>>>>> ClassNotFoundException for
>>>>>>>>>>>>> com.sun.jersey.core.util.FeaturesAndProperties,
>>>>>>>>>>>>> and if I remove /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar,
>>>>>>>>>>>>> I get a ClassNotFoundException for
>>>>>>>>>>>>> javax.ws.rs.NotFoundException.
>>>>>>>>>>>>>
>>>>>>>>>>>>> According to a `mvn ... dependency:tree` in the
>>>>>>>>>>>>> zeppelin-server submodule,
>>>>>>>>>>>>> /usr/lib/zeppelin/lib/jersey-core-1.13.jar comes
>>>>>>>>>>>>> indirectly from zeppelin-server's direct dependency
>>>>>>>>>>>>> on com.sun.jersey:jersey-servlet:jar:1.13:compile, and
>>>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar comes from
>>>>>>>>>>>>> zeppelin-server's direct dependency on
>>>>>>>>>>>>> javax.ws.rs:javax.ws.rs-api:jar:2.0-m10:compile.
>>>>>>>>>>>>> Both of these dependencies seem to have been added a long time
>>>>>>>>>>>>> ago, and
>>>>>>>>>>>>> both seem to be used by the websocket API rather than the REST
>>>>>>>>>>>>> API (at
>>>>>>>>>>>>> least based upon the Git commit descriptions where they were
>>>>>>>>>>>>> added), so
>>>>>>>>>>>>> this confuses me even more.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does anybody from the Zeppelin side have any idea what is
>>>>>>>>>>>>> going on here?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Jonathan
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 13, 2016 at 5:34 AM Ophir Cohen <oph...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> We migrated our Zeppelin to use EMR Zeppelin. It's straight
>>>>>>>>>>>>>> forward and we were happy with the migration till we found out
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> something isn't working well with the rest API.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Calling to the interpreter API:
>>>>>>>>>>>>>> http://<my-server>:8890/api/interpreter
>>>>>>>>>>>>>> and this:
>>>>>>>>>>>>>> http://<my-server>:8890/api/interpreter/setting
>>>>>>>>>>>>>> Returns results as expected.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When trying to access the notebooks:
>>>>>>>>>>>>>> http://<my-server>:8890/api/notebook
>>>>>>>>>>>>>> Or a specific notebook:
>>>>>>>>>>>>>> http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>>>>>>>>> It fails with HTTP 500 error.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The error I can see in the logs is NoSuchMethodError (see
>>>>>>>>>>>>>> below) which suggests we might have here 'jar-hell' and the
>>>>>>>>>>>>>> wrong (probably
>>>>>>>>>>>>>> old) jar loaded instead of the need one - but I can't figure
>>>>>>>>>>>>>> that out.
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The exception:
>>>>>>>>>>>>>> WARN [2016-01-13 07:47:11,690] ({qtp716961517-38}
>>>>>>>>>>>>>> ServletHandler.java[doHandle]:590) - Error for
>>>>>>>>>>>>>> /api/notebook/2A94M5J1Z
>>>>>>>>>>>>>> java.lang.NoSuchMethodError:
>>>>>>>>>>>>>> javax.ws.rs.ClientErrorException.validate(Ljavax/ws/rs/core/Response;Ljavax/ws/rs/core/Response$Status$Family;)Ljavax/ws/rs/core/Response;
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>
