Hi Jonathan,
Thank you for the detail — I struggled to find docs regarding service
management (this is my foray into EMR services). Knowing the proper steps
is very helpful indeed.
Did the Zeppelin server process not actually die but just got stuck?
The only symptom was that I could no longer access the notebook on my
browser. Up on digging into the logs, I noticed:
1. INFO [2015-11-24 08:18:22,987] ({qtp628811122-40}
NotebookServer.java[onClose]:163) - Closed connection to 10.231.152.57 :
46619. (/var/log/zeppelin/zeppelin-zeppelin-ip-*.log)
2. 15/11/29 20:45:14 WARN AmIpFilter: Could not find proxy-user cookie, so
user will not be set (a whole bunch of these statements in
/var/log/zeppelin/zeppelin-zeppelin-*.out)
3. hadoop application UI seems to indicate the zeppelin application is
still running (!)
Not knowing enough about the internals, I figured restarting the
zeppelin-server would be a good step. Typically, if the interpreter dies, I
see the "red" disconnected button but I can still connect to http://ip:8890
— in this scenario, restarting the interpreter seems to resolve the issue.
Many thanks again,
Asif
On Mon, Nov 30, 2015 at 2:58 PM, Jonathan Kelly <jonathaka...@gmail.com>
wrote:
> Asif,
>
> EMR 4.x services are run with upstart <http://upstart.ubuntu.com/>, so
> that if they die or are killed, they will be automatically restarted. Did
> the Zeppelin server process not actually die but just got stuck? Because if
> it did die, it should have been restarted. Or was it the Zeppelin Spark
> interpreter process that died but not the Zeppelin server?
>
> Also, though zeppelin-daemon.sh stop/start seems to work on EMR
> (according to Josef's suggestion and your experience with it working), the
> correct way to restart service on EMR is to use "sudo stop <service-name>;
> sudo start <service-name>" (and in this case "<service-name>" is
> "zeppelin-server"), but I think I have experienced some bugs with this
> method for Zeppelin (in that I think I've experienced "sudo stop
> zeppelin-server" not actually stopping the server), so a more sure way of
> restarting Zeppelin is to "sudo kill -9" all processes being run by the
> zeppelin user and to let upstart restart the Zeppelin server.
>
> ~ Jonathan
>
> On Sun, Nov 29, 2015 at 11:31 PM, Asif Imran <covariantmon...@gmail.com>
> wrote:
>
>> Hi Joseph,
>>
>> Thanks so much for the fast response. Till now, I could not locate where
>> the daemon scripts are located (on EMR cluster).
>>
>> But your recommendation definitely did the trick. I am back in action
>>
>> Cheers
>> Asif
>>
>> On Sun, Nov 29, 2015 at 11:20 PM, Josef A. Habdank <jahabd...@gmail.com>
>> wrote:
>>
>>> Did you try restarting the Zeppelin daemon?
>>>
>>> On Amazon EMR you do it by:
>>>
>>> sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh stop
>>> sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh start
>>>
>>> Josef
>>>
>>> On 30 November 2015 at 07:55, Asif Imran <covariantmon...@gmail.com>
>>> wrote:
>>>
>>>> The zeppelin server running on AWS EMR suddenly died from what I can
>>>> tell. From the zeppelin logs:
>>>>
>>>> INFO [2015-11-24 08:18:22,987] ({qtp628811122-40}
>>>> NotebookServer.java[onClose]:163) - Closed connection to 10.231.152.57 :
>>>> 46619.
>>>>
>>>> This was the *default* installation of zeppelin that came with the
>>>> cluster. I would rather not restart the entire cluster.
>>>>
>>>> Any one run into similar situation? I don't see any obvious way to
>>>> stop/start the application.
>>>>
>>>> Thanks
>>>> Asif
>>>>
>>>> FYI, tha hadoop application UI seems to indicate the zeppelin
>>>> application is still running.
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
