Thanks a lot for the update Tomasz and keep up posted if it happens again. Kostas
On Fri, Jul 24, 2020 at 6:37 PM Tomasz Dudziak <t.dudz...@mwam.com> wrote: > > Yes, the job was running and the REST server as well. No JobMaster failures > noticed. > I used a test cluster deployed on a bunch of VM's and bare metal boxes. > I am afraid, I can no longer reproduce this issue. It occurred a couple days > ago and lasted for an entire day with jobs being quite often erratically > reported as Not Found. As I said, I noticed that another query immediately > after the one that returned Not Found consistently returned a correct result. > It had never occurred before and I am afraid now I could no longer observe it > again. I appreciate it does not give too much information so I will come back > with more info on this thread if it happens again. > > -----Original Message----- > From: Kostas Kloudas <kklou...@gmail.com> > Sent: 24 July 2020 15:46 > To: Tomasz Dudziak <t.dudz...@mwam.com> > Cc: user@flink.apache.org; Chesnay Schepler <ches...@apache.org> > Subject: Re: REST API randomly returns Not Found for an existing job > > Hi Tomasz, > > Thanks a lot for reporting this issue. If you have verified that the job is > running AND that the REST server is also up and running (e.g. > check the overview page) then I think that this should not be happening. I am > cc'ing Chesnay who may have an additional opinion on this. > > Cheers, > Kostas > > On Thu, Jul 23, 2020 at 12:59 PM Tomasz Dudziak <t.dudz...@mwam.com> wrote: > > > > Hi, > > > > > > > > I have come across an issue related to GET /job/:jobId endpoint from > > monitoring REST API in Flink 1.9.0. A few seconds after successfully > > starting a job and confirming its status as RUNNING, that endpoint would > > return 404 (Not Found). Interestingly, querying immediately again > > (literally a millisecond later) would return a valid result. I later > > noticed a similar behaviour in regard to a finished job as well. At certain > > points in time that endpoint would arbitrarily return 404, but similarly > > querying again would succeed. I saw this strange behaviour only recently > > and it used to work fine before. > > > > > > > > Do you know what could be the root cause of this? At the moment, as a > > workaround I just query a job a couple of times in a row to ensure > > whether it definitely does not exist or it is just being misreported > > as non-existent, but this feels a bit like cottage industry… > > > > > > > > Kind regards, > > > > Tomasz > > > > > > > > Tomasz Dudziak | Marshall Wace LLP, George House, 131 Sloane Street, > > London, SW1X 9AT | E-mail: t.dudz...@mwam.com | Tel: +44 207 024 7061 > > > > > > > > > > > > This e-mail and any attachments are confidential to the addressee(s) and > > may contain information that is legally privileged and/or confidential. > > Please refer to http://www.mwam.com/email-disclaimer-uk for important > > disclosures regarding this email. If we collect and use your personal data > > we will use it in accordance with our privacy policy, which can be reviewed > > at https://www.mwam.com/privacy-policy. > > > > Marshall Wace LLP is authorised and regulated by the Financial Conduct > > Authority. Marshall Wace LLP is a limited liability partnership registered > > in England and Wales with registered number OC302228 and registered office > > at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving > > this e-mail as a client, or an investor in an investment vehicle, managed > > or advised by Marshall Wace North America L.P., the sender of this e-mail > > is communicating with you in the sender's capacity as an associated or > > related person of Marshall Wace North America L.P., which is registered > > with the US Securities and Exchange Commission as an investment adviser. > > This e-mail and any attachments are confidential to the addressee(s) and may > contain information that is legally privileged and/or confidential. Please > refer to http://www.mwam.com/email-disclaimer-uk for important disclosures > regarding this email. If we collect and use your personal data we will use it > in accordance with our privacy policy, which can be reviewed at > https://www.mwam.com/privacy-policy . > > Marshall Wace LLP is authorised and regulated by the Financial Conduct > Authority. Marshall Wace LLP is a limited liability partnership registered in > England and Wales with registered number OC302228 and registered office at > George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this > e-mail as a client, or an investor in an investment vehicle, managed or > advised by Marshall Wace North America L.P., the sender of this e-mail is > communicating with you in the sender's capacity as an associated or related > person of Marshall Wace North America L.P., which is registered with the US > Securities and Exchange Commission as an investment adviser.
