Hi Roger, In fact I am implementing another use case than the one you know about, with more sources than Kafka: we now also use Flink in the BI team (which I belong to).
The problem with the web interface is that it is not easily scriptable and to my understanding it does not allow cleaning code to be called upon cancellation. I would have liked to integrate with the standard BI production environment of my company, which requires to be able to call start, status & stop scripts. I think I will implement such a mechanism by periodically testing in my Source for the existence of a specific “heart beet” HDFS file, and quit the run() method if this file no longer exists because it has been deleted by a stop script. Arnaud De : Robert Metzger [mailto:rmetz...@apache.org] Envoyé : jeudi 2 juillet 2015 09:48 À : user@flink.apache.org Objet : Re: How to cancel a Flink DataSource from the driver code? Hi Arnaud, when using the PersistentKafkaSource, you can always cancel the job in the web interface and start it again. We will continue reading from Kafka where you left off. You can probably also send the cancel request manually to the web interface, to that URL: http://localhost:8081/jobsInfo?get=cancel&job=68c53a77f11d34695ac1aea4f098af82 But I don't think there is a way to submit a topology in a non-blocking way, so that env.execute() returns immediately with the JobId. On Thu, Jul 2, 2015 at 9:35 AM, LINZ, Arnaud <al...@bouyguestelecom.fr<mailto:al...@bouyguestelecom.fr>> wrote: Hi Stephan, I think that clean shutdown is a major feature to build a complex persistent service that use Flink Streaming for a data-quality critical task, and I’ll mark my code with a // FIXME comment waiting for this feature to be available ! Greetings, Arnaud De : ewenstep...@gmail.com<mailto:ewenstep...@gmail.com> [mailto:ewenstep...@gmail.com<mailto:ewenstep...@gmail.com>] De la part de Stephan Ewen Envoyé : mercredi 1 juillet 2015 15:58 À : user@flink.apache.org<mailto:user@flink.apache.org> Objet : Re: How to cancel a Flink DataSource from the driver code? Hi Arnaud! There is a pending issue and pull request that is adding a "cancel()" call to the command line interface. https://github.com/apache/flink/pull/750 It would be possible to extend that such that the driver can also cancel the program. Greetings, Stephan On Wed, Jul 1, 2015 at 3:33 PM, LINZ, Arnaud <al...@bouyguestelecom.fr<mailto:al...@bouyguestelecom.fr>> wrote: Hello, I really looked in the documentation but unfortunately I could not find the answer: how do you cancel your data SourceFunction from your “driver” code (i.e., from a monitoring thread that can initiate a proper shutdown) ? Calling “cancel()” on the object passed to the addSource() has no effect since it does not apply to the marshalled distributed object(s). Best regards, Arnaud ________________________________ L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
