Re: [DISCUSS] PIP-208: HTTP Sink

> > I think we could shrink the connectors a lot by removing from the NAR > archives dependencies that are already present in the > pulsar-functions-instance. > I mean java-instance.jar Le mer. 19 oct. 2022 à 12:29, Christophe Bornet a écrit : > The pulsar-all docker image is pretty big. I assum

Re: [DISCUSS] PIP-208: HTTP Sink

> > The pulsar-all docker image is pretty big. I assume we will continue > to build and package additional connectors. It would be great to > figure out how to make it smaller at some point. > I think we could shrink the connectors a lot by removing from the NAR archives dependencies that are alrea

Re: [DISCUSS] PIP-208: HTTP Sink

I just tested and the HTTP Sink works fine with the ClickHouse HTTP ingest API. Create table in ClickHouse: CREATE TABLE test ( col1 String ) ENGINE = MergeTree() PRIMARY KEY (col1) Create Sink: bin/pulsar-admin sinks create -a pulsar-io/http/target/pulsar-io-http-2.11.0-SNAPSHOT.nar -i my-to

Re: [DISCUSS] PIP-208: HTTP Sink

Great discussion. I have one minor comment that is tangentially related. > On building the project `pulsar-io-http-{version}.nar` will be built and > added to the `pulsar-all` distribution. The pulsar-all docker image is pretty big. I assume we will continue to build and package additional connec

Re: [DISCUSS] PIP-208: HTTP Sink

Sure you can test with the Sink of my PR branch. Otherwise I'll do the test after ApacheCon. Le mar. 27 sept. 2022 à 12:57, tison a écrit : > Yes. It's a potential use case for validating the implementation. If you > don't have time to try it out, I can schedule some time to demo it with a > pro

Re: [DISCUSS] PIP-208: HTTP Sink

Yes. It's a potential use case for validating the implementation. If you don't have time to try it out, I can schedule some time to demo it with a prototype HTTP sink or after the patch gets merged :) Best, tison. Christophe Bornet 于2022年9月27日周二 18:51写道： > Hi Tison, > > Very interesting and sh

Re: [DISCUSS] PIP-208: HTTP Sink

Hi Tison, Very interesting and shows the value of such a HTTP Sink. The Pulsar HTTP Sink should work OOTB with ClickHouse. I don't have time to do the test right now, so would someone want to do it ? Best regards. Christophe Bornet Le mar. 27 sept. 2022 à 12:31, tison a écrit : > Hi Christoph

Re: [DISCUSS] PIP-208: HTTP Sink

Hi Christophe, Thanks for starting this proposal. It looks cool. I'd suggest one real-world integration test you can make use of: https://clickhouse.com/docs/en/integrations/kafka/kafka-connect-http (replace source kafka with pulsar). Best, tison. Enrico Olivelli 于2022年9月27日周二 18:04写道： > Tha

Re: [DISCUSS] PIP-208: HTTP Sink

Thanks for your answers. I am fine with the current proposal. We can enhance it as follow up work Enrico Il giorno ven 23 set 2022 alle ore 19:20 Christophe Bornet ha scritto: > > Thanks for your feedback Enrico. > My answers to your comments below > > BR > > Christophe > > Le mar. 20 sept. 2022

Re: [DISCUSS] PIP-208: HTTP Sink

Thanks for your feedback Alexander. My answers to your comments below. BR Christophe Le mer. 21 sept. 2022 à 15:58, Alexander Preuss a écrit : > Hi Christophe, > > I think this is a very good idea! > > I agree with Enrico that the body should depend on the record schema, but > it could also be

Re: [DISCUSS] PIP-208: HTTP Sink

Thanks for your feedback Enrico. My answers to your comments below BR Christophe Le mar. 20 sept. 2022 à 14:16, Enrico Olivelli a écrit : > Christophe, > very good initiative! > > I support it > Some comments inline below > > > Enrico > > Il giorno lun 19 set 2022 alle ore 19:10 Christophe Bor

Re: [DISCUSS] PIP-208: HTTP Sink

Hi Christophe, I think this is a very good idea! I agree with Enrico that the body should depend on the record schema, but it could also be done as a follow-up task. Another thing to think about could be an optional batching mechanism that would take a batch of records and send them as a list of

Re: [DISCUSS] PIP-208: HTTP Sink

Christophe, very good initiative! I support it Some comments inline below Enrico Il giorno lun 19 set 2022 alle ore 19:10 Christophe Bornet ha scritto: > > Hi all, > > I have drafted PIP-208: HTTP Sink > > PIP link: > https://github.com/apache/pulsar/issues/17719 > > Here's a copy of the conte

[DISCUSS] PIP-208: HTTP Sink

Hi all, I have drafted PIP-208: HTTP Sink PIP link: https://github.com/apache/pulsar/issues/17719 Here's a copy of the contents of the GH issue for your references: ### Motivation Currently, when you want to consume from Pulsar topics in applications written in languages that don't have a Puls