Re: [DISCUSS] Move Pulsar SQL to a separated repository?
Thanks for picking up this task. The decision to move Pulsar SQL out of apache/pulsar repository has been made over 2 years ago in April 2020 with PIP-62, https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories . It's not only about moving Pulsar SQL out of apache/pulsar repository, but also includes Pulsar IO connectors and Pulsar adapters (already moved to https://github.com/apache/pulsar-adapters). > 1. Moving out the source code. > 2. Adapt Pulsar distribution logic so that we no longer includes Pulsar SQL > libs. > 3. Adapt Docker build logic for the same purpose. > 4. Define a release strategy for Pulsar SQL especially Pulsar compatibility > policy. > 5. Build Pulsar SQL release tarball and Docker image. > 6. Wire integration tests with the new repo model. regarding "6. Wire integration tests with the new repo model.", the integration tests referencing Pulsar SQL should also be moved out of apache/pulsar repository so that there's no dependency on Pulsar SQL in apache/pulsar. This also applies to "3. Adapt Docker build logic for the same purpose.", nothing in apache/pulsar repository docker images should depend on Pulsar SQL. I guess that the challenge is releasing something that is equivalent to the current "pulsar-all" docker image. It should not be handled in the apache/pulsar repository. We would need a new repository "pulsar-all-distribution" (or "pulsar-distribution" to make the name shorter). That repository could include the docker building logic and the integration tests that require the pulsar-all docker image that includes both Pulsar SQL and Pulsar IO Connectors. It might not be an optimal solution to run the integration tests against pulsar-all image only in the pulsar-all-distribution. It would be better if Pulsar SQL integration tests could run in the pulsar-sql repository and run with a docker image that can be quickly built to support Pulsar SQL integration tests. A possible solution could be that the same integration tests are run against the actual pulsar-all docker image in the "pulsar-distribution" repository to ensure that the tests also pass in full integration. This might be useful for validating the pulsar-all docker image release. We also would like to remove Pulsar IO from apache/pulsar and move it to a separate repository. This goal should be considered when we start making the changes. To summarize: when we are removing Pulsar SQL from apache/pulsar, it also means that the pulsar-all docker image is no more built as part of apache/pulsar builds and it requires a replacement (or dropping pulsar-all docker image completely). -Lari On 2022/07/15 12:11:32 tison wrote: > Hi devs, > > *Backgorund* > > In the past few weeks I pay some time on Pulsar SQL or name it Pulsar Trino > connector. > > I noticed in Trino community our committer Marvin (@xxc) ever submitted a > PR to contribute the connector to upstream[1]. However, due to the huge > version gap and lack of time to spend on that topic, it stalls about ten > months ago. > > At the moment the latest Pulsar version was 2.8.0 while we're preparing > 2.11.0 now. Besides, in Pulsar main repo we make several changes for the > connector, especially add Decimal support incompatible with upstream > changes. > > Then I have an idea to bump the version of Trino to migrate code from > Pulsar side[2]. Matteo reached out to me that it can be a worthy work > to remove the hard-dependency on Presto and if it’s going to take longer > time to get Trino to accept and merge it, move the Presto connector to a > separate repository, still within the Pulsar project. > > I try to prototype the work of this idea this week[3] and get more insights > about it. So I'd like to start this discussion to find other contributors > interested in this topic, figure out if moving Pulsar SQL to a separated > repository is a good idea, and discuss a few concrete challenges. > > *Motivation* > > The strongest motivation to move Pulsar SQL to a separated repository is > that even if a Pulsar user never uses Pulsar SQL, those libs are included > in the release tarball and take over 50% space of the distribution. This is > similar to the Pulsar Docker image. > > Another motivation is that we can simplify the codebase of main repo with > this movement. I found repo pulsar-connectors[4] and pulsar-presto[5] > exists but we didn't push them forward. > > Pulsar SQL modules are relatively stable and deserved for its own life. > > *Scope* > > The technique part for moving Pulsar SQL to a separated repository includes: > > 1. Moving out the source code. > 2. Adapt Pulsar distribution logic so that we no longer includes Pulsar SQL > libs. > 3. Adapt Docker build logic for the same purpose. > 4. Define a release strategy for Pulsar SQL especially Pulsar compatibility > policy. > 5. Build Pulsar SQL release tarball and Docker image. > 6. Wire integration tests
[GitHub] [pulsar] tisonkun added a comment to the discussion: What's the best practice to build java-test-image?
GitHub user tisonkun added a comment to the discussion: What's the best practice to build java-test-image? Thank you @lhotari! I'll try this script later. GitHub link: https://github.com/apache/pulsar/discussions/16633#discussioncomment-3169266 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [DISCUSS] Apache Pulsar 2.11.0 Release
Thanks Penghui for the reminder. I'd like to also include PIP: 181 Pulsar shell if the time permits. I believe that is a good idea to start testing the code freeze proposed by PIP-175 (https://github.com/apache/pulsar/issues/15966). Even if not officially approved, we discussed it many times and agreed to the usefulness of the code freezing. So a plan could be to try to merge the work in progress targeted for 2.11 by the mid of August and then start the code freezing as described in the PIP. Also, I volunteer for driving the release if nobody else is interested Thanks, Nicolò Boschi Il giorno lun 18 lug 2022 alle ore 06:59 Yunze Xu ha scritto: > In addition to #16202, there is a following PR to support the correct > ACK implementation for chunked messages. It should depend on #16202 > But I think I can submit an initial PR this week and change the tests > after #16202 is merged. > > Thanks, > Yunze > > > > > > 2022年7月18日 11:22，PengHui Li 写道： > > > > Hi all, > > > > We released 2.10.0 three months ago. And there are many great changes in > > the master branch, > > including new features and performance improvements. > > > > - PIP 74: apply client memory to consumer > > https://github.com/apache/pulsar/pull/15216 > > - PIP 143: Support split bundles by specified boundaries > > https://github.com/apache/pulsar/pull/13796 > > - PIP 145: regex subscription improvements > > https://github.com/apache/pulsar/pull/16062 > > - PIP 160: transaction performance improvements (still in progress and > > merged some PRs) > > - PIP 161: new exclusive producer mode support > > https://github.com/apache/pulsar/pull/15488 > > - PIP 182: Provide new load balance placement strategy implementation for > > ModularLoadManagerStrategy https://github.com/apache/pulsar/pull/16281 > > Add Pulsar Auth support for the Pulsar SQL > > https://github.com/apache/pulsar/pull/15571 > > > > And some features are blocked in the review stage, but they are powerful > > improvements for Pulsar > > > > PIP 37: Support chunking with Shared subscription > > https://github.com/apache/pulsar/pull/16202 > > PIP-166: Function add MANUAL delivery semantics > > https://github.com/apache/pulsar/pull/16279 > > > > You can find the complete change list in 2.11.0 at > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+milestone%3A2.11.0+-label%3Arelease%2F2.10.1+-label%3Arelease%2F2.10.2 > > > > And maybe I missed some important in-progress PRs, please let me know if > it > > should be a blocker of the 2.11.0 release. > > > > It's a good time to discuss the target time of the 2.11.0 release. > > I think we can leave 2 weeks to complete the in-progress PRs and 2 weeks > to > > accept bug fixes. > > And target the 2.11.0 release in mid-August. > > > > Please let me know what you think. > > > > Thanks, > > Penghui > >
Re: [DISCUSS] Apache Pulsar 2.11.0 Release
Nicolò, Il Lun 18 Lug 2022, 10:00 Nicolò Boschi ha scritto: > Thanks Penghui for the reminder. > I'd like to also include PIP: 181 Pulsar shell if the time permits. > > I believe that is a good idea to start testing the code freeze proposed by > PIP-175 (https://github.com/apache/pulsar/issues/15966). Even if not > officially approved, we discussed it many times and agreed to the > usefulness of the code freezing. > Great idea! We should really try it So a plan could be to try to merge the work in progress targeted for 2.11 > by the mid of August and then start the code freezing as described in the > PIP. > > Also, I volunteer for driving the release if nobody else is interested > Thanks for volunteering Enrico > Thanks, > Nicolò Boschi > > > Il giorno lun 18 lug 2022 alle ore 06:59 Yunze Xu > ha scritto: > > > In addition to #16202, there is a following PR to support the correct > > ACK implementation for chunked messages. It should depend on #16202 > > But I think I can submit an initial PR this week and change the tests > > after #16202 is merged. > > > > Thanks, > > Yunze > > > > > > > > > > > 2022年7月18日 11:22，PengHui Li 写道： > > > > > > Hi all, > > > > > > We released 2.10.0 three months ago. And there are many great changes > in > > > the master branch, > > > including new features and performance improvements. > > > > > > - PIP 74: apply client memory to consumer > > > https://github.com/apache/pulsar/pull/15216 > > > - PIP 143: Support split bundles by specified boundaries > > > https://github.com/apache/pulsar/pull/13796 > > > - PIP 145: regex subscription improvements > > > https://github.com/apache/pulsar/pull/16062 > > > - PIP 160: transaction performance improvements (still in progress and > > > merged some PRs) > > > - PIP 161: new exclusive producer mode support > > > https://github.com/apache/pulsar/pull/15488 > > > - PIP 182: Provide new load balance placement strategy implementation > for > > > ModularLoadManagerStrategy https://github.com/apache/pulsar/pull/16281 > > > Add Pulsar Auth support for the Pulsar SQL > > > https://github.com/apache/pulsar/pull/15571 > > > > > > And some features are blocked in the review stage, but they are > powerful > > > improvements for Pulsar > > > > > > PIP 37: Support chunking with Shared subscription > > > https://github.com/apache/pulsar/pull/16202 > > > PIP-166: Function add MANUAL delivery semantics > > > https://github.com/apache/pulsar/pull/16279 > > > > > > You can find the complete change list in 2.11.0 at > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+milestone%3A2.11.0+-label%3Arelease%2F2.10.1+-label%3Arelease%2F2.10.2 > > > > > > And maybe I missed some important in-progress PRs, please let me know > if > > it > > > should be a blocker of the 2.11.0 release. > > > > > > It's a good time to discuss the target time of the 2.11.0 release. > > > I think we can leave 2 weeks to complete the in-progress PRs and 2 > weeks > > to > > > accept bug fixes. > > > And target the 2.11.0 release in mid-August. > > > > > > Please let me know what you think. > > > > > > Thanks, > > > Penghui > > > > >
Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value
Any comments ?
FYI I have updated the PIP after addressing some feedback on the PR:
- no more need to create a dummy Dispatcher
- now we are reading entries in batches
I would like to see this in 2.11 please
Enrico
Il giorno gio 14 lug 2022 alle ore 09:34 Enrico Olivelli
ha scritto:
>
> Hello,
> this is a PIP to implement a tool to analyse the subscription backlog
>
> Link: https://github.com/apache/pulsar/issues/16597
> Prototype: https://github.com/apache/pulsar/pull/16545
>
> Below you can find the proposal (I will amend the GH issue while we
> discuss, as usual)
>
> Enrico
>
> Motivation
>
> Currently there is no way to have a accurate backlog for a subscription:
>
> you have only the number of "entries", not messages
> server side filters (PIP-105) may filter out some messages
>
> Having the number of entries is sometimes not enough because with
> batch messages the amount of work on the Consumers is proportional to
> the number of messages, that may vary from entry to entry.
>
> Goal
>
> The idea of this patch is to provide a dedicate API (REST,
> pulsar-admin, and Java PulsarAdmin) to "analise" a subscription and
> provide detailed information about that is expected to be delivered to
> Consumers.
>
> The operation will be quite expensive because we have to load the
> messages from storage and pass them to the filters, but due to the
> dynamic nature of Pulsar subscriptions there is no other way to have
> this value.
>
> One good strategy to do monitoring/alerting is to setup alerts on the
> usual "stats" and use this new API to inspect the subscription deeper,
> typically be issuing a manual command.
>
> API Changes
>
> internal ManagedCursor API:
>
> CompletableFuture scan(Predicate condition, long
> maxEntries, long timeOutMs);
>
> This method scans the Cursor from the lastMarkDelete position to the tail.
> There is a time limit and a maxEntries limit, these are needed in
> order to prevent huge (and useless) scans.
> The Predicate can stop the scan, if it doesn't want to continue the
> processing for some reasons.
>
> New REST API:
>
> @GET
>
> @Path("/{tenant}/{namespace}/{topic}/subscription/{subName}/analiseBacklog")
> @ApiOperation(value = "Analyse a subscription, by scanning all the
> unprocessed messages")
>
> public void analiseSubscriptionBacklog(
>@Suspended final AsyncResponse asyncResponse,
> @ApiParam(value = "Specify the tenant", required = true)
> @PathParam("tenant") String tenant,
> @ApiParam(value = "Specify the namespace", required = true)
> @PathParam("namespace") String namespace,
> @ApiParam(value = "Specify topic name", required = true)
> @PathParam("topic") @Encoded String encodedTopic,
> @ApiParam(value = "Subscription", required = true)
> @PathParam("subName") String encodedSubName,
> @ApiParam(value = "Is authentication required to perform
> this operation")
> @QueryParam("authoritative") @DefaultValue("false")
> boolean authoritative) {
>
> API response model:
>
> public class AnaliseSubscriptionBacklogResult {
> private long entries;
> private long messages;
>
> private long filterRejectedEntries;
> private long filterAcceptedEntries;
> private long filterRescheduledEntries;
>
> private long filterRejectedMessages;
> private long filterAcceptedMessages;
> private long filterRescheduledMessages;
>
> private boolean aborted;
>
> The response contains "aborted=true" is the request has been aborted
> by some internal limitations, like a timeout or the scan hit the max
> number of entries.
> We are not going to provide more details about the reason of the stop.
> It will make the API too detailed and harder to maintain. Also, in the
> logs of the broker you will find the details.
>
> New PulsarAdmin API:
>
> /**
> * Analise subscription backlog.
> * This is a potentially expensive operation, as it requires
> * to read the messages from storage.
> * This function takes into consideration batch messages
> * and also Subscription filters.
> * @param topic
> *Topic name
> * @param subscriptionName
> *the subscription
> * @return an accurate analysis of the backlog
> * @throws PulsarAdminException
> *Unexpected error
> */
> AnaliseSubscriptionBacklogResult analiseSubscriptionBacklog(String
> topic, String subscriptionName)
> throws PulsarAdminException;
>
> /**
> * Analise subscription backlog.
> * This is a potentially expensive operation, as it requires
> * to read the messages from storage.
> * This function takes into consideration batch messages
> * and also Subscription filters.
> * @param topic
> *Topic name
> * @param subscriptionName
> *the subscription
> * @return an accurate analysis of th
[GitHub] [pulsar] tisonkun added a comment to the discussion: How to build java-test-image on Apple M1?
GitHub user tisonkun added a comment to the discussion: How to build java-test-image on Apple M1? It seems that upstream doesn't support Apple M1 and the dockerfile-maven-plugin has been abandon. See also https://github.com/spotify/dockerfile-maven/issues/394. It would be better to switch to another plugin. GitHub link: https://github.com/apache/pulsar/discussions/16642#discussioncomment-3169994 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value
This should be really helpful to monitor real backlog in production
environments.
Currently we are struck at "entries" count but doesn't describe well how
much real message lag we have.
+1 excited to see it work on our production environments
Diego Salvi
Il giorno lun 18 lug 2022 alle ore 11:18 Enrico Olivelli <
eolive...@gmail.com> ha scritto:
> Any comments ?
>
> FYI I have updated the PIP after addressing some feedback on the PR:
> - no more need to create a dummy Dispatcher
> - now we are reading entries in batches
>
> I would like to see this in 2.11 please
>
> Enrico
>
> Il giorno gio 14 lug 2022 alle ore 09:34 Enrico Olivelli
> ha scritto:
> >
> > Hello,
> > this is a PIP to implement a tool to analyse the subscription backlog
> >
> > Link: https://github.com/apache/pulsar/issues/16597
> > Prototype: https://github.com/apache/pulsar/pull/16545
> >
> > Below you can find the proposal (I will amend the GH issue while we
> > discuss, as usual)
> >
> > Enrico
> >
> > Motivation
> >
> > Currently there is no way to have a accurate backlog for a subscription:
> >
> > you have only the number of "entries", not messages
> > server side filters (PIP-105) may filter out some messages
> >
> > Having the number of entries is sometimes not enough because with
> > batch messages the amount of work on the Consumers is proportional to
> > the number of messages, that may vary from entry to entry.
> >
> > Goal
> >
> > The idea of this patch is to provide a dedicate API (REST,
> > pulsar-admin, and Java PulsarAdmin) to "analise" a subscription and
> > provide detailed information about that is expected to be delivered to
> > Consumers.
> >
> > The operation will be quite expensive because we have to load the
> > messages from storage and pass them to the filters, but due to the
> > dynamic nature of Pulsar subscriptions there is no other way to have
> > this value.
> >
> > One good strategy to do monitoring/alerting is to setup alerts on the
> > usual "stats" and use this new API to inspect the subscription deeper,
> > typically be issuing a manual command.
> >
> > API Changes
> >
> > internal ManagedCursor API:
> >
> > CompletableFuture scan(Predicate condition, long
> > maxEntries, long timeOutMs);
> >
> > This method scans the Cursor from the lastMarkDelete position to the
> tail.
> > There is a time limit and a maxEntries limit, these are needed in
> > order to prevent huge (and useless) scans.
> > The Predicate can stop the scan, if it doesn't want to continue the
> > processing for some reasons.
> >
> > New REST API:
> >
> > @GET
> >
> @Path("/{tenant}/{namespace}/{topic}/subscription/{subName}/analiseBacklog")
> > @ApiOperation(value = "Analyse a subscription, by scanning all the
> > unprocessed messages")
> >
> > public void analiseSubscriptionBacklog(
> >@Suspended final AsyncResponse asyncResponse,
> > @ApiParam(value = "Specify the tenant", required = true)
> > @PathParam("tenant") String tenant,
> > @ApiParam(value = "Specify the namespace", required = true)
> > @PathParam("namespace") String namespace,
> > @ApiParam(value = "Specify topic name", required = true)
> > @PathParam("topic") @Encoded String encodedTopic,
> > @ApiParam(value = "Subscription", required = true)
> > @PathParam("subName") String encodedSubName,
> > @ApiParam(value = "Is authentication required to perform
> > this operation")
> > @QueryParam("authoritative") @DefaultValue("false")
> > boolean authoritative) {
> >
> > API response model:
> >
> > public class AnaliseSubscriptionBacklogResult {
> > private long entries;
> > private long messages;
> >
> > private long filterRejectedEntries;
> > private long filterAcceptedEntries;
> > private long filterRescheduledEntries;
> >
> > private long filterRejectedMessages;
> > private long filterAcceptedMessages;
> > private long filterRescheduledMessages;
> >
> > private boolean aborted;
> >
> > The response contains "aborted=true" is the request has been aborted
> > by some internal limitations, like a timeout or the scan hit the max
> > number of entries.
> > We are not going to provide more details about the reason of the stop.
> > It will make the API too detailed and harder to maintain. Also, in the
> > logs of the broker you will find the details.
> >
> > New PulsarAdmin API:
> >
> > /**
> > * Analise subscription backlog.
> > * This is a potentially expensive operation, as it requires
> > * to read the messages from storage.
> > * This function takes into consideration batch messages
> > * and also Subscription filters.
> > * @param topic
> > *Topic name
> > * @param subscriptionName
> > *the subscription
> > * @return an accurate analysis of the backlog
> > * @throws PulsarAdminException
> > *Unexpected erro
Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value
+1, good work, I think it's very valuable for users for monitoring
purposes.
Nicolò Boschi
Il giorno lun 18 lug 2022 alle ore 11:47 Lothruin Mirwen <
lothruin.mir...@gmail.com> ha scritto:
> This should be really helpful to monitor real backlog in production
> environments.
> Currently we are struck at "entries" count but doesn't describe well how
> much real message lag we have.
>
> +1 excited to see it work on our production environments
>
> Diego Salvi
>
> Il giorno lun 18 lug 2022 alle ore 11:18 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
> > Any comments ?
> >
> > FYI I have updated the PIP after addressing some feedback on the PR:
> > - no more need to create a dummy Dispatcher
> > - now we are reading entries in batches
> >
> > I would like to see this in 2.11 please
> >
> > Enrico
> >
> > Il giorno gio 14 lug 2022 alle ore 09:34 Enrico Olivelli
> > ha scritto:
> > >
> > > Hello,
> > > this is a PIP to implement a tool to analyse the subscription backlog
> > >
> > > Link: https://github.com/apache/pulsar/issues/16597
> > > Prototype: https://github.com/apache/pulsar/pull/16545
> > >
> > > Below you can find the proposal (I will amend the GH issue while we
> > > discuss, as usual)
> > >
> > > Enrico
> > >
> > > Motivation
> > >
> > > Currently there is no way to have a accurate backlog for a
> subscription:
> > >
> > > you have only the number of "entries", not messages
> > > server side filters (PIP-105) may filter out some messages
> > >
> > > Having the number of entries is sometimes not enough because with
> > > batch messages the amount of work on the Consumers is proportional to
> > > the number of messages, that may vary from entry to entry.
> > >
> > > Goal
> > >
> > > The idea of this patch is to provide a dedicate API (REST,
> > > pulsar-admin, and Java PulsarAdmin) to "analise" a subscription and
> > > provide detailed information about that is expected to be delivered to
> > > Consumers.
> > >
> > > The operation will be quite expensive because we have to load the
> > > messages from storage and pass them to the filters, but due to the
> > > dynamic nature of Pulsar subscriptions there is no other way to have
> > > this value.
> > >
> > > One good strategy to do monitoring/alerting is to setup alerts on the
> > > usual "stats" and use this new API to inspect the subscription deeper,
> > > typically be issuing a manual command.
> > >
> > > API Changes
> > >
> > > internal ManagedCursor API:
> > >
> > > CompletableFuture scan(Predicate condition, long
> > > maxEntries, long timeOutMs);
> > >
> > > This method scans the Cursor from the lastMarkDelete position to the
> > tail.
> > > There is a time limit and a maxEntries limit, these are needed in
> > > order to prevent huge (and useless) scans.
> > > The Predicate can stop the scan, if it doesn't want to continue the
> > > processing for some reasons.
> > >
> > > New REST API:
> > >
> > > @GET
> > >
> >
> @Path("/{tenant}/{namespace}/{topic}/subscription/{subName}/analiseBacklog")
> > > @ApiOperation(value = "Analyse a subscription, by scanning all the
> > > unprocessed messages")
> > >
> > > public void analiseSubscriptionBacklog(
> > >@Suspended final AsyncResponse asyncResponse,
> > > @ApiParam(value = "Specify the tenant", required = true)
> > > @PathParam("tenant") String tenant,
> > > @ApiParam(value = "Specify the namespace", required = true)
> > > @PathParam("namespace") String namespace,
> > > @ApiParam(value = "Specify topic name", required = true)
> > > @PathParam("topic") @Encoded String encodedTopic,
> > > @ApiParam(value = "Subscription", required = true)
> > > @PathParam("subName") String encodedSubName,
> > > @ApiParam(value = "Is authentication required to perform
> > > this operation")
> > > @QueryParam("authoritative") @DefaultValue("false")
> > > boolean authoritative) {
> > >
> > > API response model:
> > >
> > > public class AnaliseSubscriptionBacklogResult {
> > > private long entries;
> > > private long messages;
> > >
> > > private long filterRejectedEntries;
> > > private long filterAcceptedEntries;
> > > private long filterRescheduledEntries;
> > >
> > > private long filterRejectedMessages;
> > > private long filterAcceptedMessages;
> > > private long filterRescheduledMessages;
> > >
> > > private boolean aborted;
> > >
> > > The response contains "aborted=true" is the request has been aborted
> > > by some internal limitations, like a timeout or the scan hit the max
> > > number of entries.
> > > We are not going to provide more details about the reason of the stop.
> > > It will make the API too detailed and harder to maintain. Also, in the
> > > logs of the broker you will find the details.
> > >
> > > New PulsarAdmin API:
> > >
> > > /**
> > > * Analise subscription backlog.
> > > * This is a potentially expe
Re: [DISCUSS] PIP-180: Shadow Topic, an alternative way to support readonly topic ownership.
> > For an extra field in Command Send, as a matter of fact, there is already > an > extra field called marker introduced in Command Send to improved logic for > pausing replicated subscription snapshots when no traffic (PR #11922). I > believe it's introduced by similar reason. I looked at that PR and I couldn't find a protocol change there. In general, that approach is not sustainable. If for every feature we will allow ourselves to change the public protocol, in 2-3 years' time we'll end up with 5 extra fields, all intended for internal purposes. I understand it's more engineering time and more code, but in my opinion: 1. It will be easier to reason when reading the code. It's easier to grasp consuming a topic in the shadow broker and applicatively filling the cache with it, than using a replicator that was intended for the purposes of data replication e2e and modifying it only to go to the cache, and not write it. It's less entanglement IMO. 2. You keep the encapsulation and make it easier for client implementations and people reasoning about the protocol. IMO changing the protocol requires "hitting a wall" and having no other solution. On Sun, Jul 17, 2022 at 5:47 PM Haiting Jiang wrote: > Hi Asafa, > > On 2022/07/13 08:15:59 Asaf Mesika wrote: > > Thanks for the detailed answer. I have a few follow-up questions: > > > > 1. Can you clarify how data is flowing exactly. Specifically, from source > > topic broker to shadow topic broker? I tried to follow the PIP again to > > understand that, especially the overview but couldn't figure it out. > > Also if you can while you're at it, how do you do flow control? I mean, > you > > stream all messages from source topic to shadow topic broker - there will > > be some cases it will not have any more room in memory. > > Once data arrives at the shadow topic, where does it go exactly? > > A simple answer to these questions would be it's the same as > geo-replication > except that shadow topic don't actually write entry to BK. > The normal data flow goes by this: > - User producer client > - Source topic broker cache > - Bookkeeper (Normal topic finishes here) > - Shadow replicator in source topic broker (As a consumer and re-producer) > - Shadow topic broker cache. > > The flow control settings for replicator can be configured by > `replicatorDispatchRate` > in `Policies` and `TopicPolicies`. > > We don't need to worry about the impact of memory management here, since > the > EntryCache implementation has its cache eviction policy, by time and space. > > > > > 2. You wrote it was the easiest way to populate the entry cache, but > there > > is a very big penalty to pay here: contaminating the protocol. > > The way I see it, the protocol should be sacred. > > Most importantly, the internal details of a broker-to-broker should never > > leak to the client. > > In this case, the Command Send which is what the producer's clients are > > using now has an extra field, called shadow_message_id. This field is an > > internal detail of broker-to-broker communication and thus should never > > leak outside. > > > > I'm not that fluent in the Pulsar codebase to give a decent alternative > of > > implementation. I can only point out that this IMO is too high a price to > > pay. > > > > One possible solution comes to mind for brokers to broker RPC without > > interfering with the protocol: open a consumer in shadow topic broker and > > consume the messages and from there fill up the cache of managed ledger? > > > > > > > Maybe it's a bit of over exaggerated here. Software development is all > about > trade-off and resources are constantly limited. I totally agree that we > should > minimize the change to the protocol and keep it clean and simple. IMO, > there > would be no meaning talking about the price is too high without considering > the cost of the alternatives. In this case, the alternative would be open a > consumer in shadow topic. It's quite a classic case of choosing "push vs > pull" > model to solve message syncing problems. And I believe both could solve it > with about the same effort if we are building this from scratch. But the > currently situation is that the "push" model is already well implemented > and > verified under a lot of production usages. If we ignore the code > reusability > here, then we need to manage the whole package of the consumers, like the > life > cycle, the configurations, running-threads, buffer size limitations, flow > controls, metrics, stats, etc. I think this price would be too high to > implement another similar "pull" model for broker-to-broker communication. > > > For an extra field in Command Send, as a matter of fact, there is already > an > extra field called marker introduced in Command Send to improved logic for > pausing replicated subscription snapshots when no traffic (PR #11922). I > believe it's introduced by similar reason. > > > > 3. I understand that you want to minimize consumption latency, hence > you're >
Re: [DISCUSS] PIP-180: Shadow Topic, an alternative way to support readonly topic ownership.
Hi Penghui, > Can we use `message_id` directly? It's a message ID from the source > cluster, it can be used > by the shadow topic feature but can also be used by other features in the > feature. > Sure, `message_id` is better. > > there are not much difference to reset the cursor in persistent > replicator or > in the raw reader. > > Do you want to control it in the shadow topic? or by users? > Maybe we can make some changes to the replicator to support skip backlogs. > +1. We can add new configuration like `shadowReplicatorAutoResetBacklogEntries`. Once backlog entry number exceeded this threshold, the shadow replicator will reset the cursor to LATEST automatically. Thanks, Haiting On 2022/07/18 00:38:37 PengHui Li wrote: > Hi Haiting, > > > For this point, like I wrote in the reply to Asaf, I still think the cost > of > maintaining the consumers would be a little too high, especially the > lifecycle > management part. It's quite complicated in replicator already. > > Yes, I see it. Makes sense. Without the replicator, it looks like we are > trying to implement > a very similar function to replicate data. The only difference is which > broker to > maintain the consumer. > > Can we use `message_id` directly? It's a message ID from the source > cluster, it can be used > by the shadow topic feature but can also be used by other features in the > feature. > > > there are not much difference to reset the cursor in persistent > replicator or > in the raw reader. > > Do you want to control it in the shadow topic? or by users? > Maybe we can make some changes to the replicator to support skip backlogs. > > Thanks, > Penghui > > On Sun, Jul 17, 2022 at 10:57 PM Haiting Jiang > wrote: > > > Hi Penghui, > > > > > I had a conversation with Asaf, looks like we can use the RawReader for a > > > shadow topic internally to consume > > > messages from the original topic. So that we don't need to introduce > > > `shadow_message_id`, and don't need to introduce > > > changes to the replicator. > > > > For this point, like I wrote in the reply to Asaf, I still think the cost > > of > > maintaining the consumers would be a little too high, especially the > > lifecycle > > management part. It's quite complicated in replicator already. > > > > > > > > > > And using a persistent replicator might cause the very old messages to > > add > > > to the entry cache of the shadow topic? > > > Maybe there are some backlogs of the replicator. > > > > Yes, we should skip old backlogs for shadow replicator. But it seems that > > there are not much difference to reset the cursor in persistent replicator > > or > > in the raw reader. > > > > BR, > > Haiting > > > > > > On 2022/07/13 08:29:34 PengHui Li wrote: > > > Hi Haiting, > > > > > > I had a conversation with Asaf, looks like we can use the RawReader for a > > > shadow topic internally to consume > > > messages from the original topic. So that we don't need to introduce > > > `shadow_message_id`, and don't need to introduce > > > changes to the replicator. > > > > > > And using a persistent replicator might cause the very old messages to > > add > > > to the entry cache of the shadow topic? > > > Maybe there are some backlogs of the replicator. > > > > > > Thanks, > > > Penghui > > > > > > On Wed, Jul 13, 2022 at 4:16 PM Asaf Mesika > > wrote: > > > > > > > Thanks for the detailed answer. I have a few follow-up questions: > > > > > > > > 1. Can you clarify how data is flowing exactly. Specifically, from > > source > > > > topic broker to shadow topic broker? I tried to follow the PIP again to > > > > understand that, especially the overview but couldn't figure it out. > > > > Also if you can while you're at it, how do you do flow control? I > > mean, you > > > > stream all messages from source topic to shadow topic broker - there > > will > > > > be some cases it will not have any more room in memory. > > > > Once data arrives at the shadow topic, where does it go exactly? > > > > > > > > 2. You wrote it was the easiest way to populate the entry cache, but > > there > > > > is a very big penalty to pay here: contaminating the protocol. > > > > The way I see it, the protocol should be sacred. > > > > Most importantly, the internal details of a broker-to-broker should > > never > > > > leak to the client. > > > > In this case, the Command Send which is what the producer's clients are > > > > using now has an extra field, called shadow_message_id. This field is > > an > > > > internal detail of broker-to-broker communication and thus should never > > > > leak outside. > > > > > > > > I'm not that fluent in the Pulsar codebase to give a decent > > alternative of > > > > implementation. I can only point out that this IMO is too high a price > > to > > > > pay. > > > > > > > > One possible solution comes to mind for brokers to broker RPC without > > > > interfering with the protocol: open a consumer in shadow topic broker > > and > > > > consume the messages and from th
[GitHub] [pulsar] tisonkun edited a discussion: How to build java-test-image on Apple M1?
GitHub user tisonkun edited a discussion: How to build java-test-image on Apple M1? cc @lhotari Confirmed as a potential improvement work. Currently, we don't support build the image on Apple M1. See also #16652 for more information. Today I'm trying to build java-test-image on Apple M1. Run `build/build_java_test_image.sh` and get: ``` [ERROR] Failed to execute goal com.spotify:dockerfile-maven-plugin:1.4.13:build (default) on project pulsar-docker-image: Could not build image: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.UnsatisfiedLinkError: could not load FFI provider com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.Provider: ExceptionInInitializerError: Can't overwrite cause with java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/n2/rnpcrm6x3mb2c1jn1dkdb33cgn/T/jffi13338179526049207391.dylib [ERROR] at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393) [ERROR] at java.base/java.lang.Runtime.load0(Runtime.java:755) [ERROR] at java.base/java.lang.System.load(System.java:1953) [ERROR] at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:371) [ERROR] at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258) [ERROR] at com.kenai.jffi.internal.StubLoader.(StubLoader.java:444) [ERROR] at java.base/java.lang.Class.forName0(Native Method) [ERROR] at java.base/java.lang.Class.forName(Class.java:467) [ERROR] at com.kenai.jffi.Init.load(Init.java:68) [ERROR] at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49) [ERROR] at com.kenai.jffi.Foreign$InstanceHolder.(Foreign.java:45) [ERROR] at com.kenai.jffi.Foreign.getInstance(Foreign.java:103) [ERROR] at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242) [ERROR] at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237) [ERROR] at com.kenai.jffi.Type.resolveSize(Type.java:155) [ERROR] at com.kenai.jffi.Type.size(Type.java:138) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:187) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.AbstractRuntime.(AbstractRuntime.java:48) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.NativeRuntime.(NativeRuntime.java:66) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.NativeRuntime.(NativeRuntime.java:41) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.(NativeRuntime.java:62) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:58) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.jffi.Provider.(Provider.java:29) [ERROR] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) [ERROR] at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [ERROR] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) [ERROR] at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128) [ERROR] at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347) [ERROR] at java.base/java.lang.Class.newInstance(Class.java:645) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.(FFIProvider.java:57) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35) [ERROR] at com.spotify.docker.client.shaded.jnr.ffi.LibraryLoader.create(LibraryLoader.java:73) [ERROR] at com.spotify.docker.client.shaded.jnr.unixsocket.Native.(Native.java:76) [ERROR] at com.spotify.docker.client.shaded.jnr.unixsocket.UnixSocketChannel.(UnixSocketChannel.java:101) [ERROR] at com.spotify.docker.client.shaded.jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60) [ERROR] at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:69) [ERROR] at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:44) [ERROR] at com.spotify.docker.client.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118) [ERROR] at co
[GitHub] [pulsar] tisonkun added a comment to the discussion: What's the best practice to build java-test-image?
GitHub user tisonkun added a comment to the discussion: What's the best practice to build java-test-image? The first script works! Thank you. GitHub link: https://github.com/apache/pulsar/discussions/16633#discussioncomment-3170863 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: What's the best practice to build java-test-image?
GitHub user tisonkun added a comment to the discussion: What's the best practice to build java-test-image? @lhotari the next time you can answer in a different thread so that I can mark your response as the answer :) GitHub link: https://github.com/apache/pulsar/discussions/16633#discussioncomment-3170869 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on a diff in pull request #277: Bump Apache Pulsar version to 2.9.3
lhotari commented on code in PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277#discussion_r923368906 ## charts/pulsar/Chart.yaml: ## @@ -18,7 +18,7 @@ # apiVersion: v2 -appVersion: "2.9.2" +appVersion: "2.9.3" description: Apache Pulsar Helm chart for Kubernetes name: pulsar version: 2.9.3 Review Comment: The chart version will also have to be bumped. The chart version is independent of Pulsar's version. The current version scheme for the chart version is confusing since it seems to match the Pulsar version, but that shouldn't be the goal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on a diff in pull request #277: Bump Apache Pulsar version to 2.9.3
lhotari commented on code in PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277#discussion_r923370490 ## charts/pulsar/Chart.yaml: ## @@ -18,7 +18,7 @@ # apiVersion: v2 -appVersion: "2.9.2" +appVersion: "2.9.3" description: Apache Pulsar Helm chart for Kubernetes name: pulsar version: 2.9.3 Review Comment: There's PR #200 which improves the situation, but that stalled since I didn't receive feedback from @sijie . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] mattisonchao commented on a diff in pull request #277: Bump Apache Pulsar version to 2.9.3
mattisonchao commented on code in PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277#discussion_r923376333 ## charts/pulsar/Chart.yaml: ## @@ -18,7 +18,7 @@ # apiVersion: v2 -appVersion: "2.9.2" +appVersion: "2.9.3" description: Apache Pulsar Helm chart for Kubernetes name: pulsar version: 2.9.3 Review Comment: Thanks for your explanation. I changed the chart version to 2.9.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] mattisonchao commented on pull request #277: Bump Apache Pulsar version to 2.9.3
mattisonchao commented on PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277#issuecomment-1187442014 > pulsar_metadata.image.tag fixed, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on pull request #277: Bump Apache Pulsar version to 2.9.3
lhotari commented on PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277#issuecomment-1187443859 I guess tags referencing `2.9.2` should also be updated in `.ci/clusters/values-pulsar-image.yaml` file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] mattisonchao commented on pull request #277: Bump Apache Pulsar version to 2.9.3
mattisonchao commented on PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277#issuecomment-1187452537 > I guess tags referencing `2.9.2` should also be updated in `.ci/clusters/values-pulsar-image.yaml` file. It might be better to remove the tags from that file so that it's not necessary to update the file for each release. I'm not sure about it. maybe I can write the shell to help bump the version with one command. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] mattisonchao merged pull request #277: Bump Apache Pulsar version to 2.9.3
mattisonchao merged PR #277: URL: https://github.com/apache/pulsar-helm-chart/pull/277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new pull request, #148: Remove locales we don't support yet
tisonkun opened a new pull request, #148: URL: https://github.com/apache/pulsar-site/pull/148 Otherwise when a user change the language but found the result is just English, it's not a good experience. If we don't support it, we don't mention it. We can also add them back when there is real translated contents. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun commented on pull request #148: Remove locales we don't support yet
tisonkun commented on PR #148: URL: https://github.com/apache/pulsar-site/pull/148#issuecomment-1188492252 @Anonymitaet thanks for your reaction. Since we don't run any CI tasks to dry run and verify the build, may I ask how to verify all changes looks good locally? I've run `./preview.sh`, but it only generates English version. cc @urfreespace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on pull request #148: Remove locales we don't support yet
Anonymitaet commented on PR #148: URL: https://github.com/apache/pulsar-site/pull/148#issuecomment-1188567099 @tisonkun double check: do you want to preview localized pages? Seems that they can not be previewed, correct? @urfreespace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun commented on pull request #148: Remove locales we don't support yet
tisonkun commented on PR #148: URL: https://github.com/apache/pulsar-site/pull/148#issuecomment-1188570313 @Anonymitaet I make this patch with intention to avoid display language options which we don't have a real translated content. However, I don't know whether I change it correctly. I'd like to verify it locally, no matter is "localized pages" or other approach. For example, when I execute `/preview.sh`, I get: https://user-images.githubusercontent.com/18818196/179662100-bbefdc2d-d296-4251-ad81-a13344864ab8.png";> .. it seems other options are populated in a following step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] youzipi opened a new issue, #473: fail to create environment as broker throw error: `io.netty.handler.codec.TooLongFrameException`
youzipi opened a new issue, #473: URL: https://github.com/apache/pulsar-manager/issues/473 when i create new environment, got error: `This environment is error. Please check it` container log: ``` pulsar_1 | 2022-07-19T04:57:04,910+ [pulsar-io-31-4] WARN org.apache.pulsar.broker.service.ServerCnx - [/172.31.0.3:40930] Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 1195725860 - discarded pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:507) pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:493) pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:377) pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:423) pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:333) pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) pulsar_1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) pulsar_1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) pulsar_1 | at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) pulsar_1 | at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) pulsar_1 | at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) pulsar_1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) pulsar_1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) pulsar_1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) pulsar_1 | at java.base/java.lang.Thread.run(Thread.java:829) ``` i found this issue: https://github.com/apache/pulsar/issues/15688 ``` # The maximum netty frame size in bytes. Any message received larger than this will be rejected. The default value is 5MB. nettyMaxFrameSizeBytes=5253120 5,253,120 1,195,725,860 # 1G ?? ``` is manager backend call the wrong interface? why the data size is 1G? this is our test environment with 100 topic and not too much msgs. https://github.com/apache/pulsar-manager/blob/master/src/main/java/org/apache/pulsar/manager/controller/EnvironmentsController.java#L124 ``` pulsarAdminService.clusters(environmentEntity.getBroker()).getClusters(); ``` my configuration: ```yaml # docker-compose.yml --- version: "3.7" services: pulsar: image: apachepulsar/pulsar:2.10.1 environment: - PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g - PULSAR_PREFIX_defaultRetentionTimeInMinutes=14400 - PULSAR_PREFIX_defaultRetentionSizeInMB=1000 # - PULSAR_PREFIX_httpServerEnabled=true - PULSAR_PREFIX_bookkeeperClientExposeStatsToPrometheus=true - PULSAR_PREFIX_transactionCoordinatorEnabled=true # - PULSAR_PREFIX_nettyMaxFrameSizeBytes=5,253,120 # default # - PULSAR_PREFIX_nettyMaxFrameSizeBytes=1,195,725,860 # command: bin/pulsar standalone command: > /bin/bash -c "bin/apply-config-from-env.py conf/standalone.conf && bin/pulsar standalone" hostname: pulsar ports: - "8090:8080" - "6650:6650" restart: unless-stopped volumes: - "./data/:/pulsar/data" dashboard: image: apachepulsar/pulsar-manager:v0.3.0 ports: - "8091:9527" - "7750:7750" depends_on:
[GitHub] [pulsar-site] Anonymitaet commented on pull request #148: Remove locales we don't support yet
Anonymitaet commented on PR #148: URL: https://github.com/apache/pulsar-site/pull/148#issuecomment-1188605008 If you want to preview changes locally, then `./preview.sh` is OK and enough -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun commented on pull request #148: Remove locales we don't support yet
tisonkun commented on PR #148: URL: https://github.com/apache/pulsar-site/pull/148#issuecomment-1188619227 @Anonymitaet thank you. Then I'm waiting for more review comments. The original purpose to make this change is "If we don't support it, we don't mention it". Even if we _have a plan_ to offer those translated contents, we can re-add the language switch option when they're present. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] youzipi commented on issue #473: fail to create environment as broker throw error: `io.netty.handler.codec.TooLongFrameException`
youzipi commented on issue #473: URL: https://github.com/apache/pulsar-manager/issues/473#issuecomment-1188671403 use the wrong url found this in manager backend log: ```js Create Pulsar Admin instance. url=http://pulsar:6650, authPlugin=, authParams=, tlsAllowInsecureConnection=false, tlsTrustCertsFilePath=, tlsEnableHostnameVerification=false ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
