Ignore this, looked further and it actually does double resolution. sheesh
On Wed, Dec 15, 2021 at 10:14 AM Ivan Kelly wrote:
>
> Maybe this has been discussed elsewhere, but if possible, we should
> add a rule that to block any future log4shell type attack area.
> In human ter
Maybe this has been discussed elsewhere, but if possible, we should
add a rule that to block any future log4shell type attack area.
In human terms, the rule should be:
Disallow any {log,logger,LOG}.* with a non-const first argument.
-Ivan
On Wed, Dec 15, 2021 at 7:15 AM Enrico Olivelli wrote:
>
Hi Tom,
The images seem to be stripped by the mailing list. Perhaps post them to
the PR.
Cheers,
Ivan
On Tue, Sep 7, 2021 at 1:30 PM tom lee wrote:
> Thanks @Penghui for your comments.
>
> I added timezone in log pattern. Do other guys have any questions or
> concerns about this change? Lookin
responses inline.
> Marking a worker as unschedulable in a persisted manner is non trivial.
> We would have to introduce a new internal topic to track this or leverage
> something like ZK. I didn't think it was worthwhile for us to add these
> new resources / potential dependencies for this feat
> When the leader receives a request to drain a worker, it must first mark
> the worker as in the process to be drained i.e. blacklist the worker so
> that no new assignments can be assigned to it. We can perhaps just save the
> blacklist in memory. The worker should then create a new scheduling in
> My doubt here is if that alone is enough of a reason to require users
> to switch, or if we should instead wait for JDK 17 and fast track to
> that.
The biggest thing isn't to get people on to a new one. It's to wean
them off of jdk8. jdk11 achieves this.
-Ivan
> * Pulsar client should remain on JDK 8
I'm sure this was discussed in the past, but who is the JDK8 support
being maintained for?
And wouldn't they just continue to use the 2.8 client which does support 8?
-Ivan
> Sorry for not being clear. My question is, when do we start requiring
> JDK11+ so that we can start using newer Java features in the Pulsar code
> base?
I think this is using jdk11+ for the build and the broker should be fine.
The client, and modules depended on by the client may be a different
> My two cents as a mere contributor who once wrote a PIP. I would love to
> see PIPs as a versioned document in git instead of the Wiki. This would
> provide history and context, make it easy to comment and propose fixes and
> enhancements which is currently not possible with the Wiki. The PMC wou
@Matteo Merli
I've been playing around with where to put the atInfo() and event ID
and had the following thoughts.
Putting atInfo(Enum event) at the point of event creation is a
suggestion so that if that event is not enabled at that level, then a
dummy attribute accumulator will be returned and t
Hi folks,
Rajan's opinion would be particularly useful here as they appears to
have done most of the TooManyRequests work in the past.
So, the original TooManyRequests works well in the case that you have
a lot of topics and loads of lookups and PMR come in to the broker. In
this case, most reque
b.com/apache/pulsar/pull/11627/>Thanks,Rajan*
>
> On Mon, Aug 9, 2021 at 12:54 AM Ivan Kelly wrote:
>
> > > Suppose you have about a million topics and your Pulsar cluster goes down
> > > (say, ZK down). ..many millions of producers and consumers are now
> > >
tto:
>
> > +1
> >
> > It makes sense to me. I also encountered TooManyRequests in topic lookup
> > when there’re a lot of topics.
> >
> > It should be retriable instead of a simple error response to client.
> >
> > Thanks,
> > Yunze
> >
>
Hi folks,
I've created a PIP to do some rework on lookup timeouts and retries.
We've had major client incidents recently due to a client with many
many producers, which triggered a herding effect on broker restarts.
This PIP aims to alleviate some of the issues we saw.
In summary, we want to retr
> Suppose you have about a million topics and your Pulsar cluster goes down
> (say, ZK down). ..many millions of producers and consumers are now
> anxiously awaiting the cluster to comeback. .. fun experience for the first
> broker that comes up. Don't ask me how I know, ref blame
> ServerCnx.ja
Inline
> // trimmed discussion about using annotations for documentation
>
> I was checking and it's not possible to have annotations on the enum
> values. We should still have an annotation for the Events enum types
> so that we can both:
> 1. Add context for that event group (eg: broker events,
Inline
> In that scenario,
> should we block or fail fast and let the application decide which is what
> we do today? Also, should we distinguish between the two scenarios, i.e.
> broker sends the error vs client internally throws the error?
While I agree that the client limit is different to the
request to complete. This is set to 30s by default.
However, in this case, it can be much much more. I suspect that fixing
this will result in a lot more people seeing timeouts though :/
-Ivan
On Thu, Aug 5, 2021 at 7:07 PM Ivan Kelly wrote:
>
> Hi folks,
>
> I'm currently digging
Thanks for the detailed feedback Matteo,
Responses inline.
> 1. I like the idea of using enum to identify an event, but I fear
> that the enum name itself will not be, in many cases, as expressive of
> the context as a complete phrase.
I'm not convinced by this. From just eyeballing one of our
Seems like all the feedback is positive. What's the process for moving
the PIP to approved? I don't see anything on the wiki. Nor in the
bylaws (where are they? they're not linked on site menu).
-Ivan
On Wed, Aug 4, 2021 at 5:11 PM Ivan Kelly wrote:
>
> > I wonder how we
Hi folks,
I'm currently digging into a customer issue we've seen where the retry
logic isn't working well. Basically, they have a ton of producers and
a load of partitions, and when they all connect at the same time, they
bury the brokers for a few minutes. I'm looking at this from the
consumer po
> I wonder how we will be able to enforce such way of logging once we are
> done with the switch.
No all logging needs to be done this way. This will coexist with
traditional slf4j logging.
However, hopefully this will prove useful enough that a string
interpolated log will
start to stick out like
Congrats Enrico!
On Tue, Aug 3, 2021 at 12:19 PM Jia Zhai wrote:
>
> Congrats Enrico!
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Tue, Aug 3, 2021 at 6:17 PM Huanli Meng
> wrote:
>
> > Congrats to Enrico.
> >
> > BR// Huanli Meng
> >
> > > On Aug
Hi folks,
I've just created a PIP in the wiki to add structured documented
logging for Pulsar.
More often than not, when debugging an incident in Pulsar we find that
information we need is not logged, at which point we have to resort to
digging into zookeeper directly or rely on gut feelings of pe
> Creating a topic will first check whether the topic already exists.
> The verification will read all topics under the namespace, and then
> traverse these topics to see if the topic already exists.
> When there are a large number of topics under the namespace(about 300,000
> topics),
> less than
> It's a multifaceted issue it's becoming a bottleneck to support the growing
> list of committers.
Please list the facets. Is the issue speed of CI or speed on local builds?
> The build has issues like python2, python 3, c++ being built serially in
> containers separately before being copied to
> If we believe coordinating multiple repos is too difficult. We can
> consider migrate to bazel as a build system with a hosted cache. That can
> give us build speed benefits while maintaining the mono repo structure.
What is the problem motivating this change? From this comment it seems
that it
> To enable a faster development cadence I am proposing the following steps
> to reorg the current pulsar monorepo and test harness.
How is the current repo impeding development cadence?
> Milestone 1
>
> Move doc artifacts to an independent repo.
Moving the documents to a different repo will ac
+1 for moving to actions. It looks to be azure pipelines in
background. It should be able to give us more hermetic testing.
Jenkins has always had issues of old builds hanging around the cluster
and causing unreproducible flakes.
FYI, infra are giving their blessing to people trying it out.
http:/
Joe, Matteo
What was the conclusion of this discussion? Was there a conclusion?
Has the algorithm used in the implementation changed from the initial
draft?
-Ivan
On Fri, May 3, 2019 at 4:00 AM Joe F wrote:
>
> Let me try to explain my suggestion better.
>
> First, about positions in an ordered
+1 from me
✓ sha512 & package sigs are correct
✓ licenses are correct
✓ rat check passes
✓ maven artifacts sigs are correct
✓ unit tests
✓ integration tests
One thing to note is the -src package contains pom.xml.versionsBackup
files which should not be there.
Good work Jerry!
-Ivan
On Wed, May
Hi folks,
+1 from me (binding)
✓ tag matches sha
✓ tag is signed
✓ check sha512
✓ check gpg
✓ test licenses
✓ rat check
✓ -src compiles
✓ tag compiles
✓ compiled tag matches -src
✓ maven artifacts sigs are correct
? unit tests, I have the same flake on
UnAcknowledgedMessagesTimeoutTest,
-1 (binding)
I'm seeing a couple of issues with the RC:
Two tests in pulsar-broker fail consistently:
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
org.apache.pulsar.client.impl.UnAcknowledgedMessagesTimeoutTest.testFailoverSingleAckedPartitionedTopic(org.apache.pulsar.client.imp
> > > > "streaming".
> > > > The "microbatching" vs "streaming" are usually found in the context
> > > > talking about spark streaming vs storm/flink.
> > > > The context is more about how computing engine "scheduli
typo when talking about PIPs. s/sever/several/
-Ivan
On Wed, Mar 13, 2019 at 6:47 AM Matteo Merli wrote:
>
> ## Description:
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-on
Hi folks,
I'm planning to cut a release for 4.8.2 before the end of the week.
The primary motivation is to have an official release with the dlog
read ahead fix [1], which has been requested by a customer.
Does anyone else have changes they'd like pushed into this release?
Best regards,
Ivan
[
> > My replies inline assume the above, so if you have a different view of
> > the general shape let me know.
> >
>
> Yes. We are on the same view of the general shape. I will write down the
> details of my proposal and will share it with the community tomorrow.
Please do. I think there's a lot of
I think we agree on the general shape of the design as follows.
- Transactions store message data somewhere that it is not
materialized to the client immediately.
- On commit, a single message is written to some log
- Commit messages are then written to the topics of the logs in the
transaction, w
> > Transactional acknowledgement also needs to be taken into account
>
> I don't think we have to treat `transactional acknowledgement` as a special
> case. currently `acknowledgment` are actually "append" operations into
> cursor ledgers.
> So the problem set can be reduced as `atomic append` to
Hi Richard,
This is something that has been discussed a few times, but the
outcomes of the discussions never written down. Thanks for starting
the conversation.
Responses to your PIP inline.
> Motivation: Pulsar currently could improve upon their system of sending
> packets of data by implementi
> If so, I would suggest pulling the proposal out of the thread and do a vote
> to make sure everyone in the community are on the same page.
>
> It is okay we don't have to document at this time. We can let you manage
> the process in next bugfix release. Once it is done, propagate the process
> to
> Your proposal works for me.
If it helps, I can volunteer to do the next bugfix release also.
-Ivan
> When it comes to release a minor release, he should :
>
> 1) go through all the issues in current master and pick the issues that are
> required to cherry-pick to branches, and label them with `cherry-pick`
> label.
> 2) go through all the issues label with `cherry-pick` label, and
> cherry-pick
> Can anyone propose a change to current release management? Otherwise it is
> not a scalable solution because not every committer knows how to handle
> this situation clearly.
I agree that not every committer knows this.
What has worked well for me in the past is to have _every_ commit go
to mast
+1 (binding) from me
I checked the following:
- sigs, sha (GOOD)
- rat, compile, test (GOOD)
- licenses (GOOD)
- start standalone (GOOD)
- rpms install (GOOD)
- debs install (GOOD)
- Tested cassandra connector (GOOD)
Great work everyone!
-Ivan
On Fri, Feb 15, 2019 at 9:10 AM Matteo Merli wrote
> Correct. PIP-26 can be implemented in Functions. I believe the last
> discussion in PIP-26 thread kind of agree on functions approach.
> If the community is okay with PIP-26 in functions, I think that is probably
> a good approach to start.
+1 for doing it in functions.
-Ivan
One thing missing from this discussion is details on the motivating
use-case. How many delayed messages per second are we expecting? And
what is the payload size?
> If consumer control the delayed message specific execution time we must
> trust clock of consumer, this can cause delayed message pro
y since it is on a public IP - hence we authenticate at proxy
> level and authorize at the broker, but that’s not the ideal configuration.
>
> On Wed, Jan 9, 2019 at 3:10 AM Ivan Kelly wrote:
>
> > > Assume a role/principal R has permissions to produce on a namespace.
>
> Assume a role/principal R has permissions to produce on a namespace. If
> we don't authenticate at the proxy then anyone (attacker) can say that they
> belong to role R and connect to the proxy, the proxy will forward the role
> name to the broker which will authorize it and allow access. Ins
Hi folks,
The pulsar proxy allows authorization to be configured, which checks
if a role has access to a resource it is trying to access. If it does,
the request is forwarded to the broker. At the broker, authorization
is checked again. So my question is, what is the point of having
authorization
Welcome Penghui!
On Mon, Dec 10, 2018 at 12:11 PM Sijie Guo wrote:
>
> The Apache Pulsar PMC recently extended committer karma to Penghui Li and
> he has accepted.
>
> Penghui has been very active in Pulsar community and also contributed large
> features like Interceptors, Dead Lead Topic, and Del
On Tue, Nov 13, 2018 at 8:23 PM, Rajan Dhabalia wrote:
>> Is there any documentation on how to use it?
>
> I have documented peer-cluster setup at:
> https://github.com/apache/pulsar/wiki/PIP-8:-Pulsar-beyond-1M-topics#how-to-setup-peer-clusters
Great! We should turn this into a cookbook at some
> After, PIP-8, It is now possible to run multiple Pulsar clusters under one
> service URL (going beyond the limits of a what a single ZK can support).
I wasn't aware of that feature. Is there any documentation on how to use it?
-Ivan
Hi Koushik,
> I just got introduced to Apache pulsar and studying it to solve a problem at
> hand.
> As I understand that Pulsar supports 1M of topics, Is there any limitation on
> no.of producers and subscribers that brokers can handle? I have the following
> use cases.
The limit of how many
> Decoded Payload: {"sub":"my-test-subject"}
And sub is role? I guess you didn't use "role" since "sub" is a jwt builtin?
>> key? Where should the private key be stored? Is it PSK?
>>
>
> I would leave that outside the scope of this plugin.
It would be good to mention it in the PIP to give a lit
Looks like a great scheme, but I'd like some more concrete details
about how you see this interacting with Pulsar.
A couple of questions:
How will the token be passed in HTTP? As a header?
What does the concrete payload look like?
How is the server configured to accept this? Is there a public/pr
+1
On Wed, Oct 10, 2018 at 6:51 AM, Sijie Guo wrote:
> +1 for making it mandatory.
>
> - Sijie
>
> On Tue, Oct 9, 2018 at 9:36 PM Matteo Merli wrote:
>
>> Should we ask infra to make the integration tests check to be mandatory?
>>
>> On Tue, Oct 9, 2018 at 9:26 PM Sijie Guo wrote:
>>
>> > Hi al
Hi folks,
I was looking into a user bug, "Offload fails",
https://github.com/apache/pulsar/issues/2697 and it seems to be a
documentation issue.
Namely, the latest stable version of the docs, i.e. what I get when I
click through for 2.1.1-incubating docs
(http://pulsar.apache.org/docs/en/cookbook
Could it be a problem with the cpp job creating everything as root so
then jenkins isn't able to clean up?
-Ivan
On Mon, Oct 1, 2018 at 8:34 PM, Joe Francis wrote:
> We should cleanup all the bk/zk dirs created by unit tests. Almost 16gig of
> junk
>
> On Mon, Oct 1, 2018, 11:31 AM Matteo Merli
> I was looking for information on how to become moderator on a list but
> couldn't find any. Do you have to add new moderators, or is there some
> guide that I'm missing?
You need to open a ticket with INFRA to change moderators.
-Ivan
60 matches
Mail list logo
