2020-02-25 12:00:49 UTC - Roman Popenov: @Sijie Guo yeah, those are run individually. ---- 2020-02-25 14:14:10 UTC - RAMG: @RAMG has joined the channel ---- 2020-02-25 15:47:46 UTC - Graham: @Graham has joined the channel ---- 2020-02-25 17:59:55 UTC - Sijie Guo: ```00:00:17.341 [bookkeeper-ml-workers-OrderedExecutor-1-0:org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl@406] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [schema-validation-enforced/enable-has-schema-mismatch/persistent/test] Created ledger 3 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/Users/roman.popenov/.m2/repository/com/google/protobuf/protobuf-java/3.5.1/protobuf-java-3.5.1.jar) to field java.nio.Buffer.address WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release``` ---- 2020-02-25 18:00:21 UTC - Sijie Guo: I think this is related. ---- 2020-02-25 18:01:01 UTC - Roman Popenov: Ok, I’ll ponder on how to make it work in Java 11 ---- 2020-02-25 18:01:19 UTC - Roman Popenov: I know of a way, but it DOES break in Java 12 and onwards ---- 2020-02-25 18:02:24 UTC - Roman Popenov: I’ll take a closer look later and maybe open a task to revamp the test to be compatible with java 11 and onwards ---- 2020-02-25 18:06:26 UTC - Sijie Guo: thanks! ---- 2020-02-25 20:45:18 UTC - matt_innerspace.io: @matt_innerspace.io has joined the channel ---- 2020-02-25 20:46:56 UTC - matt_innerspace.io: simple question - is it possible to use wildcards to specify multiple topics as input for a function? In my scenario, I want a single function to process thousands of topics. Is there a better way to do this? ---- 2020-02-25 20:47:21 UTC - Devin G. Bost: Yes. You can use the regex feature. ---- 2020-02-25 20:48:12 UTC - Devin G. Bost: Are you using a broker proxy? ---- 2020-02-25 20:48:41 UTC - matt_innerspace.io: currently using standalone to test it out. ---- 2020-02-25 20:48:59 UTC - matt_innerspace.io: regex to specify input for a function? hmm.. didn't notice it in the docs.. i'll try. ---- 2020-02-25 20:49:19 UTC - Devin G. Bost: Okay. There’s an open issue when using the regex feature when using a proxy for the brokers, but it’s being worked on. I don’t think you will be impacted by it. ---- 2020-02-25 20:51:05 UTC - matt_innerspace.io: is there an example for the regex syntax when adding through the pulsar-admin cli? i'm not having any luck with it. ---- 2020-02-25 20:51:25 UTC - Devin G. Bost: There are tests that show the syntax. +1 : matt_innerspace.io ---- 2020-02-25 20:52:25 UTC - Devin G. Bost: e.g. <https://github.com/apache/pulsar/blob/e9083f5592035b4812e086040d4a758379fdd349/pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyTest.java#L193> ---- 2020-02-25 21:02:46 UTC - matt_innerspace.io: i see it, replace `inputs` with `topics-pattern` in the `pulsar-admin` cli tool. i missed it because i'm new and wasn't thinking to look for _topics-pattern_ . +1 : Devin G. Bost ---- 2020-02-25 21:07:12 UTC - matt_innerspace.io: thanks for your help @Devin G. Bost! ---- 2020-02-25 21:35:03 UTC - Devin G. Bost: :slightly_smiling_face: ---- 2020-02-25 22:23:20 UTC - Eugen: Regarding consumption of thousands of topics, that may consume a lot of resources, have a look at my question here: <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1579903006163200> ---- 2020-02-26 02:58:47 UTC - Eugen: Imagine a reader that is interested in only receiving real time messages of some topic, but that also wants to get the latest value as of the time of connecting, for times where there are no new messages for an extended period of time, so that it does not start out empty-handed, waiting for the first real-time message to come in. Currently, we would have to implement it something like this: ```consumer = client.newConsumer()... .startMessageIdInclusive() .subscribe(); MessageId id = consumer.getLastMessageId(); consumer.seek(id);``` I don't think it is possible to do (purely) with `Reader`, because Reader does not have a `getLastMessageId()`. Ideally, we'd like to have a way to do it like this: `Reader<byte[]> reader = pulsarClient.newReader().startMessageId(MessageId.latestInclusive).create();` This would also make it easier to make it available for readers connecting via WebSocket. Does this sound like a realistic feature request, worthy of a Pulsar issue? ---- 2020-02-26 03:01:50 UTC - Eugen: N.B. `startMessageIdInclusive()` only works in combination with calls to `seek()` ---- 2020-02-26 03:55:42 UTC - Devin G. Bost: I’m not sure that I understand the use case. ---- 2020-02-26 04:02:07 UTC - Eugen: Assume every message in a topic contains some kind of state, and we are at any point in time interested only in the current state, i.e. the value of the latest message in the topic. We could use a reader with `startMessageId==MessageId.latest`, however, if the first message to the topic after a reader subscription arrives only, say, 30 seconds later, the reader will not have a value for 30 seconds. That's why I'd like to get the latest message at the time of subscription. ---- 2020-02-26 06:04:11 UTC - Joe Francis: At this point you need a database. Essentially what you are asking for is a current record, on a record that is updated frequently. ---- 2020-02-26 08:09:09 UTC - Eugen: @Joe Francis consumers seem to be able to do `consumer.getLastMessageId();` without any database ---- 2020-02-26 08:31:09 UTC - Devin G. Bost: Oh, I think I understand now what you’re trying to do. You’re trying to use a Pulsar topic like a state variable that allows you to subscribe for changes. That’s basically what Zookeeper does. You’re wanting to check the current value and then watch for changes of the value and be notified as soon as there’s a change. That’s equivalent to checking a value and setting a watch in Zookeeper (though with Zookeeper, you must set the watch again every time you’re notified of a change.) It’s definitely a new type of use case for Pulsar that I hadn’t considered. Are there any other services/softwares that allow you to subscribe to changes of a state variable like that? Perhaps that could be done in Kafka if retention has been set. However, if I’m understanding you correctly, if at any point there actually hadn’t yet been a value written to the topic, then the consumer would still need to be able to set a default value (or throw an exception); so, I’m not sure we’d be gaining anything by trying to guarantee that a first value could always be retained until it could be read. Am I missing or misunderstanding something? ---- 2020-02-26 08:31:30 UTC - Devin G. Bost: @Eugen ----
