Il lun 2 ott 2017, 18:33 Sijie Guo <guosi...@gmail.com> ha scritto:
> On Sun, Oct 1, 2017 at 10:11 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
> > Il lun 2 ott 2017, 04:06 Sijie Guo <guosi...@gmail.com> ha scritto:
> >
> > > On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eolive...@gmail.com>
> wrote:
> > >
> > > 2017-09-28 15:34 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> > >
> > > > - these interfaces are more an admin interface. you can get an
> > bookkeeper
> > > > admin object and then access ledger manager to fetch metadata, does
> > that
> > > > work for you?
> > > >
> > >
> > > Yes as a workaround I am taking exactly this approch...listLedgers ->
> > > getLedgerMetadata (using BK 4.5)
> > >
> > > As the project I am working on will use 4.6 and the new
> > > org.apache.bookkeeper.client.api as I need volatily durability ledgers
> > it
> > > is a pity to me that I will be stuck to use BookKeeperAdmin and
> > > LedgerMetadata
> > > only for listing ledgers
> > >
> > >
> > > We can propose the interface for bk admin. But I don't see a reason
> that
> > > listing ledgers being as part of bookkeeper.
> > >
> >
> > Ok
> >
> >
> > >
> > >
> > > > - metadata filter is just a wrapper over ledger iterator or async
> > process
> > > > ledger of ledger manager. most likely you will be the logic
> > implementer.
> > > I
> > > > don't see any strong reason we need to expose this filter as well. I
> > > would
> > > > suggesting deferring this kind of requirements when we really need
> it.
> > > >
> > >
> > > we could at least add listLedgers + getLedgerMetadata API in the new
> API
> > so
> > > that new clients won't need BookKeeperAdmin, which takes a direct ZK
> > client
> > >
> > >
> > > -1 for any direct zk access. It should be hidden behind a ledger
> manager.
> > >
> >
> > Sorry maybe I was not clear, I did not propose to make zk more visibile.
> > I wrote that I would like that new clients do not use legacy BookKeeper
> > Admin class which has explicit deps on zk
> >
> > >
> > > or a legacy o.a.b.client.BookKeper object
> > >
> > > I have a good use case, to access the list of ledgers which satisfy a
> > given
> > > condition on custom metadata, we introduced custom metadata in order to
> > > mark ledgers and to meta operations, classifications....
> > >
> > >
> > > If metadata store doesn't support predicate pushdown, you still iterate
> > all
> > > the ledgers and do the filter at your side. So I don't see a reason to
> > > support filter.
> > >
> >
> > In case of a ledgermanager which lives inside the bookie and in presence
> of
> > a big number of ledgers, say 100000, we will save a lot of network by
> > returning only a sublist of ledgers to the client.
> > In the new scenario of thin client this makes sense to me
> >
>
> It will save the network bandwidth in the thin client case. but it still
> doesn't save any things when talking to zookeeper directly.
> I am not against filtering approach. I am suggesting it doesn't buy
> applications any benefits at this moment. I would prefer proposing
> such interface after we have a better metadata store.
>
At the moment we can wait, there is no hurry.
In the future we will have a new admin API, but now let's wait for thin
client and http
Thank you all
Enrico
> >
> > >
> > > And if you are filtering custom metadata, this logic sounds tight to
> > > application logic, no?
> > >
> >
> > Yes but we can give basic support for simple filtering. For instance I am
> > usibg metadata in order to:
> > - define the application which created the ledger
> > - define a group if semantically bound ledgers
> > - add explicit references to aplication ids
> >
> > We introduced custom metada as key value pairs and I see it very simple
> to
> > create a basic but effective API to use them
> >
>
> Currently it is not more effective than using the 'ledger range iterator'
> or 'async process ledger' in the ledger manager.
> since there is already an interface for iterating ledgers and there isn't a
> metadata store supports filtering effectively, I would
> suggest doing any metadata filter api after we have a better metadata
> store.
>
> It is going be just a filter wrapper on top of 'ledger range iterator' at
> current implementation. It doesn't really buy you any benefits at this
> moment.
>
>
> >
> > >
> > >
> > > It is not blocker but if you are ok I will create an issue, but before
> > > creating an issue I would like to draft a starting idea which would
> work
> > > for the community as usual
> > >
> > >
> > > I don't see a strong value having a metadata filter, given it doesn't
> buy
> > > you any benefits at current metadata store. So I am -0 to this idea.
> > >
> >
> > As I wrote above, I will see the benefits in new implementations, like
> the
> > http API and the thin client
> >
>
> In my view, you have to design http API or thin client with filtering
> semantic first in a consistent way. Then you will know the requirement for
> what can be improved in the ledger metadata interface.
> I am seeing this is done in a reverse way.
>
>
> >
> > Enrico
> >
> > >
> > >
> > > Thanks
> > > Enrico
> > >
> > >
> > >
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eolive...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > 2017-09-28 14:55 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> > > > >
> > > > > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eolive...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I am looking for an API to read ledger metadata without actually
> > > > opening
> > > > > a
> > > > > > ledger.
> > > > > >
> > > > > > Currently opening a ledger sets a watch on ZK and this is really
> > > > > expensive
> > > > > > and in any case the desired action is not to "open a ledger" but
> to
> > > > > access
> > > > > > meta information about the ledger.
> > > > > >
> > > > > > My real usecase is that I want to list ledgers and filter the
> > results
> > > > > using
> > > > > > custom metadata.
> > > > > >
> > > > > > If there is no available solution I would like to file and issue
> > and
> > > a
> > > > > > proposal to have a new method in the new API like
> > > > > >
> > > > > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > > > > public long getId();
> > > > > > public Map<String, byte[]> getCustomMetadata();
> > > > > > public long getCtime();
> > > > > > }
> > > > > >
> > > > > >
> > > > > > I am fine with an interface for ledger metadata. It can help hide
> > the
> > > > > > implementation details.
> > > > > >
> > > > > >
> > > > > > org.apache.bookkeeper.client.api.BookKeeper {
> > > > > >
> > > > > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > > > > void listLedgers(String query, Consumer<LedgerMetadata>
> > > consumer)
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > > I am not sure if we want this. Now, you can access the ledger
> > manager
> > > > for
> > > > > > reading and listing metadata. And this is how it was used by bk
> > > shell.
> > > > I
> > > > > > would not suggest adding this to the public API until we really
> > see a
> > > > > value
> > > > > > there.
> > > > > >
> > > > >
> > > > > BookKeeper object does not expose LedgerManager, I don't know if we
> > > want
> > > > to
> > > > > expose it in the new API
> > > > >
> > > > > we could add an API like:
> > > > >
> > > > > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > > > > // marker interface
> > > > > }
> > > > >
> > > > > class org.apache.bookkeeper.client.api.CustomMetadataFilter
> > implements
> > > > > LedgerMetadataFilter {
> > > > > // filter custom metadata field == value
> > > > > CustomMetadataFilter(String key, byte[] value);
> > > > > }
> > > > >
> > > > > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> > > > implements
> > > > > LedgerMetadataFilter {
> > > > > // apply a sequence of filters in AND
> > > > > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > > > > }
> > > > >
> > > > >
> > > > > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata>
> > > consumer)
> > > > >
> > > > > Does this sound better to you ? it is more simpler but clear and
> the
> > > > > implemenation will be really easy
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > for the "query" we can define a simple "expression language"
> > > > > >
> > > > > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or
> > metadata.type =
> > > > > > 'zzzz')
> > > > > >
> > > > > >
> > > > > > I am not a fan of having a query language for such purpose in bk.
> > At
> > > > > least,
> > > > > > I don't see a lot of use cases that need this kind of query
> > > > capabilities.
> > > > > > If you can share more use cases about this, that would be good
> for
> > > > > > discussions.
> > > > > >
> > > > > >
> > > > > > we can provide a basic implementation of that language which
> > actually
> > > > > works
> > > > > > on LedgerMetadata objects and internally we will loop over the
> > > ledgers
> > > > > and
> > > > > > apply the expression to every ledger
> > > > > > the basic implementation can leverage standard expression
> languages
> > > > like
> > > > > EL
> > > > > >
> > > > > > smarter implementations of LedgerManager will be able to narrow
> the
> > > > > search
> > > > > > and save resources
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > -- Enrico
> > > > > >
> > > > >
> > > >
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
--
-- Enrico Olivelli
