On Sun, Oct 1, 2017 at 10:11 PM, Enrico Olivelli <eolive...@gmail.com>
wrote:
> Il lun 2 ott 2017, 04:06 Sijie Guo <guosi...@gmail.com> ha scritto:
>
> > On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote:
> >
> > 2017-09-28 15:34 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> >
> > > - these interfaces are more an admin interface. you can get an
> bookkeeper
> > > admin object and then access ledger manager to fetch metadata, does
> that
> > > work for you?
> > >
> >
> > Yes as a workaround I am taking exactly this approch...listLedgers ->
> > getLedgerMetadata (using BK 4.5)
> >
> > As the project I am working on will use 4.6 and the new
> > org.apache.bookkeeper.client.api as I need volatily durability ledgers
> it
> > is a pity to me that I will be stuck to use BookKeeperAdmin and
> > LedgerMetadata
> > only for listing ledgers
> >
> >
> > We can propose the interface for bk admin. But I don't see a reason that
> > listing ledgers being as part of bookkeeper.
> >
>
> Ok
>
>
> >
> >
> > > - metadata filter is just a wrapper over ledger iterator or async
> process
> > > ledger of ledger manager. most likely you will be the logic
> implementer.
> > I
> > > don't see any strong reason we need to expose this filter as well. I
> > would
> > > suggesting deferring this kind of requirements when we really need it.
> > >
> >
> > we could at least add listLedgers + getLedgerMetadata API in the new API
> so
> > that new clients won't need BookKeeperAdmin, which takes a direct ZK
> client
> >
> >
> > -1 for any direct zk access. It should be hidden behind a ledger manager.
> >
>
> Sorry maybe I was not clear, I did not propose to make zk more visibile.
> I wrote that I would like that new clients do not use legacy BookKeeper
> Admin class which has explicit deps on zk
>
> >
> > or a legacy o.a.b.client.BookKeper object
> >
> > I have a good use case, to access the list of ledgers which satisfy a
> given
> > condition on custom metadata, we introduced custom metadata in order to
> > mark ledgers and to meta operations, classifications....
> >
> >
> > If metadata store doesn't support predicate pushdown, you still iterate
> all
> > the ledgers and do the filter at your side. So I don't see a reason to
> > support filter.
> >
>
> In case of a ledgermanager which lives inside the bookie and in presence of
> a big number of ledgers, say 100000, we will save a lot of network by
> returning only a sublist of ledgers to the client.
> In the new scenario of thin client this makes sense to me
>
It will save the network bandwidth in the thin client case. but it still
doesn't save any things when talking to zookeeper directly.
I am not against filtering approach. I am suggesting it doesn't buy
applications any benefits at this moment. I would prefer proposing
such interface after we have a better metadata store.
>
> >
> > And if you are filtering custom metadata, this logic sounds tight to
> > application logic, no?
> >
>
> Yes but we can give basic support for simple filtering. For instance I am
> usibg metadata in order to:
> - define the application which created the ledger
> - define a group if semantically bound ledgers
> - add explicit references to aplication ids
>
> We introduced custom metada as key value pairs and I see it very simple to
> create a basic but effective API to use them
>
Currently it is not more effective than using the 'ledger range iterator'
or 'async process ledger' in the ledger manager.
since there is already an interface for iterating ledgers and there isn't a
metadata store supports filtering effectively, I would
suggest doing any metadata filter api after we have a better metadata store.
It is going be just a filter wrapper on top of 'ledger range iterator' at
current implementation. It doesn't really buy you any benefits at this
moment.
>
> >
> >
> > It is not blocker but if you are ok I will create an issue, but before
> > creating an issue I would like to draft a starting idea which would work
> > for the community as usual
> >
> >
> > I don't see a strong value having a metadata filter, given it doesn't buy
> > you any benefits at current metadata store. So I am -0 to this idea.
> >
>
> As I wrote above, I will see the benefits in new implementations, like the
> http API and the thin client
>
In my view, you have to design http API or thin client with filtering
semantic first in a consistent way. Then you will know the requirement for
what can be improved in the ledger metadata interface.
I am seeing this is done in a reverse way.
>
> Enrico
>
> >
> >
> > Thanks
> > Enrico
> >
> >
> >
> >
> >
> > >
> > >
> > >
> > > On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eolive...@gmail.com>
> > > wrote:
> > >
> > > > 2017-09-28 14:55 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> > > >
> > > > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eolive...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi,
> > > > > I am looking for an API to read ledger metadata without actually
> > > opening
> > > > a
> > > > > ledger.
> > > > >
> > > > > Currently opening a ledger sets a watch on ZK and this is really
> > > > expensive
> > > > > and in any case the desired action is not to "open a ledger" but to
> > > > access
> > > > > meta information about the ledger.
> > > > >
> > > > > My real usecase is that I want to list ledgers and filter the
> results
> > > > using
> > > > > custom metadata.
> > > > >
> > > > > If there is no available solution I would like to file and issue
> and
> > a
> > > > > proposal to have a new method in the new API like
> > > > >
> > > > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > > > public long getId();
> > > > > public Map<String, byte[]> getCustomMetadata();
> > > > > public long getCtime();
> > > > > }
> > > > >
> > > > >
> > > > > I am fine with an interface for ledger metadata. It can help hide
> the
> > > > > implementation details.
> > > > >
> > > > >
> > > > > org.apache.bookkeeper.client.api.BookKeeper {
> > > > >
> > > > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > > > void listLedgers(String query, Consumer<LedgerMetadata>
> > consumer)
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > > I am not sure if we want this. Now, you can access the ledger
> manager
> > > for
> > > > > reading and listing metadata. And this is how it was used by bk
> > shell.
> > > I
> > > > > would not suggest adding this to the public API until we really
> see a
> > > > value
> > > > > there.
> > > > >
> > > >
> > > > BookKeeper object does not expose LedgerManager, I don't know if we
> > want
> > > to
> > > > expose it in the new API
> > > >
> > > > we could add an API like:
> > > >
> > > > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > > > // marker interface
> > > > }
> > > >
> > > > class org.apache.bookkeeper.client.api.CustomMetadataFilter
> implements
> > > > LedgerMetadataFilter {
> > > > // filter custom metadata field == value
> > > > CustomMetadataFilter(String key, byte[] value);
> > > > }
> > > >
> > > > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> > > implements
> > > > LedgerMetadataFilter {
> > > > // apply a sequence of filters in AND
> > > > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > > > }
> > > >
> > > >
> > > > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata>
> > consumer)
> > > >
> > > > Does this sound better to you ? it is more simpler but clear and the
> > > > implemenation will be really easy
> > > >
> > > >
> > > >
> > > > >
> > > > >
> > > > > for the "query" we can define a simple "expression language"
> > > > >
> > > > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or
> metadata.type =
> > > > > 'zzzz')
> > > > >
> > > > >
> > > > > I am not a fan of having a query language for such purpose in bk.
> At
> > > > least,
> > > > > I don't see a lot of use cases that need this kind of query
> > > capabilities.
> > > > > If you can share more use cases about this, that would be good for
> > > > > discussions.
> > > > >
> > > > >
> > > > > we can provide a basic implementation of that language which
> actually
> > > > works
> > > > > on LedgerMetadata objects and internally we will loop over the
> > ledgers
> > > > and
> > > > > apply the expression to every ledger
> > > > > the basic implementation can leverage standard expression languages
> > > like
> > > > EL
> > > > >
> > > > > smarter implementations of LedgerManager will be able to narrow the
> > > > search
> > > > > and save resources
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > -- Enrico
> > > > >
> > > >
> > >
> >
> --
>
>
> -- Enrico Olivelli
>
