On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote:
2017-09-28 15:34 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> - these interfaces are more an admin interface. you can get an bookkeeper
> admin object and then access ledger manager to fetch metadata, does that
> work for you?
>
Yes as a workaround I am taking exactly this approch...listLedgers ->
getLedgerMetadata (using BK 4.5)
As the project I am working on will use 4.6 and the new
org.apache.bookkeeper.client.api as I need volatily durability ledgers it
is a pity to me that I will be stuck to use BookKeeperAdmin and
LedgerMetadata
only for listing ledgers
We can propose the interface for bk admin. But I don't see a reason that
listing ledgers being as part of bookkeeper.
> - metadata filter is just a wrapper over ledger iterator or async process
> ledger of ledger manager. most likely you will be the logic implementer. I
> don't see any strong reason we need to expose this filter as well. I would
> suggesting deferring this kind of requirements when we really need it.
>
we could at least add listLedgers + getLedgerMetadata API in the new API so
that new clients won't need BookKeeperAdmin, which takes a direct ZK client
-1 for any direct zk access. It should be hidden behind a ledger manager.
or a legacy o.a.b.client.BookKeper object
I have a good use case, to access the list of ledgers which satisfy a given
condition on custom metadata, we introduced custom metadata in order to
mark ledgers and to meta operations, classifications....
If metadata store doesn't support predicate pushdown, you still iterate all
the ledgers and do the filter at your side. So I don't see a reason to
support filter.
And if you are filtering custom metadata, this logic sounds tight to
application logic, no?
It is not blocker but if you are ok I will create an issue, but before
creating an issue I would like to draft a starting idea which would work
for the community as usual
I don't see a strong value having a metadata filter, given it doesn't buy
you any benefits at current metadata store. So I am -0 to this idea.
Thanks
Enrico
>
>
>
> On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
> > 2017-09-28 14:55 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> >
> > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eolive...@gmail.com>
> wrote:
> > >
> > > Hi,
> > > I am looking for an API to read ledger metadata without actually
> opening
> > a
> > > ledger.
> > >
> > > Currently opening a ledger sets a watch on ZK and this is really
> > expensive
> > > and in any case the desired action is not to "open a ledger" but to
> > access
> > > meta information about the ledger.
> > >
> > > My real usecase is that I want to list ledgers and filter the results
> > using
> > > custom metadata.
> > >
> > > If there is no available solution I would like to file and issue and a
> > > proposal to have a new method in the new API like
> > >
> > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > public long getId();
> > > public Map<String, byte[]> getCustomMetadata();
> > > public long getCtime();
> > > }
> > >
> > >
> > > I am fine with an interface for ledger metadata. It can help hide the
> > > implementation details.
> > >
> > >
> > > org.apache.bookkeeper.client.api.BookKeeper {
> > >
> > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > void listLedgers(String query, Consumer<LedgerMetadata>
consumer)
> > >
> > > }
> > >
> > >
> > > I am not sure if we want this. Now, you can access the ledger manager
> for
> > > reading and listing metadata. And this is how it was used by bk shell.
> I
> > > would not suggest adding this to the public API until we really see a
> > value
> > > there.
> > >
> >
> > BookKeeper object does not expose LedgerManager, I don't know if we want
> to
> > expose it in the new API
> >
> > we could add an API like:
> >
> > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > // marker interface
> > }
> >
> > class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
> > LedgerMetadataFilter {
> > // filter custom metadata field == value
> > CustomMetadataFilter(String key, byte[] value);
> > }
> >
> > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> implements
> > LedgerMetadataFilter {
> > // apply a sequence of filters in AND
> > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > }
> >
> >
> > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata> consumer)
> >
> > Does this sound better to you ? it is more simpler but clear and the
> > implemenation will be really easy
> >
> >
> >
> > >
> > >
> > > for the "query" we can define a simple "expression language"
> > >
> > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> > > 'zzzz')
> > >
> > >
> > > I am not a fan of having a query language for such purpose in bk. At
> > least,
> > > I don't see a lot of use cases that need this kind of query
> capabilities.
> > > If you can share more use cases about this, that would be good for
> > > discussions.
> > >
> > >
> > > we can provide a basic implementation of that language which actually
> > works
> > > on LedgerMetadata objects and internally we will loop over the ledgers
> > and
> > > apply the expression to every ledger
> > > the basic implementation can leverage standard expression languages
> like
> > EL
> > >
> > > smarter implementations of LedgerManager will be able to narrow the
> > search
> > > and save resources
> > >
> > > Thoughts ?
> > >
> > > -- Enrico
> > >
> >
>
