Re: Embedded Version of H2
Hi Steve, I am afraid that window functions and OVER keyword will not work properly in new Ignite version with upgraded H2. Could you share some simple query with OVER keyword you are going to execute? Then I can check it against current master version. чт, 17 окт. 2019 г. в 14:44, steve.hostett...@gmail.com : > > Hello, > > Actually I found the answer with the following ticket : > https://issues.apache.org/jira/browse/IGNITE-10801 > > Seems to be planned for 2.8.0 > > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ -- Best regards, Ivan Pavlukhin
Re: Embedded Version of H2
A folow-up, there is an interesting ticket [1]. Need to check if window functions are covered. [1] https://issues.apache.org/jira/browse/IGNITE-7664 пт, 18 окт. 2019 г. в 10:01, Ivan Pavlukhin : > > Hi Steve, > > I am afraid that window functions and OVER keyword will not work > properly in new Ignite version with upgraded H2. Could you share some > simple query with OVER keyword you are going to execute? Then I can > check it against current master version. > > чт, 17 окт. 2019 г. в 14:44, steve.hostett...@gmail.com > : > > > > Hello, > > > > Actually I found the answer with the following ticket : > > https://issues.apache.org/jira/browse/IGNITE-10801 > > > > Seems to be planned for 2.8.0 > > > > > > > > > > -- > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Binary object format KB article
Great job, I think we should have details like this in documentation, not only in wiki Denis, what do you think? Best Regards, Igor On Wed, Oct 16, 2019 at 7:19 PM Ivan Pavlukhin wrote: > Sergey, > > Thank you for a review! > > > It seems to me that document tries to focus on details of the format > itself but other aspects of this functionality leak into the explanation > and confuses reader. > > You are absolutely right, it was an original idea. I will try to > define a terminology and clarify things about schemas. > > ср, 16 окт. 2019 г. в 16:49, Sergey Chugunov : > > > > Then I would suggest to define good terminology at the very beginning of > > the article. > > > > Right in introduction section I see a lot of terms like "Binary object > > format", "Binary object container format" (is it the same thing?), > "Binary > > serialization format". In the next section "binary type" pops up. What > are > > the relations between them? > > > > Schemes part needs more examples. What is scheme? How it is related to > > binary type? Is it a one-to-one relationship? One-to-many? When a new > > scheme is created? Why type and scheme should be registered on a receiver > > side? And if the receiver exists then who is the sender? > > > > It seems to me that document tries to focus on details of the format > itself > > but other aspects of this functionality leak into the explanation and > > confuses reader. > > > > On Wed, Oct 16, 2019 at 2:52 PM Ivan Pavlukhin > wrote: > > > > > Pavel, Sergey, > > > > > > Thank you for your feedback! > > > > > > To be exact the document does not describe broad picture (including > > > metadata exchange) and is not a formal format specification > > > intentionally. I wanted to create a lightweight article giving an > > > intuition about binary object structure to a reader. And yes, > > > intuition about metadata registration is definitely an important, > > > related but slightly different subject. > > > > > > ср, 16 окт. 2019 г. в 14:23, Sergey Chugunov < > sergey.chugu...@gmail.com>: > > > > > > > > Ivan, thank you for documenting this functionality, agree with Pavel > > > here. > > > > > > > > I think this document is a good starting point and contains a lot of > > > > low-level details and great examples but from my perspective it > doesn't > > > > show how binary objects fit into a broader picture. > > > > > > > > It worth adding higher-level details and restructure the document > into a > > > > top-down article starting from where binary format is used > > > (representation > > > > of objects in cache, binary protocol for communications with thin > > > clients) > > > > and down to lower details like binary metadata exchange and > serialization > > > > and container formats. > > > > > > > > Another option would be to leave the document focused on a low-level > > > > details as it is now but build around it drafts for documents > describing > > > > other aspects of Binary Objects. > > > > This will make our documentation much more solid and useful for > readers. > > > > > > > > On Wed, Oct 16, 2019 at 2:07 PM Pavel Tupitsyn > > > > wrote: > > > > > > > > > Ivan, great job, thanks for putting this together. > > > > > > > > > > I think we also need a more formal description of the format, > including > > > > > binary metadata exchange mechanics. > > > > > It was done (partially) for IEP-9 Thin Client Protocol, we should > > > probably > > > > > copy from there: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol#IEP-9ThinClientProtocol-BinaryObjects > > > > > > > > > > > > > > > > > > > > On Wed, Oct 16, 2019 at 11:49 AM Ivan Pavlukhin < > vololo...@gmail.com> > > > > > wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > I published a document about Binary format in cwiki [1]. Please > share > > > > > > your feedback. I feel that there is a lack of pictures on the > page. > > > > > > Need to figure out what aspects will be more clear with pictures. > > > > > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format > > > > > > > > > > > > -- > > > > > > Best regards, > > > > > > Ivan Pavlukhin > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > -- > Best regards, > Ivan Pavlukhin >
[jira] [Created] (IGNITE-12303) Change comment for an enumeration item CACHE_DESTROY
Surkov Aleksandr created IGNITE-12303: - Summary: Change comment for an enumeration item CACHE_DESTROY Key: IGNITE-12303 URL: https://issues.apache.org/jira/browse/IGNITE-12303 Project: Ignite Issue Type: Wish Components: documentation, security Reporter: Surkov Aleksandr For the _org.apache.ignite.plugin.security.SecurityPermission#CACHE_DESTROY_ enumeration element it would be worth changing the comment. "Cache create permission." it's not very good. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: TDE Master key rotation (Phase-2)
Hello, Nikita. Thank you. I will take a look. shortly. чт, 17 окт. 2019 г. в 18:23, Maxim Muzafarov : > Nikita, > > > Can we include it into a 2.8 release scope? > I think it is possible since the release scope freeze date has not > happened yet. > > On Thu, 17 Oct 2019 at 17:36, Nikita Amelchev > wrote: > > > > Hi, Igniters! > > > > I have implemented the master key change process [1] for TDE as > > described in the design [2]. > > > > I have prepared PR [3] and created the Upsource review branch [4]. > > > > Could anyone take a look at my changes? > > > > Can we include it into a 2.8 release scope? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12186 > > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > [3] https://github.com/apache/ignite/pull/6937 > > [4] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-1067 > > > > пн, 23 сент. 2019 г. в 17:13, Nikolay Izhikov : > > > > > > Hello, Nikita. > > > > > > > A node creates the ChangeMasterKeyMessage message and sent it by > discovery as a custom event. > > > > The goal is to verify that all nodes have the same master key. > > > ... > > > > The ChangeMasterKeyFinishMessage action message is sent by discovery > as a custom event. > > > > New master key id. > > > > > > 1. We should store locally new key id and new key hash after > processing of ChangeMasterKeyMessage > > > 2. We should send new key hash in ChangeMasterKeyFinishMessage > > > 3. We should ensure that both ChangeMasterKeyMessage and > ChangeMasterKeyFinishMessage contains the same key id and key hash before > executing a change process. > > > > > > I think we should rename: > > > > > > > Node left during key rotation process(was not starting re-encrypt > cache keys) > > > > > > Node was down during key rotation. ChangeMasterKeyRecord was not found. > > > > > > > Node left during key rotation process(was starting re-encrypt cache > keys) > > > > > > Node was down during key rotation. ChangeMasterKeyRecord found. > > > > > > We should add a description of changes in the cluster join process. > > > A node should not try to join to the cluster before the process of > ChangeMasterKeyRecord. > > > > > > It's not clear for me how we differ two cases: > > > > > > 1. ChangeMasterKeyRecord found in WAL and key rotation was finished > successfully. > > > 2. ChangeMasterKeyRecord found in WAL and key rotation was NOT > finished successfully. > > > > > > > Meta storage will store master key id. > > > > > > We should add that key id from metastorage has a higher priority to > key id from IgniteConfiguration. > > > > > > > > > В Пт, 20/09/2019 в 14:06 +0300, Nikita Amelchev пишет: > > > > Nikolay, > > > > > > > > you are right in many ways. I updated the design on the wiki. [1] > > > > > > > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381 > > > > > > > > пт, 20 сент. 2019 г. в 13:49, Nikolay Izhikov : > > > > > > > > > > Nikita > > > > > > > > > > > I suggested the implementation where the encryption manager is > > > > > > responsible for storing the master key id. > > > > > > > > > > I don't think it's a right proposal. > > > > > > > > > > 1. EncryptionSpi implementation becomes more complicated. > Developer of it should be aware of Ignite deployment scenarious, etc. > > > > > Imagine implementation when EncryptionSpi send master key id to > some external storage over network(it's happen in Discovery thread) > > > > > > > > > > 2. Implementation would be duplicate features(saving master key id) > > > > > > > > > > 3. We already store cache keys in metastore. For me it's a native > approach to store master key id in the same place. > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > В Пт, 20/09/2019 в 13:39 +0300, Nikita Amelchev пишет: > > > > > > Nikolay, > > > > > > > > > > > > because I suggested the implementation where the encryption > manager is > > > > > > responsible for storing the master key id. > > > > > > To implement this logic in the EncryptionSpi, we will need to > > > > > > introduce the methods look like this: > > > > > > > > > > > > setMasterKeyId(String masterKeyId) // Sets "current" master key > id > > > > > > String getMasterKeyId() // Gets "current" master key id > > > > > > > > > > > > Follow methods will work with master key that setted by previous > method: > > > > > > > > > > > > byte[] masterKeyDigest() > > > > > > byte[] encryptKey(Serializable key) > > > > > > Serializable decryptKey(byte[] key) > > > > > > > > > > > > If such implementation is more reasonable, I will do so. > > > > > > > > > > > > пт, 20 сент. 2019 г. в 13:04, Nikolay Izhikov < > nizhi...@apache.org>: > > > > > > > > > > > > > > Why do we need "defaultMasterKeyId" instead of *current* > master key id that can be obtained with > `KeystoreEncryptionSpi#getMasterKeyName()`? > > > > > > > > > > > > > > В Пт, 20/09/2019 в 12:56 +0300, Nikita Amelchev пишет: > > > > > > > > Nikolay, > > > > > > > > > > > > > > > >
Thin clients: WithExpiryPolicy
Igniters, I would like to add WithExpirePolicy support to thin clients. [1] For a thick client, we can obtain a reference to a cache wrapper instance and use cache API through it. At the same time, the thin client protocol is stateless, we do not hold a reference to a cache but rather a cache name identifier is used for a server to create an appropriate cache instance. We could extend the protocol as we did with WithKeepBinaryMethod: every time we need to call some API on a cache with expiration, a serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This approach works well, but things could get worse if we decided to add a few more WithSomething* methods. Initially, I was thinking about introducing some state context to a protocol, similar to a QueryCursor API. For instance, we can save an expire policy configuration for the first call and use some hash value based on an ExpiryPolicy for further calls, just as we do for cache names. I.e. newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId, )] But this approach complicates logic and leads to additional memory consumption. I think it's ok for now to use the first approach with ExpiryPolicy serialization. But any ideas are welcome. [1] - https://issues.apache.org/jira/browse/IGNITE-9033
Re: Ignite diagnostic (SQL system views)
Alex, Igniters, Who of us was contributing this feature? I don’t see any documentation, not clear how the users are expected to benefit from the capability and how everybody will be aware of the feature existence. We need to close the gap and spread the word. Denis On Thursday, October 17, 2019, Alex Plehanov wrote: > Denis, > > Views engine and some views were released in AI 2.7. > In 2.8 they will be moved to the new engine and new views will be added (as > part of IEP-35) > > > пт, 18 окт. 2019 г. в 00:50, Denis Magda : > > > Anton, Maxim, > > > > Are we planning to release the views as part of 2.8? Don't see them > listed > > in the important features section: > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8# > ApacheIgnite2.8-Themostimportantreleasetasks > > > > - > > Denis > > > > > > On Wed, Feb 14, 2018 at 1:49 AM Anton Vinogradov < > avinogra...@gridgain.com > > > > > wrote: > > > > > Vova, > > > > > > Could you confirm https://issues.apache.org/jira/browse/IGNITE-7527 > > ready > > > to be merged? > > > > > > On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov < > voze...@gridgain.com> > > > wrote: > > > > > > > I would start with NODES and NODE_ATTRIBUTES as the most simple > thing. > > > > > > > > On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda > > wrote: > > > > > > > > > Alex P, sounds like a good plan for me. > > > > > > > > > > Vladimir, do you have any suggestions or corrections? > > > > > > > > > > — > > > > > Denis > > > > > > > > > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov < > > plehanov.a...@gmail.com> > > > > > wrote: > > > > > > > > > > > > The views engine and the first view are almost ready to merge > > (review > > > > > > comments are resolved). Which views should we take next? My > > proposal > > > - > > > > > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and > > NODE_ADDRESSES, > > > > > since > > > > > > these views are clear and all topology data available on each > node. > > > > > > Any objections? > > > > > > > > > > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov < > plehanov.a...@gmail.com > > >: > > > > > > > > > > > >> Anton, Vladimir, I've made some fixes. There is only one view > left > > > and > > > > > >> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'. > > > > > >> > > > > > >> High level design of solution: > > > > > >> When IgniteH2Indexing is starting, it create and start > > > > > >> new GridH2SysViewProcessor, which create and register in H2 (via > > its > > > > own > > > > > >> table engine) all implementations of system views. Each system > > view > > > > > >> implementation extends base abstract class GridH2SysView. View > > > > > >> implementation describes columns, their types and indexes in > > > > constructor > > > > > >> and must override method getRows for data retrieval (this method > > > > called > > > > > by > > > > > >> H2-compatible table and index implementations for ignite system > > > > views). > > > > > >> Almost no fixes to existing parsing engine was made, except some > > > > places, > > > > > >> where GridH2Table instance was expected, but for system views > > there > > > is > > > > > >> another class. > > > > > >> > > > > > >> New PR: [1]. Please have a look. > > > > > >> > > > > > >> [1] https://github.com/apache/ignite/pull/3433 > > > > > >> > > > > > >> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov < > > > avinogra...@gridgain.com > > > > >: > > > > > >> > > > > > >>> I've created IEP-13 [1] to cover all cases. > > > > > >>> Feel free to create issues. > > > > > >>> > > > > > >>> [1] > > > > > >>> https://cwiki.apache.org/confluence/pages/viewpage. > > > > > action?pageId=75962769 > > > > > >>> > > > > > >>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov < > > > > voze...@gridgain.com > > > > > > > > > > > >>> wrote: > > > > > >>> > > > > > Let's start with a single and the most simple view, e.g. > > > > > LOCAL_TRANSACTIONS. We will review and merge it along with > > > necessary > > > > > infrastructure. Then will handle the rest view in separate > > tickets > > > > and > > > > > separate focused discussions. > > > > > > > > > > On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov < > > > > > plehanov.a...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > 1) It’s not a principal point, I can change schema. The > > > > > INFORMATION_SCHEMA > > > > > > was used because it’s already exists and usually used for > > > metadata > > > > > >>> tables > > > > > > and views. Your proposal is to use schema “IGNITE”, am I > > > understand > > > > > >>> you > > > > > > right? BTW, for now, we can’t query another (H2) meta tables > > from > > > > the > > > > > > INFORMATION_SCHEMA, so, “Ignite system views” is only > available > > > > views > > > > > >>> to > > > > > > query from this schema. > > > > > > 2) Exactly for this reason the IGNITE_INSTANCE view is > useful: > > to > > > > > determine > > > > > > which node we are con
Re: Ignite diagnostic (SQL system views)
Denis, AFAIK we doesn't have documentation for the SQL System Views existing in Ignite. I have plans to write a documentation about metrics and syste views that will cover SQL System View. It will be available till 2.8 release. В Пт, 18/10/2019 в 07:16 -0700, Denis Magda пишет: > Alex, Igniters, > > Who of us was contributing this feature? I don’t see any documentation, not > clear how the users are expected to benefit from the capability and how > everybody will be aware of the feature existence. > > We need to close the gap and spread the word. > > Denis > > On Thursday, October 17, 2019, Alex Plehanov > wrote: > > > Denis, > > > > Views engine and some views were released in AI 2.7. > > In 2.8 they will be moved to the new engine and new views will be added (as > > part of IEP-35) > > > > > > пт, 18 окт. 2019 г. в 00:50, Denis Magda : > > > > > Anton, Maxim, > > > > > > Are we planning to release the views as part of 2.8? Don't see them > > > > listed > > > in the important features section: > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8# > > > > ApacheIgnite2.8-Themostimportantreleasetasks > > > > > > - > > > Denis > > > > > > > > > On Wed, Feb 14, 2018 at 1:49 AM Anton Vinogradov < > > > > avinogra...@gridgain.com > > > > > > > > > > wrote: > > > > > > > Vova, > > > > > > > > Could you confirm https://issues.apache.org/jira/browse/IGNITE-7527 > > > > > > ready > > > > to be merged? > > > > > > > > On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > wrote: > > > > > > > > > I would start with NODES and NODE_ATTRIBUTES as the most simple > > > > thing. > > > > > > > > > > On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda > > > > > > wrote: > > > > > > > > > > > Alex P, sounds like a good plan for me. > > > > > > > > > > > > Vladimir, do you have any suggestions or corrections? > > > > > > > > > > > > — > > > > > > Denis > > > > > > > > > > > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov < > > > > > > plehanov.a...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > The views engine and the first view are almost ready to merge > > > > > > (review > > > > > > > comments are resolved). Which views should we take next? My > > > > > > proposal > > > > - > > > > > > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and > > > > > > NODE_ADDRESSES, > > > > > > since > > > > > > > these views are clear and all topology data available on each > > > > node. > > > > > > > Any objections? > > > > > > > > > > > > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov < > > > > plehanov.a...@gmail.com > > > > : > > > > > > > > > > > > > > > Anton, Vladimir, I've made some fixes. There is only one view > > > > left > > > > and > > > > > > > > it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'. > > > > > > > > > > > > > > > > High level design of solution: > > > > > > > > When IgniteH2Indexing is starting, it create and start > > > > > > > > new GridH2SysViewProcessor, which create and register in H2 (via > > > > > > its > > > > > own > > > > > > > > table engine) all implementations of system views. Each system > > > > > > view > > > > > > > > implementation extends base abstract class GridH2SysView. View > > > > > > > > implementation describes columns, their types and indexes in > > > > > > > > > > constructor > > > > > > > > and must override method getRows for data retrieval (this method > > > > > > > > > > called > > > > > > by > > > > > > > > H2-compatible table and index implementations for ignite system > > > > > > > > > > views). > > > > > > > > Almost no fixes to existing parsing engine was made, except some > > > > > > > > > > places, > > > > > > > > where GridH2Table instance was expected, but for system views > > > > > > there > > > > is > > > > > > > > another class. > > > > > > > > > > > > > > > > New PR: [1]. Please have a look. > > > > > > > > > > > > > > > > [1] https://github.com/apache/ignite/pull/3433 > > > > > > > > > > > > > > > > 2018-01-24 19:12 GMT+03:00 Anton Vinogradov < > > > > > > > > avinogra...@gridgain.com > > > > > > : > > > > > > > > > > > > > > > > > I've created IEP-13 [1] to cover all cases. > > > > > > > > > Feel free to create issues. > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage. > > > > > > > > > > > > action?pageId=75962769 > > > > > > > > > > > > > > > > > > On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov < > > > > > > > > > > voze...@gridgain.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Let's start with a single and the most simple view, e.g. > > > > > > > > > > LOCAL_TRANSACTIONS. We will review and merge it along with > > > > > > > > necessary > > > > > > > > > > infrastructure. Then will handle the rest view in separate > > > > > > tickets > > > > > and > > > > > > > > > > separate focused discussions. > > > > > > > > > > > > > > > > > > > > On
Re: Ignite diagnostic (SQL system views)
Documentation for views released in 2.7 is here: https://apacheignite-sql.readme.io/docs/system-views пт, 18 окт. 2019 г. в 17:24, Nikolay Izhikov : > Denis, AFAIK we doesn't have documentation for the SQL System Views > existing in Ignite. > > I have plans to write a documentation about metrics and syste views that > will cover SQL System View. > It will be available till 2.8 release. > > > > В Пт, 18/10/2019 в 07:16 -0700, Denis Magda пишет: > > Alex, Igniters, > > > > Who of us was contributing this feature? I don’t see any documentation, > not > > clear how the users are expected to benefit from the capability and how > > everybody will be aware of the feature existence. > > > > We need to close the gap and spread the word. > > > > Denis > > > > On Thursday, October 17, 2019, Alex Plehanov > > wrote: > > > > > Denis, > > > > > > Views engine and some views were released in AI 2.7. > > > In 2.8 they will be moved to the new engine and new views will be > added (as > > > part of IEP-35) > > > > > > > > > пт, 18 окт. 2019 г. в 00:50, Denis Magda : > > > > > > > Anton, Maxim, > > > > > > > > Are we planning to release the views as part of 2.8? Don't see them > > > > > > listed > > > > in the important features section: > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8# > > > > > > ApacheIgnite2.8-Themostimportantreleasetasks > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Wed, Feb 14, 2018 at 1:49 AM Anton Vinogradov < > > > > > > avinogra...@gridgain.com > > > > > > > > > > > > > wrote: > > > > > > > > > Vova, > > > > > > > > > > Could you confirm > https://issues.apache.org/jira/browse/IGNITE-7527 > > > > > > > > ready > > > > > to be merged? > > > > > > > > > > On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov < > > > > > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > I would start with NODES and NODE_ATTRIBUTES as the most simple > > > > > > thing. > > > > > > > > > > > > On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda > > > > > > > > wrote: > > > > > > > > > > > > > Alex P, sounds like a good plan for me. > > > > > > > > > > > > > > Vladimir, do you have any suggestions or corrections? > > > > > > > > > > > > > > — > > > > > > > Denis > > > > > > > > > > > > > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov < > > > > > > > > plehanov.a...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > The views engine and the first view are almost ready to merge > > > > > > > > (review > > > > > > > > comments are resolved). Which views should we take next? My > > > > > > > > proposal > > > > > - > > > > > > > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and > > > > > > > > NODE_ADDRESSES, > > > > > > > since > > > > > > > > these views are clear and all topology data available on each > > > > > > node. > > > > > > > > Any objections? > > > > > > > > > > > > > > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov < > > > > > > plehanov.a...@gmail.com > > > > > : > > > > > > > > > > > > > > > > > Anton, Vladimir, I've made some fixes. There is only one > view > > > > > > left > > > > > and > > > > > > > > > it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'. > > > > > > > > > > > > > > > > > > High level design of solution: > > > > > > > > > When IgniteH2Indexing is starting, it create and start > > > > > > > > > new GridH2SysViewProcessor, which create and register in > H2 (via > > > > > > > > its > > > > > > own > > > > > > > > > table engine) all implementations of system views. Each > system > > > > > > > > view > > > > > > > > > implementation extends base abstract class GridH2SysView. > View > > > > > > > > > implementation describes columns, their types and indexes > in > > > > > > > > > > > > constructor > > > > > > > > > and must override method getRows for data retrieval (this > method > > > > > > > > > > > > called > > > > > > > by > > > > > > > > > H2-compatible table and index implementations for ignite > system > > > > > > > > > > > > views). > > > > > > > > > Almost no fixes to existing parsing engine was made, > except some > > > > > > > > > > > > places, > > > > > > > > > where GridH2Table instance was expected, but for system > views > > > > > > > > there > > > > > is > > > > > > > > > another class. > > > > > > > > > > > > > > > > > > New PR: [1]. Please have a look. > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/ignite/pull/3433 > > > > > > > > > > > > > > > > > > 2018-01-24 19:12 GMT+03:00 Anton Vinogradov < > > > > > > > > > > avinogra...@gridgain.com > > > > > > > : > > > > > > > > > > > > > > > > > > > I've created IEP-13 [1] to cover all cases. > > > > > > > > > > Feel free to create issues. > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage. > > > > > > > > > > > > > > action?pageId=75962769 > > > > > > > > > > > > > > > > > > > > On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov < > > > > > > > > > > > > voze...@gridgain.com > > > > > > >
Re: Thin clients: WithExpiryPolicy
Stateless approach looks a lot better to me. We have a choice: * Keep expiry policy on server and send an ID with every request (like a query cursor ID - 8 bytes) * Send full expiry policy with every expiry-enabled request (24 bytes - or maybe less? We should think about the format) Stateful approach will bring a lot of complexity if we consider Affinity Awareness [1] mechanism (and also automatic reconnect). We would have to keep ExpiryPolicyId for every server and choose the right one based on the affinity for every operation. This can easily negate any performance gain from saving 16 bytes. And there is always CacheConfiguration.ExpiryPolicyFactory, which allows us to set up default expiry policy. > things could get worse if we decided to add a few more WithSomething* methods I don't think this is a good argument - we should decide on case by case basis. Anyway, other With* methods don't have any parameters, so they carry only 1 bit of information. https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin wrote: > Igniters, > > I would like to add WithExpirePolicy support to thin clients. [1] > For a thick client, we can obtain a reference to a cache wrapper instance > and use cache API through it. At the same time, the thin client protocol is > stateless, we do not hold a reference to a cache but rather a cache name > identifier is used for a server to create an appropriate cache instance. > > We could extend the protocol as we did with WithKeepBinaryMethod: > every time we need to call some API on a cache with expiration, a > serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This approach > works well, but things could get worse if we decided to add a few more > WithSomething* methods. > > Initially, I was thinking about introducing some state context to a > protocol, similar to a QueryCursor API. For instance, we can save an expire > policy configuration for the first call and use some hash value based on an > ExpiryPolicy for further calls, just as we do for cache names. I.e. > newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId, > )] But this approach complicates logic and leads to additional memory > consumption. > > I think it's ok for now to use the first approach with ExpiryPolicy > serialization. > But any ideas are welcome. > > [1] - https://issues.apache.org/jira/browse/IGNITE-9033 > >
RE: Thin clients: WithExpiryPolicy
Pavel, Thanks for sharing your thoughts. Right now I see that 2 reserved bytes come with every cache request: cacheId and flags. The first one is obvious, but the second one is used only for java thin client with enabled KeepBinary mode. I think we could use the flags byte and write an expiration policy flag when required. Whenever the server sees that there is a request with expiration flag, we deserialize a policy and apply it to the request. From: Pavel Tupitsyn Sent: Friday, October 18, 2019 7:05 PM To: dev Subject: Re: Thin clients: WithExpiryPolicy Stateless approach looks a lot better to me. We have a choice: * Keep expiry policy on server and send an ID with every request (like a query cursor ID - 8 bytes) * Send full expiry policy with every expiry-enabled request (24 bytes - or maybe less? We should think about the format) Stateful approach will bring a lot of complexity if we consider Affinity Awareness [1] mechanism (and also automatic reconnect). We would have to keep ExpiryPolicyId for every server and choose the right one based on the affinity for every operation. This can easily negate any performance gain from saving 16 bytes. And there is always CacheConfiguration.ExpiryPolicyFactory, which allows us to set up default expiry policy. > things could get worse if we decided to add a few more WithSomething* methods I don't think this is a good argument - we should decide on case by case basis. Anyway, other With* methods don't have any parameters, so they carry only 1 bit of information. https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients On Fri, Oct 18, 2019 at 5:14 PM Alexandr Shapkin wrote: > Igniters, > > I would like to add WithExpirePolicy support to thin clients. [1] > For a thick client, we can obtain a reference to a cache wrapper instance > and use cache API through it. At the same time, the thin client protocol is > stateless, we do not hold a reference to a cache but rather a cache name > identifier is used for a server to create an appropriate cache instance. > > We could extend the protocol as we did with WithKeepBinaryMethod: > every time we need to call some API on a cache with expiration, a > serialized ExpiryPolicy (additional 3*8 bytes) would be sent. This approach > works well, but things could get worse if we decided to add a few more > WithSomething* methods. > > Initially, I was thinking about introducing some state context to a > protocol, similar to a QueryCursor API. For instance, we can save an expire > policy configuration for the first call and use some hash value based on an > ExpiryPolicy for further calls, just as we do for cache names. I.e. > newCacheId = [cacheId, new AdditionalValues(expiryPolicyId, binaryModeId, > )] But this approach complicates logic and leads to additional memory > consumption. > > I think it's ok for now to use the first approach with ExpiryPolicy > serialization. > But any ideas are welcome. > > [1] - https://issues.apache.org/jira/browse/IGNITE-9033 > >
Re: Ignite diagnostic (SQL system views)
Alex, Nikolay, thanks! Forgive me for my poor googling skills ;) - Denis On Fri, Oct 18, 2019 at 7:26 AM Alex Plehanov wrote: > Documentation for views released in 2.7 is here: > https://apacheignite-sql.readme.io/docs/system-views > > пт, 18 окт. 2019 г. в 17:24, Nikolay Izhikov : > > > Denis, AFAIK we doesn't have documentation for the SQL System Views > > existing in Ignite. > > > > I have plans to write a documentation about metrics and syste views that > > will cover SQL System View. > > It will be available till 2.8 release. > > > > > > > > В Пт, 18/10/2019 в 07:16 -0700, Denis Magda пишет: > > > Alex, Igniters, > > > > > > Who of us was contributing this feature? I don’t see any documentation, > > not > > > clear how the users are expected to benefit from the capability and how > > > everybody will be aware of the feature existence. > > > > > > We need to close the gap and spread the word. > > > > > > Denis > > > > > > On Thursday, October 17, 2019, Alex Plehanov > > > wrote: > > > > > > > Denis, > > > > > > > > Views engine and some views were released in AI 2.7. > > > > In 2.8 they will be moved to the new engine and new views will be > > added (as > > > > part of IEP-35) > > > > > > > > > > > > пт, 18 окт. 2019 г. в 00:50, Denis Magda : > > > > > > > > > Anton, Maxim, > > > > > > > > > > Are we planning to release the views as part of 2.8? Don't see them > > > > > > > > listed > > > > > in the important features section: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8# > > > > > > > > ApacheIgnite2.8-Themostimportantreleasetasks > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Wed, Feb 14, 2018 at 1:49 AM Anton Vinogradov < > > > > > > > > avinogra...@gridgain.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > Vova, > > > > > > > > > > > > Could you confirm > > https://issues.apache.org/jira/browse/IGNITE-7527 > > > > > > > > > > ready > > > > > > to be merged? > > > > > > > > > > > > On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov < > > > > > > > > voze...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > I would start with NODES and NODE_ATTRIBUTES as the most simple > > > > > > > > thing. > > > > > > > > > > > > > > On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda < > dma...@apache.org> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Alex P, sounds like a good plan for me. > > > > > > > > > > > > > > > > Vladimir, do you have any suggestions or corrections? > > > > > > > > > > > > > > > > — > > > > > > > > Denis > > > > > > > > > > > > > > > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov < > > > > > > > > > > plehanov.a...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > The views engine and the first view are almost ready to > merge > > > > > > > > > > (review > > > > > > > > > comments are resolved). Which views should we take next? My > > > > > > > > > > proposal > > > > > > - > > > > > > > > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and > > > > > > > > > > NODE_ADDRESSES, > > > > > > > > since > > > > > > > > > these views are clear and all topology data available on > each > > > > > > > > node. > > > > > > > > > Any objections? > > > > > > > > > > > > > > > > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov < > > > > > > > > plehanov.a...@gmail.com > > > > > > : > > > > > > > > > > > > > > > > > > > Anton, Vladimir, I've made some fixes. There is only one > > view > > > > > > > > left > > > > > > and > > > > > > > > > > it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'. > > > > > > > > > > > > > > > > > > > > High level design of solution: > > > > > > > > > > When IgniteH2Indexing is starting, it create and start > > > > > > > > > > new GridH2SysViewProcessor, which create and register in > > H2 (via > > > > > > > > > > its > > > > > > > own > > > > > > > > > > table engine) all implementations of system views. Each > > system > > > > > > > > > > view > > > > > > > > > > implementation extends base abstract class GridH2SysView. > > View > > > > > > > > > > implementation describes columns, their types and indexes > > in > > > > > > > > > > > > > > constructor > > > > > > > > > > and must override method getRows for data retrieval (this > > method > > > > > > > > > > > > > > called > > > > > > > > by > > > > > > > > > > H2-compatible table and index implementations for ignite > > system > > > > > > > > > > > > > > views). > > > > > > > > > > Almost no fixes to existing parsing engine was made, > > except some > > > > > > > > > > > > > > places, > > > > > > > > > > where GridH2Table instance was expected, but for system > > views > > > > > > > > > > there > > > > > > is > > > > > > > > > > another class. > > > > > > > > > > > > > > > > > > > > New PR: [1]. Please have a look. > > > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/ignite/pull/3433 > > > > > > > > > > > > > > > > > > > > 2018-01-24 19:12 GMT+03:00 Anton Vinogradov < > > > > > > > > > > > >
Re: [DISCUSS] Pub/Sub Streamer Implementation
Hello Denis, Sure, sounds good. I will create a separate discussion thread to get community feedback on whether we should join the Bahir or kick-off "Ignite Extensions" project. Regards, Saikat On Thu, Oct 17, 2019 at 2:21 PM Denis Magda wrote: > Folks, > > The concept of Apache Bahir is precisely what we need! Saikat, thanks for > doing the research. Why I believe the * idea* fits us well: > >- All integrations can be stored in separate Github repositories and >have their dev lifecycles. We've not obliged to couple all the > integrations >in a single repo. For instance, Spark can be located in a dedicated repo >while streaming integrations might be in a single one. It's up to us. >- All the repositories and integrations belong to ASF. We're not dumping >them on Github but rather keep supporting and developing in accordance > with >ASF vision and practices. >- There is a way to reward contributors via committership and PMC >membership. You won't get this if a project is just one of the millions > of >Github projects. > > Saikat, as Ignite PMC member, could you please kick-off a separate dev-list > discussion to involve more community members and referring to this thread? > I think the primary question the community needs to answer whether we > should join the Bahir or kick-off "Ignite Extensions" project? > > - > Denis > > > On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev > wrote: > > > Maybe we could move all our Streaming Integrations there, but what is > about > > maintaining and committer permissions to the new repositories? > > I see the list of the committers and PMC members there > > https://bahir.apache.org/community-members/ > > > > Could somebody from Ignite community be added to this list as a > > PMC/committer to maintain Ignite integrations? > > If yes, I'd happy to join this project and collaborate with these guys > > together > > If no, and we should start from the zero level with external PRs - hmmm, > > it's better to have our own external repository with ApacheBahirr > ideology. > > > > Also, I agree, that all connectors could be moved there (in ApacheBahirr > > like repository), but not all modules from the page > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > because > > they use different parts of core modules and could be developed with > > different velocity. > > > > In reality, before creation of new repositories we need wide discussion > > based on the document mentioned above. > > > > P.S. Of course, we could experiment in the next release 2.8 with one or > two > > integrations like twitter or ZeroMQ > > Also, I'd like Denis Magda idea for the separate repositories in Apache > > Github to maintain them by anybody. > > > > > > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra : > > > > > Hi Denis, Emmanouil > > > > > > Thank you for your email. I think creating a separate integration > project > > > in github and also proposing it as an apache incubator project will be > a > > > good move. The other separate integration modules > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > can > > > be moved to this new apache incubator project. > > > > > > There are similar integration available for Flink and Spark in Apache > > Bahir > > > https://bahir.apache.org/ > > > > > > Do you think we should reach out to Apache Bahir community with a > > proposal > > > or we should plan to create a separate Apache Incubator project? > > > > > > Regards, > > > Saikat > > > > > > > > > > > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < > > > gkatzio...@gmail.com> > > > wrote: > > > > > > > Hi all! > > > > > > > > Based on the discussions most probably this is going to be on another > > > > GitHub repo. Therefore I will prepare a standalone project with the > > > Pub/Sub > > > > module and once the repository is created a pull request shall be > > issued. > > > > If there is anything else I can do in order to facilitate this please > > let > > > > me know. > > > > > > > > If a ticket is created for this issue, I shall be able to create a > pull > > > > request and thus a build will take place on ignite's CI. > > > > If everyone is aligned with adding a Pub/Sub implementation may I > > create > > > a > > > > ticket on JIRA? > > > > > > > > My next question has to do on how Ignite configures the build jobs in > > > Team > > > > City. The unit tests for my implementation need a local Pub/Sub > server > > up > > > > and running (they provide one for testing purposes). Will this take > > > effect > > > > in a form of a build script inside my implementation or is it > something > > > > that should be configured on Team City? > > > > > > > > Kind regards, > > > > Emmanouil > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > https://egkatzioura.com/ | > > > > https://www.linkedin
[DISCUSS] Proposal for Ignite Extensions as a separate Bahir module or Incubator project
Hello, We wanted to discuss on a proposal to move and support the Apache Ignite integrations as separate Ignite Extensions as discussed here http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Pub-Sub-Streamer-Implementation-td43944.html . The reason we wanted to move our Apache Ignite integration as separate Extensions is this will help us to manage and maintain separate lifecycle for Apache Ignite integrations. All the integrations will continue to be part of ASF and we will keep supporting and developing in accordance with ASF vision and practices. We are considering following two choices for moving to Apache Ignite Extensions: 1. Reach out to Apache Bahir community and propose to make Ignite Extensions a separate module as part of Apache Bahir project. https://bahir.apache.org/ https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces96 2. Reach out to Apache Incubator community and request for a new project for Ignite Extensions. Please review and share feedback on our proposal. Warm Regards, Saikat
