Re: [DISCUSS] Pub/Sub Streamer Implementation

[Alexey Zinoviev]

Maybe we could move all our Streaming Integrations there, but what is about
maintaining and committer permissions to the new repositories?
I see the list of the committers and PMC members there
https://bahir.apache.org/community-members/

Could somebody from Ignite community be added to this list as a
PMC/committer to maintain Ignite integrations?
If yes, I'd happy to join this project and collaborate with these guys
together
If no, and we should start from the zero level with external PRs - hmmm,
it's better to have our own external repository with ApacheBahirr ideology.

Also, I agree, that all connectors could be moved there (in ApacheBahirr
like repository), but not all modules from the page
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
because
they use different parts of core modules and could be developed with
different velocity.

In reality, before creation of new repositories we need wide discussion
based on the document mentioned above.

P.S. Of course, we could experiment in the next release 2.8 with one or two
integrations like twitter or ZeroMQ
Also, I'd like Denis Magda idea for the separate repositories in Apache
Github to maintain them by anybody.


чт, 17 окт. 2019 г. в 05:02, Saikat Maitra :

> Hi Denis, Emmanouil
>
> Thank you for your email. I think creating a separate integration project
> in github and also proposing it as an apache incubator project will be a
> good move. The other separate integration modules
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> can
> be moved to this new apache incubator project.
>
> There are similar integration available for Flink and Spark in Apache Bahir
> https://bahir.apache.org/
>
> Do you think we should reach out to Apache Bahir community with a proposal
> or we should plan to create a separate Apache Incubator project?
>
> Regards,
> Saikat
>
>
>
> On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> gkatzio...@gmail.com>
> wrote:
>
> > Hi all!
> >
> > Based on the discussions most probably this is going to be on another
> > GitHub repo. Therefore I will prepare a standalone project with the
> Pub/Sub
> > module and once the repository is created a pull request shall be issued.
> > If there is anything else I can do in order to facilitate this please let
> > me know.
> >
> > If a ticket is created for this issue, I shall be able to create a pull
> > request and thus a build will take place on ignite's CI.
> > If everyone is aligned with adding a Pub/Sub implementation may I create
> a
> > ticket on JIRA?
> >
> > My next question has to do on how Ignite configures the build jobs in
> Team
> > City. The unit tests for my implementation need a local Pub/Sub server up
> > and running (they provide one for testing purposes). Will this take
> effect
> > in a form of a build script inside my implementation or is it something
> > that should be configured on Team City?
> >
> > Kind regards,
> > Emmanouil
> >
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Tue, 15 Oct 2019 at 23:27, Denis Magda  wrote:
> >
> > > Emmanouil, Saikat,
> > >
> > > After contemplating on this, let me propose another way with this
> > > particular integration that is aligned with our modularization
> endeavor.
> > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > independent integrations. The first two are to be located in Ignite
> > > repositories and to be tested/updated by the community for every
> release.
> > > The modules are a subset of existing integrations that are adopted
> widely
> > > and require community attention.
> > >
> > > As for the independent integrations [1], those are supposed to be
> located
> > > in a separate GitHub repo. I would advise us to start implementing this
> > > concept with the Pub/Sub integration. Even though it will be located
> in a
> > > separate Github repo, it will be listed among officially available
> > > integrations on the Ignite website featuring Emmanouil as a
> contributor.
> > >
> > > What do you think?
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >
> > >
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra  >
> > > wrote:
> > >
> > > > Hi Emmanouil,
> > > >
> > > > The changes looks good to me. I wanted to check if you are also able
> to
> > > > validate that the once data is added to Ignite cluster you are also
> > able
> > > to
> > > > access it using another Ignite client or using rest endpoints.
> > > >
> > > > We use Teamcity for CI process, the details are as follows
> > > >
> > > > Trigger validation of those test suites that have been affected by
> your
> > > > cha

[jira] [Created] (IGNITE-12298) Write tombstones on incomplete baseline to get rid of partition cleanup

[Pavel Kovalenko (Jira)]

Pavel Kovalenko created IGNITE-12298:


 Summary: Write tombstones on incomplete baseline to get rid of 
partition cleanup
 Key: IGNITE-12298
 URL: https://issues.apache.org/jira/browse/IGNITE-12298
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.8
Reporter: Pavel Kovalenko
 Fix For: 2.9


After tombstone objects are introduced 
https://issues.apache.org/jira/browse/IGNITE-11704
we can write tombstones on OWNING nodes if the baseline is incomplete (some of 
the backup nodes are left). When baseline completes and old nodes return back 
we can avoid partition cleanup on those nodes before rebalance. We can 
translate the whole OWNING partition state including tombstones that will clear 
the data that was removed when node was offline.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12299) Store tombstone links into separate BPlus tree to avoid partition full-scan during tombstones remove

[Pavel Kovalenko (Jira)]

Pavel Kovalenko created IGNITE-12299:


 Summary: Store tombstone links into separate BPlus tree to avoid 
partition full-scan during tombstones remove
 Key: IGNITE-12299
 URL: https://issues.apache.org/jira/browse/IGNITE-12299
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.8
Reporter: Pavel Kovalenko
 Fix For: 2.9


Currently, we can't identify which keys are tombstones in the partition fastly. 
To collect tombstones we need to make a full-scan BPlus tree. It can slowdown 
node performance when rebalance is finished and tombstones cleanup is needed. 
We can introduce a separate BPlus tree (like for TTL) inside partition where we 
can store links to tombstone keys. When tombstones cleanup is needed we can 
make a fast scan for tombstones using the only a subset of the keys stored to 
this tree.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Pub/Sub Streamer Implementation

[Ilya Kasnacheev]

Hello!

Historically I support some integrations, so I can final review/merge
integrations commit until someone else steps in.

Regards,
-- 
Ilya Kasnacheev


чт, 17 окт. 2019 г. в 12:54, Alexey Zinoviev :

> Maybe we could move all our Streaming Integrations there, but what is about
> maintaining and committer permissions to the new repositories?
> I see the list of the committers and PMC members there
> https://bahir.apache.org/community-members/
>
> Could somebody from Ignite community be added to this list as a
> PMC/committer to maintain Ignite integrations?
> If yes, I'd happy to join this project and collaborate with these guys
> together
> If no, and we should start from the zero level with external PRs - hmmm,
> it's better to have our own external repository with ApacheBahirr ideology.
>
> Also, I agree, that all connectors could be moved there (in ApacheBahirr
> like repository), but not all modules from the page
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> because
> they use different parts of core modules and could be developed with
> different velocity.
>
> In reality, before creation of new repositories we need wide discussion
> based on the document mentioned above.
>
> P.S. Of course, we could experiment in the next release 2.8 with one or two
> integrations like twitter or ZeroMQ
> Also, I'd like Denis Magda idea for the separate repositories in Apache
> Github to maintain them by anybody.
>
>
> чт, 17 окт. 2019 г. в 05:02, Saikat Maitra :
>
> > Hi Denis, Emmanouil
> >
> > Thank you for your email. I think creating a separate integration project
> > in github and also proposing it as an apache incubator project will be a
> > good move. The other separate integration modules
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > can
> > be moved to this new apache incubator project.
> >
> > There are similar integration available for Flink and Spark in Apache
> Bahir
> > https://bahir.apache.org/
> >
> > Do you think we should reach out to Apache Bahir community with a
> proposal
> > or we should plan to create a separate Apache Incubator project?
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > gkatzio...@gmail.com>
> > wrote:
> >
> > > Hi all!
> > >
> > > Based on the discussions most probably this is going to be on another
> > > GitHub repo. Therefore I will prepare a standalone project with the
> > Pub/Sub
> > > module and once the repository is created a pull request shall be
> issued.
> > > If there is anything else I can do in order to facilitate this please
> let
> > > me know.
> > >
> > > If a ticket is created for this issue, I shall be able to create a pull
> > > request and thus a build will take place on ignite's CI.
> > > If everyone is aligned with adding a Pub/Sub implementation may I
> create
> > a
> > > ticket on JIRA?
> > >
> > > My next question has to do on how Ignite configures the build jobs in
> > Team
> > > City. The unit tests for my implementation need a local Pub/Sub server
> up
> > > and running (they provide one for testing purposes). Will this take
> > effect
> > > in a form of a build script inside my implementation or is it something
> > > that should be configured on Team City?
> > >
> > > Kind regards,
> > > Emmanouil
> > >
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Tue, 15 Oct 2019 at 23:27, Denis Magda  wrote:
> > >
> > > > Emmanouil, Saikat,
> > > >
> > > > After contemplating on this, let me propose another way with this
> > > > particular integration that is aligned with our modularization
> > endeavor.
> > > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > > independent integrations. The first two are to be located in Ignite
> > > > repositories and to be tested/updated by the community for every
> > release.
> > > > The modules are a subset of existing integrations that are adopted
> > widely
> > > > and require community attention.
> > > >
> > > > As for the independent integrations [1], those are supposed to be
> > located
> > > > in a separate GitHub repo. I would advise us to start implementing
> this
> > > > concept with the Pub/Sub integration. Even though it will be located
> > in a
> > > > separate Github repo, it will be listed among officially available
> > > > integrations on the Ignite website featuring Emmanouil as a
> > contributor.
> > > >
> > > > What do you think?
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >
> > > >
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra

Embedded Version of H2

[steve.hostett...@gmail.com]

Hello,

we would like to use new H2 features (especially the OVER keyword)
introduced in 1.4.198. Currently, 2.8.0-SNAPSHOT is relying on 1.4.197. Is
there any plan to upgrade and if not, why? Do you expect any major issue
with that upgrade or the use of the OVER keyword?

Steve



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Embedded Version of H2

[steve.hostett...@gmail.com]

Hello, 

Actually I found the answer with the following ticket :
https://issues.apache.org/jira/browse/IGNITE-10801

Seems to be planned for 2.8.0




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-12300) ComputeJob#cancel executes with wrong SecurityContext

[Denis Garus (Jira)]

Denis Garus created IGNITE-12300:


 Summary: ComputeJob#cancel executes with wrong SecurityContext
 Key: IGNITE-12300
 URL: https://issues.apache.org/jira/browse/IGNITE-12300
 Project: Ignite
  Issue Type: Bug
Reporter: Denis Garus


ComputeJob#cancel executes with context of current node rather then should be 
executed with context of node that initiate ComputeJob.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

[Yuriy Shuliga]

  Andrey,

Per you request, I created ticket
https://issues.apache.org/jira/browse/IGNITE-12291   linked to
https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12189

Could you please proceed with PR merge ?

BR,
Yuriy Shuliha

ср, 9 жовт. 2019 о 12:52 Andrey Mashenkov  пише:

> Hi Yuri,
>
> To get access to TC Bot you should register as TeamCity user [1], if you
> didn't do this already.
> Then you will be able to authorize on Ignite TC Bot page with same
> credentials.
>
> [1] https://ci.ignite.apache.org/registerUser.html
>
> On Fri, Oct 4, 2019 at 3:10 PM Yuriy Shuliga  wrote:
>
>> Andrew,
>>
>> I have corrected PR according to your notes. Please review.
>> What will be the next steps in order to merge in?
>>
>> Y.
>>
>> чт, 3 жовт. 2019 о 17:47 Andrey Mashenkov 
>> пише:
>>
>> > Yuri,
>> >
>> > I've done with review.
>> > No crime found, but trivial compatibility bug.
>> >
>> > On Thu, Oct 3, 2019 at 3:54 PM Yuriy Shuliga  wrote:
>> >
>> > > Denis,
>> > >
>> > > Thank you for your attention to this.
>> > > as for now, the https://issues.apache.org/jira/browse/IGNITE-12189
>> > ticket
>> > > is still pending review.
>> > > Do we have a chance to move it forward somehow?
>> > >
>> > > BR,
>> > > Yuriy Shuliha
>> > >
>> > > пн, 30 вер. 2019 о 23:35 Denis Magda  пише:
>> > >
>> > > > Yuriy,
>> > > >
>> > > > I've seen you opening a pull-request with the first changes:
>> > > > https://issues.apache.org/jira/browse/IGNITE-12189
>> > > >
>> > > > Alex Scherbakov and Ivan are you the right guys to do the review?
>> > > >
>> > > > -
>> > > > Denis
>> > > >
>> > > >
>> > > > On Fri, Sep 27, 2019 at 8:48 AM Павлухин Иван 
>> > > wrote:
>> > > >
>> > > > > Yuriy,
>> > > > >
>> > > > > Thank you for providing details! Quite interesting.
>> > > > >
>> > > > > Yes, we already have support of distributed limit and merging
>> sorted
>> > > > > subresults for SQL queries. E.g. ReduceIndexSorted and
>> > > > > MergeStreamIterator are used for merging sorted streams.
>> > > > >
>> > > > > Could you please also clarify about score/relevance? Is it
>> provided
>> > by
>> > > > > Lucene engine for each query result? I am thinking how to do
>> sorted
>> > > > > merge properly in this case.
>> > > > >
>> > > > > ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga :
>> > > > > >
>> > > > > > Ivan,
>> > > > > >
>> > > > > > Thank you for interesting question!
>> > > > > >
>> > > > > > Text searches (or full text searches) are mostly human-oriented.
>> > And
>> > > > the
>> > > > > > point of user's interest is topmost part of response.
>> > > > > > Then user can read it, evaluate and use the given records for
>> > further
>> > > > > > purposes.
>> > > > > >
>> > > > > > Particularly in our case, we use Ignite for operations with
>> > financial
>> > > > > data,
>> > > > > > and there lots of text stuff like assets names, fin.
>> instruments,
>> > > > > companies
>> > > > > > etc.
>> > > > > > In order to operate with this quickly and reliably, users used
>> to
>> > > work
>> > > > > with
>> > > > > > text search, type-ahead completions, suggestions.
>> > > > > >
>> > > > > > For this purposes we are indexing particular string data in
>> > separate
>> > > > > caches.
>> > > > > >
>> > > > > > Sorting capabilities and response size limitations are very
>> > important
>> > > > > > there. As our API have to provide most relevant information in
>> view
>> > > of
>> > > > > > limited size.
>> > > > > >
>> > > > > > Now let me comment some Ignite/Lucene perspective.
>> > > > > > Actually Ignite queries and Lucene returns *TopDocs.scoresDocs
>> > > *already
>> > > > > > sorted by *score *(relevance). So most relevant documents are on
>> > the
>> > > > top.
>> > > > > > And currently distributed queries responses from different nodes
>> > are
>> > > > > merged
>> > > > > > into final query cursor queue in arbitrary way.
>> > > > > > So in fact we already have the score order ruined here. Also
>> Ignite
>> > > > > > requests all possible documents from Lucene that is redundant
>> and
>> > not
>> > > > > good
>> > > > > > for performance.
>> > > > > >
>> > > > > > I'm implementing *limit* parameter to be part of *TextQuery *and
>> > have
>> > > > to
>> > > > > > notice that we still have to add sorting for text queries
>> > processing
>> > > in
>> > > > > > order to have applicable results.
>> > > > > >
>> > > > > > *Limit* parameter itself should improve the part of issues from
>> > > above,
>> > > > > but
>> > > > > > definitely, sorting by document score at least  should be
>> > implemented
>> > > > > along
>> > > > > > with limit.
>> > > > > >
>> > > > > > This is a pretty short commentary if you still have any
>> questions,
>> > > > please
>> > > > > > ask, do not hesitate)
>> > > > > >
>> > > > > > BR,
>> > > > > > Yuriy Shuliha
>> > > > > >
>> > > > > > чт, 19 вер. 2019 о 11:38 Павлухин Иван 
>> пише:
>> > > > > >
>> > > > > > > Yuriy,
>> > > > > > >
>> > > > > > > Greatly appreciate your interest.
>> > > > > > >
>> > > > > > > Co

Re: Review needed for IGNITE-11410 Sandbox for user-defined code

[Denis Garus]

Hello, Pavel!

Thank you for the feedback!

I've created IEP-38 that describes the Ignite Sandbox [1].

Yes, the issue requires documentation (there is the flag "Docs equired"),
but common practice is to write documentation in the end.

>> 1) Why do you run resource injection through security and how it tested?
>> 2) Why do you check security at *dumpThreads* and *wrapThreadLoader
*methods?
>>  These methods are needed only for internal node processes.
>> 4) There are suspicious security checks at:
>> CacheOperationContext:37
>> GridCacheDefaultAffinityKeyMapper:86
>> PageMemoryImpl:874
>> I'm not following why they are needed.

These questions have a common answer.
A user-defined code can call any operation through the public API of
Ignite. But he may don't have permissions to execute this operation
successfully.
For example, to put a value into a cache, it requires permissions for
accessing to reflection API and reading system property
IGNITE_ALLOW_ATOMIC_OPS_IN_TX.
In that case, we have to use AccessController#doPrirvelged call to exclude
a user-defined code from checking of permissions.
SecurityUtils#doPriveleged does calling AccessController#doPrirvelged a
more convenient way.

>> 3) Have you tested security if a compute job is canceled?
You are right; we should add a test for the cancel case.
But, for now, we have the issue [2] with the current SecurityContext for
the canceling of ComputeJob.

1. https://cwiki.apache.org/confluence/display/IGNITE/IEP-38%3A+Sandbox
2. https://issues.apache.org/jira/browse/IGNITE-12300

пн, 14 окт. 2019 г. в 16:16, Pavel Kovalenko :

> Denis,
>
> The idea of having a sandbox for running a user-defined code is useful, but
> I don't fully understand the implementation approach.
> There is no detailed description of the ticket about what public API
> methods or configuration parameters should be covered.
> There is no description of what have done in initial PR and how.
> First of all, there should be an umbrella ticket that should contain all
> public API points and configuration parameters where used defined code may
> be run.
> Without a full list of all possible user-defined code injections, we can't
> track what was covered and where are a possible security lacks.
> I've checked PR and I have the following questions:
> 1) Why do you run resource injection through security and how it tested?
> 2) Why do you check security at *dumpThreads* and *wrapThreadLoader
> *methods?
> These methods are needed only for internal node processes.
> 3) Have you tested security if a compute job is canceled?
> 4) There are suspicious security checks at:
> CacheOperationContext:37
> GridCacheDefaultAffinityKeyMapper:86
> PageMemoryImpl:874
> I'm not following why they are needed.
>
>
>
>
> пн, 14 окт. 2019 г. в 12:19, Anton Vinogradov :
>
> > Fully agree with the benchmark's importance.
> > Currently, we're not able to perform proper benchmarking.
> > Slava, Is it possible to ask you to check the solution using GridGain's
> > benchmarking environment?
> >
> > On Mon, Oct 14, 2019 at 12:07 PM Вячеслав Коптилин <
> > slava.kopti...@gmail.com>
> > wrote:
> >
> > > Hello Anton,
> > >
> > > > We should avoid heavy merges if possible.
> > > Why it should be avoided? To be honest, I don't see any reason for
> that.
> > > Every pull request can be and should be reviewed when it is done and
> > ready
> > > to be merged into the epic branch (IEP branch).
> > > So, the final review of the entire IEP is just a technical/trivial
> task,
> > in
> > > my opinion.
> > >
> > > If I am not mistaken, we are at the stage of preparing a new release
> > (2.8),
> > > right?
> > > And we are trying to add a new feature that may impact the performance.
> > > For example, affinity function, which can be overridden by the
> end-user,
> > > and therefore should be covered by `sandbox`.
> > > On the other hand, affinity function is a crucial component that is
> used
> > > very often.
> > > Are we really sure that the proposed change does not affect the
> > > performance? Do we have a benchmark?
> > >
> > > Please don't get me wrong, guys. I am not against the feature itself.
> > > Moreover, it is a great feature and improvement of security.
> > > I just want to say that we need to be sure that we are on the right way
> > of
> > > implementing this without affecting other developers.
> > >
> > > PS: This is just my opinion, which may be wrong.
> > >
> > > Thanks,
> > > S.
> > >
> > > пн, 14 окт. 2019 г. в 09:09, Anton Vinogradov :
> > >
> > > > Folks,
> > > >
> > > > We should avoid heavy merges if possible.
> > > > I'm ok with IEP to keep tasks properly, but strictly against
> all-in-one
> > > > "+27000,-18200" merges.
> > > > This task implements Sandbox (API + core) which covered by tests and
> by
> > > > some integrations with existing components, which is enough to merge.
> > > > The most important thing here is that we will be able to speed-up
> > Sandbox
> > > > coverage development once its core menged t

[jira] [Created] (IGNITE-12301) Free-lists system view

[Aleksey Plekhanov (Jira)]

Aleksey Plekhanov created IGNITE-12301:
--

 Summary: Free-lists system view
 Key: IGNITE-12301
 URL: https://issues.apache.org/jira/browse/IGNITE-12301
 Project: Ignite
  Issue Type: Sub-task
Reporter: Aleksey Plekhanov
Assignee: Aleksey Plekhanov


Implement a system view for free-lists monitoring.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: TDE Master key rotation (Phase-2)

[Nikita Amelchev]

Hi, Igniters!

I have implemented the master key change process [1] for TDE as
described in the design [2].

I have prepared PR [3] and created the Upsource review branch [4].

Could anyone take a look at my changes?

Can we include it into a 2.8 release scope?

[1] https://issues.apache.org/jira/browse/IGNITE-12186
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381
[3] https://github.com/apache/ignite/pull/6937
[4] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-1067

пн, 23 сент. 2019 г. в 17:13, Nikolay Izhikov :
>
> Hello, Nikita.
>
> > A node creates the ChangeMasterKeyMessage message and sent it by discovery 
> > as a custom event.
> > The goal is to verify that all nodes have the same master key.
> ...
> > The ChangeMasterKeyFinishMessage action message is sent by discovery as a 
> > custom event.
> > New master key id.
>
> 1. We should store locally new key id and new key hash after processing of 
> ChangeMasterKeyMessage
> 2. We should send new key hash in ChangeMasterKeyFinishMessage
> 3. We should ensure that both ChangeMasterKeyMessage and 
> ChangeMasterKeyFinishMessage contains the same key id and key hash before 
> executing a change process.
>
> I think we should rename:
>
> > Node left during key rotation process(was not starting re-encrypt cache 
> > keys)
>
> Node was down during key rotation. ChangeMasterKeyRecord was not found.
>
> > Node left during key rotation process(was starting re-encrypt cache keys)
>
> Node was down during key rotation. ChangeMasterKeyRecord found.
>
> We should add a description of changes in the cluster join process.
> A node should not try to join to the cluster before the process of 
> ChangeMasterKeyRecord.
>
> It's not clear for me how we differ two cases:
>
> 1. ChangeMasterKeyRecord found in WAL and key rotation was finished 
> successfully.
> 2. ChangeMasterKeyRecord found in WAL and key rotation was NOT finished 
> successfully.
>
> > Meta storage will store master key id.
>
> We should add that key id from metastorage has a higher priority to key id 
> from IgniteConfiguration.
>
>
> В Пт, 20/09/2019 в 14:06 +0300, Nikita Amelchev пишет:
> > Nikolay,
> >
> > you are right in many ways. I updated the design on the wiki. [1]
> >
> > [1] 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381
> >
> > пт, 20 сент. 2019 г. в 13:49, Nikolay Izhikov :
> > >
> > > Nikita
> > >
> > > > I suggested the implementation where the encryption manager is
> > > > responsible for storing the master key id.
> > >
> > > I don't think it's a right proposal.
> > >
> > > 1. EncryptionSpi implementation becomes more complicated. Developer of it 
> > > should be aware of Ignite deployment scenarious, etc.
> > > Imagine implementation when EncryptionSpi send master key id to some 
> > > external storage over network(it's happen in Discovery thread)
> > >
> > > 2. Implementation would be duplicate features(saving master key id)
> > >
> > > 3. We already store cache keys in metastore. For me it's a native 
> > > approach to store master key id in the same place.
> > >
> > > What do you think?
> > >
> > >
> > > В Пт, 20/09/2019 в 13:39 +0300, Nikita Amelchev пишет:
> > > > Nikolay,
> > > >
> > > > because I suggested the implementation where the encryption manager is
> > > > responsible for storing the master key id.
> > > > To implement this logic in the EncryptionSpi, we will need to
> > > > introduce the methods look like this:
> > > >
> > > > setMasterKeyId(String masterKeyId) // Sets "current" master key id
> > > > String getMasterKeyId() // Gets "current" master key id
> > > >
> > > > Follow methods will work with master key that setted by previous method:
> > > >
> > > > byte[] masterKeyDigest()
> > > > byte[] encryptKey(Serializable key)
> > > > Serializable decryptKey(byte[] key)
> > > >
> > > > If such implementation is more reasonable, I will do so.
> > > >
> > > > пт, 20 сент. 2019 г. в 13:04, Nikolay Izhikov :
> > > > >
> > > > > Why do we need "defaultMasterKeyId" instead of *current* master key 
> > > > > id that can be obtained with 
> > > > > `KeystoreEncryptionSpi#getMasterKeyName()`?
> > > > >
> > > > > В Пт, 20/09/2019 в 12:56 +0300, Nikita Amelchev пишет:
> > > > > > Nikolay,
> > > > > >
> > > > > > Thanks for the proposal, I like it.
> > > > > >
> > > > > > The GridEncryptionManager will control the process of master key
> > > > > > rotation, so we should provide him master key id at startup. Seems 
> > > > > > we
> > > > > > should get it from some configuration for encryption.
> > > > > >
> > > > > > I suggest just adding the String defaultMasterKeyId() method into 
> > > > > > the
> > > > > > EncryptionSpi interface. This method gets default master key id used
> > > > > > on first cluster start.
> > > > > >
> > > > > > The specific implementation will be responsible for setting this 
> > > > > > value.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > пт, 20 сент. 2019 г. в 10:44,

Re: Ignite-Spark integration meeting

[Alexey Zinoviev]

The meeting notes and results of discussion are next:

   1. We are going to release Apache Ignite 2.8 with limited support of
   Spark 2.4.4 (known issues are listed here
   https://issues.apache.org/jira/browse/IGNITE-12054)
   2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
   fixed bugs
   3. We will create the developer-profiles spark_2_3 and spark_2_4
   separately from current scala profile
   4. We are ready to move the Ignite-Spark integration to the separate
   module after release 2.8 with support of most actual Spark versions and
   with independent release cycle
   5. We will work together on the next release roadmap with Nikolay and
   all interested igniters to improve the current solution (add new features
   as thin client support, working with caches via binary object and so on)
   6. Nikolay is ready to help with the review of tickets as an author of
   this component

Thanks



пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev :

> Yes, let's use the Russian for short discussion, the meeting notes will be
> presented in English
>
> пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov :
>
>> Hello, Alexey.
>>
>> I'm ready to take part in the meeting.
>> It will be in Russian, isn't it?
>>
>> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
>> > Hi, Igniters, and especially Nikolay Izhikov
>> >
>> > I suggest to arrange the meeting via ASF-slack in
>> #ignite-spark-integration
>> > channel at 17 October, in 16 MSK Time.
>> > The language of the meeting: Russian
>> >
>> > *The topics to discuss:*
>> >
>> >- Review of next tickets
>> >
>> >
>> >1. [Spark][Bug] IgniteSpark integration forget to close the
>> >   IgniteContext and stops the client node in case if error during
>> >   PairFunction logic
>> >   
>> >   2. [Spark] Add initial support of Spark 2.4
>> >   
>> >   3. [Spark] Add joins to Spark Dataframe examples
>> >   
>> >
>> >
>> >- Known issues of migration to 2.4 version
>> >- Organization of multiple version support for Apache Spark
>> >
>> > P.S. meeting notes will be shared with the community
>> > P.P.S Nikolay, if you have any objections about the time or topics,
>> please
>> > let me know
>>
>

Re: TDE Master key rotation (Phase-2)

[Maxim Muzafarov]

Nikita,

> Can we include it into a 2.8 release scope?
I think it is possible since the release scope freeze date has not happened yet.

On Thu, 17 Oct 2019 at 17:36, Nikita Amelchev  wrote:
>
> Hi, Igniters!
>
> I have implemented the master key change process [1] for TDE as
> described in the design [2].
>
> I have prepared PR [3] and created the Upsource review branch [4].
>
> Could anyone take a look at my changes?
>
> Can we include it into a 2.8 release scope?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12186
> [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381
> [3] https://github.com/apache/ignite/pull/6937
> [4] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-1067
>
> пн, 23 сент. 2019 г. в 17:13, Nikolay Izhikov :
> >
> > Hello, Nikita.
> >
> > > A node creates the ChangeMasterKeyMessage message and sent it by 
> > > discovery as a custom event.
> > > The goal is to verify that all nodes have the same master key.
> > ...
> > > The ChangeMasterKeyFinishMessage action message is sent by discovery as a 
> > > custom event.
> > > New master key id.
> >
> > 1. We should store locally new key id and new key hash after processing of 
> > ChangeMasterKeyMessage
> > 2. We should send new key hash in ChangeMasterKeyFinishMessage
> > 3. We should ensure that both ChangeMasterKeyMessage and 
> > ChangeMasterKeyFinishMessage contains the same key id and key hash before 
> > executing a change process.
> >
> > I think we should rename:
> >
> > > Node left during key rotation process(was not starting re-encrypt cache 
> > > keys)
> >
> > Node was down during key rotation. ChangeMasterKeyRecord was not found.
> >
> > > Node left during key rotation process(was starting re-encrypt cache keys)
> >
> > Node was down during key rotation. ChangeMasterKeyRecord found.
> >
> > We should add a description of changes in the cluster join process.
> > A node should not try to join to the cluster before the process of 
> > ChangeMasterKeyRecord.
> >
> > It's not clear for me how we differ two cases:
> >
> > 1. ChangeMasterKeyRecord found in WAL and key rotation was finished 
> > successfully.
> > 2. ChangeMasterKeyRecord found in WAL and key rotation was NOT finished 
> > successfully.
> >
> > > Meta storage will store master key id.
> >
> > We should add that key id from metastorage has a higher priority to key id 
> > from IgniteConfiguration.
> >
> >
> > В Пт, 20/09/2019 в 14:06 +0300, Nikita Amelchev пишет:
> > > Nikolay,
> > >
> > > you are right in many ways. I updated the design on the wiki. [1]
> > >
> > > [1] 
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652381
> > >
> > > пт, 20 сент. 2019 г. в 13:49, Nikolay Izhikov :
> > > >
> > > > Nikita
> > > >
> > > > > I suggested the implementation where the encryption manager is
> > > > > responsible for storing the master key id.
> > > >
> > > > I don't think it's a right proposal.
> > > >
> > > > 1. EncryptionSpi implementation becomes more complicated. Developer of 
> > > > it should be aware of Ignite deployment scenarious, etc.
> > > > Imagine implementation when EncryptionSpi send master key id to some 
> > > > external storage over network(it's happen in Discovery thread)
> > > >
> > > > 2. Implementation would be duplicate features(saving master key id)
> > > >
> > > > 3. We already store cache keys in metastore. For me it's a native 
> > > > approach to store master key id in the same place.
> > > >
> > > > What do you think?
> > > >
> > > >
> > > > В Пт, 20/09/2019 в 13:39 +0300, Nikita Amelchev пишет:
> > > > > Nikolay,
> > > > >
> > > > > because I suggested the implementation where the encryption manager is
> > > > > responsible for storing the master key id.
> > > > > To implement this logic in the EncryptionSpi, we will need to
> > > > > introduce the methods look like this:
> > > > >
> > > > > setMasterKeyId(String masterKeyId) // Sets "current" master key id
> > > > > String getMasterKeyId() // Gets "current" master key id
> > > > >
> > > > > Follow methods will work with master key that setted by previous 
> > > > > method:
> > > > >
> > > > > byte[] masterKeyDigest()
> > > > > byte[] encryptKey(Serializable key)
> > > > > Serializable decryptKey(byte[] key)
> > > > >
> > > > > If such implementation is more reasonable, I will do so.
> > > > >
> > > > > пт, 20 сент. 2019 г. в 13:04, Nikolay Izhikov :
> > > > > >
> > > > > > Why do we need "defaultMasterKeyId" instead of *current* master key 
> > > > > > id that can be obtained with 
> > > > > > `KeystoreEncryptionSpi#getMasterKeyName()`?
> > > > > >
> > > > > > В Пт, 20/09/2019 в 12:56 +0300, Nikita Amelchev пишет:
> > > > > > > Nikolay,
> > > > > > >
> > > > > > > Thanks for the proposal, I like it.
> > > > > > >
> > > > > > > The GridEncryptionManager will control the process of master key
> > > > > > > rotation, so we should provide him master key id at startup. 
> > > > > > > Seems we
> > > > > > > should get it from some configu

Fwd: You are invited to a Persistent Memory Programming Performance Guidelines Webinar 11/7

[Denis Magda]

Alex, Igniters,

This webinar might be relevant as long as we're looking into the
integration with Optane.

-
Denis




Persistent Memory Programming Performance Guidelines
Click here to view this email as a web page.

[image: Intel]
Webinar: Persistent Memory Programming Performance Guidelines

Thursday, November 7, 9:00 am (PT)
Register Today


Denis,

You are invited to attend this webinar. In this session we explain how to
write persistent memory optimized applications in a way that maximizes
performance. We will also be sharing techniques of achieving fail-safety
without compromising performance that are used throughout the Persistent
Memory Development Kit codebase.
You will learn

   - Important performance characteristics of Intel® Optane™ DC Persistent
   Memory.
   - How to design your algorithms in a way that optimizes performance on
   persistent memory.
   - High performance techniques for achieving power-fail safety in your
   code.

Register



[image: Facebook]

[image: LinkedIn]

[image: Twitter]

[image: YouTube]

[image: Instagram]

[image: AddThis]




This was sent to *dma...@gridgain.com * because you
are subscribed to *Webinars*. To view and manag

[jira] [Created] (IGNITE-12302) Test ZookeeperDiscoveryTopologyChangeAndReconnectTest.testDuplicatedNodeId is broken.

[Amelchev Nikita (Jira)]

Amelchev Nikita created IGNITE-12302:


 Summary: Test 
ZookeeperDiscoveryTopologyChangeAndReconnectTest.testDuplicatedNodeId is broken.
 Key: IGNITE-12302
 URL: https://issues.apache.org/jira/browse/IGNITE-12302
 Project: Ignite
  Issue Type: Bug
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita
 Fix For: 2.8


Test checks that a new node will not be started with the same node id. 
Newly added SystemView creates table on node startup and fails with error:

{noformat}
java.lang.AssertionError: Unexpected exception
at 
org.apache.ignite.testframework.GridTestUtils.fail(GridTestUtils.java:622)
at 
org.apache.ignite.testframework.GridTestUtils.assertThrowsAnyCause(GridTestUtils.java:465)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryTopologyChangeAndReconnectTest.testDuplicatedNodeId(ZookeeperDiscoveryTopologyChangeAndReconnectTest.java:582)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2090)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to register 
system view.
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1401)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:615)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:983)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:924)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:912)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:878)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryTopologyChangeAndReconnectTest.lambda$testDuplicatedNodeId$0(ZookeeperDiscoveryTopologyChangeAndReconnectTest.java:583)
at 
org.apache.ignite.testframework.GridTestUtils.assertThrowsAnyCause(GridTestUtils.java:449)
... 11 more
Caused by: class org.apache.ignite.IgniteException: Failed to register system 
view.
at 
org.apache.ignite.internal.processors.query.h2.SchemaManager.createSystemView(SchemaManager.java:238)
at 
org.apache.ignite.internal.processors.query.h2.SchemaManager.createSystemViews(SchemaManager.java:247)
at 
org.apache.ignite.internal.processors.query.h2.SchemaManager.start(SchemaManager.java:195)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.start(IgniteH2Indexing.java:2083)
[2019-10-17 21:50:42,017][INFO ][main][root] >>> Stopping test: 
ZookeeperDiscoveryTopologyChangeAndReconnectTest#testDuplicatedNodeId in 261 ms 
<<<
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.start(GridQueryProcessor.java:250)
at 
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1977)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1213)
... 21 more
Caused by: org.h2.jdbc.JdbcSQLException: Таблица "NODE_ATTRIBUTES" уже 
существует
Table "NODE_ATTRIBUTES" already exists; SQL statement:
CREATE TABLE NODE_ATTRIBUTES(NODE_ID UUID, NAME VARCHAR, VALUE VARCHAR) ENGINE 
"org.apache.ignite.internal.processors.query.h2.sys.SqlSystemTableEngine" 
[42101-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.command.ddl.CreateTable.update(CreateTable.java:86)
at org.h2.command.CommandContainer.update(CommandContainer.java:102)
at org.h2.command.Command.executeUpdate(Command.java:261)
at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement

Re: [DISCUSS] Pub/Sub Streamer Implementation

[Denis Magda]

Folks,

The concept of Apache Bahir is precisely what we need! Saikat, thanks for
doing the research. Why I believe the * idea* fits us well:

   - All integrations can be stored in separate Github repositories and
   have their dev lifecycles. We've not obliged to couple all the integrations
   in a single repo. For instance, Spark can be located in a dedicated repo
   while streaming integrations might be in a single one. It's up to us.
   - All the repositories and integrations belong to ASF. We're not dumping
   them on Github but rather keep supporting and developing in accordance with
   ASF vision and practices.
   - There is a way to reward contributors via committership and PMC
   membership. You won't get this if a project is just one of the millions of
   Github projects.

Saikat, as Ignite PMC member, could you please kick-off a separate dev-list
discussion to involve more community members and referring to this thread?
I think the primary question the community needs to answer whether we
should join the Bahir or kick-off "Ignite Extensions" project?

-
Denis


On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev 
wrote:

> Maybe we could move all our Streaming Integrations there, but what is about
> maintaining and committer permissions to the new repositories?
> I see the list of the committers and PMC members there
> https://bahir.apache.org/community-members/
>
> Could somebody from Ignite community be added to this list as a
> PMC/committer to maintain Ignite integrations?
> If yes, I'd happy to join this project and collaborate with these guys
> together
> If no, and we should start from the zero level with external PRs - hmmm,
> it's better to have our own external repository with ApacheBahirr ideology.
>
> Also, I agree, that all connectors could be moved there (in ApacheBahirr
> like repository), but not all modules from the page
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> because
> they use different parts of core modules and could be developed with
> different velocity.
>
> In reality, before creation of new repositories we need wide discussion
> based on the document mentioned above.
>
> P.S. Of course, we could experiment in the next release 2.8 with one or two
> integrations like twitter or ZeroMQ
> Also, I'd like Denis Magda idea for the separate repositories in Apache
> Github to maintain them by anybody.
>
>
> чт, 17 окт. 2019 г. в 05:02, Saikat Maitra :
>
> > Hi Denis, Emmanouil
> >
> > Thank you for your email. I think creating a separate integration project
> > in github and also proposing it as an apache incubator project will be a
> > good move. The other separate integration modules
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > can
> > be moved to this new apache incubator project.
> >
> > There are similar integration available for Flink and Spark in Apache
> Bahir
> > https://bahir.apache.org/
> >
> > Do you think we should reach out to Apache Bahir community with a
> proposal
> > or we should plan to create a separate Apache Incubator project?
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > gkatzio...@gmail.com>
> > wrote:
> >
> > > Hi all!
> > >
> > > Based on the discussions most probably this is going to be on another
> > > GitHub repo. Therefore I will prepare a standalone project with the
> > Pub/Sub
> > > module and once the repository is created a pull request shall be
> issued.
> > > If there is anything else I can do in order to facilitate this please
> let
> > > me know.
> > >
> > > If a ticket is created for this issue, I shall be able to create a pull
> > > request and thus a build will take place on ignite's CI.
> > > If everyone is aligned with adding a Pub/Sub implementation may I
> create
> > a
> > > ticket on JIRA?
> > >
> > > My next question has to do on how Ignite configures the build jobs in
> > Team
> > > City. The unit tests for my implementation need a local Pub/Sub server
> up
> > > and running (they provide one for testing purposes). Will this take
> > effect
> > > in a form of a build script inside my implementation or is it something
> > > that should be configured on Team City?
> > >
> > > Kind regards,
> > > Emmanouil
> > >
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Tue, 15 Oct 2019 at 23:27, Denis Magda  wrote:
> > >
> > > > Emmanouil, Saikat,
> > > >
> > > > After contemplating on this, let me propose another way with this
> > > > particular integration that is aligned with our modularization
> > endeavor.
> > > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > > independent integrations. The first two are to be located in Ignite
> > > > repositories and 

Re: Ignite-Spark integration meeting

[Denis Magda]

>
>  1. We are going to release Apache Ignite 2.8 with limited support of
>Spark 2.4.4 (known issues are listed here
>https://issues.apache.org/jira/browse/IGNITE-12054)


What does limited support mean? Is it about regressions (something that
worked before but will fail) or capabilities unsupported even for the
presently released version of the integration?

-
Denis


On Thu, Oct 17, 2019 at 7:55 AM Alexey Zinoviev 
wrote:

> The meeting notes and results of discussion are next:
>
>1. We are going to release Apache Ignite 2.8 with limited support of
>Spark 2.4.4 (known issues are listed here
>https://issues.apache.org/jira/browse/IGNITE-12054)
>2. The AI 2.8 will support both, Spark 2.3 and Spark 2.4.4 with known
>fixed bugs
>3. We will create the developer-profiles spark_2_3 and spark_2_4
>separately from current scala profile
>4. We are ready to move the Ignite-Spark integration to the separate
>module after release 2.8 with support of most actual Spark versions and
>with independent release cycle
>5. We will work together on the next release roadmap with Nikolay and
>all interested igniters to improve the current solution (add new
> features
>as thin client support, working with caches via binary object and so on)
>6. Nikolay is ready to help with the review of tickets as an author of
>this component
>
> Thanks
>
>
>
> пн, 14 окт. 2019 г. в 14:36, Alexey Zinoviev :
>
> > Yes, let's use the Russian for short discussion, the meeting notes will
> be
> > presented in English
> >
> > пн, 14 окт. 2019 г. в 14:08, Nikolay Izhikov :
> >
> >> Hello, Alexey.
> >>
> >> I'm ready to take part in the meeting.
> >> It will be in Russian, isn't it?
> >>
> >> В Пн, 14/10/2019 в 14:00 +0300, Alexey Zinoviev пишет:
> >> > Hi, Igniters, and especially Nikolay Izhikov
> >> >
> >> > I suggest to arrange the meeting via ASF-slack in
> >> #ignite-spark-integration
> >> > channel at 17 October, in 16 MSK Time.
> >> > The language of the meeting: Russian
> >> >
> >> > *The topics to discuss:*
> >> >
> >> >- Review of next tickets
> >> >
> >> >
> >> >1. [Spark][Bug] IgniteSpark integration forget to close the
> >> >   IgniteContext and stops the client node in case if error during
> >> >   PairFunction logic
> >> >   
> >> >   2. [Spark] Add initial support of Spark 2.4
> >> >   
> >> >   3. [Spark] Add joins to Spark Dataframe examples
> >> >   
> >> >
> >> >
> >> >- Known issues of migration to 2.4 version
> >> >- Organization of multiple version support for Apache Spark
> >> >
> >> > P.S. meeting notes will be shared with the community
> >> > P.P.S Nikolay, if you have any objections about the time or topics,
> >> please
> >> > let me know
> >>
> >
>

Re: Ignite community is building Calcite-based prototype

[Denis Magda]

Igniters,

I came across good introductory slides to Calcite that might be useful to
those of you who are newbies to it as I am :)
https://www.slideshare.net/JordanHalterman/introduction-to-apache-calcite

Added a reference to the IEP.

-
Denis


On Wed, Oct 9, 2019 at 4:03 PM Stamatis Zampetakis 
wrote:

> Hi Denis,
>
> Great initiative! It will be a great opportunity for the two communities to
> learn from each other.
> Personally, I am too far to attend physically but I find Julian's idea for
> an online meeting very appealing.
> If there will be such an option I will try to join.
>
> Best,
> Stamatis
>
> On Tue, Oct 8, 2019 at 9:54 PM Denis Magda  wrote:
>
> > (Looping in the dev lists as suggested by Julian.)
> >
> > Julian, Calcite community, let me know what you think about this option.
> > We're planning to host an Apache Ignite meetup with our group [1]
> > on November 13th or 14th in the San Francisco Bay Area. The desired topic
> > is the new Ignite SQL engine based on Calcite. Alex Goncharuk will be
> > representing the Ignite community doing the presentation.
> >
> > Is there anybody from the Calcite community (Julian or mister X) who can
> > join the meetup and do a Calcite-intro talk? Alex will be presenting
> next.
> > We would also invite the members of Calcite Bay Area Group. Ignite
> > community will take care of all org hurdles.
> >
> > [1] https://www.meetup.com/Bay-Area-In-Memory-Computing/
> > [2] https://www.meetup.com/Apache-Calcite/
> >
> > -
> > Denis
> >
> >
> > On Mon, Oct 7, 2019 at 1:07 PM Julian Hyde  wrote:
> >
> > > I’m just about to let that meetup group lapse. I have never had the
> time
> > &
> > > energy to organize meetups.
> > >
> > > Maybe we could do an online meeting, that the community could attend?
> > > (Subject to the limits set by Zoom or Hangouts or whatever.) Feel free
> to
> > > suggest something on the dev list.
> > >
> > > Julian
> > >
> > >
> > > On Oct 3, 2019, at 4:16 PM, Denis Magda  wrote:
> > >
> > > Julian,
> > >
> > > I just found out that you run an Apache Calcite meetup in Palo Alto:
> > > https://www.meetup.com/Apache-Calcite/
> > >
> > > What do you think if you schedule a meetup in the mid of November? I'm
> > > based in Silicon Valley and Alex Goncharuk (one of Ignite veterans and
> > > architects) is visiting that month. Alex and I can talk about the
> reasons
> > > why Calcite was selected by Ignite community, covering our architecture
> > of
> > > today and how it's planned to be changed with Calcite. Plus, the group
> > can
> > > give valuable feedback.
> > >
> > > -
> > > Denis
> > >
> > >
> > > -- Forwarded message -
> > > From: Denis Magda 
> > > Date: Wed, Oct 2, 2019 at 3:32 PM
> > > Subject: Re: Ignite community is building Calcite-based prototype
> > > To: dev 
> > > Cc: dev , dev 
> > >
> > >
> > > Julian,
> > >
> > > Thanks a lot for the references and guidance! I do believe that from
> now
> > > on our community guys will become frequent visitors of yours ;)
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Wed, Oct 2, 2019 at 12:40 PM Julian Hyde  wrote:
> > >
> > >> Denis,
> > >>
> > >> I’ve been a follower and admirer of Ignite for several years, so I am
> > >> delighted that you are considering Calcite.
> > >>
> > >> Ask questions on the dev list, log JIRA cases if you find them, and
> > we’ll
> > >> do our best to help.
> > >>
> > >> I’d like to bring to your attention RelBuilder. Some people want to go
> > >> from SQL text to executable plan, but others want to drop in at the
> > >> relational algebra level, and RelBuilder is a convenient interface for
> > the
> > >> latter.
> > >>
> > >> (The other) Julian
> > >>
> > >>
> > >> > On Oct 1, 2019, at 3:43 PM, Denis Magda  wrote:
> > >> >
> > >> > Hi Julian,
> > >> >
> > >> > Nice to e-meet you and thanks for being ready to help! Hopefully,
> the
> > >> > Ignite community will be able to contribute valuable changes back to
> > >> > Calcite as part of this activity - "pay good for good" :)
> > >> >
> > >> > You are right that distributed computing, massive-parallel
> processing,
> > >> and
> > >> > calculations/querying at scale is what Ignite is targeted for.
> > However,
> > >> > while Drill is designed for analytics and IoTDB is for time-series,
> > >> Ignite
> > >> > is primarily used for OLTP with an increasing number of real-time
> > >> analytics
> > >> > use cases (no adhoc).
> > >> >
> > >> > Let's stay in touch!
> > >> >
> > >> > -
> > >> > Denis
> > >> >
> > >> >
> > >> > On Tue, Oct 1, 2019 at 6:42 AM Julian Feinauer <
> > >> j.feina...@pragmaticminds.de>
> > >> > wrote:
> > >> >
> > >> >> Hi Igor,
> > >> >>
> > >> >> I agree that it should be rather similar to what Drill did as
> > >> distributed
> > >> >> computing also is a big concern for Ignite, I guess, right?
> > >> >>
> > >> >> Julian
> > >> >>
> > >> >> Am 01.10.19, 15:06 schrieb "Seliverstov Igor" <
> gvvinbl...@gmail.com
> > >:
> > >> >>
> > >> >>Guys,
> > >> >>
> > >> >>The better

Re: Ignite diagnostic (SQL system views)

[Denis Magda]

Anton, Maxim,

Are we planning to release the views as part of 2.8? Don't see them listed
in the important features section:
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Themostimportantreleasetasks

-
Denis


On Wed, Feb 14, 2018 at 1:49 AM Anton Vinogradov 
wrote:

> Vova,
>
> Could you confirm https://issues.apache.org/jira/browse/IGNITE-7527 ready
> to be merged?
>
> On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov 
> wrote:
>
> > I would start with NODES and NODE_ATTRIBUTES as the most simple thing.
> >
> > On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda  wrote:
> >
> > > Alex P, sounds like a good plan for me.
> > >
> > > Vladimir, do you have any suggestions or corrections?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov 
> > > wrote:
> > > >
> > > > The views engine and the first view are almost ready to merge (review
> > > > comments are resolved). Which views should we take next? My proposal
> -
> > > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and NODE_ADDRESSES,
> > > since
> > > > these views are clear and all topology data available on each node.
> > > > Any objections?
> > > >
> > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov :
> > > >
> > > >> Anton, Vladimir, I've made some fixes. There is only one view left
> and
> > > >> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'.
> > > >>
> > > >> High level design of solution:
> > > >> When IgniteH2Indexing is starting, it create and start
> > > >> new GridH2SysViewProcessor, which create and register in H2 (via its
> > own
> > > >> table engine) all implementations of system views. Each system view
> > > >> implementation extends base abstract class GridH2SysView. View
> > > >> implementation describes columns, their types and indexes in
> > constructor
> > > >> and must override method getRows for data retrieval (this method
> > called
> > > by
> > > >> H2-compatible table and index implementations for ignite system
> > views).
> > > >> Almost no fixes to existing parsing engine was made, except some
> > places,
> > > >> where GridH2Table instance was expected, but for system views there
> is
> > > >> another class.
> > > >>
> > > >> New PR: [1].  Please have a look.
> > > >>
> > > >> [1] https://github.com/apache/ignite/pull/3433
> > > >>
> > > >> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov <
> avinogra...@gridgain.com
> > >:
> > > >>
> > > >>> I've created IEP-13 [1] to cover all cases.
> > > >>> Feel free to create issues.
> > > >>>
> > > >>> [1]
> > > >>> https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=75962769
> > > >>>
> > > >>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > >>> wrote:
> > > >>>
> > >  Let's start with a single and the most simple view, e.g.
> > >  LOCAL_TRANSACTIONS. We will review and merge it along with
> necessary
> > >  infrastructure. Then will handle the rest view in separate tickets
> > and
> > >  separate focused discussions.
> > > 
> > >  On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov <
> > > plehanov.a...@gmail.com
> > > 
> > >  wrote:
> > > 
> > > > 1) It’s not a principal point, I can change schema. The
> > >  INFORMATION_SCHEMA
> > > > was used because it’s already exists and usually used for
> metadata
> > > >>> tables
> > > > and views. Your proposal is to use schema “IGNITE”, am I
> understand
> > > >>> you
> > > > right? BTW, for now, we can’t query another (H2) meta tables from
> > the
> > > > INFORMATION_SCHEMA, so, “Ignite system views” is only available
> > views
> > > >>> to
> > > > query from this schema.
> > > > 2) Exactly for this reason the IGNITE_INSTANCE view is useful: to
> > >  determine
> > > > which node we are connected to.
> > > > 3) As the first phase, in my opinion, local views will be enough.
> > > > Performance and caching of distributed views should be discussed
> at
> > > >>> next
> > > > phases, when distributed views implementation will be planned. In
> > > >>> current
> > > > implementation I tried to use indexing for local views wherever
> > it’s
> > > > possible.
> > > > 4) I don’t think, that JVM info is more critical information
> than,
> > > for
> > > > example, caches or nodes information. When authorization
> > capabilities
> > > > planned to implement?
> > > >
> > > > About local data: yes, we can rename all currently implemented
> > views
> > > >>> for
> > > > the local node data as LOCAL_..., and create (someday) new whole
> > > >>> cluster
> > > > views (which use distributed requests) without prefix or, for
> > > example,
> > >  with
> > > > CLUSTER_ prefix. But some views can show all cluster information
> > > using
> > >  only
> > > > local node data, without distributed requests (for example
> > > > IGNITE_NODE_METRICS, IGNITE_PART_ASSIGNMENT,
> > IGNITE_PART_ALLOCATION,
> > > > I

Re: Adding experimental support for Intel Optane DC Persistent Memory

[Dohrmann, Steve]

Hi Denis,

Having read through the alternate proposal that would use JEP-352, it all 
sounds very plausible and may be a great fit in this case, assuming the 
dependency on JDK14 is ok.  We are familiar with the persistent ByteBuffer JEP 
and think it is a great idea.

Best regards,
Steve

On Oct 15, 2019, at 3:43 PM, Kaczmarek, Eric 
mailto:eric.kaczma...@intel.com>> wrote:

Steve,

Need your help to respond to all of this.

From: Denis Magda mailto:dma...@apache.org>>
Date: Tuesday, October 15, 2019 at 3:10 PM
To: dev mailto:dev@ignite.apache.org>>, "Mammo, 
Mulugeta" mailto:mulugeta.ma...@intel.com>>, 
"Kaczmarek, Eric" mailto:eric.kaczma...@intel.com>>
Subject: Re: Adding experimental support for Intel Optane DC Persistent Memory

Alex, thanks for an extensive summary and proposing different implementation 
trails. Presently, the JEP-352 approach seems the most reasonable - JDK folks 
will take care of the integration and will be testing/maintaining the feature 
going forward, while the Ignite community will reuse what JDK provides. 
However, let's see what Eric and Mulugeta of Intel (both copied) think about 
this and other approaches.

Ivan, I'm not sure if any performance testing was done for the given 
pull-request. Anyway, it might be premature considering Alex's summary.

-
Denis


On Mon, Oct 14, 2019 at 11:50 PM Ivan Pavlukhin 
mailto:vololo...@gmail.com>> wrote:
By the way, did we have a change to run some existing benchmarks
against proposed implementation?

пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk 
mailto:alexey.goncha...@gmail.com>>:
>
> Hello Mulugeta, Igniters.
>
> Thanks for your interest and efforts in integrating the persistent memory
> to Ignite. Here are my thoughts on the PR:
>
> First of all I was questioning whether we want to use the integration with
> the pmem library via JNI. Java 14 will have native support for NVME via
> native mapped byte buffers [1] since the corresponding tickets are already
> resolved [2],[3]. The suggested integration uses JNI calls quite often (if
> I read the PR right, there will be tens of JNI calls per a single
> operation), which may and most likely will undermine the benefits of using
> PM. On the other hand, there is a PoC/project [4] which successfully
> demonstrates how the new integration can be used to natively access
> persistent memory from Java. Additionally, new Unsafe have several
> performance issues - each put* methods has a map lookup, and CAS method
> uses a global lock. Removing these issues will change the PR and
> architecture dramatically.
>
> Other than performance and an extra library dependency, there are
> additional reasons to use custom implementation of PM persistence
> primitives:
>
>- Ignite significantly benefits from having WAL as a single point of
>data for both failure recovery and historical rebalance. As far as I can
>guess, the suggested library also uses some sort of journaling in order to
>support crash recovery. Since we do not want give up on the historical
>rebalance, we would need to write an additional journal, which then raises
>a question of how to transact between this journal.
>- Ignite is supposed to work with large indexes, a I not aware how LLPL
>handles them. Several researches [5],[6] show that since PM memory has
>larger access latency than regular RAM, it is beneficial to buffer index
>pages in regular memory and store leaf pages in PM. There is a generic
>approach to convert in-memory indexes to PM ones [7], I think it makes
>sense to investigate whether our existing indexes can be converted to PM.
>- There is an active IEP which is targeted for a file-based rebalancing.
>In the suggested implementation this optimization would not work. A better
>integration should allow to take a snapshot of a partition and transfer it
>to another node for fast rebalancing.
>
> Overall, I think we may take the following path:
>
>- Target JEP352 APIs; for now keep using the whole region msync for flush
>- Introduce PM-based isolated components to support WAL iterator and
>transaction management. We can start with WAL, then implement a PM-based
>index based on current research, then a tuple heap with free space
>management. Each of the components may be developed and tested 
> independently
>- Do Ignite internals refactoring to properly abstract storage layer
>from transactions/communication. Currently, there is an abstraction leak in
>the storage layer, which makes it aware of particular Ignite implementation
>details (Binary Objects, partition eviction, etc)
>- Once the storage is separated, we should be able to plug the new
>components instead of the "classical" ones
>
> WDYT? I believe we should create a detailed IEP with concrete proposals on
> both PM structures design and Ignite internals changes before making any
> further code changes.
>
> [1] https://openjdk.java.net/jeps/352
> [2] https://bug

Re: Adding experimental support for Intel Optane DC Persistent Memory

[Denis Magda]

Hi Steve, thanks for joining the discussion and sharing your expertise
knowledge! The JDK14 dependency is not a matter of our concern.

Steve, Erick, is there anybody from the Intel engineering who can join the
community and take part in the development? Ignite committers will help
that person(s) gain all required knowledge about Ignite internals
throughout the journey. We definitely need your help with a B+tree
implementation for Optane.

Alex Goncharuk, would you mind accepting the role of the architect for this
integration? We’ll form a team of Ignite contributors who will take part in
the development under your architectural leadership. You can count on me,
will be the first one in that group )

Igniters, is there anybody else who’s ready to play with the new exciting
storage technology developed by Intel, gain advance knowledge of Ignite
memory management and other internals? Who’s ready to contribute as a
developer?

Denis

On Thursday, October 17, 2019, Dohrmann, Steve 
wrote:

> Hi Denis,
>
> Having read through the alternate proposal that would use JEP-352, it all
> sounds very plausible and may be a great fit in this case, assuming the
> dependency on JDK14 is ok.  We are familiar with the persistent ByteBuffer
> JEP and think it is a great idea.
>
> Best regards,
> Steve
>
> On Oct 15, 2019, at 3:43 PM, Kaczmarek, Eric 
> wrote:
>
> Steve,
>
> Need your help to respond to all of this.
>
> *From: *Denis Magda 
> *Date: *Tuesday, October 15, 2019 at 3:10 PM
> *To: *dev , "Mammo, Mulugeta" <
> mulugeta.ma...@intel.com>, "Kaczmarek, Eric" 
> *Subject: *Re: Adding experimental support for Intel Optane DC Persistent
> Memory
>
> Alex, thanks for an extensive summary and proposing different
> implementation trails. Presently, the JEP-352 approach seems the most
> reasonable - JDK folks will take care of the integration and will be
> testing/maintaining the feature going forward, while the Ignite community
> will reuse what JDK provides. However, let's see what Eric and Mulugeta of
> Intel (both copied) think about this and other approaches.
>
> Ivan, I'm not sure if any performance testing was done for the given
> pull-request. Anyway, it might be premature considering Alex's summary.
>
> -
> Denis
>
>
> On Mon, Oct 14, 2019 at 11:50 PM Ivan Pavlukhin 
> wrote:
>
> By the way, did we have a change to run some existing benchmarks
> against proposed implementation?
>
> пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk  >:
> >
> > Hello Mulugeta, Igniters.
> >
> > Thanks for your interest and efforts in integrating the persistent memory
> > to Ignite. Here are my thoughts on the PR:
> >
> > First of all I was questioning whether we want to use the integration
> with
> > the pmem library via JNI. Java 14 will have native support for NVME via
> > native mapped byte buffers [1] since the corresponding tickets are
> already
> > resolved [2],[3]. The suggested integration uses JNI calls quite often
> (if
> > I read the PR right, there will be tens of JNI calls per a single
> > operation), which may and most likely will undermine the benefits of
> using
> > PM. On the other hand, there is a PoC/project [4] which successfully
> > demonstrates how the new integration can be used to natively access
> > persistent memory from Java. Additionally, new Unsafe have several
> > performance issues - each put* methods has a map lookup, and CAS method
> > uses a global lock. Removing these issues will change the PR and
> > architecture dramatically.
> >
> > Other than performance and an extra library dependency, there are
> > additional reasons to use custom implementation of PM persistence
> > primitives:
> >
> >- Ignite significantly benefits from having WAL as a single point of
> >data for both failure recovery and historical rebalance. As far as I
> can
> >guess, the suggested library also uses some sort of journaling in
> order to
> >support crash recovery. Since we do not want give up on the historical
> >rebalance, we would need to write an additional journal, which then
> raises
> >a question of how to transact between this journal.
> >- Ignite is supposed to work with large indexes, a I not aware how
> LLPL
> >handles them. Several researches [5],[6] show that since PM memory has
> >larger access latency than regular RAM, it is beneficial to buffer
> index
> >pages in regular memory and store leaf pages in PM. There is a generic
> >approach to convert in-memory indexes to PM ones [7], I think it makes
> >sense to investigate whether our existing indexes can be converted to
> PM.
> >- There is an active IEP which is targeted for a file-based
> rebalancing.
> >In the suggested implementation this optimization would not work. A
> better
> >integration should allow to take a snapshot of a partition and
> transfer it
> >to another node for fast rebalancing.
> >
> > Overall, I think we may take the following path:
> >
> >- Target JEP352 APIs; for now keep u

[MTCGA]: new failures in builds [4704446] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New test failure in master 
CacheSerializableTransactionsTest.testGetRemoveTxNearCache1 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-5096358285353064325&branch=%3Cdefault%3E&tab=testDetails
 Changes may lead to failure were done by 
 - pavel kovalenko  
https://ci.ignite.apache.org/viewModification.html?modId=891495

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 07:23:55 18-10-2019 

Re: [jira] [Created] (IGNITE-12259) Create new module for support spring-5.2.X and spring-data-2.2.X

[Surkov]

Hi everybody.
Please, review my changes.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Ignite diagnostic (SQL system views)

[Alex Plehanov]

Denis,

Views engine and some views were released in AI 2.7.
In 2.8 they will be moved to the new engine and new views will be added (as
part of IEP-35)


пт, 18 окт. 2019 г. в 00:50, Denis Magda :

> Anton, Maxim,
>
> Are we planning to release the views as part of 2.8? Don't see them listed
> in the important features section:
>
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Themostimportantreleasetasks
>
> -
> Denis
>
>
> On Wed, Feb 14, 2018 at 1:49 AM Anton Vinogradov  >
> wrote:
>
> > Vova,
> >
> > Could you confirm https://issues.apache.org/jira/browse/IGNITE-7527
> ready
> > to be merged?
> >
> > On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov 
> > wrote:
> >
> > > I would start with NODES and NODE_ATTRIBUTES as the most simple thing.
> > >
> > > On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda 
> wrote:
> > >
> > > > Alex P, sounds like a good plan for me.
> > > >
> > > > Vladimir, do you have any suggestions or corrections?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov <
> plehanov.a...@gmail.com>
> > > > wrote:
> > > > >
> > > > > The views engine and the first view are almost ready to merge
> (review
> > > > > comments are resolved). Which views should we take next? My
> proposal
> > -
> > > > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and
> NODE_ADDRESSES,
> > > > since
> > > > > these views are clear and all topology data available on each node.
> > > > > Any objections?
> > > > >
> > > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov  >:
> > > > >
> > > > >> Anton, Vladimir, I've made some fixes. There is only one view left
> > and
> > > > >> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'.
> > > > >>
> > > > >> High level design of solution:
> > > > >> When IgniteH2Indexing is starting, it create and start
> > > > >> new GridH2SysViewProcessor, which create and register in H2 (via
> its
> > > own
> > > > >> table engine) all implementations of system views. Each system
> view
> > > > >> implementation extends base abstract class GridH2SysView. View
> > > > >> implementation describes columns, their types and indexes in
> > > constructor
> > > > >> and must override method getRows for data retrieval (this method
> > > called
> > > > by
> > > > >> H2-compatible table and index implementations for ignite system
> > > views).
> > > > >> Almost no fixes to existing parsing engine was made, except some
> > > places,
> > > > >> where GridH2Table instance was expected, but for system views
> there
> > is
> > > > >> another class.
> > > > >>
> > > > >> New PR: [1].  Please have a look.
> > > > >>
> > > > >> [1] https://github.com/apache/ignite/pull/3433
> > > > >>
> > > > >> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov <
> > avinogra...@gridgain.com
> > > >:
> > > > >>
> > > > >>> I've created IEP-13 [1] to cover all cases.
> > > > >>> Feel free to create issues.
> > > > >>>
> > > > >>> [1]
> > > > >>> https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=75962769
> > > > >>>
> > > > >>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > >  Let's start with a single and the most simple view, e.g.
> > > >  LOCAL_TRANSACTIONS. We will review and merge it along with
> > necessary
> > > >  infrastructure. Then will handle the rest view in separate
> tickets
> > > and
> > > >  separate focused discussions.
> > > > 
> > > >  On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov <
> > > > plehanov.a...@gmail.com
> > > > 
> > > >  wrote:
> > > > 
> > > > > 1) It’s not a principal point, I can change schema. The
> > > >  INFORMATION_SCHEMA
> > > > > was used because it’s already exists and usually used for
> > metadata
> > > > >>> tables
> > > > > and views. Your proposal is to use schema “IGNITE”, am I
> > understand
> > > > >>> you
> > > > > right? BTW, for now, we can’t query another (H2) meta tables
> from
> > > the
> > > > > INFORMATION_SCHEMA, so, “Ignite system views” is only available
> > > views
> > > > >>> to
> > > > > query from this schema.
> > > > > 2) Exactly for this reason the IGNITE_INSTANCE view is useful:
> to
> > > >  determine
> > > > > which node we are connected to.
> > > > > 3) As the first phase, in my opinion, local views will be
> enough.
> > > > > Performance and caching of distributed views should be
> discussed
> > at
> > > > >>> next
> > > > > phases, when distributed views implementation will be planned.
> In
> > > > >>> current
> > > > > implementation I tried to use indexing for local views wherever
> > > it’s
> > > > > possible.
> > > > > 4) I don’t think, that JVM info is more critical information
> > than,
> > > > for
> > > > > example, caches or nodes information. When authorization
> > > capabilities
> > > > > planned to implement?
> > > > >
> > > > > About local data: yes, we can rename