[jira] [Created] (IGNITE-11962) [TC Bot] Test threated as recently contributed even if there were runs outside 21 days
Dmitriy Pavlov created IGNITE-11962: --- Summary: [TC Bot] Test threated as recently contributed even if there were runs outside 21 days Key: IGNITE-11962 URL: https://issues.apache.org/jira/browse/IGNITE-11962 Project: Ignite Issue Type: Bug Reporter: Dmitriy Pavlov Assignee: Dmitriy Pavlov Before notifying about recently contributed test it is necessary to double-check - if there were runs outside 21 days limit - if test failures count is not equal to the suite execution count - every time test was executed, the suite was executed as well (this may protect from flaking appearing-disappearing tests init errors). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11963) Remove ContinuousQueryDeserializationErrorOnNodeJoinTest
Ivan Bessonov created IGNITE-11963: -- Summary: Remove ContinuousQueryDeserializationErrorOnNodeJoinTest Key: IGNITE-11963 URL: https://issues.apache.org/jira/browse/IGNITE-11963 Project: Ignite Issue Type: Test Reporter: Ivan Bessonov Assignee: Ivan Bessonov Test ContinuousQueryDeserializationErrorOnNodeJoinTest is invalid after IGNITE-11914 and should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Read Repair (ex. Consistency Check) - review request #2
Folks, Just a minor update. RunAll [1] with enabled ReadRepair proxy is almost green now (~10 tests left, started with 6k :)). During the analisys, I've found some tests with - unexpected repairs at tx caches - inconsistent state after the test finished (different entries across the topology) For example, - testInvokeAllAppliedOnceOnBinaryTypeRegistration generates obsolete versions on backups in case of retry, fixed [2] - initial cache load generates not equal versions on backups, fixed [3] - testAccountTxNodeRestart causes unexpected repairs (entries have different versions), to be investigated. What's next? 1) Going to merge the solution once "RunAll with ReadRepair enabled" becomes fully green. 2) Going to add special check after each test which will ensure caches content after the test is consistent. 2.1) The Same check can (should?) be injected to awaitPartitionMapExchange() and similar methods. 3) Update Jepsen tests with RR checks. 4) Introduce per partition RR. So, the final goal is to be sure that Ignite produces only consistent data and to have a feature to solve consistency in case we gain inconsistent state somehow. Limitations? Currently, RR has some limitations, but they are not related to real production cases. In case someone interested to support, for example, MVCC or near caches, please, feel free to contribute. [1] https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/6575/head&action=Latest [2] https://github.com/apache/ignite/pull/5656/commits/6f6ec4434095e692af209c61833a350f3013408c [3] https://github.com/apache/ignite/pull/5656/commits/255e552b474839e470c66a77e74e3c807bc76f13 On Fri, Jun 28, 2019 at 2:41 PM Anton Vinogradov wrote: > Slava, > > >> I will take a look at your pull request if you don't mind. > Great news! > > >> In any way, could you please update the IEP page with the list of > >> constraints/limitations of the proposed approach, TODOs, etc? > Not sure we should keep this at IEP until list (#4 from original letter) > is not confirmed. > > Currently, I'm checking RunAll with RR enabled to almost each get request. > "Almost" means: readRepair = !ctx.readThrough() && > ctx.config().getBackups() > 0 && !ctx.isNear() && !ctx.mvccEnabled() > For now I have 60 failed tests and amount decreasing. > > >> For instance, I would like to see all these limitations on the IEP page > as > >> JIRA tickets. Perhaps, it would be good to create an epic/umbrella > ticket > >> in order to track all activities related to `Read Repair` feature. > Let's do this at merge day to be sure useless issues will not be created. > > On Fri, Jun 28, 2019 at 2:01 PM Вячеслав Коптилин < > slava.kopti...@gmail.com> wrote: > >> Hi Anton, >> >> I will take a look at your pull request if you don't mind. >> >> In any way, could you please update the IEP page with the list of >> constraints/limitations of the proposed approach, TODOs, etc? >> For instance, I would like to see all these limitations on the IEP page as >> JIRA tickets. Perhaps, it would be good to create an epic/umbrella ticket >> in order to track all activities related to `Read Repair` feature. >> >> Thanks, >> S. >> >> чт, 20 июн. 2019 г. в 14:15, Anton Vinogradov : >> >> > Igniters, >> > I'm glad to introduce Read Repair feature [0] provides additional >> > consistency guarantee for Ignite. >> > >> > 1) Why we need it? >> > The detailed explanation can be found at IEP-31 [1]. >> > In short, because of bugs, it's possible to gain an inconsistent state. >> > We need additional features to handle this case. >> > >> > Currently we able to check cluster using Idle_verify [2] feature, but it >> > will not fix the data, will not even tell which entries are broken. >> > Read Repair is a feature to understand which entries are broken and to >> fix >> > them. >> > >> > 1) How it works? >> > IgniteCache now able to provide special proxy [3] withReadRepair(). >> > This proxy guarantee that data will be gained from all owners and >> compared. >> > In the case of consistency violation situation, data will be recovered >> and >> > a special event recorded. >> > >> > 3) Naming? >> > Feature name based on Cassandra's Read Repair feature [4], which is >> pretty >> > similar. >> > >> > 4) Limitations which can be fixed in the future? >> > * MVCC and Near caches are not supported. >> > * Atomic caches can be checked (false positive case is possible on >> this >> > check), but can't be recovered. >> > * Partial entry removal can't be recovered. >> > * Entries streamed using data streamer (using not a "cache.put" based >> > updater) and loaded by cache.load >> > are perceived as inconsistent since they may have different versions >> for >> > same keys. >> > * Only explicit get operations are supported (getAndReplace, >> getAndPut, >> > etc can be supported in future). >> > >> > 5) What's left? >> > * SQL/ThinClient/etc support. >> > * Metrics (found/repaired). >> > * Simple per-partition r
Re: Performance drop with New Monitoring.
Fix merged to master. Guys, thanks for the help! В Вт, 02/07/2019 в 18:25 +0300, Alexey Goncharuk пишет: > Hello Nikolay, > > I will take a look shortly. > > вт, 2 июл. 2019 г. в 14:30, Nikolay Izhikov : > > > Guys. > > > > I've prepared PR to fix this issue. > > Who want to review it? > > > > https://github.com/apache/ignite/pull/6656 > > > > В Пн, 01/07/2019 в 15:17 +0300, Nikolay Izhikov пишет: > > > Hello, Igniter. > > > > > > I think we should fix the issue with slight changes in internal API (was > > > > not released yet). > > > > > > Changes in API was independently proposed by me and Alexey Plekhanov [1]: > > > > > > 1. We should introduce `MetricGroup` - collection of metric of one > > > > specific Ignite entity(io subsystem, data region, cache, etc.). > > > > > > 2. We can remove entire `MetricGroup` (on cache or cache group destroy) > > > > with complexity of O(1). > > > > > > 3. We can remove metric from `MetricGroup` with complexity of O(1). > > > > > > [1] > > > > https://issues.apache.org/jira/browse/IGNITE-11920?jql=labels%20%3D%20IEP-35%20ORDER%20BY%20status%20ASC > > > > > > В Пт, 28/06/2019 в 19:10 +0300, Nikolay Izhikov пишет: > > > > Hello, Maksim. > > > > > > > > I will take a look shortly. > > > > > > > > В Пт, 28/06/2019 в 19:02 +0300, Maksim Stepachev пишет: > > > > > Hello, Igniters. > > > > > > > > > > I see that the ticket - "[IGNITE-11848] New Monitoring. Phase1" > > > > breaks cache stop performance. It affects next tests: > > testParallelStartAndStop, testStartManyCaches > > > > > > > > > > The spent time was increased 6 times:( > > > > > > > > > > Look at: > > > > https://ci.ignite.apache.org/project.html?tab=testDetails&branch_IgniteTests24Java8=%3Cdefault%3E&projectId=IgniteTests24Java8&testNameId=-145777872614811891&page=4 > > signature.asc Description: This is a digitally signed message part
[jira] [Created] (IGNITE-11964) Code style failed
Vladimir Malinovskiy created IGNITE-11964: - Summary: Code style failed Key: IGNITE-11964 URL: https://issues.apache.org/jira/browse/IGNITE-11964 Project: Ignite Issue Type: Bug Reporter: Vladimir Malinovskiy Assignee: Vladimir Malinovskiy [https://ci.ignite.apache.org/viewLog.html?buildId=4258304&buildTypeId=IgniteTests24Java8_CheckCodeStyle] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[MTCGA]: new failures in builds [4258304] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New Trusted Suite failure in master [Check Code Style] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_CheckCodeStyle&branch=%3Cdefault%3E&tab=buildTypeStatusDiv Changes may lead to failure were done by - vmalinovs...@gridgain.com https://ci.ignite.apache.org/viewModification.html?modId=887168 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 14:56:13 04-07-2019
Re: [MTCGA]: new failures in builds [4258304] needs to be handled
I've just added this suite to list of trusted. This means that exit code (and any other build problem) in this suite is a reason to notify. For all other suites, only some specific build problems (OOME, Timeout, Crash) causes notification. Thanks to N. Izikov, this issue is already solved. чт, 4 июл. 2019 г. в 14:56, : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were > a volunteer to make the contribution to this project, but things change and > you may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and > fix test failures or step down and some committer may revert you commit. > > *New Trusted Suite failure in master [Check Code Style] > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_CheckCodeStyle&branch=%3Cdefault%3E&tab=buildTypeStatusDiv > Changes may lead to failure were done by > - vmalinovs...@gridgain.com > https://ci.ignite.apache.org/viewModification.html?modId=887168 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact > dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 14:56:13 04-07-2019 >
Re: [MTCGA]: new failures in builds [4255938] needs to be handled
I discussed this issue with Denis and we decided to remove this test completely as it is invalid after my changes. Corresponding issue: [1] [1] https://issues.apache.org/jira/browse/IGNITE-11963 чт, 4 июл. 2019 г. в 08:25, Павлухин Иван : > Seems to be an interference of two commits related to continuous > queries [1, 2]. In one a test was added and the test was broken by > another commit. > > Ivan, Denis, could please take a look? > > [1] > https://github.com/apache/ignite/commit/db74e92dcd56bc5a45ed9d3a382be8bcfa2a76dc > [2] > https://github.com/apache/ignite/commit/2cea7e245927e8e54b93f7adea7524ce68c6d226 > > чт, 4 июл. 2019 г. в 03:57, : > > > > Hi Igniters, > > > > I've detected some new issue on TeamCity to be handled. You are more > than welcomed to help. > > > > If your changes can lead to this failure(s): We're grateful that you > were a volunteer to make the contribution to this project, but things > change and you may no longer be able to finalize your contribution. > > Could you respond to this email and indicate if you wish to continue > and fix test failures or step down and some committer may revert you commit. > > > > *New test failure in master > ContinuousQueryDeserializationErrorOnNodeJoinTest.testDeserializationErrorOnJoiningNode > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=2327531993912363014&branch=%3Cdefault%3E&tab=testDetails > > Changes may lead to failure were done by > > - dchu...@gridgain.com > https://ci.ignite.apache.org/viewModification.html?modId=887157 > > - bessonov...@gmail.com > https://ci.ignite.apache.org/viewModification.html?modId=887155 > > > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > > - Should you have any questions please contact > dev@ignite.apache.org > > > > Best Regards, > > Apache Ignite TeamCity Bot > > https://github.com/apache/ignite-teamcity-bot > > Notification generated at 03:57:01 04-07-2019 > > > > -- > Best regards, > Ivan Pavlukhin > -- Sincerely yours, Ivan Bessonov
Re: [MTCGA]: new failures in builds [4172317, 4172313, 4172278] needs to be handled
Dmitry, I'm aware of failing tests, will check in nearest time. пн, 1 июл. 2019 г. в 22:04, Dmitriy Pavlov : > I've checked Pavel's change and it is .NET only > https://issues.apache.org/jira/browse/IGNITE-11867 > I guess this change may have an effect on other tests > > пн, 1 июл. 2019 г. в 21:24, Dmitriy Pavlov : > > > Alexey, Pavel, could you please double check reasons for the failure? > > > > пн, 1 июл. 2019 г. в 21:11, : > > > >> Hi Igniters, > >> > >> I've detected some new issue on TeamCity to be handled. You are more > >> than welcomed to help. > >> > >> If your changes can lead to this failure(s): We're grateful that you > >> were a volunteer to make the contribution to this project, but things > >> change and you may no longer be able to finalize your contribution. > >> Could you respond to this email and indicate if you wish to continue > and > >> fix test failures or step down and some committer may revert you commit. > >> > >> *New stable failure of a flaky test in master > >> LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-6585115376754732686&branch=%3Cdefault%3E&tab=testDetails > >> Changes may lead to failure were done by > >> - alexey.scherbak...@gmail.com > >> https://ci.ignite.apache.org/viewModification.html?modId=886764 > >> - ptupit...@apache.org > >> https://ci.ignite.apache.org/viewModification.html?modId=886762 > >> > >> *New stable failure of a flaky test in master > >> > GridCacheRebalancingWithAsyncClearingMvccTest.testPartitionClearingNotBlockExchange > >> > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-7007912051428984819&branch=%3Cdefault%3E&tab=testDetails > >> Changes may lead to failure were done by > >> - alexey.scherbak...@gmail.com > >> https://ci.ignite.apache.org/viewModification.html?modId=886764 > >> - ptupit...@apache.org > >> https://ci.ignite.apache.org/viewModification.html?modId=886762 > >> > >> *New stable failure of a flaky test in master > >> GridCacheRebalancingAsyncSelfTest.testComplexRebalancing > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-8829809273565657995&branch=%3Cdefault%3E&tab=testDetails > >> Changes may lead to failure were done by > >> - alexey.scherbak...@gmail.com > >> https://ci.ignite.apache.org/viewModification.html?modId=886764 > >> - ptupit...@apache.org > >> https://ci.ignite.apache.org/viewModification.html?modId=886762 > >> > >> - Here's a reminder of what contributors were agreed to do > >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > >> - Should you have any questions please contact > >> dev@ignite.apache.org > >> > >> Best Regards, > >> Apache Ignite TeamCity Bot > >> https://github.com/apache/ignite-teamcity-bot > >> Notification generated at 21:10:24 01-07-2019 > >> > > > -- Best regards, Alexei Scherbakov
Re: [DISCUSSION][IEP-35] Metrics configuration
Igniters, I rethought the issue and I see some problems: 1. It seems that in most cases bucket boundaries configuration will be problem for user. Absolute values for latency boundaries it is very odd choice. 2. Also seems that latency for most caches (if we configure cache metrics fro example) will be similar. 3. I can't imagine that adequate values will be chosen on project setup stage. So chosen values should be changed in the future. Solution with configuration file looks unnatural and creates more problems than could solve. My proposal is adding API for boundaries configuration to the metrics framework and expose it via JMX (at this step). It still provides configuration possibility but don't force user to do it. Also we should chose default values for bucket boundaries. And it is most complex problem at the moment :) Let's discuss it. On Wed, Jul 3, 2019 at 4:49 PM Andrey Gura wrote: > > Nikolai, > > Metric is disabled if it doesn't allocate any memory and doesn't > update any variable because doesn't have any value. Ideally disabling > metrics for some cache should be equal to cache stopping. > > On Fri, Jun 28, 2019 at 1:02 PM Nikolay Izhikov wrote: > > > > Hello, Alexey. > > > > Thanks for the feedback! > > > > > My only concert is that we should have the metrics framework configuration > > > as the first-citizen of the framework itself > > > > Yes. I planned to add `void configure(String param)` method to the metric > > API. > > > > > but change the metrics parameters in > > > runtime from JMX or command-line, etc. > > > > I've add requirement of JMX method to the ticket: > > > > https://issues.apache.org/jira/browse/IGNITE-11927 > > > > > Another concern is to have an > > > ability to disable/enable metrics per metrics group/prefix. > > > > Yes, we discusss it. > > But, let's make it clear: > > > > *What is disabling metric?* > > > > Looks like exporter filter solve this task. > > > > В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет: > > > Nikolay, > > > > > > My only concert is that we should have the metrics framework configuration > > > as the first-citizen of the framework itself. This way, we can configure > > > the metrics not only from file, but change the metrics parameters in > > > runtime from JMX or command-line, etc. Another concern is to have an > > > ability to disable/enable metrics per metrics group/prefix. > > > > > > The logger-like configuration meets these suggestions given that the > > > configuration is generalized into the metrics framework. > > > > > > What do you think? > > > > > > чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov : > > > > > > > Hello, Igniters. > > > > > > > > As you may know, I've contributed Phase1 [1] for IEP-35 [2]. > > > > Now we have metrics subsystem and can create and export any metrics from > > > > Ignite. > > > > > > > > I think user(administrator of Ignite) should be able to configure some > > > > metrics params in a common way [3] > > > > > > > > I propose to use the same way from logging frameworks. > > > > We should define some file format Ignite can understand. > > > > An administrator fills configuration file to configure one or several > > > > metrics. > > > > Ignite will analyze the file and use provided params during metrics > > > > creation. > > > > > > > > For now, we have 2 types of metrics that should be configured: > > > > > > > > * HistrogramMetric [4] > > > > This metric is a count of measurement that falls into > > > > predefined intervals. > > > > An example is "Request processing time distribution". > > > > We want to calculate a count of requests processed > > > > quicker > > > > then 50ms, 50-100, 100-250, 250-500 and slower. > > > > > > > > * HitRateMetric [5] > > > > This metric is a count of events in the last time > > > > interval. > > > > An example is the "Count of requests processed in the > > > > last > > > > 5 seconds". > > > > > > > > Example of file content: > > > > > > > > > > > > cache.my-cahe.GetLatency=50,100,250,500 #Params for the histogram metric > > > > with the name `cache.my-cahe.get` > > > > cache.my-cache.RebalancingKeysRate=6 #Param for existing > > > > HitRateMetric > > > > that hold "Estimated rebalancing speed in keys". > > > > > > > > > > > > Please, share your vision. > > > > > > > > [1] > > > > https://github.com/apache/ignite/commit/fdaa310430aefff07994eb35510d3416886b5bbe > > > > [2] > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-11927 > > > > [4] > > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HistogramMetric.java > > > > [5] > > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HitRateMetric.java > > > >
[jira] [Created] (IGNITE-11965) Pyton client: Expiration policies are missed
Andrey Aleksandrov created IGNITE-11965: --- Summary: Pyton client: Expiration policies are missed Key: IGNITE-11965 URL: https://issues.apache.org/jira/browse/IGNITE-11965 Project: Ignite Issue Type: Bug Components: thin client Affects Versions: 2.7.5, 2.7 Reporter: Andrey Aleksandrov Fix For: 2.8 [https://apacheignite.readme.io/docs/expiry-policies] are missed but [https://apacheignite.readme.io/docs/expiry-policies#section-eager-ttl] can be set. Should be added. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11966) Using AdaptiveLoadBalancingSpi without IgniteConfiguration.setIncludeEventTypes(EventType.EVT_TASK_FINISHED, EventType.EVT_TASK_FAILED) leads to memory leak
Andrey Aleksandrov created IGNITE-11966:
---
Summary: Using AdaptiveLoadBalancingSpi without
IgniteConfiguration.setIncludeEventTypes(EventType.EVT_TASK_FINISHED,
EventType.EVT_TASK_FAILED) leads to memory leak
Key: IGNITE-11966
URL: https://issues.apache.org/jira/browse/IGNITE-11966
Project: Ignite
Issue Type: Bug
Affects Versions: 2.7.5
Reporter: Andrey Aleksandrov
Fix For: 2.8
Attachments: Top_Consumers.zip
You can use in debug mode any of tests related to AdaptiveLoadBalancingSpi like
GridAdaptiveLoadBalancingSpiSelfTest.
In case if you don't subscribe on EventType.EVT_TASK_FINISHED,
EventType.EVT_TASK_FAILED then next code will be never called:
case EVT_TASK_FINISHED:
case EVT_TASK_FAILED: {
TaskEvent taskEvt = (TaskEvent)evt;
taskTops.remove(taskEvt.taskSessionId()); //this
As a result, you will see the growth of memory usage.
[^Top_Consumers.zip]
if (log.isDebugEnabled())
log.debug("Removed task topology from topology cache for session: " +
taskEvt.taskSessionId());
break;
}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[discussion] using custom build of H2 for Ignite
Hi Igniters, As you know, Ignite SQL engine is tightly coupled with the H2 database that provides basic parsing and query execution capabilities. This synergy has worked well for a while until Ignite SQL engine got a much broader adoption for all sort of use cases. Presently, there is a list of impactful issues and limitations related to memory management, distributed engine optimization, and queries planning that require changes in H2. We've tried to contribute to H2 directly with no significant luck - what's needed for our distributed engine is of no interest to H2 community. At the same time, we can't leave the things as is, as long as these limitations keep Ignite SQL engine from gradual evolution. As a solution, we created an H2 fork [1] and did all of the required changes there. We would be happy to include the fork into Ignite source base, but H2's license (available under dual MPL 2.0 and EPL 1.0) is not compliant with Apache 2.0. However, if Ignite starts using our maven artifacts instead of the standard H2's ones, then the licensing issue is solved. Is the community ready to accept this solution and swap the standard H2 artifact with the one prepared by GridGain? Presently, all of those improvements are available to GridGain customers, but GridGain wants to make all of them be available for Ignite community. And that's the only legal way we've come up with... [1] https://github.com/gridgain/gridgain/tree/master/modules/h2 -- - Denis
Re: [discussion] using custom build of H2 for Ignite
Hi Denis, As you know, some time ago I've started a discussion about removing dependence from gridgain:shmem. Ignite community seems to be not so much interested in this removal, for now. So once added it could stay here forever. Reverse dependency direction seems to be more natural. It is like the open-core model. I feel more comfortable if all Ignite dependencies are released as part of the Ignite code base, or some open governed project with a license from Category A https://www.apache.org/legal/resolved.html. It is true that H2 has Category B license, so derivative can't be committed into ASF repository. What if we consult with le...@apache.org to find additional ways to donate forked version into ASF codebase? We anyway need their approval because gridgain/h2 has a non-standard license, so we should approve including non-standard licensed component it the product. Sincerely, Dmitriy Pavlov чт, 4 июл. 2019 г. в 18:57, Denis Magda : > Hi Igniters, > > As you know, Ignite SQL engine is tightly coupled with the H2 database that > provides basic parsing and query execution capabilities. This synergy has > worked well for a while until Ignite SQL engine got a much broader adoption > for all sort of use cases. > > Presently, there is a list of impactful issues and limitations related to > memory management, distributed engine optimization, and queries planning > that require changes in H2. We've tried to contribute to H2 directly with > no significant luck - what's needed for our distributed engine is of no > interest to H2 community. At the same time, we can't leave the things as > is, as long as these limitations keep Ignite SQL engine from gradual > evolution. > > As a solution, we created an H2 fork [1] and did all of the required > changes there. We would be happy to include the fork into Ignite source > base, but H2's license (available under dual MPL 2.0 and EPL 1.0) is not > compliant with Apache 2.0. However, if Ignite starts using our maven > artifacts instead of the standard H2's ones, then the licensing issue is > solved. > > Is the community ready to accept this solution and swap the standard H2 > artifact with the one prepared by GridGain? Presently, all of those > improvements are available to GridGain customers, but GridGain wants to > make all of them be available for Ignite community. And that's the only > legal way we've come up with... > > > [1] https://github.com/gridgain/gridgain/tree/master/modules/h2 > > > > -- > - > Denis >
Re: [discussion] using custom build of H2 for Ignite
Hello, Denis. Who can contribute to gridgain/h2? Who are commiters of gridgain/h2? В Чт, 04/07/2019 в 19:26 +0300, Dmitriy Pavlov пишет: > Hi Denis, > > As you know, some time ago I've started a discussion about removing > dependence from gridgain:shmem. Ignite community seems to be not so much > interested in this removal, for now. So once added it could stay here > forever. Reverse dependency direction seems to be more natural. It is like > the open-core model. > > I feel more comfortable if all Ignite dependencies are released as part of > the Ignite code base, or some open governed project with a license from > Category A https://www.apache.org/legal/resolved.html. > > It is true that H2 has Category B license, so derivative can't be committed > into ASF repository. > > What if we consult with le...@apache.org to find additional ways to donate > forked version into ASF codebase? We anyway need their approval because > gridgain/h2 has a non-standard license, so we should approve including > non-standard licensed component it the product. > > Sincerely, > Dmitriy Pavlov > > чт, 4 июл. 2019 г. в 18:57, Denis Magda : > > > Hi Igniters, > > > > As you know, Ignite SQL engine is tightly coupled with the H2 database that > > provides basic parsing and query execution capabilities. This synergy has > > worked well for a while until Ignite SQL engine got a much broader adoption > > for all sort of use cases. > > > > Presently, there is a list of impactful issues and limitations related to > > memory management, distributed engine optimization, and queries planning > > that require changes in H2. We've tried to contribute to H2 directly with > > no significant luck - what's needed for our distributed engine is of no > > interest to H2 community. At the same time, we can't leave the things as > > is, as long as these limitations keep Ignite SQL engine from gradual > > evolution. > > > > As a solution, we created an H2 fork [1] and did all of the required > > changes there. We would be happy to include the fork into Ignite source > > base, but H2's license (available under dual MPL 2.0 and EPL 1.0) is not > > compliant with Apache 2.0. However, if Ignite starts using our maven > > artifacts instead of the standard H2's ones, then the licensing issue is > > solved. > > > > Is the community ready to accept this solution and swap the standard H2 > > artifact with the one prepared by GridGain? Presently, all of those > > improvements are available to GridGain customers, but GridGain wants to > > make all of them be available for Ignite community. And that's the only > > legal way we've come up with... > > > > > > [1] https://github.com/gridgain/gridgain/tree/master/modules/h2 > > > > > > > > -- > > - > > Denis > > signature.asc Description: This is a digitally signed message part
Re: [DISCUSSION][IEP-35] Metrics configuration
Hello, Andrey. > 3. I can't imagine that adequate values will be chosen on project > setup stage. Configuration file required in the case we adds new node or replace existing to the cluster. Use can have parameters similar to Ignite configuration, log configuration files. > My proposal is adding API for boundaries configuration to the metrics > framework and expose it via JMX Agree. I think we should have both: 1. Configuration file. 2. JMX API to change bounaries of histogram *and HitRateMetric params*. But, if you and other community member are against config file, let's have only JMX. Seems, JMX will provide required level of configurability for metrics. В Чт, 04/07/2019 в 17:53 +0300, Andrey Gura пишет: > Igniters, > > I rethought the issue and I see some problems: > > 1. It seems that in most cases bucket boundaries configuration will be > problem for user. Absolute values for latency boundaries it is very > odd choice. > 2. Also seems that latency for most caches (if we configure cache > metrics fro example) will be similar. > 3. I can't imagine that adequate values will be chosen on project > setup stage. So chosen values should be changed in the future. > > Solution with configuration file looks unnatural and creates more > problems than could solve. > > My proposal is adding API for boundaries configuration to the metrics > framework and expose it via JMX (at this step). It still provides > configuration possibility but don't force user to do it. > > Also we should chose default values for bucket boundaries. And it is > most complex problem at the moment :) Let's discuss it. > > > > > > On Wed, Jul 3, 2019 at 4:49 PM Andrey Gura wrote: > > > > Nikolai, > > > > Metric is disabled if it doesn't allocate any memory and doesn't > > update any variable because doesn't have any value. Ideally disabling > > metrics for some cache should be equal to cache stopping. > > > > On Fri, Jun 28, 2019 at 1:02 PM Nikolay Izhikov wrote: > > > > > > Hello, Alexey. > > > > > > Thanks for the feedback! > > > > > > > My only concert is that we should have the metrics framework > > > > configuration > > > > as the first-citizen of the framework itself > > > > > > Yes. I planned to add `void configure(String param)` method to the metric > > > API. > > > > > > > but change the metrics parameters in > > > > runtime from JMX or command-line, etc. > > > > > > I've add requirement of JMX method to the ticket: > > > > > > https://issues.apache.org/jira/browse/IGNITE-11927 > > > > > > > Another concern is to have an > > > > ability to disable/enable metrics per metrics group/prefix. > > > > > > Yes, we discusss it. > > > But, let's make it clear: > > > > > > *What is disabling metric?* > > > > > > Looks like exporter filter solve this task. > > > > > > В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет: > > > > Nikolay, > > > > > > > > My only concert is that we should have the metrics framework > > > > configuration > > > > as the first-citizen of the framework itself. This way, we can configure > > > > the metrics not only from file, but change the metrics parameters in > > > > runtime from JMX or command-line, etc. Another concern is to have an > > > > ability to disable/enable metrics per metrics group/prefix. > > > > > > > > The logger-like configuration meets these suggestions given that the > > > > configuration is generalized into the metrics framework. > > > > > > > > What do you think? > > > > > > > > чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov : > > > > > > > > > Hello, Igniters. > > > > > > > > > > As you may know, I've contributed Phase1 [1] for IEP-35 [2]. > > > > > Now we have metrics subsystem and can create and export any metrics > > > > > from > > > > > Ignite. > > > > > > > > > > I think user(administrator of Ignite) should be able to configure some > > > > > metrics params in a common way [3] > > > > > > > > > > I propose to use the same way from logging frameworks. > > > > > We should define some file format Ignite can understand. > > > > > An administrator fills configuration file to configure one or several > > > > > metrics. > > > > > Ignite will analyze the file and use provided params during metrics > > > > > creation. > > > > > > > > > > For now, we have 2 types of metrics that should be configured: > > > > > > > > > > * HistrogramMetric [4] > > > > > This metric is a count of measurement that falls into > > > > > predefined intervals. > > > > > An example is "Request processing time distribution". > > > > > We want to calculate a count of requests processed > > > > > quicker > > > > > then 50ms, 50-100, 100-250, 250-500 and slower. > > > > > > > > > > * HitRateMetric [5] > > > > > This metric is a count of events in the last time > > > > > interval. > > > > > An example is the "Count of requests processed in the > > > > > last > > > >
