Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Vladimir Ozerov
Vyacheslav, Metadata is already stored separately. Object only contain 4 bytes reference to that metadata (aka "schema ID") and offsets to be able to find fields quickly. But if separate row format from binary format, we may be able to reduce it event further to some extent. On Tue, Nov 20, 2018

Re: Service grid redesign

2018-11-21 Thread Vladimir Ozerov
I do not know what is correct term :-) What I said is that "exchange" is counter intuitive here. There is no "exchange", instead nodes send information to coordinator that they finished some operation. E.g. we do the same for schema changes (index creation), and as Denis suggested, Request-Response

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Vladimir Ozerov
Hi Alexey, Yes, this looks really similar to Postgres format as welд - bitset, fixed fields, varlen fields. Most probably we need something similar. On Wed, Nov 21, 2018 at 10:10 AM Alexey Zinoviev wrote: > I'd like @Vyacheslav Daradur approach. > > Maybe somebody could have a look at UnsafeRow

Re: Service grid redesign

2018-11-21 Thread Vyacheslav Daradur
I think request-response is not suitable terms. Nodes send to coordinator maps of actual service deployments which contains what count of instances of each service node hosts locally. Coordinator sends to the cluster the full map of deployments across the cluster. On Wed, Nov 21, 2018 at 11:04 A

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Vladimir Ozerov
Sergi, Changing filed name to change it's type is not user-friendly approach. Because it will prevent operations like, which are perfectly normal from user perspective: ALTER TABLE my_table MODIFY COLUMN x BIGINT; // Was INT previously. Command above is way more simpler than: 1. ALTER TABLE my_ta

Re: Service grid redesign

2018-11-21 Thread Nikolay Izhikov
Hello, Vladimir. > There is no "exchange", instead nodes send information to coordinator that they finished some operation 1. Each node sends to coordinator local deployments results. 2. Coordinator sends to each node full deployment map. This is the same process we have in PME for me. What have

Re: Service grid redesign

2018-11-21 Thread Vladimir Ozerov
Hi Vyacheslav, Could you please explain in what situation coordinator needs to collect service deployments info from all nodes and share it with the cluster? I cannot remember from our design discussion when it is needed. Global state normally shared through discovery and only on node join, In thi

[jira] [Created] (IGNITE-10357) Check web console in version 2.7

2018-11-21 Thread Pavel Konstantinov (JIRA)
Pavel Konstantinov created IGNITE-10357: --- Summary: Check web console in version 2.7 Key: IGNITE-10357 URL: https://issues.apache.org/jira/browse/IGNITE-10357 Project: Ignite Issue Type:

Re: [MTCGA]: new failures in builds [2364043] needs to be handled

2018-11-21 Thread Alexey Goncharuk
Looking into this, looks like it is related to 'set consistent ID in tests' change. ср, 21 нояб. 2018 г. в 09:38, : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful th

Re: [MTCGA]: new failures in builds [2364187] needs to be handled

2018-11-21 Thread Alexey Goncharuk
Looking into this, looks like it is related to 'set consistent ID in tests' change. ср, 21 нояб. 2018 г. в 07:08, : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful th

Re: [MTCGA]: new failures in builds [2364039] needs to be handled

2018-11-21 Thread Alexey Goncharuk
Looking into this, looks like it is related to 'set consistent ID in tests' change. ср, 21 нояб. 2018 г. в 08:23, : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful th

[jira] [Created] (IGNITE-10358) thin python: put collections have no data type specification

2018-11-21 Thread Stepan Pilschikov (JIRA)
Stepan Pilschikov created IGNITE-10358: -- Summary: thin python: put collections have no data type specification Key: IGNITE-10358 URL: https://issues.apache.org/jira/browse/IGNITE-10358 Project: I

Re: JdbcColumnMeta

2018-11-21 Thread Vladimir Ozerov
Hi Peter, We will be able to fix this in consistent way for both "thin" and "thick" JDBC drivers once IGNITE-6173 [1] is merged. In this ticket we ensure that metadata for all schema objects are present on all nodes, so that we can simply delegate metadata request to underlying H2 table. [1] http

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Denis Mekhanikov
People often ask about possibility to store their data in that format, that they use in their applications. If you use Avro everywhere in your application, then why not store data in the same format in Ignite? So, how about making an interface, that would enlist all operations we need, and use this

Re: Time to remove automated messages from the devlist?

2018-11-21 Thread Павлухин Иван
Dmitriy, let's proceed with it. вт, 20 нояб. 2018 г. в 23:20, Dmitriy Pavlov : > > One more thing I want to emphasize here. We can't just remove messages, it > _must_ be sent to some list, which is why we need some additional list, > e.g. notifications@ for this. > > So only one option to proceed h

Re: Service grid redesign

2018-11-21 Thread Denis Mekhanikov
Vyacheslav, Actually, the service assignment is implemented in a way, that allows every node calculate the assignment itself, so no information needs to be shared. The only data, that is sent between nodes is deployment results, and I don't see an analogy with exchange here. Denis ср, 21 нояб. 2

[ANNOUNCE] Welcome Pavel Kovalenko as a new committer

2018-11-21 Thread Alexey Goncharuk
The Apache Ignite Project Management Committee (PMC) has invited Pavel Kovalenko to become a new committer and are happy to announce that he has accepted. Pavel was actively investigating and improving the speed of PME during node join/leave scenarios and achieved great progress in these tasks. P

[GitHub] ignite pull request #5074: IGNITE-9622: MVCC Cache API: prohibit non PESSIMI...

2018-11-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5074 ---

Re: [ANNOUNCE] Welcome Pavel Kovalenko as a new committer

2018-11-21 Thread Vyacheslav Daradur
Congrats Pavel, I'm looking forward to future PME optimizations! On Wed, Nov 21, 2018 at 11:44 AM Alexey Goncharuk wrote: > > The Apache Ignite Project Management Committee (PMC) has invited Pavel > Kovalenko to become a new committer and are happy to announce that he has > accepted. > > Pavel was

Re: Service grid redesign

2018-11-21 Thread Vyacheslav Daradur
During each deployment (exchange) process nodes send to the coordinator (p2p) deployments maps, caused: - users requests (deploy/undeploy) - affinity change (if affinity services present) - topology change (if services present) - activation (if services descriptors present) Also, single node deplo

Re: Thin clients all in one

2018-11-21 Thread Stepan Pilschikov
Dmitry, Alexey Thank you for help, this answers help me a lot with understanding how clients are work Not so long time ago i met problem which is have expected behavior, but its may broke some workflows in future for some users Its all about not specified data types in collections and map's All

Re: Service grid redesign

2018-11-21 Thread Vyacheslav Daradur
Denis, I suggested new names above in the thread. Please, look at PME document [1] is should be quiet actual to show the same flow. [1] https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood On Wed, Nov 21, 2018 at 11:43 AM Denis Mekhanikov wrote: > >

Re: [ANNOUNCE] Welcome Pavel Kovalenko as a new committer

2018-11-21 Thread Ivan Rakov
Pavel, please accept my congratulations. PME wouldn't be this fast without your efforts, keep the pace! Best Regards, Ivan Rakov On 21.11.2018 11:46, Vyacheslav Daradur wrote: Congrats Pavel, I'm looking forward to future PME optimizations! On Wed, Nov 21, 2018 at 11:44 AM Alexey Goncharuk wro

[jira] [Created] (IGNITE-10359) MVCC: P2P deployment for EntryProcessor looks broken.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10359: - Summary: MVCC: P2P deployment for EntryProcessor looks broken. Key: IGNITE-10359 URL: https://issues.apache.org/jira/browse/IGNITE-10359 Project: Ignite

Re: TcpCommunicationSpi extension to ignore docker bridge network

2018-11-21 Thread Denis Mekhanikov
David, This is a problem, that many people encounter. And I think, it's about time to deal with it. We have the following method, that collects all local addresses, that will be sent with node attributes: *IgniteUtils#resolveLocalAddresses()*. And it iterates over all local interfaces, that the m

Re: [ANNOUNCE] Welcome Pavel Kovalenko as a new committer

2018-11-21 Thread Nikita Amelchev
Congratulations, Pavel! ср, 21 нояб. 2018 г. в 11:47, Vyacheslav Daradur : > > Congrats Pavel, I'm looking forward to future PME optimizations! > On Wed, Nov 21, 2018 at 11:44 AM Alexey Goncharuk > wrote: > > > > The Apache Ignite Project Management Committee (PMC) has invited Pavel > > Kovalenko

[GitHub] ignite pull request #5455: Ignite 2.7 with tde fixes

2018-11-21 Thread nizhikov
GitHub user nizhikov opened a pull request: https://github.com/apache/ignite/pull/5455 Ignite 2.7 with tde fixes You can merge this pull request into a Git repository by running: $ git pull https://github.com/nizhikov/ignite ignite-2.7-with-tde-fixes Alternatively you can rev

Re: [ANNOUNCE] Welcome Pavel Kovalenko as a new committer

2018-11-21 Thread Dmitriy Pavlov
Congrats, Pavel. Well deserved achievement. ср, 21 нояб. 2018 г., 12:09 Nikita Amelchev : > Congratulations, Pavel! > ср, 21 нояб. 2018 г. в 11:47, Vyacheslav Daradur : > > > > Congrats Pavel, I'm looking forward to future PME optimizations! > > On Wed, Nov 21, 2018 at 11:44 AM Alexey Goncharuk

[jira] [Created] (IGNITE-10360) MVCC: Refactor MvccUpdateDataRow status handling.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10360: - Summary: MVCC: Refactor MvccUpdateDataRow status handling. Key: IGNITE-10360 URL: https://issues.apache.org/jira/browse/IGNITE-10360 Project: Ignite

[GitHub] ignite pull request #5456: IGNITE-10250

2018-11-21 Thread xtern
GitHub user xtern opened a pull request: https://github.com/apache/ignite/pull/5456 IGNITE-10250 …f write permits. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xtern/ignite IGNITE-10250 Alternatively you can review and apply

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Igor Sapego
I want to offer several optimizations: 1. If we store fields metadata anyway, and are going to store bitmasks for null fields, should we also exclude "header" byte from object field? As we can get field type info from a metadata. 2. If we have subsequent fields of fixed length we can avoid storin

Re: Service grid redesign

2018-11-21 Thread Vladimir Ozerov
Vyacheslav, I looked at the document and failed to find explanation why full maps are needed. Could you point me to a place where it is explained? I ask this because my impression from last discussion was that it is never needed. Service status change is initiated by user action, then all nodes pe

[jira] [Created] (IGNITE-10361) MTCGA: Remove duplicate tests.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10361: - Summary: MTCGA: Remove duplicate tests. Key: IGNITE-10361 URL: https://issues.apache.org/jira/browse/IGNITE-10361 Project: Ignite Issue Type: Impro

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Vladimir Ozerov
Denis, Could you please clarify - are you talking about storage, e.g. how objects are stored in Ignite, or about serialization as a whole? I'd like to better understand whether the use case you described is relevant to my idea of splitting binary objects from underlying storage format. My vision w

Re: Service grid redesign

2018-11-21 Thread Vyacheslav Daradur
The full map is needed: 1) to propagate deployment results which could be different from locally calculated in case of any errors; 2) to transfer deployment errors across the cluster; 3) to undeploy exceeded the number of service instances if needed; 4) to get know other nodes that deployment proce

Re: Service grid redesign

2018-11-21 Thread Vladimir Ozerov
Hi Vyacheslav, Still not clear enough for me. I do not see a reason to send another over a ring in case of successful execution. The only reason is an error on a node which require correction (re-deploy to other node, full service undeploy, etc). I think it makes sense to organize another call to

[jira] [Created] (IGNITE-10362) NoneRebalanceModeSelfTest looks check nothing.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10362: - Summary: NoneRebalanceModeSelfTest looks check nothing. Key: IGNITE-10362 URL: https://issues.apache.org/jira/browse/IGNITE-10362 Project: Ignite I

[GitHub] ignite pull request #5457: IGNITE-10352 Fix cache get request can be mapped ...

2018-11-21 Thread dgovorukhin
GitHub user dgovorukhin opened a pull request: https://github.com/apache/ignite/pull/5457 IGNITE-10352 Fix cache get request can be mapped to node while partition in MOVING state You can merge this pull request into a Git repository by running: $ git pull https://github.com/g

Re: TcpCommunicationSpi extension to ignore docker bridge network

2018-11-21 Thread Alexey Goncharuk
David, Skipping connection to an address that is declared as local by the local node is an absolutely valid idea. Though, will it be always the case that nodes will have the same IP for this bogus interface? We have a similar check for loopback addresses, I see nothing wrong with adding the same c

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Denis Mekhanikov
Vladimir, Thank you for the clarification. I didn't see this distinction first. I meant using customizable formats for all serialization, not only for storage. The idea behind my proposal is to avoid data conversion, when loading data into Ignite. It will complicate usage of thin clients though,

[jira] [Created] (IGNITE-10363) Web console: unexpected unsaved changes confirmation

2018-11-21 Thread Pavel Konstantinov (JIRA)
Pavel Konstantinov created IGNITE-10363: --- Summary: Web console: unexpected unsaved changes confirmation Key: IGNITE-10363 URL: https://issues.apache.org/jira/browse/IGNITE-10363 Project: Ignite

[GitHub] ignite pull request #5415: IGNITE-10234: Inference workflow for ML

2018-11-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5415 ---

[GitHub] ignite pull request #5430: IGNITE-9872 [Test falied] IgniteSinkConnectorTest...

2018-11-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5430 ---

[jira] [Created] (IGNITE-10364) .Net: FailureHandlerTest should be reworked

2018-11-21 Thread Vyacheslav Koptilin (JIRA)
Vyacheslav Koptilin created IGNITE-10364: Summary: .Net: FailureHandlerTest should be reworked Key: IGNITE-10364 URL: https://issues.apache.org/jira/browse/IGNITE-10364 Project: Ignite

Re: JdbcColumnMeta

2018-11-21 Thread Peter Borissow
Hello Vladimir,    Is there a ticket for the issue I described in JIRA or should we create one and mark it blocked pending #6173? Thanks,Peter On Wednesday, November 21, 2018, 3:30:21 AM EST, Vladimir Ozerov wrote: Hi Peter, We will be able to fix this in consistent way for both "th

Improve lazy mode SQL query execution (IGNITE-9171)

2018-11-21 Thread Taras Ledkov
Hi community, We will enhance lazy mode for SQL query execution. Lazy mode overview: Lazy mode is related to H2 lazy mode when the all query results are not copied to the RAM in some cases. The mode it is applicable for SELECTs that doesn't not require materialize all results in memory, e.g. 

[GitHub] ignite pull request #5458: IGNIE-10339 Fix connection to cluster failed when...

2018-11-21 Thread ivandasch
GitHub user ivandasch opened a pull request: https://github.com/apache/ignite/pull/5458 IGNIE-10339 Fix connection to cluster failed when invoking validate indexes closure from control.sh You can merge this pull request into a Git repository by running: $ git pull https://githu

[GitHub] ignite pull request #5412: IGNITE-10260: MVCC: GetAndPutIfAbsent operation r...

2018-11-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5412 ---

Re: [ANNOUNCE] Welcome Pavel Kovalenko as a new committer

2018-11-21 Thread Ilya Lantukh
Congratulations! On Wed, Nov 21, 2018 at 12:30 PM Dmitriy Pavlov wrote: > Congrats, Pavel. Well deserved achievement. > > ср, 21 нояб. 2018 г., 12:09 Nikita Amelchev : > > > Congratulations, Pavel! > > ср, 21 нояб. 2018 г. в 11:47, Vyacheslav Daradur : > > > > > > Congrats Pavel, I'm looking for

[GitHub] ignite pull request #5459: IGNITE-8391

2018-11-21 Thread vldpyatkov
GitHub user vldpyatkov opened a pull request: https://github.com/apache/ignite/pull/5459 IGNITE-8391 Removing some WAL history segments leads to WAL rebalance hanging You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache

[GitHub] ignite pull request #5460: IGNITE-10191 Incorrect comparison of lists in Ren...

2018-11-21 Thread oignatenko
GitHub user oignatenko opened a pull request: https://github.com/apache/ignite/pull/5460 IGNITE-10191 Incorrect comparison of lists in RendezvousAffinityFunctionSimpleBenchmark#testAffinityCompatibility You can merge this pull request into a Git repository by running: $ git p

Re: proposed realization KILL QUERY command

2018-11-21 Thread Юрий
Denis, Let's consider parameter of KILL QUERY just a string with some query id, without any meaning for user. User just need to get the id and pass as parameter to KILL QUERY command. Even if query is distributed it have single query id from user perspective and will killed on all nodes. User jus

Re: proposed realization KILL QUERY command

2018-11-21 Thread Vladimir Ozerov
Denis, Space is bad candidate because it is a whitespace. Without whitespaces we can have syntax without quotes at all. Any non-whitespace delimiter will work, though: KILL QUERY 45.1 KILL QUERY 45-1 KILL QUERY 45:1 On Wed, Nov 21, 2018 at 3:06 PM Юрий wrote: > Denis, > > Let's consider parame

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Vladimir Ozerov
Denis, In theory data conversion could be avoided in certain cases. E.g. consider a case of loading data through streamer. We know the cache, we know it's metadata and row format. So instead of doing "user object" -> "binary object" -> "row", we can do "user object" -> "row". On Wed, Nov 21, 2018

[jira] [Created] (IGNITE-10366) MVCC: Create "Cache 1" test suite for MVCC mode.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10366: - Summary: MVCC: Create "Cache 1" test suite for MVCC mode. Key: IGNITE-10366 URL: https://issues.apache.org/jira/browse/IGNITE-10366 Project: Ignite

[jira] [Created] (IGNITE-10365) MVCC: Create "Cache 6" test suite for MVCC mode.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10365: - Summary: MVCC: Create "Cache 6" test suite for MVCC mode. Key: IGNITE-10365 URL: https://issues.apache.org/jira/browse/IGNITE-10365 Project: Ignite

[GitHub] ignite pull request #5057: IGNITE-9928

2018-11-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5057 ---

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Pavel Tupitsyn
Vladimir, IMO the issue is that we allow any type of data in the cache (put Person, then put int to the same cache). Are we going to address this in 3.0 and enforce key/value types according to cache configuration? This will provide more space for optimizations. On Wed, Nov 21, 2018 at 3:14 PM Vl

[jira] [Created] (IGNITE-10367) MVCC: Create "Cache 8" test suite for MVCC mode.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10367: - Summary: MVCC: Create "Cache 8" test suite for MVCC mode. Key: IGNITE-10367 URL: https://issues.apache.org/jira/browse/IGNITE-10367 Project: Ignite

Re: [ANNOUNCE] Welcome Igor Seliverstov as a new committer

2018-11-21 Thread Andrey Mashenkov
Congratulations, Igor! On Wed, Nov 21, 2018 at 10:11 AM Alexey Zinoviev wrote: > Great news, Igor, hope to use this great feature in ML module! > > вт, 20 нояб. 2018 г. в 11:08, Павлухин Иван : > > > Igor my congratulations! Keep going! > > > > 2018-11-20 9:01 GMT+01:00, Vladimir Ozerov : > > >

[jira] [Created] (IGNITE-10368) MVCC: Create "Cache 9" test suite for MVCC mode.

2018-11-21 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-10368: - Summary: MVCC: Create "Cache 9" test suite for MVCC mode. Key: IGNITE-10368 URL: https://issues.apache.org/jira/browse/IGNITE-10368 Project: Ignite

Re: Time to remove automated messages from the devlist?

2018-11-21 Thread Dmitriy Pavlov
Please start a vote according to https://www.apache.org/foundation/voting.html Anyone can start a vote, you don't need to be a PMC. I don't feel it is a very important issue to remove notifications from the list, as it can be easily filtered out using mail setup. But if someone feels it is really

Re: Improve lazy mode SQL query execution (IGNITE-9171)

2018-11-21 Thread Vladimir Ozerov
Taras, Thank you for analysis. I'd like to add that there is another solution - PME :-) In this case DDL will be blocked until current SELECTs and updates are finished, and do not allow new operations to be executed. This approach solves all issues - it avoids deadlocks between query threads and D

[jira] [Created] (IGNITE-10369) PDS 4 hangs on TC

2018-11-21 Thread Ivan Bessonov (JIRA)
Ivan Bessonov created IGNITE-10369: -- Summary: PDS 4 hangs on TC Key: IGNITE-10369 URL: https://issues.apache.org/jira/browse/IGNITE-10369 Project: Ignite Issue Type: Test Reporte

[GitHub] SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching.

2018-11-21 Thread GitBox
SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching. URL: https://github.com/apache/ignite-teamcity-bot/pull/76#discussion_r235369173 ## File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/TcHelperDb.java ## @@ -93,6

[GitHub] SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching.

2018-11-21 Thread GitBox
SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching. URL: https://github.com/apache/ignite-teamcity-bot/pull/76#discussion_r235369350 ## File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/TcHelperDb.java ## @@ -93,6

[GitHub] SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching.

2018-11-21 Thread GitBox
SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching. URL: https://github.com/apache/ignite-teamcity-bot/pull/76#discussion_r235369867 ## File path: ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/visa/TcBotTriggerAndSignOffService

[jira] [Created] (IGNITE-10370) ML: TensorFlowLocalInferenceExample fails on Windows

2018-11-21 Thread Anton Dmitriev (JIRA)
Anton Dmitriev created IGNITE-10370: --- Summary: ML: TensorFlowLocalInferenceExample fails on Windows Key: IGNITE-10370 URL: https://issues.apache.org/jira/browse/IGNITE-10370 Project: Ignite

[GitHub] ignite pull request #5461: IGNITE-10370: Fix TensorFlowLocalInferenceExample...

2018-11-21 Thread dmitrievanthony
GitHub user dmitrievanthony opened a pull request: https://github.com/apache/ignite/pull/5461 IGNITE-10370: Fix TensorFlowLocalInferenceExample fail on Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignit

[GitHub] ignite pull request #4812: IGNITE-9309 added runLocal

2018-11-21 Thread oleg-ostanin
Github user oleg-ostanin closed the pull request at: https://github.com/apache/ignite/pull/4812 ---

Re: Improve lazy mode SQL query execution (IGNITE-9171)

2018-11-21 Thread Taras Ledkov
Vladimir, The query protocol will be changed in both solution. The tables versions must be added to the 'GridQueryNextPageResponse' message and must be compared on the reduce node too. Because the DLL / DML race may be happens on different nodes. 21.11.2018 15:24, Vladimir Ozerov пишет: Tara

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Vladimir Ozerov
Pavel, This could be solved with aforementioned "RowFormat". We will be able to configure cache as follows: "this is a cache with strict type checks, first one is A, with fields A1, A2, A3, second is B with fields B1, B2". So it will be possible to serialize anything into binary object, but when i

Re: JdbcColumnMeta

2018-11-21 Thread Vladimir Ozerov
Hi Peter, I am not aware of such ticket. On Wed, Nov 21, 2018 at 2:23 PM Peter Borissow wrote: > Hello Vladimir,Is there a ticket for the issue I described in JIRA or > should we create one and mark it blocked pending #6173? > > Thanks,Peter > > On Wednesday, November 21, 2018, 3:30:21

Re: Service grid redesign

2018-11-21 Thread Dmitriy Pavlov
Hi Vyacheslav, Vladimir, Could you please invite me, if you will set up a call. ср, 21 нояб. 2018 г. в 13:08, Vladimir Ozerov : > Hi Vyacheslav, > > Still not clear enough for me. I do not see a reason to send another over a > ring in case of successful execution. The only reason is an error on

Re: Improve lazy mode SQL query execution (IGNITE-9171)

2018-11-21 Thread Vladimir Ozerov
Hi Taras, Unfortunately, this will not be easy to implement. We already have AffinityTopologyVersion which is updated on certain schema change operations, such as CREATE/DROP TABLE. Moreover, it is already passed in query request messages (GridH2QueryRequest.topVer). Unfortunately, it is not updat

New API for changing configuration of persistent caches

2018-11-21 Thread Eduard Shangareev
Igniters, I propose new public API to change the cache configuration of persistent caches with keeping data. It would look like: Ignite ignite = ...; ignite.restartCaches(cfg1, ... cfgN); where cfgX is a new cache configuration, which contains the name of an existing persistent cache. The obvi

Re: Ignite web console on kubernetes with ingress

2018-11-21 Thread Павлухин Иван
Hi Vishwas, Are you going to do some contribution related to your question? If not then you could ask the same question on u...@ignite.apache.org User list exists exactly for asking questions related to usability problems and most likely you will get an answer there. ср, 21 нояб. 2018 г. в 10:26,

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Vladimir Ozerov
Hi Ed, Several questions from my side: 1) If we do not allow to change the most demanded by users things like affinity or persistence/in-memory, then what kind of configuration properties do we expect to be changed? Can we have several examples? 2) How will it interact with PME? 3) How will it int

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Dmitriy Pavlov
Hi Eduard, Can we just change cache parameters after calling cache() or getOrCreateCache() by user? The method is blocked until restart finished. We can set up new cache parameters according to the provided configuration. If it is not possible to change some parameter, a method will fail. I gues

[GitHub] ignite pull request #5462: IGNITE-10369 PDS 4 hangs on TC

2018-11-21 Thread ibessonov
GitHub user ibessonov opened a pull request: https://github.com/apache/ignite/pull/5462 IGNITE-10369 PDS 4 hangs on TC You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10369 Alternatively you can rev

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Eduard Shangareev
Vladimir, 1) Affinity could be changed, but count of partition couldn't be. 2) So it would trigger 2 PME. Dynamic start and stop. 3) In theory, should cancel them and new setting should be applied. How it works now? Create an index and stop node, for example. On Wed, Nov 21, 2018 at 4:56 PM Vladi

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Eduard Shangareev
Dmitriy, >From my point of view, it's not good to have such unexpected behavior (and change old one) for cache and getOrCreateCache method. It should be a conscious decision, not an accident. On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev wrote: > Vladimir, > > 1) Affinity could be changed,

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Pavel Tupitsyn
Makes sense. I'm trying to grasp this from usability POV. Having two ways of storing data with different behavior can be confusing. In .NET we already have this issue with DateTime when if you want SQL, you get subtly different behavior. So IMO we should enable strict type checks for all caches,

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Dmitriy Pavlov
Sure. thanks for the reply. I think in this case we should mention a new method in all error messages like 'cache create failed because of different configs'. So a user can easily find a new method to apply. So product will point itself to a method how to get the evolution of cache configs without

Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Stephen Darlington
Possibly heading into wishlist rather than practical territory here, but you did ask... > What we need is to introduce an interface which will convert a pair of > key-value objects into a row. This row will be used to store data and to > get fields from it. Rather than mapping objects to a row,

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Vladimir Ozerov
Eduard, Got it. Please take the following things in count during design: 1) Two distinct PMEs might not work well. Consider a situation w1hen I decided to move a cache with index "MY_INDEX" from schema A to schema B. While cache was stopped, another cache with index "MY_INDEX" was created in sche

MVCC test coverage.

2018-11-21 Thread Andrey Mashenkov
Hi Igniters, As you may already know, MVCC transaction feature will be available in upcoming Ignite-2.7. However, MVCC Tx feature is released for beta testing and has many limitations and we a going to improve stability and integrations with other features in future releases. We can reuse existed

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Eduard Shangareev
Vladimir, I propose not to change cache configuration in runtime but restart cache with the new compatible configuration on data which we have underfoot. What we could change: -backup count; -TRANSACTIONAL <-> ATOMIC; -REPLICATED - PARTITIONED; -other settings. So, yeah, it would be great to hav

[GitHub] dspavlov opened a new pull request #77: IGNITE-10336: Build chain collection refactored to avoid missing builds

2018-11-21 Thread GitBox
dspavlov opened a new pull request #77: IGNITE-10336: Build chain collection refactored to avoid missing builds URL: https://github.com/apache/ignite-teamcity-bot/pull/77 This is an automated message from the Apache Git Serv

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Vladimir Ozerov
Eduard, Simple != correct. Let’s consider a simple use case: user want to change PARTITIONED -> REPLICATED from .NET, but do not some classes from CacheConfiguration. How do we solve this? Vladimir. ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev : > Vladimir, > > I propose not to change cache

[GitHub] zzzadruga opened a new pull request #78: IGNITE-10203 Support for alternative configurations for PR testing

2018-11-21 Thread GitBox
zzzadruga opened a new pull request #78: IGNITE-10203 Support for alternative configurations for PR testing URL: https://github.com/apache/ignite-teamcity-bot/pull/78 This is an automated message from the Apache Git Service.

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Alexey Kuznetsov
Vova, Ed I think in case when classes not available we can sent a string with cache XML configuration (bean) to server node that has all classes in classpath. What do you think? On Wed, Nov 21, 2018 at 10:10 PM Vladimir Ozerov wrote: > Eduard, > > Simple != correct. Let’s consider a simple use

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Vladimir Ozerov
Alex, Spring XML is not easily available from API. Sometimes it simply doesn’t exist. Why not use builder instead? Ignite.changeCache(“my_cache”, Builder.create().setCacheMode(REPLICATED).build()) Advantages: 1) Available on all platforms 2) Expose only properties we support 3) Avoid race condit

[GitHub] ignite pull request #5463: IGNITE-10301 GridToStringBuilder is broken for cl...

2018-11-21 Thread SomeFire
GitHub user SomeFire opened a pull request: https://github.com/apache/ignite/pull/5463 IGNITE-10301 GridToStringBuilder is broken for classes with inheritance You can merge this pull request into a Git repository by running: $ git pull https://github.com/SomeFire/ignite IGNITE

[GitHub] ignite pull request #5464: IGNITE-9902: ScanQuery doesn't take lost partitio...

2018-11-21 Thread avplatonov
GitHub user avplatonov opened a pull request: https://github.com/apache/ignite/pull/5464 IGNITE-9902: ScanQuery doesn't take lost partitions into account You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite igni

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Vladimir Ozerov
Correction: “do not *have* some classes”. ср, 21 нояб. 2018 г. в 18:09, Vladimir Ozerov : > Eduard, > > Simple != correct. Let’s consider a simple use case: user want to change > PARTITIONED -> REPLICATED from .NET, but do not some classes from > CacheConfiguration. How do we solve this? > > Vlad

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Eduard Shangareev
Vladimir, How does .Net user start caches right now? On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov wrote: > Eduard, > > Simple != correct. Let’s consider a simple use case: user want to change > PARTITIONED -> REPLICATED from .NET, but do not some classes from > CacheConfiguration. How do we

[GitHub] ignite pull request #5396: IGNITE-10226 Fixed wrong partition state recovery

2018-11-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5396 ---

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Vladimir Ozerov
Ed, Caches in .NET could be started programmatically, from XML which .NET API has no access to, or dynamically from remote nodes (eg Java node). ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev : > Vladimir, > > How does .Net user start caches right now? > > On Wed, Nov 21, 2018 at 6:10 PM Vladim

Re: New API for changing configuration of persistent caches

2018-11-21 Thread Eduard Shangareev
Vladimir, I don't see any difference here. The same possibilities would be available as with normal cache start: -XML; -remote node. >3) Avoid race condition when configuration changes between configuration read and method call (what could lead to a number of strange effects). Well, we could ad

  1   2   >