[DISCUSSION] IGNITE-13152 Introduce spring-data repository configuration properties

[Sergey Moldachev]

Hello, Igniters!

I'd like to discuss with you changes for *ignite-spring-data*. You can read
about motivation and the idea in [1].

[1] https://issues.apache.org/jira/browse/IGNITE-13152


-- 
Regards,
Sergey

Re: Slim binary release and docker image for Apache Ignite

[Ilya Kasnacheev]

Hello!

I have just merged slim binary release to master.

I will now try to tweak nightly builds TC suite to build this package also.
It may be broken for some brief period of time.

Regards,
-- 
Ilya Kasnacheev


вт, 10 мар. 2020 г. в 18:24, Ilya Kasnacheev :

> Hello!
>
> I understand that procedures are courtesy Apache Ignite, but I assume that
> you went through them and can now repeat them reproducibly.
>
> Thank you!
> --
> Ilya Kasnacheev
>
>
> вт, 10 мар. 2020 г. в 18:12, Maxim Muzafarov :
>
>> Ilya,
>>
>> It is not "mine" generic release procedures they are "ours" :-)
>> I've created the issue [1] based on current discussion thread.
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-12765
>>
>> On Tue, 10 Mar 2020 at 13:31, Ilya Kasnacheev 
>> wrote:
>> >
>> > Hello!
>> >
>> > It is currently included.
>> >
>> > Maxim, can you prepare a slim release package based on your generic
>> release
>> > procedures? We could take a look at it and then perhaps add it to
>> downloads
>> > page officially.
>> >
>> > What do you think?
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > пт, 6 мар. 2020 г. в 20:48, Maxim Muzafarov :
>> >
>> > > Ilya,
>> > >
>> > > `ignite-compress` is necessary for `wal page snapshot compression` [1]
>> > > which in turn shows very good performance results. So, I suppose, it's
>> > > better to include it to the "slim" binary.
>> > >
>> > > [1] https://issues.apache.org/jira/browse/IGNITE-11336
>> > >
>> > > On Fri, 6 Mar 2020 at 13:31, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hello!
>> > > >
>> > > > I added these because they are infrastructural to Ignite, as
>> opposed to
>> > > > integrations. They are also both very slim.
>> > > >
>> > > > Regards,
>> > > > --
>> > > > Ilya Kasnacheev
>> > > >
>> > > >
>> > > > пт, 6 мар. 2020 г. в 13:25, Stephen Darlington <
>> > > > stephen.darling...@gridgain.com>:
>> > > >
>> > > > > Why ignite-jta and ignite-urideploy? Anecdotally at least, I know
>> very
>> > > few
>> > > > > people who use either.
>> > > > >
>> > > > > > On 6 Mar 2020, at 11:09, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > Hello!
>> > > > > >
>> > > > > > Re-posting from *[DISCUSSION] Release Apache Ignite 2.8.0 RC1*
>> > > > > >
>> > > > > > I have prepared assemblies for Apache Ignite slim packaging:
>> > > > > > https://github.com/apache/ignite/tree/ignite-slim
>> > > > > >
>> > > > > > It is based on ignite-2.8
>> > > > > >
>> > > > > > You can build it with mvn initialize -Prelease,lgpl
>> > > > > -Dignite.edition=apache-
>> > > > > > ignite-slim after a normal release build.
>> > > > > >
>> > > > > > Please consider the contents of resulting
>> > > > > > target/bin/apache-ignite-slim-2.8.0-bin.zip
>> > > > > > It will be a 65M download as opposed to main 455M
>> apache-ignite-2.8.0
>> > > > > > distribution.
>> > > > > >
>> > > > > > My suggestion is that we can publish it as a post-release step
>> since
>> > > it
>> > > > > > does not affect the release in any way. If we do, we should
>> probably
>> > > > > > indicate size for every kind of artifact in our download
>> section, so
>> > > our
>> > > > > > users can choose based on that information.
>> > > > > >
>> > > > > > The following modules are included:
>> > > > > >
>> > > > > > libs:
>> > > > > > core/shmem/jcache
>> > > > > > ignite-indexing
>> > > > > > ignite-spring
>> > > > > >
>> > > > > > libs/optional:
>> > > > > > ignite-compress  ignite-kubernetes  ignite-log4j2
>> > > ignite-rest-http
>> > > > > > ignite-spring-data_2.2
>> > > > > > ignite-jta   ignite-log4j   ignite-opencensus
>> ignite-slf4j
>> > > > > > ignite-urideploy
>> > > > > >
>> > > > > > I have kept examples, but removed benchmarks. sqlline still
>> present,
>> > > of
>> > > > > > course.
>> > > > > >
>> > > > > > ignite-zookeeper has a lot of dependencies (8M) which we do not
>> > > update
>> > > > > > often enough (such as guava, curator, jackson), and which may
>> form an
>> > > > > > attack surface.
>> > > > > >
>> > > > > > Not a pressing problem for 'integrated' ignite-zookeeper users,
>> since
>> > > > > they
>> > > > > > can re-import these dependencies with more recent versions using
>> > > maven or
>> > > > > > gradle.
>> > > > > > But for our users who rely on binary package for all JARs,
>> outdated
>> > > > > > dependencies may pose a problem.
>> > > > > >
>> > > > > > Therefore my opinion is to exclude this dependency and not put
>> our
>> > > faith
>> > > > > on
>> > > > > > zookeeper dependency version.
>> > > > > >
>> > > > > > The same can be put for ignite-compress, and indeed, I'm not
>> sure if
>> > > we
>> > > > > > should keep it.
>> > > > > >
>> > > > > > We can have an ad-hoc vote here.
>> > > > > >
>> > > > > > I would like to hear arguments for both inclusion and exclusion
>> of
>> > > > > > ignite-zookeeper and ignite-compress into slim package (in any
>> > > > > combination).
>> > > > > >
>> > > 

[jira] [Created] (IGNITE-13158) Get rid Externalizable interface at IgniteTxAdapter

[Vyacheslav Koptilin (Jira)]

Vyacheslav Koptilin created IGNITE-13158:


 Summary: Get rid Externalizable interface at IgniteTxAdapter
 Key: IGNITE-13158
 URL: https://issues.apache.org/jira/browse/IGNITE-13158
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Vyacheslav Koptilin
Assignee: Vyacheslav Koptilin
 Fix For: 2.9


It seems that {{IgniteTxAdapter}} implements {{Externalizable}} without any 
reason. Transaction instances are not transferred via network, and so I see no 
reason for implementing {{Externalizable}} interface.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13159) Calcite integration. Decorrelator support SOME and ALL operator.

[Andrey Mashenkov (Jira)]

Andrey Mashenkov created IGNITE-13159:
-

 Summary: Calcite integration. Decorrelator support SOME and ALL 
operator.
 Key: IGNITE-13159
 URL: https://issues.apache.org/jira/browse/IGNITE-13159
 Project: Ignite
  Issue Type: Bug
Reporter: Andrey Mashenkov


Now Calcite SqlToRelConverter throws an exception if found a subquery in 
ALL\SOME operator. See SubqueryRewriteRuleTest.testNonInToSemiAntiJoinRule test 
and find stack trace below.

We have to either set flag "expand=false" and implement LogicalCorrelate 
converter
or some rules to overcome the issue.

However, setting flag "expand=false" makes query pass, but with wrong join type 
(left and inner instead of anti) and moreover it brakes other queries due to 
weird decorrelator behavior. Possibly there is bug in Calcite.

 

 
{code:java}
Caused by: java.lang.RuntimeException: ALL is only supported if expand = 
falseCaused by: java.lang.RuntimeException: ALL is only supported if expand = 
false at 
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4814)
 at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertWhere(SqlToRelConverter.java:1016)
 at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:662)
 at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:640)
 at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3384)
 at 
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:566)
 at 
org.apache.ignite.internal.processors.query.calcite.prepare.IgnitePlanner.rel(IgnitePlanner.java:203)
 at 
org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.optimize(ExecutionServiceImpl.java:618)
 at 
org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.prepareExplain(ExecutionServiceImpl.java:644)
 at 
org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.prepareSingle(ExecutionServiceImpl.java:573)
 at 
org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.prepare0(ExecutionServiceImpl.java:531)
 ... 16 more
{code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Evgenii Zhuravlev]

Hi,

+1 for both moving control.sh to the separate module and adding
autocompletion.

Will API remain the same in control.sh?

Evgenii

пт, 5 июн. 2020 г. в 01:59, ткаленко кирилл :

> Folks have created a ticket [1].
>
> 1 - https://issues.apache.org/jira/browse/IGNITE-13120
>
> 02.06.2020, 16:48, "ткаленко кирилл" :
> > Maxim I suggested moving control.sh in a separate module, are we talking
> about the same thing?
> >
> > 02.06.2020, 16:15, "Maxim Muzafarov" :
> >>  Folks,
> >>
> >>  +1
> >>
> >>  However, AFAIK control.sh is the part of the ignite-core module with
> >>  zero dependencies from external resources.
> >>  Would it be better going the `control.sh` extensions-way?
> >>
> >>  By the way, according to README.md [1] the picocli is already using by
> >>  the Apache Ignite, right? :-)
> >>
> >>>   Picocli is used in the Apache Hadoop Ozone/HDDS command line tools,
> the Apache Hive benchmark CLI, has ** Apache [Ignite TensorFlow] **, and
> Apache Sling.
> >>
> >>  [1] https://github.com/remkop/picocli/blame/master/README.md#L199
> >>
> >>  On Tue, 2 Jun 2020 at 16:09, Ivan Daschinsky 
> wrote:
> >>>   +1 But this is not only usability improvement, but also a huge code
> >>>   improvement. With picocli developers can add custom command without
> writing
> >>>   a lot of boilerplate and error prone code to do a trivial task
> >>>   of parsing CLI arguments. Cleaner code, less bugs also matter.
> >>>
> >>>   вт, 2 июн. 2020 г. в 16:02, Sergey Antonov <
> antonovserge...@gmail.com>:
> >>>
> >>>   > It would be a great usability improvement!
> >>>   >
> >>>   > +1 From me.
> >>>   >
> >>>   > вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky
>  >>>   > >:
> >>>   >
> >>>   > >
> >>>   > >
> >>>   > > good catch ! it`s a little bit pain for now to working with it.
> >>>   > >
> >>>   > >
> >>>   > > >Hi, Igniters!
> >>>   > > >
> >>>   > > >At the moment to work with the control.sh we need to know
> exactly what
> >>>   > > the name of the command and its options are and so the user can
> often
> >>>   > make
> >>>   > > mistakes when using it. So I think it would be useful to do
> control.sh
> >>>   > more
> >>>   > > user-friendly by adding autocomplete as in modern command-line
> utilities.
> >>>   > > >
> >>>   > > >For this purpose, I suggest using framework [1] and to do this,
> take out
> >>>   > > control.sh together with its associated classes in a separate
> module such
> >>>   > > as "modules/control-utility".
> >>>   > > >
> >>>   > > >Comments, suggestions?
> >>>   > > >
> >>>   > > >[1] - https://picocli.info/
> >>>   > >
> >>>   > >
> >>>   > >
> >>>   > >
> >>>   >
> >>>   >
> >>>   >
> >>>   > --
> >>>   > BR, Sergey Antonov
> >>>   >
> >>>
> >>>   --
> >>>   Sincerely yours, Ivan Daschinskiy
>

[jira] [Created] (IGNITE-13160) .NET: wrong affinity key registration with AffinityKeyMapped attribute

[Alexandr Shapkin (Jira)]

Alexandr Shapkin created IGNITE-13160:
-

 Summary: .NET: wrong affinity key registration with 
AffinityKeyMapped attribute
 Key: IGNITE-13160
 URL: https://issues.apache.org/jira/browse/IGNITE-13160
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Alexandr Shapkin


When QueryEntities is set alongside a custom key that utilizes the 
AffinityKeyMapped attribute, the field won't be taken into account for some 
reason. When QueryEntities configuration gets removed, all works fine, setting 
KeyConfiguration manually helps as well. At the same time, the BinaryType 
registration itself looks fine.

After the investigation, it turned out that affinityKey field is not being 
registered on cache registration, and null is being cached instead. Later on 
Ignite correctly tries to register a custom class as a key, but it's never 
being updated internally case the real value is already cached.

 

Reproducer (add to the AffinityTest.cs):
{code:java}
/// 
/// Tests AffinityKeyMapped attribute should map to the same partitions
/// for the same field value.
/// 
[Test]
public void TestCustomAffinity()
{
IIgnite g = Ignition.GetIgnite("grid-0");

var cacheCfg = new CacheConfiguration("mycache")
{
// Without QueryEntities tests passes.
QueryEntities = new List()
{
new QueryEntity(typeof(MyKey), typeof(int))
}
};
g.GetOrCreateCache(cacheCfg);

var key1 = new MyKey {Data = "data1", AffinityKey = 1};
var key2 = new MyKey {Data = "data2", AffinityKey = 1};

ICacheAffinity aff = g.GetAffinity(cacheCfg.Name);
Assert.AreEqual(aff.GetPartition(key1), aff.GetPartition(key2));
}

public class MyKey
{
[QuerySqlField]
public string Data { get; set; }
[AffinityKeyMapped]
public long AffinityKey { get; set; }
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] IGNITE-13152 Introduce spring-data repository configuration properties

[Denis Magda]

Hi Sergey,

In my experience, Spring Repositories configuration stays minimalistic, and
one of two approaches is followed to create a schema in an underlying
storage engine.

The first approach is when we can execute schema initialization scripts
that will pre-create all SQL tables, indexes, etc. This approach works for
a relational database as well as Ignite. Once the schema is set, we can
start our Spring application with Repositories and other entities in place.
An Ignite caches/tables can also be pre-established via CacheConfiguration
with all required expiration policies and other parameters.

The second method is to let Spring Data initialize the schema dynamically
based on the configuration of your @Entity classes. That's probably what
you're suggesting to do by adding custom configuration parameters
to @Repositories. If we want to support this second dynamic approach, then
we should do this via @Entity classes and not through @Repositories. It
should be possible to introspect @Entity specific annotations and produce
DDL commands or CacheConfigurations that will create Ignite caches/tables
as soon as your Spring Data application starts.

Let me know if I miss anything.

-
Denis


On Wed, Jun 17, 2020 at 6:17 AM Sergey Moldachev 
wrote:

> Hello, Igniters!
>
> I'd like to discuss with you changes for *ignite-spring-data*. You can read
> about motivation and the idea in [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13152
> 
>
> --
> Regards,
> Sergey
>

Re: Nightly snapshots build path

[Saikat Maitra]

Thank you, I will check if I can pull dependencies from teamcity.

Regards
Saikat

On Mon, 15 Jun 2020 at 3:21 AM, Petr Ivanov  wrote:

> Hi, Saikat.
>
>
> AFAIK, this [1] build configuration never had an option of uploading
> artifacts to some staging Maven repository.
> Perhaps, that should be fixed, but I do not have access to those
> repositories for upload.
>
>
>
> [1]
> https://ci.ignite.apache.org/buildConfiguration/Releases_NightlyRelease_RunApacheIgniteNightlyRelease?mode=builds
>
> > On 13 Jun 2020, at 20:29, Saikat Maitra  wrote:
> >
> > Hi All,
> >
> > We have enabled the travis build for ignite-extensions (
> > https://travis-ci.org/github/apache/ignite-extensions) but it has
> > dependency on ignite-core 2.9.0-snapshot jars. I was looking into our
> > nightly snapshots repository paths and could not find the latest
> snapshots
> > build.
> >
> >
> https://repository.apache.org/content/repositories/snapshots/org/apache/ignite/ignite-core/
> >
> > In teamcity we have been able to resolve this dependency since
> > ignite-extensions teamcity build agent was able to pull nightly snapshots
> > from main ignite build.
> >
> > Can you please let me know if the nightly snapshots repository path has
> > changed for ignite-core?
> >
> > Regards,
> > Saikat
>
>