Re: Apache Ignite 3 Alpha 2 webinar follow up questions

[Andrey Mashenkov]

Courtney,

I have a short update.
>   3. The Lucene API enabled more flexibility and removed a network
>  round trip from our queries.
>  4. Given Calcite's ability to support custom SQL functions, I'd love
>  to have the ability to define custom functions that Lucene was
answering
As we won't support IndexingSPI extension point anymore.

Seems ContinuousQuery might be a good choice to keep the local Lucene index
in sync with the data.

SQL functions will be supported, if it will be possible to do smth like
this "Select * From T1 Where T1.key in
(TextQueryFunction("text_query_string"))",
where "TextQueryFunction" is a custom SQL function that might be able to
access some static method or local Ignite service or whatever, then
will this work for you?


On Fri, Jul 23, 2021 at 1:10 AM Andrey Mashenkov 
wrote:

> Hi Courtney,
>
> Thanks for your feedback.
>
> I've gone through the questions and have no the whole picture of your use
> case.
> Would you please clarify how you exactly use the Ignite? what are the
> integration points?
> and maybe share some experience with using Ignite SPIs?
>
> We'll keep the information in mind while developing the Ignite,
> because this may help us to make a better product.
>
> By the way, I'll try to answer the questions.
>
> >   1. Schema change - does that include the ability to change the types of
> >   fields/columns?
> Yes, we plan to support transparent conversion to a wider type on-fly
> (e.g. 'int' to 'long').
> This is a major point of our Live-schema concept.
> In fact, there is no need to convert data on all the nodes in a
> synchronous way as old SQL databases do (if one supports though),
> we are going to support multiple schema versions and convert data
> on-demand on a per-row basis to the latest version,
> then write-back the row.
>
> More complex things like 'String' -> 'int' are out of scope for now
> because it requires the execution of a user code on the critical path.
> The limitation here is column MUST NOT be indexed, because an index over
> the data of different kinds is impossible.
>
>
>  >  2. Will the new guaranteed consistency between APIs also mean SQL will
>  >  gain transaction support?
> Yes, we plan to have Transactional SQL.
> DDL will be non-transactional though, and I wonder if the one supports
> this.
>
> Ignite 3 will operate with Rows underneath, but classic Table API and
> Key-value will be available to a user
> at the same time and with all consistency guarantees.
>
>
> >  3. Has there been any decision about how much of Calcite will be exposed
> >   to the client? When using thick clients, it'll be hugely beneficial to
> be
> >   able to work with Calcite APIs directly to provide custom rules and
> >  optimizations to better suit organization needs
> As of now, we have no plans to expose any Calcite API to a user.
> AFAIK, we have our custom Calcite convention, custom rules that are aware
> of distributed environment,
> and additional AST nodes. The rules MUST correctly propagate internal
> information about data distribution,
> so I'm not sure want to give low-level access to them.
>
> > We Index into Solr and use the Solr indices
> Ignite 1-2 has poor support for TEXT queries, which is totally
> unconfigurable.
> Also, Lucene indices underneath are NOT persistent that requires too much
> effort to fix it.
> GeoSpatial index has the same issues, we decided to drop them along with
> Indexing SPI at all.
>
> However, you can find the activity on dev-list on the Index Query topic.
> Guys are going to add IndexQuery (a scan query over the sorted index which
> can use simple conditions) in Ignite 2.
> We also plan to have the same functionality, maybe it is possible to add
> full-text search support here.
> Will it work for you, what do you think?
>
>
> >4. Will the unified storage model enable different versions of Ignite
> to
> >   be in the cluster when persistence is enabled so that rolling restarts
> can
> >   be done?
> I'm not sure a rolling upgrade (RU) will be available because too much
> compatibility issues should be resolved
> to make RU possible under the load without downtime.
>
> Maybe it makes sense to provide some grid mode (maintenance mode) for RU
> purposes that will block all the user load
> but allow upgrade the grid. E.g. for the pure in-memory case.
>
> Persistence compatibility should be preserved as it works for Ignite 2.
>
>
> >5. Will it be possible to provide a custom cache store still and will
> >   these changes enable custom cache stores to be queryable from SQL?
> I'm not sure I fully understand this.
> 1. Usually, SQL is about indices. Ignite can't perform a query over the
> unindexed data.
>
> 2. Fullscan over the cache that contains only part of data + scan the
> CacheStore, then merging the results is a pain.
> Most likely, running a query over CacheStore directly will be a simpler
> way, and even more performant.
> Shared CacheStore (same for all nodes) will definitely kill the
> 

Re: Text Queries Support

[Atri Sharma]

Hello,

An update, please. I am working through persistence of Lucene index using
Ignite Dictionary, and will be asking some questions soon.

I had one doubt - - where does this change go? Ignite 3?

Also, I know we want to build native support for text searches in Ignite 3.
Is the work I am proposing here part of that, or will that be a separate
effort?

On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev, 
wrote:

> Hello!
>
> I think that number one is the most important one, then maybe it will see
> more use and other deficiencies become more apparent, leading to more
> tickets and visibility.
>
> Maybe 2. and 3. will even use a different approach when persistence is
> implemented.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 28 июн. 2021 г. в 14:34, Atri Sharma :
>
> > Hello Again!
> >
> > I have been looking into the aforementioned and here are my follow up
> > thoughts:
> >
> > 1. Support persistence of Lucene indexes.
> > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of
> > moving partitions first)
> > 3. Figure out how to return scores from nodes and use them as sort
> > parameters on the coordinator node
> > (https://issues.apache.org/jira/browse/IGNITE-12291)
> >
> > Please let me know if this looks ok to make text queries functional?
> >
> > Atri
> >
> > On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov
> >  wrote:
> > >
> > > Hi.
> > >
> > > One of the biggest issues with text queries is a lack of support for
> > lucene
> > > indices persistence, which makes this functionality useless if a
> > > persistence is enabled.
> > >
> > > I would first take care of it.
> > >
> > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin :
> > >
> > > > Hi, Atri!
> > > >
> > > > You're right, Actually there is a lack of support for TextQueries.
> For
> > the
> > > > last ticket I'm doing I see some obvious issues with them (no page
> size
> > > > support, for example). I'm glad that somebody wants to maintain this
> > > > functionality. Thanks a lot!
> > > >
> > > > For the MergeSort algorithm there is already a patch for that [1].
> It's
> > > > currently on review. This patch introduces an abstract reducer for
> > > > CacheQueries with 2 implementations (unordered, merge-sort). Then
> > TextQuery
> > > > leverages on MergeSort to order results from multiple nodes by score.
> > This
> > > > patch also fixes the pageSize issue, I've mentioned before. Could you
> > > > please check if it fully matches your idea? Any issues or comments
> are
> > > > welcome.
> > > >
> > > > I've prepared this ticket, because I need the MergeSort algorithm for
> > the
> > > > new type of queries I'm implementing (IndexQuery, it should also
> > provide
> > > > ordered results over multiple nodes). Currently I'm not planning to
> go
> > > > further with TextQuery, so if you're going to support this it'll be a
> > great
> > > > contribution, I think.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703
> > > > [2] https://github.com/apache/ignite/pull/9081
> > > >
> > > >
> > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma 
> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I have been looking into our text queries support and see that it
> has
> > > > > limited community support.
> > > > >
> > > > > Therefore, I volunteer to be the maintainer of the module and work
> on
> > > > > enhancing it further.
> > > > >
> > > > > First goal would be to move to Lucene 8.x, then work on sorted
> reduce
> > > > > - merge across nodes. Fundamentally, this is doable since Lucene
> > ranks
> > > > > documents according to their score, and documents are returned in
> the
> > > > > order of their score. Since the scoring function is homogeneous,
> this
> > > > > means that across nodes, we can compare scores and merge sort.
> > > > >
> > > > > Please let me know if I can take this up.
> > > > >
> > > > > Atri
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >
>

Re: Apache Ignite Repo 403 Forbidden

[Petr Ivanov]

No mirror.

Also http://apache.org/dist/ignite/deb/dists/apache-ignite/InRelease 
 is redirected 
to Bintray which is closed.
The only way to get packages now — get source code for corresponding release 
version and build package manually (there is script for that inside).


Regards,
Petr Ivanov
Head of IT
IT & Development Solutions | GRIDGAIN SYSTEMS
+7 (911) 945-00-59

> On 23 Jul 2021, at 06:09, Mustafa Sunka  wrote:
> 
> Is there a mirror available? I have an app I am trying to rebuild and am 
> unable to proceed without this install.
> 
> On Thu, Jul 22, 2021 at 5:09 PM Ilya Kasnacheev  > wrote:
> Hello!
> 
> + Petr
> 
> Bintray went away, and I guess we have to re-publish these to jfrog.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> чт, 22 июл. 2021 г. в 20:39, Mustafa Sunka  >:
> 
> > This looks very similar to an issue from last year:
> >
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_Apache-2DIgnite-2Ddownloads-2Dare-2Dredirecting-2Dfrom-2Dhttps-2Dto-2Dhttp-2Dtd31428.html&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=Oy3AYmpDoeav2VjaPGM-zWUKLgzX42RUO1GsDRsdy44&e=
> >  
> > 
> >  
> >
> > When I attempt to run `apt-get update` on Ubuntu 18.04 with the Apache
> > Ignite repository in /etc/apt/sources.list, the update fails due to an
> > https to http redirect.  Here's the output from stdout:
> >
> >
> > WARNING: apt does not have a stable CLI interface. Use with caution in
> > scripts.
> >
> > Hit:1 
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
> >  
> > 
> >   bionic InRelease
> > Get:2 
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
> >  
> > 
> >   bionic-updates InRelease
> > [88.7 kB]
> > Get:4 
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
> >  
> > 
> >   bionic-backports InRelease
> > [74.6 kB]
> > Hit:5 
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__security.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=G6_zzVnIHvOkW-Y3DmaEkgIomjN4ruRMTMjrrzK1IFU&e=
> >  
> > 
> >   bionic-security InRelease
> > Hit:6 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__packages.microsoft.com_ubuntu_18.04_prod&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=k2leSDqZA4M0XmZUsXQjt1dz4ItJlOnVUu-jXnZdTes&e=
> >  
> > 
> >   bionic InRelease
> > Err:3 
> > https://urldefense.proofpoint.c

Re: Text Queries Support

[Andrey Mashenkov]

Hi Atri,

All the Jira tickets we have on the Full-text search (FTS) thing are
targeted to Ignite 2.

AFAIK, we want, but we have NOT committed to FTS support in Ignite 3, yet.
By the way, we are getting requests for this thing from the user side, and
definitely,
FTS would be a valuable feature for Ignite.

It will be great if the one wants to drive it, any help will be appreciated.


On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma  wrote:

> Hello,
>
> An update, please. I am working through persistence of Lucene index using
> Ignite Dictionary, and will be asking some questions soon.
>
> I had one doubt - - where does this change go? Ignite 3?
>
> Also, I know we want to build native support for text searches in Ignite 3.
> Is the work I am proposing here part of that, or will that be a separate
> effort?
>
> On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev, 
> wrote:
>
> > Hello!
> >
> > I think that number one is the most important one, then maybe it will see
> > more use and other deficiencies become more apparent, leading to more
> > tickets and visibility.
> >
> > Maybe 2. and 3. will even use a different approach when persistence is
> > implemented.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 28 июн. 2021 г. в 14:34, Atri Sharma :
> >
> > > Hello Again!
> > >
> > > I have been looking into the aforementioned and here are my follow up
> > > thoughts:
> > >
> > > 1. Support persistence of Lucene indexes.
> > > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of
> > > moving partitions first)
> > > 3. Figure out how to return scores from nodes and use them as sort
> > > parameters on the coordinator node
> > > (https://issues.apache.org/jira/browse/IGNITE-12291)
> > >
> > > Please let me know if this looks ok to make text queries functional?
> > >
> > > Atri
> > >
> > > On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov
> > >  wrote:
> > > >
> > > > Hi.
> > > >
> > > > One of the biggest issues with text queries is a lack of support for
> > > lucene
> > > > indices persistence, which makes this functionality useless if a
> > > > persistence is enabled.
> > > >
> > > > I would first take care of it.
> > > >
> > > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin  >:
> > > >
> > > > > Hi, Atri!
> > > > >
> > > > > You're right, Actually there is a lack of support for TextQueries.
> > For
> > > the
> > > > > last ticket I'm doing I see some obvious issues with them (no page
> > size
> > > > > support, for example). I'm glad that somebody wants to maintain
> this
> > > > > functionality. Thanks a lot!
> > > > >
> > > > > For the MergeSort algorithm there is already a patch for that [1].
> > It's
> > > > > currently on review. This patch introduces an abstract reducer for
> > > > > CacheQueries with 2 implementations (unordered, merge-sort). Then
> > > TextQuery
> > > > > leverages on MergeSort to order results from multiple nodes by
> score.
> > > This
> > > > > patch also fixes the pageSize issue, I've mentioned before. Could
> you
> > > > > please check if it fully matches your idea? Any issues or comments
> > are
> > > > > welcome.
> > > > >
> > > > > I've prepared this ticket, because I need the MergeSort algorithm
> for
> > > the
> > > > > new type of queries I'm implementing (IndexQuery, it should also
> > > provide
> > > > > ordered results over multiple nodes). Currently I'm not planning to
> > go
> > > > > further with TextQuery, so if you're going to support this it'll
> be a
> > > great
> > > > > contribution, I think.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703
> > > > > [2] https://github.com/apache/ignite/pull/9081
> > > > >
> > > > >
> > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma 
> > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I have been looking into our text queries support and see that it
> > has
> > > > > > limited community support.
> > > > > >
> > > > > > Therefore, I volunteer to be the maintainer of the module and
> work
> > on
> > > > > > enhancing it further.
> > > > > >
> > > > > > First goal would be to move to Lucene 8.x, then work on sorted
> > reduce
> > > > > > - merge across nodes. Fundamentally, this is doable since Lucene
> > > ranks
> > > > > > documents according to their score, and documents are returned in
> > the
> > > > > > order of their score. Since the scoring function is homogeneous,
> > this
> > > > > > means that across nodes, we can compare scores and merge sort.
> > > > > >
> > > > > > Please let me know if I can take this up.
> > > > > >
> > > > > > Atri
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Atri
> > > > > > Apache Concerted
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > >
> > > --
> > > Regards,
> > >
> > > Atri
> > > Apache Concerted
> > >
> >
>


-- 
Best regards,
Andrey V. Mashenkov

Re: Text Queries Support

[Atri Sharma]

I am actually happy to drive the feature for Ignite 3. FTS is very
important for me and I think Ignite users will benefit from it
greatly.

If it makes sense to be focusing on Ignite 3 for this capability, I am
eager to contribute there and lead the development.

Please share your thoughts.

On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov
 wrote:
>
> Hi Atri,
>
> All the Jira tickets we have on the Full-text search (FTS) thing are
> targeted to Ignite 2.
>
> AFAIK, we want, but we have NOT committed to FTS support in Ignite 3, yet.
> By the way, we are getting requests for this thing from the user side, and
> definitely,
> FTS would be a valuable feature for Ignite.
>
> It will be great if the one wants to drive it, any help will be appreciated.
>
>
> On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma  wrote:
>
> > Hello,
> >
> > An update, please. I am working through persistence of Lucene index using
> > Ignite Dictionary, and will be asking some questions soon.
> >
> > I had one doubt - - where does this change go? Ignite 3?
> >
> > Also, I know we want to build native support for text searches in Ignite 3.
> > Is the work I am proposing here part of that, or will that be a separate
> > effort?
> >
> > On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev, 
> > wrote:
> >
> > > Hello!
> > >
> > > I think that number one is the most important one, then maybe it will see
> > > more use and other deficiencies become more apparent, leading to more
> > > tickets and visibility.
> > >
> > > Maybe 2. and 3. will even use a different approach when persistence is
> > > implemented.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пн, 28 июн. 2021 г. в 14:34, Atri Sharma :
> > >
> > > > Hello Again!
> > > >
> > > > I have been looking into the aforementioned and here are my follow up
> > > > thoughts:
> > > >
> > > > 1. Support persistence of Lucene indexes.
> > > > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of
> > > > moving partitions first)
> > > > 3. Figure out how to return scores from nodes and use them as sort
> > > > parameters on the coordinator node
> > > > (https://issues.apache.org/jira/browse/IGNITE-12291)
> > > >
> > > > Please let me know if this looks ok to make text queries functional?
> > > >
> > > > Atri
> > > >
> > > > On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov
> > > >  wrote:
> > > > >
> > > > > Hi.
> > > > >
> > > > > One of the biggest issues with text queries is a lack of support for
> > > > lucene
> > > > > indices persistence, which makes this functionality useless if a
> > > > > persistence is enabled.
> > > > >
> > > > > I would first take care of it.
> > > > >
> > > > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin  > >:
> > > > >
> > > > > > Hi, Atri!
> > > > > >
> > > > > > You're right, Actually there is a lack of support for TextQueries.
> > > For
> > > > the
> > > > > > last ticket I'm doing I see some obvious issues with them (no page
> > > size
> > > > > > support, for example). I'm glad that somebody wants to maintain
> > this
> > > > > > functionality. Thanks a lot!
> > > > > >
> > > > > > For the MergeSort algorithm there is already a patch for that [1].
> > > It's
> > > > > > currently on review. This patch introduces an abstract reducer for
> > > > > > CacheQueries with 2 implementations (unordered, merge-sort). Then
> > > > TextQuery
> > > > > > leverages on MergeSort to order results from multiple nodes by
> > score.
> > > > This
> > > > > > patch also fixes the pageSize issue, I've mentioned before. Could
> > you
> > > > > > please check if it fully matches your idea? Any issues or comments
> > > are
> > > > > > welcome.
> > > > > >
> > > > > > I've prepared this ticket, because I need the MergeSort algorithm
> > for
> > > > the
> > > > > > new type of queries I'm implementing (IndexQuery, it should also
> > > > provide
> > > > > > ordered results over multiple nodes). Currently I'm not planning to
> > > go
> > > > > > further with TextQuery, so if you're going to support this it'll
> > be a
> > > > great
> > > > > > contribution, I think.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703
> > > > > > [2] https://github.com/apache/ignite/pull/9081
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma 
> > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I have been looking into our text queries support and see that it
> > > has
> > > > > > > limited community support.
> > > > > > >
> > > > > > > Therefore, I volunteer to be the maintainer of the module and
> > work
> > > on
> > > > > > > enhancing it further.
> > > > > > >
> > > > > > > First goal would be to move to Lucene 8.x, then work on sorted
> > > reduce
> > > > > > > - merge across nodes. Fundamentally, this is doable since Lucene
> > > > ranks
> > > > > > > documents according to their score, and documents are returned in
> > > the
> > > > > > > order of their score. Since the scoring function is homo

Re: Apache Ignite Repo 403 Forbidden

[Maxim Muzafarov]

Folks,

Artefacts are available here. Is this what we are looking for?

https://apache.jfrog.io/artifactory/ignite-deb/
https://apache.jfrog.io/artifactory/ignite-rpm/

On Fri, 23 Jul 2021 at 12:15, Petr Ivanov  wrote:
>
> No mirror.
>
> Also http://apache.org/dist/ignite/deb/dists/apache-ignite/InRelease 
>  is 
> redirected to Bintray which is closed.
> The only way to get packages now — get source code for corresponding release 
> version and build package manually (there is script for that inside).
>
>
> Regards,
> Petr Ivanov
> Head of IT
> IT & Development Solutions | GRIDGAIN SYSTEMS
> +7 (911) 945-00-59
>
> > On 23 Jul 2021, at 06:09, Mustafa Sunka  wrote:
> >
> > Is there a mirror available? I have an app I am trying to rebuild and am 
> > unable to proceed without this install.
> >
> > On Thu, Jul 22, 2021 at 5:09 PM Ilya Kasnacheev  > > wrote:
> > Hello!
> >
> > + Petr
> >
> > Bintray went away, and I guess we have to re-publish these to jfrog.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 22 июл. 2021 г. в 20:39, Mustafa Sunka  > >:
> >
> > > This looks very similar to an issue from last year:
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_Apache-2DIgnite-2Ddownloads-2Dare-2Dredirecting-2Dfrom-2Dhttps-2Dto-2Dhttp-2Dtd31428.html&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=Oy3AYmpDoeav2VjaPGM-zWUKLgzX42RUO1GsDRsdy44&e=
> > >  
> > > 
> > >
> > > When I attempt to run `apt-get update` on Ubuntu 18.04 with the Apache
> > > Ignite repository in /etc/apt/sources.list, the update fails due to an
> > > https to http redirect.  Here's the output from stdout:
> > >
> > >
> > > WARNING: apt does not have a stable CLI interface. Use with caution in
> > > scripts.
> > >
> > > Hit:1 
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
> > >  
> > > 
> > >   bionic InRelease
> > > Get:2 
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
> > >  
> > > 
> > >   bionic-updates InRelease
> > > [88.7 kB]
> > > Get:4 
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
> > >  
> > > 
> > >   bionic-backports InRelease
> > > [74.6 kB]
> > > Hit:5 
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__security.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=G6_zzVnIHvOkW-Y3DmaEkgIomjN4ruRMTMjrrzK1IFU&e=
> > >  
> > > 
> > >   bionic-security InRelease
> > > Hit:6 
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__packages.microsoft.com_ubuntu_18.04_prod&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=k2leSDqZA4M0XmZUsXQjt1dz4ItJlOnVUu-jXnZdTes&e=

[VOTE] Release pyignite 0.5.1-rc0

[Ivan Daschinsky]

Dear Igniters!

Release candidate binaries for subj are uploaded and ready for vote
You can find them here:
https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.1-rc0

If you follow the link above, you will find source packages (*.tar.gz and
*.zip)
and binary packages (wheels) for windows (amd64) and linux (x86_64)
for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
Code signing keys can be found here --
https://downloads.apache.org/ignite/KEYS
Here you can find instructions how to verify packages
https://www.apache.org/info/verification.html

You can install binary package for specific version of python using pip
For example do this on linux for python 3.8
>> pip install pyignite-0.5.1-cp38-cp38-manylinux1_x86_64.whl

You can build and install package from source using this command:
>> pip install pyignite-0.5.1.tar.gz
You can build wheel on your platform using this command:
>> pip wheel --no-deps pyignite-0.5.1.tar.gz

For building C module, you should have python headers and C compiler
installed.
(i.e. for ubuntu sudo apt install build-essential python3-dev)
In Mac OS X xcode-tools and python from homebrew are the best option.

In order to check whether C module works, use following:
>> from pyignite import _cutils
>> print(_cutils.hashcode('test'))
>> 3556498

You can find documentation here:
https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0

You can find examples here (to check them, you should start ignite locally):
https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0/examples.html
Also, examples can be found in source archive in examples subfolder.
docker-compose.yml is supplied in order to start ignite quickly. (Use
`docker-compose up -d` to start 3 nodes cluster and `docker-compose
down` to shut down it)

Release notes:
https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=c6cbd419684cd4a97485707471bac84957b42891;hb=b48dd5dec37064b458031358c394789d15a756fc

Git release tag was created:
https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.1.rc0

The vote is formal, see voting guidelines
https://www.apache.org/foundation/voting.html

+1 - to accept pyignite-0.5.1-rc0
0 - don't care either way
-1 - DO NOT accept pyignite-0.5.1-rc0

Re: [VOTE] Release pyignite 0.5.1-rc0

[Ivan Daschinsky]

The voting finishes at 07/27/2021 12:00 UTC

пт, 23 июл. 2021 г. в 13:49, Ivan Daschinsky :

> Dear Igniters!
>
> Release candidate binaries for subj are uploaded and ready for vote
> You can find them here:
> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.1-rc0
>
> If you follow the link above, you will find source packages (*.tar.gz and
> *.zip)
> and binary packages (wheels) for windows (amd64) and linux (x86_64)
> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> Code signing keys can be found here --
> https://downloads.apache.org/ignite/KEYS
> Here you can find instructions how to verify packages
> https://www.apache.org/info/verification.html
>
> You can install binary package for specific version of python using pip
> For example do this on linux for python 3.8
> >> pip install pyignite-0.5.1-cp38-cp38-manylinux1_x86_64.whl
>
> You can build and install package from source using this command:
> >> pip install pyignite-0.5.1.tar.gz
> You can build wheel on your platform using this command:
> >> pip wheel --no-deps pyignite-0.5.1.tar.gz
>
> For building C module, you should have python headers and C compiler
> installed.
> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> In Mac OS X xcode-tools and python from homebrew are the best option.
>
> In order to check whether C module works, use following:
> >> from pyignite import _cutils
> >> print(_cutils.hashcode('test'))
> >> 3556498
>
> You can find documentation here:
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
>
> You can find examples here (to check them, you should start ignite
> locally):
>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0/examples.html
> Also, examples can be found in source archive in examples subfolder.
> docker-compose.yml is supplied in order to start ignite quickly. (Use
> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> down` to shut down it)
>
> Release notes:
>
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=c6cbd419684cd4a97485707471bac84957b42891;hb=b48dd5dec37064b458031358c394789d15a756fc
>
> Git release tag was created:
>
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.1.rc0
>
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
>
> +1 - to accept pyignite-0.5.1-rc0
> 0 - don't care either way
> -1 - DO NOT accept pyignite-0.5.1-rc0
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: Text Queries Support

[Andrey Mashenkov]

Atri,

First of all, I'd recommend going through the Ignite ticket to gather
information about the current implementation issues and users' wants.
Then look at a code to get a complete understanding of how things work now,
which may help in future decisions.

As we use the outdated Lucene version, some things may be irrelevant for
the latest Lucene version.
So, you will need expertise in the internals of modern Lucene version to
understand what capabilities, guarantees, and limitations Lucene has and
could bring to the Ignite.
The expertise could be got from the Lucene project code or Lucene project
dev-list.


As for now, the potential capabilities are not clear to me.
At first glance, I see the next topics that must be covered at first:

General questions
* How Lucene index can be split among the nodes?
* If we'll have a single index for all partitions on the particular node,
then how index records will be aware of partitioning?
This is important to filter out backup records from the results to avoid
duplicates.
* How results from several nodes can be merged on the Reduce stage?
* Does Lucene supports smth like JOIN operation or others that may require
data from another partition or index?
If so, then it likes to multistep query with merging results on
intermediate stages and requires detailed investigation and design.
It is ok if Ignite will have some limitations here, but we would like to
know about them at the early stage.
* How effectively map Lucene files to the page memory? Is it even possible?
Otherwise, how to deal with potential OOM on large queries and memory
capacity planning?

Persistence.
* How and what consistency guarantees could we have/expect?
Seems, we may not be able to write physical records for Lucene index to our
WAL. What can we do with this?

Transactions.
* Will we support transactions?
* Should Lucene be aware of Transaction and track mvcc (or whatever)
versions for the records?
* What will be consistency guarantees?

UX
* How to add FullText search queries syntax into Calcite?
* AFAIK, the Lucene index has many properties for tuning. How will the user
configure the index?
* How and where to store the settings? What are cluster-wide and what a
local to the particular node?
* Will be all the settings immutable? Can be they changed on-fly? after
node/grid restart?
* Any limitations on query syntax?

SQL
* Will we support FullText search in SQL?
* How to integrate Lucene index into Calcite? What is the cost model?
Splitting rules? Traits?
* What about consistency with DDL operations, e.g. column rename?
Ignite indices will operate column ID, so rename operation will not affect
the index.


With all of this, you can go with the IEP (or even some short summary) and
further POC and implementation.
That's a big deal, so let's discuss what could be done here.

On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma  wrote:

> I am actually happy to drive the feature for Ignite 3. FTS is very
> important for me and I think Ignite users will benefit from it
> greatly.
>
> If it makes sense to be focusing on Ignite 3 for this capability, I am
> eager to contribute there and lead the development.
>
> Please share your thoughts.
>
> On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov
>  wrote:
> >
> > Hi Atri,
> >
> > All the Jira tickets we have on the Full-text search (FTS) thing are
> > targeted to Ignite 2.
> >
> > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3,
> yet.
> > By the way, we are getting requests for this thing from the user side,
> and
> > definitely,
> > FTS would be a valuable feature for Ignite.
> >
> > It will be great if the one wants to drive it, any help will be
> appreciated.
> >
> >
> > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma  wrote:
> >
> > > Hello,
> > >
> > > An update, please. I am working through persistence of Lucene index
> using
> > > Ignite Dictionary, and will be asking some questions soon.
> > >
> > > I had one doubt - - where does this change go? Ignite 3?
> > >
> > > Also, I know we want to build native support for text searches in
> Ignite 3.
> > > Is the work I am proposing here part of that, or will that be a
> separate
> > > effort?
> > >
> > > On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev,  >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > I think that number one is the most important one, then maybe it
> will see
> > > > more use and other deficiencies become more apparent, leading to more
> > > > tickets and visibility.
> > > >
> > > > Maybe 2. and 3. will even use a different approach when persistence
> is
> > > > implemented.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пн, 28 июн. 2021 г. в 14:34, Atri Sharma :
> > > >
> > > > > Hello Again!
> > > > >
> > > > > I have been looking into the aforementioned and here are my follow
> up
> > > > > thoughts:
> > > > >
> > > > > 1. Support persistence of Lucene indexes.
> > > > > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs
> fixing of

Re: Apache Ignite Repo 403 Forbidden

[Petr Ivanov]

Yes, it is.

Seems that Apache kept the storage, or JFrog helped somehow.
Some one from committer side should update http://apache.org/dist/ignite/deb 
link to redirect correctly.


Not sure though there is still a way to deliver new packages there.



> On 23 Jul 2021, at 13:19, Maxim Muzafarov  wrote:
> 
> Folks,
> 
> Artefacts are available here. Is this what we are looking for?
> 
> https://apache.jfrog.io/artifactory/ignite-deb/
> https://apache.jfrog.io/artifactory/ignite-rpm/
> 
> On Fri, 23 Jul 2021 at 12:15, Petr Ivanov  wrote:
>> 
>> No mirror.
>> 
>> Also http://apache.org/dist/ignite/deb/dists/apache-ignite/InRelease 
>>  is 
>> redirected to Bintray which is closed.
>> The only way to get packages now — get source code for corresponding release 
>> version and build package manually (there is script for that inside).
>> 
>> 
>> Regards,
>> Petr Ivanov
>> Head of IT
>> IT & Development Solutions | GRIDGAIN SYSTEMS
>> +7 (911) 945-00-59
>> 
>>> On 23 Jul 2021, at 06:09, Mustafa Sunka  wrote:
>>> 
>>> Is there a mirror available? I have an app I am trying to rebuild and am 
>>> unable to proceed without this install.
>>> 
>>> On Thu, Jul 22, 2021 at 5:09 PM Ilya Kasnacheev >> > wrote:
>>> Hello!
>>> 
>>> + Petr
>>> 
>>> Bintray went away, and I guess we have to re-publish these to jfrog.
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> чт, 22 июл. 2021 г. в 20:39, Mustafa Sunka >> >:
>>> 
 This looks very similar to an issue from last year:
 
 https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_Apache-2DIgnite-2Ddownloads-2Dare-2Dredirecting-2Dfrom-2Dhttps-2Dto-2Dhttp-2Dtd31428.html&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=Oy3AYmpDoeav2VjaPGM-zWUKLgzX42RUO1GsDRsdy44&e=
  
 
 
 When I attempt to run `apt-get update` on Ubuntu 18.04 with the Apache
 Ignite repository in /etc/apt/sources.list, the update fails due to an
 https to http redirect.  Here's the output from stdout:
 
 
 WARNING: apt does not have a stable CLI interface. Use with caution in
 scripts.
 
 Hit:1 
 https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
  
 
   bionic InRelease
 Get:2 
 https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
  
 
   bionic-updates InRelease
 [88.7 kB]
 Get:4 
 https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
  
 
   bionic-backports InRelease
 [74.6 kB]
 Hit:5 
 https://urldefense.proofpoint.com/v2/url?u=http-3A__security.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=G6_zzVnIHvOkW-Y3DmaEkgIomjN4ruRMTMjrrzK1IFU&e=
  
 

Re: Apache Ignite Repo 403 Forbidden

[Maxim Muzafarov]

Petr,

Yes, everything has been migrated to JFrog and is available with PMC
credentials.
https://apache.jfrog.io/ui/login/

On Fri, 23 Jul 2021 at 13:58, Petr Ivanov  wrote:
>
> Yes, it is.
>
> Seems that Apache kept the storage, or JFrog helped somehow.
> Some one from committer side should update http://apache.org/dist/ignite/deb 
> link to redirect correctly.
>
>
> Not sure though there is still a way to deliver new packages there.
>
>
>
> > On 23 Jul 2021, at 13:19, Maxim Muzafarov  wrote:
> >
> > Folks,
> >
> > Artefacts are available here. Is this what we are looking for?
> >
> > https://apache.jfrog.io/artifactory/ignite-deb/
> > https://apache.jfrog.io/artifactory/ignite-rpm/
> >
> > On Fri, 23 Jul 2021 at 12:15, Petr Ivanov  wrote:
> >>
> >> No mirror.
> >>
> >> Also http://apache.org/dist/ignite/deb/dists/apache-ignite/InRelease 
> >>  is 
> >> redirected to Bintray which is closed.
> >> The only way to get packages now — get source code for corresponding 
> >> release version and build package manually (there is script for that 
> >> inside).
> >>
> >>
> >> Regards,
> >> Petr Ivanov
> >> Head of IT
> >> IT & Development Solutions | GRIDGAIN SYSTEMS
> >> +7 (911) 945-00-59
> >>
> >>> On 23 Jul 2021, at 06:09, Mustafa Sunka  wrote:
> >>>
> >>> Is there a mirror available? I have an app I am trying to rebuild and am 
> >>> unable to proceed without this install.
> >>>
> >>> On Thu, Jul 22, 2021 at 5:09 PM Ilya Kasnacheev 
> >>> mailto:ilya.kasnach...@gmail.com>> wrote:
> >>> Hello!
> >>>
> >>> + Petr
> >>>
> >>> Bintray went away, and I guess we have to re-publish these to jfrog.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> чт, 22 июл. 2021 г. в 20:39, Mustafa Sunka  >>> >:
> >>>
>  This looks very similar to an issue from last year:
> 
>  https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_Apache-2DIgnite-2Ddownloads-2Dare-2Dredirecting-2Dfrom-2Dhttps-2Dto-2Dhttp-2Dtd31428.html&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=Oy3AYmpDoeav2VjaPGM-zWUKLgzX42RUO1GsDRsdy44&e=
>   
>  
> 
>  When I attempt to run `apt-get update` on Ubuntu 18.04 with the Apache
>  Ignite repository in /etc/apt/sources.list, the update fails due to an
>  https to http redirect.  Here's the output from stdout:
> 
> 
>  WARNING: apt does not have a stable CLI interface. Use with caution in
>  scripts.
> 
>  Hit:1 
>  https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
>   
>  
>    bionic InRelease
>  Get:2 
>  https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
>   
>  
>    bionic-updates InRelease
>  [88.7 kB]
>  Get:4 
>  https://urldefense.proofpoint.com/v2/url?u=http-3A__azure.archive.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pubokT-e6V1EU_ZQrEH8b70&s=2Ov6DP1Bap0srqqc3X-UtLngbcH27ecZ8-mULSqkiTI&e=
>   
>  
>    bionic-backports InRelease
>  [74.6 kB]
>  Hit:5 
>  https://urldefense.proofpoint.com/v2/url?u=http-3A__security.ubuntu.com_ubuntu&d=DwIFaQ&c=f7K5vTjsy0ldRUVHx_C48Q&r=F_pxrwS6ECbSFnH54hVvzyOo5Yb0IvCiswhvjRVXOCs&m=XQgK5pAVCrRiCWaNfbv0pub

Re: [VOTE] Release pyignite 0.5.1-rc0

[Ivan Daschinsky]

+1 From me
1. Checked binary packages, c module and examples on windows 10 amd64 for
pythons 3.6, 3.7, 3.8, 3.9
2. Checked binary packages, c module and examples on ubuntu 20.04 amd64 for
pythons 3.6, 3.7, 3.8, 3.9
3. Checked source installation and building binary packages on ubuntu 20.04
amd 64 for pythons 3.6, 3.7, 3.8, 3.9
4. Checked documentation on
https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
5. Checked sha512 checksums and gpg signatures (signed by Igor Sapego (CODE
SIGNING KEY)  5C10 A072 2D94 7727 923C  98B5 AF35 DBD9
58FE 8DC5)
key is inside https://downloads.apache.org/ignite/KEYS)

пт, 23 июл. 2021 г. в 13:52, Ivan Daschinsky :

> The voting finishes at 07/27/2021 12:00 UTC
>
> пт, 23 июл. 2021 г. в 13:49, Ivan Daschinsky :
>
>> Dear Igniters!
>>
>> Release candidate binaries for subj are uploaded and ready for vote
>> You can find them here:
>> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.1-rc0
>>
>> If you follow the link above, you will find source packages (*.tar.gz and
>> *.zip)
>> and binary packages (wheels) for windows (amd64) and linux (x86_64)
>> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
>> Code signing keys can be found here --
>> https://downloads.apache.org/ignite/KEYS
>> Here you can find instructions how to verify packages
>> https://www.apache.org/info/verification.html
>>
>> You can install binary package for specific version of python using pip
>> For example do this on linux for python 3.8
>> >> pip install pyignite-0.5.1-cp38-cp38-manylinux1_x86_64.whl
>>
>> You can build and install package from source using this command:
>> >> pip install pyignite-0.5.1.tar.gz
>> You can build wheel on your platform using this command:
>> >> pip wheel --no-deps pyignite-0.5.1.tar.gz
>>
>> For building C module, you should have python headers and C compiler
>> installed.
>> (i.e. for ubuntu sudo apt install build-essential python3-dev)
>> In Mac OS X xcode-tools and python from homebrew are the best option.
>>
>> In order to check whether C module works, use following:
>> >> from pyignite import _cutils
>> >> print(_cutils.hashcode('test'))
>> >> 3556498
>>
>> You can find documentation here:
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
>>
>> You can find examples here (to check them, you should start ignite
>> locally):
>>
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0/examples.html
>> Also, examples can be found in source archive in examples subfolder.
>> docker-compose.yml is supplied in order to start ignite quickly. (Use
>> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
>> down` to shut down it)
>>
>> Release notes:
>>
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=c6cbd419684cd4a97485707471bac84957b42891;hb=b48dd5dec37064b458031358c394789d15a756fc
>>
>> Git release tag was created:
>>
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.1.rc0
>>
>> The vote is formal, see voting guidelines
>> https://www.apache.org/foundation/voting.html
>>
>> +1 - to accept pyignite-0.5.1-rc0
>> 0 - don't care either way
>> -1 - DO NOT accept pyignite-0.5.1-rc0
>>
>>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: Apache Ignite 3 Alpha 2 webinar follow up questions

[Courtney Robinson]

Andrey,
Thanks for the response - see my comments inline.


> I've gone through the questions and have no the whole picture of your use
> case.

Would you please clarify how you exactly use the Ignite? what are the
> integration points?
>

I'll try to clarify - we have a low/no code platform. A user designs a
model for their application and we map this model to Ignite tables and
other data sources. The model I'll describe is what we're building now and
expected to be in alpha some time in Q4 21. Our current production
architecture is different and isn't as generic, it is heavily tied to
Ignite and we've redesigned to get some flexibility where Ignite doesn't
provide what we want. Things like window functions and other SQL-99 limits.

In the next gen version we're working on you can create a model for a
Tweet(content, to) and we will create an Ignite table with content and to
columns using the type the user selects. This is the simplest case.
We are adding generic support for sources and sinks and using Calcite as a
data virtualisation layer. Ignite is one of the available source/sinks.

When a user creates a model for Tweet, we also allow them to specify how
they want to index the data. We have a copy of the calcite Elasticsearch
adapter modified for Solr.

When a source is queried (Ignite or any other that we support), we generate
SQL that Calcite executes. Calcite will push down the generated queries to
Solr and Solr produces a list of IDs (in case of Ignite) and we do a
multi-get from Ignite to produce the actual results.

Obviously there's a lot more to this but that should give you a general
idea.

and maybe share some experience with using Ignite SPIs?
>
Our evolution with Ignite started from the key value + compute APIs. We
used the SPIs then but have since moved to using only the Ignite SQL API
(we gave up transactions for this).

We originally used the indexing SPI to keep our own lucene index of data in
a cache. We did not use the Ignite FTS as it is very limited compared to
what we allow customers to do. If I remember correctly, we were using an
affinity compute job to send queries to the right Ignite node and
then doing a multi-get to pull the data from caches.
I think we used one or two other SPIs and we found them very useful to be
able to extend and customise Ignite without having to fork/change upstream
classes. We only stopped using them because we eventually concluded that
using the SQL only API was better for numerous reasons.


> We'll keep the information in mind while developing the Ignite,
> because this may help us to make a better product.
>
> By the way, I'll try to answer the questions.
>
> >   1. Schema change - does that include the ability to change the types of
> >   fields/columns?
> Yes, we plan to support transparent conversion to a wider type on-fly (e.g.
> 'int' to 'long').
> This is a major point of our Live-schema concept.
> In fact, there is no need to convert data on all the nodes in a synchronous
> way as old SQL databases do (if one supports though),
> we are going to support multiple schema versions and convert data on-demand
> on a per-row basis to the latest version,
> then write-back the row.
>

I can understand. The auto conversion to wider type makes sense.

>
> More complex things like 'String' -> 'int' are out of scope for now because
> it requires the execution of a user code on the critical path.
>

I would argue though that executing user code on the critical path
shouldn't be a blocker for custom conversions. I feel if a user is making
an advance enough integration to provide custom conversions they would be
aware that it impacts the system as a whole.

The limitation here is column MUST NOT be indexed, because an index over
> the data of different kinds is impossible.
>
 Understood - I'd make the case that indexing should be pluggable. I would
love for us to be able to take indexing away from Ignite in our impl. - I
think in Calcite, the Postgres adapter does this by having a table whose
type is "Index". The implementor would be left with the freedom to choose
how that table answers index lookups. From Ignite's perspective it wouldn't
care so long as the interface's contract is met, I could use an index that
does a lucene, ES, Solr or Redis lookup and the end result would be the
same but as the implementor I'm choosing the tradeoff I want to meet the
organisation's goals.


>
>  >  2. Will the new guaranteed consistency between APIs also mean SQL will
>  >  gain transaction support?
> Yes, we plan to have Transactional SQL.
> DDL will be non-transactional though, and I wonder if the one supports
> this.
>
I'm not sure I know of any thing that supports transactional DDL so don't
think this is an issue but I would say that a DDL statement in a
transaction shouldn't fail the transaction. I believe in Ignite 2 there is
a flag  to turn this on or off, we should definitely keep this. In our
case, it's an issue with the nature of the platform we provide, at
d

Re: [DISCUSS] Confuse default inspections.

[Alex Plehanov]

-0 for both proposals

I agree with Ivan Daschinsky, we should not mix both approaches (interfaces
with "public" keyword and without "public" keyword), but I see no big
difference between "require public keyword in interfaces" rule and
"prohibit public keyword in interfaces" rule. To switch from one rule to
another we should change thousands of lines of code, this can produce a lot
of merge conflicts (for tickets currently in development). But what for?
What are the benefits? After the switch, some "old style" code will
certainly leak to the codebase, so also a new automatic checkstyle plugin
rule will be required to avoid "new style" violations.

The same for one line statements.

> do we have an automatic inspection for it? AFAIK we don't
We have the Apache Ignite code-style plugin for IDEA, where both of these
rules are checked.

> There were a few bugs that this rule caused.
Personally, I don't remember any, but I guess these bugs were caused by
wrong indent, but not the "one line statement" rule itself (as far as I
remember, once I saw such a wrong ident after one line statement in our
code). Perhaps it's better to add an automatic indent check if it's
possible.

ср, 21 июл. 2021 г. в 11:11, ткаленко кирилл :

> +1
>
> 20.07.2021, 23:06, "Юрий" :
> > I totally agree.
> > Let's get rid of them.
> >
> > вт, 20 июл. 2021 г. в 18:11, Konstantin Orlov :
> >
> >>  + for both
> >>
> >>  --
> >>  Regards,
> >>  Konstantin Orlov
> >>
> >>  > On 20 Jul 2021, at 16:32, Pavel Tupitsyn 
> wrote:
> >>  >
> >>  > Agree with for both points
> >>  >
> >>  > On Tue, Jul 20, 2021 at 3:14 PM Alexander Polovtcev <
> >>  alexpolovt...@gmail.com>
> >>  > wrote:
> >>  >
> >>  >> this is a very welcome change for me
> >>  >>
> >>  >> On Tue, Jul 20, 2021 at 10:13 AM Ivan Pavlukhin <
> vololo...@gmail.com>
> >>  >> wrote:
> >>  >>
> >>  >>> + for both points.
> >>  >>>
> >>  >>> 2021-07-20 9:56 GMT+03:00, Ivan Daschinsky :
> >>   Hi!
> >>  
> >>   Firstly, lets talk about interfaces.
> >>  
> >>   1. First of all, do we have an automatic inspection for it? AFAIK
> we
> >>  >>> don't
> >>   2. I am for consistency. At least for production code. Nothing
> worse
> >>  is
> >>   when someone mixes both approaches.
> >>  
> >>   About a prohibition of curly brackets around one line -- I am
> strongly
> >>   against this rule, it should be removed. There were a few bugs
> that
> >>  >> this
> >>   rule caused.
> >>  
> >>   вт, 20 июл. 2021 г. в 09:23, Zhenya Stanilovsky
> >>  >>>  >>  > :
> >>  
> >>  >
> >>  > Igniters, i understand that this is very long and fundamental
> story.
> >>  >> but
> >>  > … still want to rise up this discussion, we have 2 very strange
> >>  > inspections:
> >>  > * «public» modifier in interface methods.
> >>  > * Illegal ‘{}’ for one line statement. — i found it harmful.
> >>  > I don`t want to link additional discussion about pos. 2 i think
> that
> >>  >>> harm
> >>  > is obvious.
> >>  > I suggest to get rid of them.
> >>  >
> >>  > what do you think ?
> >>  >
> >>  >
> >>  >
> >>  
> >>  
> >>  
> >>   --
> >>   Sincerely yours, Ivan Daschinskiy
> >>  
> >>  >>>
> >>  >>>
> >>  >>> --
> >>  >>>
> >>  >>> Best regards,
> >>  >>> Ivan Pavlukhin
> >>  >>>
> >>  >>
> >>  >>
> >>  >> --
> >>  >> With regards,
> >>  >> Aleksandr Polovtcev
> >>  >>
> >
> > --
> > Живи с улыбкой! :D
>

Re: Text Queries Support

[Valentin Kulichenko]

In my experience, one of the biggest usability issues with the current
support of text queries is that they are completely decoupled from SQL.
I.e. you can either execute a SQL query OR a text query. Modern databases,
on the other hand, typically allow creating text-based indexes within
regular tables and then using those indexes within regular SQL queries.
Here is an example from Oracle:
https://docs.oracle.com/cd/B10501_01/text.920/a96517/cdefault.htm

I believe this is something we can look into in the scope of Ignite 3.
Andrey, does Calcite have any support for this? What's your view on this?

-Val

On Fri, Jul 23, 2021 at 3:56 AM Andrey Mashenkov 
wrote:

> Atri,
>
> First of all, I'd recommend going through the Ignite ticket to gather
> information about the current implementation issues and users' wants.
> Then look at a code to get a complete understanding of how things work now,
> which may help in future decisions.
>
> As we use the outdated Lucene version, some things may be irrelevant for
> the latest Lucene version.
> So, you will need expertise in the internals of modern Lucene version to
> understand what capabilities, guarantees, and limitations Lucene has and
> could bring to the Ignite.
> The expertise could be got from the Lucene project code or Lucene project
> dev-list.
>
>
> As for now, the potential capabilities are not clear to me.
> At first glance, I see the next topics that must be covered at first:
>
> General questions
> * How Lucene index can be split among the nodes?
> * If we'll have a single index for all partitions on the particular node,
> then how index records will be aware of partitioning?
> This is important to filter out backup records from the results to avoid
> duplicates.
> * How results from several nodes can be merged on the Reduce stage?
> * Does Lucene supports smth like JOIN operation or others that may require
> data from another partition or index?
> If so, then it likes to multistep query with merging results on
> intermediate stages and requires detailed investigation and design.
> It is ok if Ignite will have some limitations here, but we would like to
> know about them at the early stage.
> * How effectively map Lucene files to the page memory? Is it even possible?
> Otherwise, how to deal with potential OOM on large queries and memory
> capacity planning?
>
> Persistence.
> * How and what consistency guarantees could we have/expect?
> Seems, we may not be able to write physical records for Lucene index to our
> WAL. What can we do with this?
>
> Transactions.
> * Will we support transactions?
> * Should Lucene be aware of Transaction and track mvcc (or whatever)
> versions for the records?
> * What will be consistency guarantees?
>
> UX
> * How to add FullText search queries syntax into Calcite?
> * AFAIK, the Lucene index has many properties for tuning. How will the user
> configure the index?
> * How and where to store the settings? What are cluster-wide and what a
> local to the particular node?
> * Will be all the settings immutable? Can be they changed on-fly? after
> node/grid restart?
> * Any limitations on query syntax?
>
> SQL
> * Will we support FullText search in SQL?
> * How to integrate Lucene index into Calcite? What is the cost model?
> Splitting rules? Traits?
> * What about consistency with DDL operations, e.g. column rename?
> Ignite indices will operate column ID, so rename operation will not affect
> the index.
>
>
> With all of this, you can go with the IEP (or even some short summary) and
> further POC and implementation.
> That's a big deal, so let's discuss what could be done here.
>
> On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma  wrote:
>
> > I am actually happy to drive the feature for Ignite 3. FTS is very
> > important for me and I think Ignite users will benefit from it
> > greatly.
> >
> > If it makes sense to be focusing on Ignite 3 for this capability, I am
> > eager to contribute there and lead the development.
> >
> > Please share your thoughts.
> >
> > On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov
> >  wrote:
> > >
> > > Hi Atri,
> > >
> > > All the Jira tickets we have on the Full-text search (FTS) thing are
> > > targeted to Ignite 2.
> > >
> > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3,
> > yet.
> > > By the way, we are getting requests for this thing from the user side,
> > and
> > > definitely,
> > > FTS would be a valuable feature for Ignite.
> > >
> > > It will be great if the one wants to drive it, any help will be
> > appreciated.
> > >
> > >
> > > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma  wrote:
> > >
> > > > Hello,
> > > >
> > > > An update, please. I am working through persistence of Lucene index
> > using
> > > > Ignite Dictionary, and will be asking some questions soon.
> > > >
> > > > I had one doubt - - where does this change go? Ignite 3?
> > > >
> > > > Also, I know we want to build native support for text searches in
> > Ignite 3.
> > > > Is the work I am proposing

Re: Text Queries Support

[Atri Sharma]

The standard ways to deal with text based searches in SQL are the
CONTAINS operator, the LIKE operator or specific functions
(REGEXP_MATCHES, for eg). I do not think any of these are supported by
Calcite at the moment.

On Fri, Jul 23, 2021 at 11:20 PM Valentin Kulichenko
 wrote:
>
> In my experience, one of the biggest usability issues with the current
> support of text queries is that they are completely decoupled from SQL.
> I.e. you can either execute a SQL query OR a text query. Modern databases,
> on the other hand, typically allow creating text-based indexes within
> regular tables and then using those indexes within regular SQL queries.
> Here is an example from Oracle:
> https://docs.oracle.com/cd/B10501_01/text.920/a96517/cdefault.htm
>
> I believe this is something we can look into in the scope of Ignite 3.
> Andrey, does Calcite have any support for this? What's your view on this?
>
> -Val
>
> On Fri, Jul 23, 2021 at 3:56 AM Andrey Mashenkov 
> wrote:
>
> > Atri,
> >
> > First of all, I'd recommend going through the Ignite ticket to gather
> > information about the current implementation issues and users' wants.
> > Then look at a code to get a complete understanding of how things work now,
> > which may help in future decisions.
> >
> > As we use the outdated Lucene version, some things may be irrelevant for
> > the latest Lucene version.
> > So, you will need expertise in the internals of modern Lucene version to
> > understand what capabilities, guarantees, and limitations Lucene has and
> > could bring to the Ignite.
> > The expertise could be got from the Lucene project code or Lucene project
> > dev-list.
> >
> >
> > As for now, the potential capabilities are not clear to me.
> > At first glance, I see the next topics that must be covered at first:
> >
> > General questions
> > * How Lucene index can be split among the nodes?
> > * If we'll have a single index for all partitions on the particular node,
> > then how index records will be aware of partitioning?
> > This is important to filter out backup records from the results to avoid
> > duplicates.
> > * How results from several nodes can be merged on the Reduce stage?
> > * Does Lucene supports smth like JOIN operation or others that may require
> > data from another partition or index?
> > If so, then it likes to multistep query with merging results on
> > intermediate stages and requires detailed investigation and design.
> > It is ok if Ignite will have some limitations here, but we would like to
> > know about them at the early stage.
> > * How effectively map Lucene files to the page memory? Is it even possible?
> > Otherwise, how to deal with potential OOM on large queries and memory
> > capacity planning?
> >
> > Persistence.
> > * How and what consistency guarantees could we have/expect?
> > Seems, we may not be able to write physical records for Lucene index to our
> > WAL. What can we do with this?
> >
> > Transactions.
> > * Will we support transactions?
> > * Should Lucene be aware of Transaction and track mvcc (or whatever)
> > versions for the records?
> > * What will be consistency guarantees?
> >
> > UX
> > * How to add FullText search queries syntax into Calcite?
> > * AFAIK, the Lucene index has many properties for tuning. How will the user
> > configure the index?
> > * How and where to store the settings? What are cluster-wide and what a
> > local to the particular node?
> > * Will be all the settings immutable? Can be they changed on-fly? after
> > node/grid restart?
> > * Any limitations on query syntax?
> >
> > SQL
> > * Will we support FullText search in SQL?
> > * How to integrate Lucene index into Calcite? What is the cost model?
> > Splitting rules? Traits?
> > * What about consistency with DDL operations, e.g. column rename?
> > Ignite indices will operate column ID, so rename operation will not affect
> > the index.
> >
> >
> > With all of this, you can go with the IEP (or even some short summary) and
> > further POC and implementation.
> > That's a big deal, so let's discuss what could be done here.
> >
> > On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma  wrote:
> >
> > > I am actually happy to drive the feature for Ignite 3. FTS is very
> > > important for me and I think Ignite users will benefit from it
> > > greatly.
> > >
> > > If it makes sense to be focusing on Ignite 3 for this capability, I am
> > > eager to contribute there and lead the development.
> > >
> > > Please share your thoughts.
> > >
> > > On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov
> > >  wrote:
> > > >
> > > > Hi Atri,
> > > >
> > > > All the Jira tickets we have on the Full-text search (FTS) thing are
> > > > targeted to Ignite 2.
> > > >
> > > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3,
> > > yet.
> > > > By the way, we are getting requests for this thing from the user side,
> > > and
> > > > definitely,
> > > > FTS would be a valuable feature for Ignite.
> > > >
> > > > It will be great if the one want

Re: Text Queries Support

[Atri Sharma]

Thanks Andrey.

I have collected answers or proposals to many of these questions and
would like to start a wiki page covering what we can do for Ignite 3.

Does that sound good, please?

On Fri, Jul 23, 2021 at 4:26 PM Andrey Mashenkov
 wrote:
>
> Atri,
>
> First of all, I'd recommend going through the Ignite ticket to gather
> information about the current implementation issues and users' wants.
> Then look at a code to get a complete understanding of how things work now,
> which may help in future decisions.
>
> As we use the outdated Lucene version, some things may be irrelevant for
> the latest Lucene version.
> So, you will need expertise in the internals of modern Lucene version to
> understand what capabilities, guarantees, and limitations Lucene has and
> could bring to the Ignite.
> The expertise could be got from the Lucene project code or Lucene project
> dev-list.
>
>
> As for now, the potential capabilities are not clear to me.
> At first glance, I see the next topics that must be covered at first:
>
> General questions
> * How Lucene index can be split among the nodes?
> * If we'll have a single index for all partitions on the particular node,
> then how index records will be aware of partitioning?
> This is important to filter out backup records from the results to avoid
> duplicates.
> * How results from several nodes can be merged on the Reduce stage?
> * Does Lucene supports smth like JOIN operation or others that may require
> data from another partition or index?
> If so, then it likes to multistep query with merging results on
> intermediate stages and requires detailed investigation and design.
> It is ok if Ignite will have some limitations here, but we would like to
> know about them at the early stage.
> * How effectively map Lucene files to the page memory? Is it even possible?
> Otherwise, how to deal with potential OOM on large queries and memory
> capacity planning?
>
> Persistence.
> * How and what consistency guarantees could we have/expect?
> Seems, we may not be able to write physical records for Lucene index to our
> WAL. What can we do with this?
>
> Transactions.
> * Will we support transactions?
> * Should Lucene be aware of Transaction and track mvcc (or whatever)
> versions for the records?
> * What will be consistency guarantees?
>
> UX
> * How to add FullText search queries syntax into Calcite?
> * AFAIK, the Lucene index has many properties for tuning. How will the user
> configure the index?
> * How and where to store the settings? What are cluster-wide and what a
> local to the particular node?
> * Will be all the settings immutable? Can be they changed on-fly? after
> node/grid restart?
> * Any limitations on query syntax?
>
> SQL
> * Will we support FullText search in SQL?
> * How to integrate Lucene index into Calcite? What is the cost model?
> Splitting rules? Traits?
> * What about consistency with DDL operations, e.g. column rename?
> Ignite indices will operate column ID, so rename operation will not affect
> the index.
>
>
> With all of this, you can go with the IEP (or even some short summary) and
> further POC and implementation.
> That's a big deal, so let's discuss what could be done here.
>
> On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma  wrote:
>
> > I am actually happy to drive the feature for Ignite 3. FTS is very
> > important for me and I think Ignite users will benefit from it
> > greatly.
> >
> > If it makes sense to be focusing on Ignite 3 for this capability, I am
> > eager to contribute there and lead the development.
> >
> > Please share your thoughts.
> >
> > On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov
> >  wrote:
> > >
> > > Hi Atri,
> > >
> > > All the Jira tickets we have on the Full-text search (FTS) thing are
> > > targeted to Ignite 2.
> > >
> > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3,
> > yet.
> > > By the way, we are getting requests for this thing from the user side,
> > and
> > > definitely,
> > > FTS would be a valuable feature for Ignite.
> > >
> > > It will be great if the one wants to drive it, any help will be
> > appreciated.
> > >
> > >
> > > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma  wrote:
> > >
> > > > Hello,
> > > >
> > > > An update, please. I am working through persistence of Lucene index
> > using
> > > > Ignite Dictionary, and will be asking some questions soon.
> > > >
> > > > I had one doubt - - where does this change go? Ignite 3?
> > > >
> > > > Also, I know we want to build native support for text searches in
> > Ignite 3.
> > > > Is the work I am proposing here part of that, or will that be a
> > separate
> > > > effort?
> > > >
> > > > On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev,  > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > I think that number one is the most important one, then maybe it
> > will see
> > > > > more use and other deficiencies become more apparent, leading to more
> > > > > tickets and visibility.
> > > > >
> > > > > Maybe 2. and 3. wil

Re: Text Queries Support

[Valentin Kulichenko]

It surely will not be supported out of the box, but in my understanding
Calcite provides enough flexibility for us to develop these features in a
way we would like to see them.

Andrey, please correct me if I am wrong here.

-Val

On Fri, Jul 23, 2021 at 10:58 AM Atri Sharma  wrote:

> The standard ways to deal with text based searches in SQL are the
> CONTAINS operator, the LIKE operator or specific functions
> (REGEXP_MATCHES, for eg). I do not think any of these are supported by
> Calcite at the moment.
>
> On Fri, Jul 23, 2021 at 11:20 PM Valentin Kulichenko
>  wrote:
> >
> > In my experience, one of the biggest usability issues with the current
> > support of text queries is that they are completely decoupled from SQL.
> > I.e. you can either execute a SQL query OR a text query. Modern
> databases,
> > on the other hand, typically allow creating text-based indexes within
> > regular tables and then using those indexes within regular SQL queries.
> > Here is an example from Oracle:
> > https://docs.oracle.com/cd/B10501_01/text.920/a96517/cdefault.htm
> >
> > I believe this is something we can look into in the scope of Ignite 3.
> > Andrey, does Calcite have any support for this? What's your view on this?
> >
> > -Val
> >
> > On Fri, Jul 23, 2021 at 3:56 AM Andrey Mashenkov <
> andrey.mashen...@gmail.com>
> > wrote:
> >
> > > Atri,
> > >
> > > First of all, I'd recommend going through the Ignite ticket to gather
> > > information about the current implementation issues and users' wants.
> > > Then look at a code to get a complete understanding of how things work
> now,
> > > which may help in future decisions.
> > >
> > > As we use the outdated Lucene version, some things may be irrelevant
> for
> > > the latest Lucene version.
> > > So, you will need expertise in the internals of modern Lucene version
> to
> > > understand what capabilities, guarantees, and limitations Lucene has
> and
> > > could bring to the Ignite.
> > > The expertise could be got from the Lucene project code or Lucene
> project
> > > dev-list.
> > >
> > >
> > > As for now, the potential capabilities are not clear to me.
> > > At first glance, I see the next topics that must be covered at first:
> > >
> > > General questions
> > > * How Lucene index can be split among the nodes?
> > > * If we'll have a single index for all partitions on the particular
> node,
> > > then how index records will be aware of partitioning?
> > > This is important to filter out backup records from the results to
> avoid
> > > duplicates.
> > > * How results from several nodes can be merged on the Reduce stage?
> > > * Does Lucene supports smth like JOIN operation or others that may
> require
> > > data from another partition or index?
> > > If so, then it likes to multistep query with merging results on
> > > intermediate stages and requires detailed investigation and design.
> > > It is ok if Ignite will have some limitations here, but we would like
> to
> > > know about them at the early stage.
> > > * How effectively map Lucene files to the page memory? Is it even
> possible?
> > > Otherwise, how to deal with potential OOM on large queries and memory
> > > capacity planning?
> > >
> > > Persistence.
> > > * How and what consistency guarantees could we have/expect?
> > > Seems, we may not be able to write physical records for Lucene index
> to our
> > > WAL. What can we do with this?
> > >
> > > Transactions.
> > > * Will we support transactions?
> > > * Should Lucene be aware of Transaction and track mvcc (or whatever)
> > > versions for the records?
> > > * What will be consistency guarantees?
> > >
> > > UX
> > > * How to add FullText search queries syntax into Calcite?
> > > * AFAIK, the Lucene index has many properties for tuning. How will the
> user
> > > configure the index?
> > > * How and where to store the settings? What are cluster-wide and what a
> > > local to the particular node?
> > > * Will be all the settings immutable? Can be they changed on-fly? after
> > > node/grid restart?
> > > * Any limitations on query syntax?
> > >
> > > SQL
> > > * Will we support FullText search in SQL?
> > > * How to integrate Lucene index into Calcite? What is the cost model?
> > > Splitting rules? Traits?
> > > * What about consistency with DDL operations, e.g. column rename?
> > > Ignite indices will operate column ID, so rename operation will not
> affect
> > > the index.
> > >
> > >
> > > With all of this, you can go with the IEP (or even some short summary)
> and
> > > further POC and implementation.
> > > That's a big deal, so let's discuss what could be done here.
> > >
> > > On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma  wrote:
> > >
> > > > I am actually happy to drive the feature for Ignite 3. FTS is very
> > > > important for me and I think Ignite users will benefit from it
> > > > greatly.
> > > >
> > > > If it makes sense to be focusing on Ignite 3 for this capability, I
> am
> > > > eager to contribute there and lead the development.
> > > >
> 

Re: Text Queries Support

[Valentin Kulichenko]

Atri,

Sure, go ahead. Let's put the ideas on paper and have a discussion.

-Val

On Fri, Jul 23, 2021 at 10:59 AM Atri Sharma  wrote:

> Thanks Andrey.
>
> I have collected answers or proposals to many of these questions and
> would like to start a wiki page covering what we can do for Ignite 3.
>
> Does that sound good, please?
>
> On Fri, Jul 23, 2021 at 4:26 PM Andrey Mashenkov
>  wrote:
> >
> > Atri,
> >
> > First of all, I'd recommend going through the Ignite ticket to gather
> > information about the current implementation issues and users' wants.
> > Then look at a code to get a complete understanding of how things work
> now,
> > which may help in future decisions.
> >
> > As we use the outdated Lucene version, some things may be irrelevant for
> > the latest Lucene version.
> > So, you will need expertise in the internals of modern Lucene version to
> > understand what capabilities, guarantees, and limitations Lucene has and
> > could bring to the Ignite.
> > The expertise could be got from the Lucene project code or Lucene project
> > dev-list.
> >
> >
> > As for now, the potential capabilities are not clear to me.
> > At first glance, I see the next topics that must be covered at first:
> >
> > General questions
> > * How Lucene index can be split among the nodes?
> > * If we'll have a single index for all partitions on the particular node,
> > then how index records will be aware of partitioning?
> > This is important to filter out backup records from the results to avoid
> > duplicates.
> > * How results from several nodes can be merged on the Reduce stage?
> > * Does Lucene supports smth like JOIN operation or others that may
> require
> > data from another partition or index?
> > If so, then it likes to multistep query with merging results on
> > intermediate stages and requires detailed investigation and design.
> > It is ok if Ignite will have some limitations here, but we would like to
> > know about them at the early stage.
> > * How effectively map Lucene files to the page memory? Is it even
> possible?
> > Otherwise, how to deal with potential OOM on large queries and memory
> > capacity planning?
> >
> > Persistence.
> > * How and what consistency guarantees could we have/expect?
> > Seems, we may not be able to write physical records for Lucene index to
> our
> > WAL. What can we do with this?
> >
> > Transactions.
> > * Will we support transactions?
> > * Should Lucene be aware of Transaction and track mvcc (or whatever)
> > versions for the records?
> > * What will be consistency guarantees?
> >
> > UX
> > * How to add FullText search queries syntax into Calcite?
> > * AFAIK, the Lucene index has many properties for tuning. How will the
> user
> > configure the index?
> > * How and where to store the settings? What are cluster-wide and what a
> > local to the particular node?
> > * Will be all the settings immutable? Can be they changed on-fly? after
> > node/grid restart?
> > * Any limitations on query syntax?
> >
> > SQL
> > * Will we support FullText search in SQL?
> > * How to integrate Lucene index into Calcite? What is the cost model?
> > Splitting rules? Traits?
> > * What about consistency with DDL operations, e.g. column rename?
> > Ignite indices will operate column ID, so rename operation will not
> affect
> > the index.
> >
> >
> > With all of this, you can go with the IEP (or even some short summary)
> and
> > further POC and implementation.
> > That's a big deal, so let's discuss what could be done here.
> >
> > On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma  wrote:
> >
> > > I am actually happy to drive the feature for Ignite 3. FTS is very
> > > important for me and I think Ignite users will benefit from it
> > > greatly.
> > >
> > > If it makes sense to be focusing on Ignite 3 for this capability, I am
> > > eager to contribute there and lead the development.
> > >
> > > Please share your thoughts.
> > >
> > > On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov
> > >  wrote:
> > > >
> > > > Hi Atri,
> > > >
> > > > All the Jira tickets we have on the Full-text search (FTS) thing are
> > > > targeted to Ignite 2.
> > > >
> > > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3,
> > > yet.
> > > > By the way, we are getting requests for this thing from the user
> side,
> > > and
> > > > definitely,
> > > > FTS would be a valuable feature for Ignite.
> > > >
> > > > It will be great if the one wants to drive it, any help will be
> > > appreciated.
> > > >
> > > >
> > > > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma 
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > An update, please. I am working through persistence of Lucene index
> > > using
> > > > > Ignite Dictionary, and will be asking some questions soon.
> > > > >
> > > > > I had one doubt - - where does this change go? Ignite 3?
> > > > >
> > > > > Also, I know we want to build native support for text searches in
> > > Ignite 3.
> > > > > Is the work I am proposing here part of that, or wil

Re: [ANNOUNCE] Apache Ignite spring-data-all-ext extensions 1.0.0 released

[18624049226]

Igniters,

It seems that these artifact have not been uploaded to Maven Repo? Will 
it be uploaded with ignite 2.11?


This release does not include spring-tx-ext, but the documentation of 
this artifact has been merged into the ignite-2.11 branch, What's wrong?


在 2021/7/7 下午1:49, Nikita Amelchev 写道:

The Apache Ignite Community is pleased to announce the release of
Apache Ignite Spring Data extensions 1.0.0.

The following integrations were migrated to the Apache Ignite
Extensions repository:

- Spring Data 2.2 extension.
- Spring Data 2.0 extension.
- Spring Data extension.

Release notes:
https://ignite.apache.org/releases/ext/spring-data-all-1.0.0/release_notes.html

The sources package is available here:
https://downloads.apache.org/ignite/ignite-extensions/ignite-spring-data-all-ext/1.0.0/

Please let us know if you have any problems
https://ignite.apache.org/community/resources.html#ask

[MTCGA]: new failures in builds [6095521] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *Test with high flaky rate in master-nightly 
IgniteRebalanceScheduleResendPartitionsTest.test 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=249563339269496287&branch=%3Cdefault%3E&tab=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 06:29:28 24-07-2021