Re: Ignite 2.8 documentation
I've created a draft pages on apache.readme.io and will continue my work there during next 2 weeks. Should I create any tickets for that? Or could miss that step? Will notify in this thread than the work will be done! чт, 20 февр. 2020 г. в 12:16, Alexey Zinoviev : > Yes, there are a lot of changes in ML from 2.7, I'm going to prepare new > documentation and create documentation related tickets for the ML > component. > After some consultation and review from Artem side I'll add new > documentation on readme.io. > > > > чт, 20 февр. 2020 г. в 02:34, Denis Magda : > >> Artem, >> >> Thanks for stepping in and preparing the list of top priority >> documentation tasks! How about labeling those tickets somehow and creating >> a filter similar to this one [1] but for "Required & Unresolved >> Documentation Tasks"? I would simply add this as a new section to the >> Ignite 2.8 release wiki page for ease of tracking and start working with >> the guys contributed improvements directly. Will see the names of the >> authors who need to be involved ;) >> >> *Alexey Zinoviev*, there are many ML related changes coming in the >> release. Could you check existing ML docs and suggest any changes? >> >> [1] >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Unresolveddocumentationtasks >> >> - >> Denis >> >> >> On Wed, Feb 19, 2020 at 11:14 AM Artem Budnikov < >> a.budnikov.ign...@gmail.com> wrote: >> >>> Maxim, >>> >>> One note from my side, I think we can move disk page compression [1] >>> > to the 2-nd priority, but definitely must document WAL page >>> > compression first [2] >>> >>> >>> OK, good to know. >>> >>> On Wed, Feb 19, 2020 at 6:48 PM Maxim Muzafarov >>> wrote: >>> >>> > Artem, >>> > >>> > >>> > Thank you for starting this thread. >>> > One note from my side, I think we can move disk page compression [1] >>> > to the 2-nd priority, but definitely must document WAL page >>> > compression first [2] >>> > >>> > >>> > The list of important tasks [3]. >>> > The list of documentation tasks [4]. >>> > >>> > [1] https://issues.apache.org/jira/browse/IGNITE-10330 >>> > [2] https://issues.apache.org/jira/browse/IGNITE-11336 >>> > [3] >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Themostimportantreleasetasks >>> > [4] >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Unresolveddocumentationtasks >>> > >>> > On Wed, 19 Feb 2020 at 18:15, Artem Budnikov >>> > wrote: >>> > > >>> > > Hi everyone, >>> > > >>> > > As the release of Ignite 2.8 is getting closer, let's discuss which >>> > features should be documented. I created a list of features based on >>> the >>> > release notes and the documentation tickets in jira (see below). Much >>> more >>> > has been added, but these seemed to have first priority. It's not to >>> say >>> > that other features are not important, but given the limited resources >>> a >>> > list of high-priority task would help to schedule the time of those who >>> > will help with the docs. >>> > > >>> > > Here is the list of features: >>> > > >>> > > Disk page compression >>> > > Metrics and System Views documentation >>> > > Default Ignite work dir location >>> > > Baseline auto-adjust feature >>> > > Cluster (de)activation events documentation >>> > > Remove SqlQuery documentation >>> > > Partition awareness for thin clients >>> > > Transactions support in thin clients >>> > > KILL QUERY command >>> > > Move rebalance configuration properties to the IgniteConfiguration >>> level >>> > > Renamed IGNITE schema to SYS >>> > > JDBC: Support for query cancellation >>> > > JDBC: Support for query timeout >>> > > suspend/resume for pessimistic transactions >>> > > >>> > > Now, two really big questions here: >>> > > >>> > > If you want to add anything else to the documentation, please let us >>> > know in this thread. >>> > > If you are the author of any of the features listed above, please >>> share >>> > the details on the feature. You can do it by creating a documentation >>> > ticket. If you want to contribute a page or section to the existing >>> docs, >>> > you can write directly on apacheignite.readme.io (send me a note if >>> you >>> > don't have an account there). Any help is greatly appreciated. >>> > > >>> > > -Artem >>> > > >>> > > >>> > >>> >>
Re: Unable to get the security context
HI Denis, Which version of Apache Ignite are the changes you mention in the comment( security context always not null ) above available with ? In 2.7.6 I do get security context as null in authorize method. regards, Veena. -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: .NET Near Cache - new flag in NearCacheConfiguration.java?
Sergey,
Sure, I'll add you as a reviewer.
And a thorough review will be the best contribution. The feature is rather
complex and in the works for quite some time.
On Fri, Feb 21, 2020 at 4:03 PM Ilya Kasnacheev
wrote:
> Hello!
>
> Can you please describe what is the overhead on server nodes from starting
> .Net near cache, as compared to starting "regular" near cache?
>
> Also, why do we start near cache on all server nodes - is it required by
> our architecture?
>
> Otherwise, if you add a new feature, which you have to explicitly invoke
> from application .Net code, I assume it's OK to consume some resources upon
> such invocation. We can't expect new features to consume zero resources.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 18 февр. 2020 г. в 20:46, Pavel Tupitsyn :
>
> > Ilya, looks like there is a misunderstanding.
> >
> > We don't start near cache on a subset of server nodes. Let me explain
> > again.
> > Consider the following code, executed on any node (client or server -
> does
> > not matter):
> >
> > ignite.createCache(new CacheConfiguration("c").setNearConfiguration(new
> > NearCacheConfiguration()));
> >
> > As a result, cache "c" is started on all server nodes, and corresponding
> > near cache is started on all server nodes,
> > with the same config, eviction policy, and so on.
> >
> > Now, what do we do with .NET near cache in this situation?
> > 1. Start .NET near cache as well on server nodes, always.
> > 2. Introduce a config flag, and start .NET near cache only when the flag
> is
> > set
> >
> > The problem with option (1) is that it changes the behavior of existing
> > applications,
> > increasing overhead for cache operations and increasing memory usage.
> >
> > As a user, I don't want some new features to be force-enabled on me
> > silently in a new product version.
> > That is why an opt-in config flag is required.
> >
> > Does this make sense?
> >
> > On Tue, Feb 18, 2020 at 6:42 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > I'm not sure that we can/want to create near cache on a subset of
> server
> > > nodes.
> > >
> > > If this is indeed possible, I would decouple that from .Net and
> introduce
> > > as a separate feature.
> > >
> > > Ideally we should be able to start or stop near cache on any node,
> server
> > > or client, and this should be the same cache for all platforms
> (including
> > > Java). WDYT?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вт, 18 февр. 2020 г. в 17:31, Pavel Tupitsyn :
> > >
> > > > Ilya, as noted above:
> > > >
> > > > > There are 2 kinds of Near Caches:
> > > > > - On client nodes: created per-node by calling
> ignite.CreateNearCache
> > > > > - On server nodes: created on all server nodes if
> > > > CacheConfiguration.NearCacheConfiguration is not null
> > > > >
> > > > > When user says ignite.CreateCache(new CacheConfiguration
> > > > {NearCacheConfiguration = ...}),
> > > > > the whole config is sent to all server nodes, and .NET-specific
> flag
> > > has
> > > > to be included somehow.
> > > >
> > > >
> > > >
> > > > On Tue, Feb 18, 2020 at 5:10 PM Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > Why would it waste additional memory? All nodes are also Java nodes
> > so
> > > > they
> > > > > will start near caches all right.
> > > > >
> > > > > Don't we have near cache start-up on per-node basis for clients,
> > > anyway?
> > > > >
> > > > > I'm not convinced why we need this flag. I have to admit my
> > > understanding
> > > > > of near caches is limited.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > вт, 18 февр. 2020 г. в 16:56, Pavel Tupitsyn >:
> > > > >
> > > > > > Ilya, Aleksandr,
> > > > > >
> > > > > > The flag is called platformNearCacheEnabled, my idea is to have
> > just
> > > > one
> > > > > > flag for all platforms.
> > > > > >
> > > > > > If some platform is present on the given node (.NET, C++) and it
> > > > supports
> > > > > > native near caching,
> > > > > > then the flag is honored, otherwise it is ignored. We should not
> > > throw
> > > > > > exceptions,
> > > > > > because mixed clusters are possible (.NET nodes along with Java
> > > nodes).
> > > > > >
> > > > > > > Why would enabling it affect mixed`clusters?
> > > > > > Initially the idea was to enable .NET near cache whenever
> > > > > > NearCacheConfiguration is present.
> > > > > > If we assume .NET-only clusters, it makes sense: existing code
> will
> > > > work
> > > > > > faster automatically.
> > > > > >
> > > > > > Mixed cluster is just one of the possible use cases when this may
> > be
> > > a
> > > > > bad
> > > > > > idea,
> > > > > > e.g. users have enabled near caching to speed up their Java-based
> > > > > > computations,
> > > > > > and .NET nodes are used for something else, wasting memory on
> near
> > > > > caching
> > > > > > unnecessarily.
>
Re: AWS EBS Discovery: Contributor Wanted
Hi! With regards to the PR review if there is anything I can do please tell me. Kind Regards *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Tue, 28 Jan 2020 at 08:03, Sergey Chugunov wrote: > Denis, Emmanouil, > > Sure, I'll take a look at the code shortly. > > -- > Thank you, > Sergey. > > On Mon, Jan 27, 2020 at 8:59 PM Denis Magda wrote: > > > I support the idea of triggering such tests on demand. We can create a > wiki > > page with instructions on how to run the tests. Unless there is a more > > elegant solution. > > > > Sergey, would you be able to review Emmanouil's changes in the IP Finder > > source code? > > https://issues.apache.org/jira/browse/IGNITE-8617 > > > > - > > Denis > > > > > > On Sun, Jan 26, 2020 at 2:22 AM Emmanouil Gkatziouras < > > gkatzio...@gmail.com> > > wrote: > > > > > Hi all! > > > > > > I do believe being able to execute some AWS integration tests on demand > > > would be of value, especially for reviewers who cannot have an AWS > stack > > > created on demand. > > > More than happy to help on that. > > > > > > Kind regards > > > *Emmanouil Gkatziouras* > > > https://egkatzioura.com/ | > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > https://github.com/gkatzioura > > > > > > > > > On Fri, 24 Jan 2020 at 15:15, Sergey Chugunov < > sergey.chugu...@gmail.com > > > > > > wrote: > > > > > > > Hello Emmanouil, > > > > > > > > It would be great if we have at least basic integration tests in real > > AWS > > > > environment. Even though they may require more work to keep them > green > > (I > > > > mean here AWS quotas and additional configuration/reconfiguration > > > efforts) > > > > it worth it because these tests can also be useful as an examples. > > > > > > > > As the same time as IpFinder is such a basic component I don't think > we > > > > need to include them in constantly triggered suites like Run All but > to > > > > trigger them manually before/right after merging them to master > branch > > or > > > > when developing something in related code. > > > > > > > > What do you think? > > > > > > > > -- > > > > Thank you, > > > > Sergey Chugunov. > > > > > > > > On Thu, Jan 23, 2020 at 5:32 PM Emmanouil Gkatziouras < > > > > gkatzio...@gmail.com> > > > > wrote: > > > > > > > > > Hi all! > > > > > > > > > > Yes It seems possible to get some free quota for integration tests > on > > > AWS > > > > > [1] however most probably they are not gonna last forever. > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ > > > > > > > > > > King Regards > > > > > *Emmanouil Gkatziouras* > > > > > https://egkatzioura.com/ | > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > On Wed, 22 Jan 2020 at 16:48, Denis Magda > wrote: > > > > > > > > > > > Hi Emmanouil, > > > > > > > > > > > > Thanks for preparing a pull-request for Application Load > Balancer: > > > > > > https://issues.apache.org/jira/browse/IGNITE-8617 > > > > > > > > > > > > Igniters, who is willing to step in as a primary reviewer? > > > > > > > > > > > > As for automated testing on AWS, are you aware of any sponsorship > > > > program > > > > > > of AWS for open source projects of our kind? It will be ideal to > > have > > > > > real > > > > > > testing on AWS but someone needs to pay. > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Sun, Jan 19, 2020 at 6:45 AM Emmanouil Gkatziouras < > > > > > > gkatzio...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > I have spinned up an Application Load Balancer and an > autoscaling > > > > group > > > > > > on > > > > > > > AWS and the Ignite discovery using TcpDiscoveryAlbIpFinder > works > > as > > > > > > > expected. > > > > > > > > > > > > > >- On startup nodes discover each other. > > > > > > >- On ec2 node down, connection is lost and the cluster > > > decreases. > > > > > > >- On an extra node addition the cluster size increases > > > > > > > > > > > > > > This contribution is essential since the Previous ELB based > > > discovery > > > > > > uses > > > > > > > the Classic Load Balancer which is still available however > > > > > > > AWS advices users to use the Application one. [1] > > > > > > > While my pull request gets reviewed I will also have a look at > > > > > > > the IGNITE-12398 [2] issue which has to do with the S3 > discovery. > > > > > > > Another idea would also be to implement a `TCP Load Balancer > > based` > > > > > > > discovery. > > > > > > > > > > > > > > In order to test this issue and future ones I implemented some > > > > > terraform > > > > > > > scripts (which I shall use for other issues too) [3]. > > > > > > > If some automated e2e testing on AWS is being considered they > > might > > > > b
Re: .NET Near Cache - new flag in NearCacheConfiguration.java?
Ilya, when .NET Near Cache is enabled, Java invokes callbacks to update or
evict near cache data in .NET, hence the overhead.
I've created a PR, everybody is welcome to do a preliminary review (just
ignore the minor things for now).
Core functionality is ready, put/get is supported. I'm working on other
cache operations.
[1] https://github.com/apache/ignite/pull/7468
On Sat, Feb 22, 2020 at 7:07 PM Pavel Tupitsyn wrote:
> Sergey,
>
> Sure, I'll add you as a reviewer.
> And a thorough review will be the best contribution. The feature is rather
> complex and in the works for quite some time.
>
>
> On Fri, Feb 21, 2020 at 4:03 PM Ilya Kasnacheev
> wrote:
>
>> Hello!
>>
>> Can you please describe what is the overhead on server nodes from starting
>> .Net near cache, as compared to starting "regular" near cache?
>>
>> Also, why do we start near cache on all server nodes - is it required by
>> our architecture?
>>
>> Otherwise, if you add a new feature, which you have to explicitly invoke
>> from application .Net code, I assume it's OK to consume some resources
>> upon
>> such invocation. We can't expect new features to consume zero resources.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> вт, 18 февр. 2020 г. в 20:46, Pavel Tupitsyn :
>>
>> > Ilya, looks like there is a misunderstanding.
>> >
>> > We don't start near cache on a subset of server nodes. Let me explain
>> > again.
>> > Consider the following code, executed on any node (client or server -
>> does
>> > not matter):
>> >
>> > ignite.createCache(new CacheConfiguration("c").setNearConfiguration(new
>> > NearCacheConfiguration()));
>> >
>> > As a result, cache "c" is started on all server nodes, and corresponding
>> > near cache is started on all server nodes,
>> > with the same config, eviction policy, and so on.
>> >
>> > Now, what do we do with .NET near cache in this situation?
>> > 1. Start .NET near cache as well on server nodes, always.
>> > 2. Introduce a config flag, and start .NET near cache only when the
>> flag is
>> > set
>> >
>> > The problem with option (1) is that it changes the behavior of existing
>> > applications,
>> > increasing overhead for cache operations and increasing memory usage.
>> >
>> > As a user, I don't want some new features to be force-enabled on me
>> > silently in a new product version.
>> > That is why an opt-in config flag is required.
>> >
>> > Does this make sense?
>> >
>> > On Tue, Feb 18, 2020 at 6:42 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com
>> > >
>> > wrote:
>> >
>> > > Hello!
>> > >
>> > > I'm not sure that we can/want to create near cache on a subset of
>> server
>> > > nodes.
>> > >
>> > > If this is indeed possible, I would decouple that from .Net and
>> introduce
>> > > as a separate feature.
>> > >
>> > > Ideally we should be able to start or stop near cache on any node,
>> server
>> > > or client, and this should be the same cache for all platforms
>> (including
>> > > Java). WDYT?
>> > >
>> > > Regards,
>> > > --
>> > > Ilya Kasnacheev
>> > >
>> > >
>> > > вт, 18 февр. 2020 г. в 17:31, Pavel Tupitsyn :
>> > >
>> > > > Ilya, as noted above:
>> > > >
>> > > > > There are 2 kinds of Near Caches:
>> > > > > - On client nodes: created per-node by calling
>> ignite.CreateNearCache
>> > > > > - On server nodes: created on all server nodes if
>> > > > CacheConfiguration.NearCacheConfiguration is not null
>> > > > >
>> > > > > When user says ignite.CreateCache(new CacheConfiguration
>> > > > {NearCacheConfiguration = ...}),
>> > > > > the whole config is sent to all server nodes, and .NET-specific
>> flag
>> > > has
>> > > > to be included somehow.
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Feb 18, 2020 at 5:10 PM Ilya Kasnacheev <
>> > > ilya.kasnach...@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Hello!
>> > > > >
>> > > > > Why would it waste additional memory? All nodes are also Java
>> nodes
>> > so
>> > > > they
>> > > > > will start near caches all right.
>> > > > >
>> > > > > Don't we have near cache start-up on per-node basis for clients,
>> > > anyway?
>> > > > >
>> > > > > I'm not convinced why we need this flag. I have to admit my
>> > > understanding
>> > > > > of near caches is limited.
>> > > > >
>> > > > > Regards,
>> > > > > --
>> > > > > Ilya Kasnacheev
>> > > > >
>> > > > >
>> > > > > вт, 18 февр. 2020 г. в 16:56, Pavel Tupitsyn <
>> ptupit...@apache.org>:
>> > > > >
>> > > > > > Ilya, Aleksandr,
>> > > > > >
>> > > > > > The flag is called platformNearCacheEnabled, my idea is to have
>> > just
>> > > > one
>> > > > > > flag for all platforms.
>> > > > > >
>> > > > > > If some platform is present on the given node (.NET, C++) and it
>> > > > supports
>> > > > > > native near caching,
>> > > > > > then the flag is honored, otherwise it is ignored. We should not
>> > > throw
>> > > > > > exceptions,
>> > > > > > because mixed clusters are possible (.NET nodes along with Java
>> > > nodes).
>> > > > > >
>> > > > > > > Why would enabling it affect mixed`clusters?
>>
