Write access to Apache Ignite Confluence

[Andrey Kuznetsov]

Hi, Igniters.

I'd like to add new wiki page with suggestions for future Ignite 3.0
release. We can collect any ideas there and then discuss and refine them on
dev-list.

Could someone grant me (andrey-kuznetsov) write permission?

Re: Write access to Apache Ignite Confluence

[Dmitriy Pavlov]

Hi Andrey,

I've updated permissions, please check.

Sincerely,
Dmitriy Pavlov

вт, 18 дек. 2018 г. в 12:34, Andrey Kuznetsov :

> Hi, Igniters.
>
> I'd like to add new wiki page with suggestions for future Ignite 3.0
> release. We can collect any ideas there and then discuss and refine them on
> dev-list.
>
> Could someone grant me (andrey-kuznetsov) write permission?
>

Re: Thin clients all in one

[Igor Sapego]

I've created documentation ticket for UUID type - [1].

Now about Timestamp type. Binary protocol specification only specifies
how the data type should be transmitted by the network. The purpose and
main requirements for the protocol is to be compact and fast, it is not
designed to be used by user in raw.

Thus binary protocol does not define nor mandates any type representation
for the specific thin client or its API as it is implied that different
languages
may have different practicies and paradigms.

So binary protocol is not something we should refer to when we are talking
about user API. Protocol should be compact and fast, user API should be
convenient.

[1] - https://issues.apache.org/jira/browse/IGNITE-10721.

Best Regards,
Igor


On Mon, Dec 17, 2018 at 6:18 PM Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Hi Stepan,
>
> 1) About UUID...
>
> UUID binary encoding must be defined in the Binary Protocol
> specification
> (
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-uuid-guid-
> )
> So, it's the issue against the spec first.
>
> Regarding the clients..
>
> Python client uses Python UUID class
> (https://docs.python.org/2/library/uuid.html) which, if I understand
> correctly, can operate with different encodings. Dmitry M. can correct me.
>
> NodeJs and PHP do not have standard UUID classes.
> So, the clients return/accept UUID as just a 16-byte array with binary
> encoding inside, exactly as it is read / will be written via the binary
> protocol.
> An application may convert this array to/from a required representation.
>
> 2) About Timestamp...
>
> Python, NodeJs and PHP clients behave strictly according to the Binary
> Protocol specification
> (
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-timestamp)
>
> which defines that the second field is "a nanoseconds fraction of a last
> millisecond, which value could be in a range from 0 to 99."
>
> Regards,
> -Alexey
>
> 14.12.2018 17:41, Stepan Pilschikov пишет:
> > Hello again
> >
> > Starting to check compatibility between thin clients in java/c++ and
> > py/php/nodejs and met some problems
> >
> > 1) Found that UUID mixed in a strange way if we taken it from Java/C++
> > to PY/PHP/JS client and backwards
> > Have issue about it https://issues.apache.org/jira/browse/IGNITE-10691,
> > please look at it
> >
> > 2) Is more like conversation about seamless experience between all thin
> > clients
> > PY/PHP/JS/C++/Java have Timestamp data type which is contain
> > tuple(millis, nanos)
> > Main concern about nanos
> >
> > In PHP/JS/Py Nanos count like "nanoseconds of the last millisecond" but
> > in Java and C++ its like we have millis and same millis but with nanos
> > and its strange for PHP/PY/JS if we take Java how is Base thin client
> >
> > For example in Java we have new Timestamp(11) and it
> > (fastTime=111000, nanos=11100)
> > In C++ if we getting this cache we take (millis=11, nanos=11100)
> > which is kind a right and same as in java
> > But in py/php/js its look like (millis=11, nanos=0), its kind a
> > understandable what logic is, but sill different behavior and experience
> > What you thinking about it?
> > For now i can't understand why its done how its done, looks like it
> > should be same but something going wrong
> >
> > пт, 30 нояб. 2018 г. в 01:18, Alexey Kosenchuk
> > mailto:alexey.kosenc...@nobitlost.com
> >>:
> >
> > Hi Stepan,
> >
> > pls check the Ignite cfg you use - see the comments in the jira.
> >
> > Also, the examples executors (including AuthTlsExample) are included
> > into NodeJS test suite in TeamCity which run periodically and
> > successfully passed. Eg. the latest one:
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=2426645&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_ThinClientNodeJs
> >
> > Regards,
> > -Alexey
> >
> > 28.11.2018 17:08, Stepan Pilschikov пишет:
> >  > Hello again
> >  >
> >  > If NodeJS sources found that example AuthTlsExample.js throwing
> > exception
> >  > during execution
> >  > Output and grid configuration in
> >  > https://issues.apache.org/jira/browse/IGNITE-10447
> >  >
> >  > Can someone have a look at it?
> >  >
> >  > вс, 25 нояб. 2018 г. в 19:11, Stepan Pilschikov
> > mailto:pilshchikov@gmail.com>>:
> >  >
> >  >> My bad,
> >  >> You right
> >  >>
> >  >> вс, 25 нояб. 2018 г. в 05:37, Dmitry Melnichuk <
> >  >> dmitry.melnic...@nobitlost.com
> > >:
> >  >>
> >  >>> Stepan,
> >  >>>
> >  >>> AFAIK Map type did always behave correctly on client side, as
> > it does
> >  >>> now. This is a corresponding piece of my test suite:
> >  >>>
> >  >>> ```
> >  >>> def test_put_get_map(client):
> >  >>>
> >  >>>   cache = client.get_or_create_cache('test_map_cache')
> >  >>

Re: Write access to Apache Ignite Confluence

[Andrey Kuznetsov]

Thanks, Dmitriy.

The page stub [1] is now created.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist

Best regards,
Andrey Kuznetsov.

вт, 18 дек. 2018, 12:45 Dmitriy Pavlov dpav...@apache.org:

> Hi Andrey,
>
> I've updated permissions, please check.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 18 дек. 2018 г. в 12:34, Andrey Kuznetsov :
>
> > Hi, Igniters.
> >
> > I'd like to add new wiki page with suggestions for future Ignite 3.0
> > release. We can collect any ideas there and then discuss and refine them
> on
> > dev-list.
> >
> > Could someone grant me (andrey-kuznetsov) write permission?
> >
>

Re: Pre-touch for Ignite off-heap memory

[Павлухин Иван]

Stan,

In one of your emails you wrote why you need pre-touch:
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for 
> physical page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself

As far as I see only c) can be satisfied if lazy region initialization
is enabled. Am I missing something?
пн, 17 дек. 2018 г. в 18:11, Stanislav Lukyanov :
>
> I don’t think these two issues require to be solved together, although I 
> agree there is some connection.
>
>
>
> I guess we agree that pre-touch should a be off by default.
>
> Similarly, lazy data region initialization from IGNITE-9113 can be on by 
> default.
>
> We can have two boolean properties controlling each feature, e.g.
>
> IGNITE_EAGER_DATA_REGION_INITIALIZATION
>
> IGNITE_PRE_TOUCH_OFF_HEAP_MEMORY
>
> Setting both to true will make sure that all memory is committed as early as 
> possible.
>
> Different combinations will allow for all 4 variants you’ve mentioned. I can 
> see a use case for each one:
>
> - lazy region init + no pre-touch (default) – easier configuration (at least 
> for simple use cases without a lot of regions and node filters) and faster 
> startup
>
> - lazy region init + pre-touch – allows to reuse server config on a client 
> (the goal of the IGNITE-9113) but still allows to fail-fast on cache creation
>
> - eager region init + no pre-touch – for cases when you want to eagerly 
> allocate memory but don’t care about committing physical pages; e.g. if you 
> have overcommit disabled then you don’t need to pre-touch everything to make 
> sure the memory is there
>
> - eager region init + pre-touch – to fail as early as possible if there is 
> not enough RAM
>
>
>
> Stan
>
>
>
> From: Nikolay Izhikov
> Sent: 14 декабря 2018 г. 14:42
> To: dev; vololo...@gmail.com; stanlukya...@gmail.com
> Subject: Re: Pre-touch for Ignite off-heap memory
>
>
>
> Bump.
>
>
>
> Stanislav, Ivan, can you answer my questions?
>
> Let's discuss these improvements.
>
>
>
> ср, 12 дек. 2018 г. в 22:14, Павлухин Иван :
>
>
>
> > Hi Stan,
>
> >
>
> > As I participated in discussion in current thread I would like to
>
> > leave a comment.
>
> >
>
> > Your concerns looks clear for me and if you believe that pre-touch
>
> > will help product users then I have nothing against it.
>
> > ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov :
>
> > >
>
> > > Hello, Stanislav.
>
> > >
>
> > > As far as I can see, we have controversal ticket based on previous
>
> > dev-list discussion:
>
> > >
>
> > > IGNITE-9113 - Allocate memory for a data region when first cache
>
> > assigned to this region is created [1]
>
> > >
>
> > > I planned to implement it soon.
>
> > > Looks like we should have several options of memory(data regions)
>
> > allocation:
>
> > >
>
> > > - allocate all on startup (AFAIK this is how current
>
> > implementation behave)
>
> > > - allocate all on startup AND pre touch.
>
> > > - allocate specific data region for first assignment.
>
> > > - allocate specific data region for first assignment AND pre
>
> > touch.
>
> > >
>
> > > What do you think?
>
> > >
>
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> > >
>
> > > В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
>
> > > > Igniters,
>
> > > >
>
> > > > What is being suggested here is an Ignite off-heap’s version of Java’s
>
> > -XX:+AlwaysPreTouch.
>
> > > > The latter is known to be used to guarantee that the committed memory
>
> > is backed by physical RAM.
>
> > > > This ensures that
>
> > > > a) JVM doesn’t have to do it during the actual work (avoiding overhead
>
> > for physical page allocation, possible contention with page cache, etc)
>
> > > > b) JVM fails fast if the Xmx is greater than available RAM
>
> > > > c) New processes will not be able to claim the memory JVM took for
>
> > itself
>
> > > >
>
> > > > Currently one can’t get the same benefits for Ignite because we use
>
> > off-heap as well as heap.
>
> > > > So, we can implement a similar feature for Ignite – and make sure the
>
> > users can get all the memory pre-touch benefits if they want.
>
> > > > Of course, it impacts startup time so we should enable it by a
>
> > configuration property (I’d suggest a system property for now).
>
> > > >
>
> > > > Are there any objections to implementing this?
>
> > > >
>
> > > > Thanks,
>
> > > > Stan
>
> > > >
>
> > > > From: Павлухин Иван
>
> > > > Sent: 31 октября 2018 г. 12:50
>
> > > > To: dev@ignite.apache.org
>
> > > > Subject: Re: Pre-touch for Ignite off-heap memory
>
> > > >
>
> > > > Hi,
>
> > > >
>
> > > > I did an experiment described above. I created a patch with pre-touch
>
> > [1]
>
> > > > and started a JVM with an option -XX:+AlwaysPreTouch and configured
>
> > > > Ignite with equal values for initial and max sizes for each data
>
> > region.
>
> > > > I di

[jira] [Created] (IGNITE-10728) CPP: Move networking into separate library

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-10728:


 Summary: CPP: Move networking into separate library
 Key: IGNITE-10728
 URL: https://issues.apache.org/jira/browse/IGNITE-10728
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.7
Reporter: Igor Sapego
Assignee: Igor Sapego
 Fix For: 2.8


Currently, there are two very similar parts in ODBC and C++ Thin Client, which 
implement networking functionality. We need to move this functionality into 
separate library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10729) MVCC TX: Improvements.

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-10729:
-

 Summary: MVCC TX: Improvements.
 Key: IGNITE-10729
 URL: https://issues.apache.org/jira/browse/IGNITE-10729
 Project: Ignite
  Issue Type: Improvement
  Components: mvcc
Reporter: Igor Seliverstov


Currently there are several problems:
1) vacuum doesn't have change set, this means it travers all data to find 
invisible entries; hanse it breaks read statistics and make all data set "hot" 
- we should travers data entries instead, and only those entries, which was 
updated (linked to newer versions), moreover, vacuum should travers only those 
data pages, which were updated after last successful vacuum (at least one entry 
on the data page was linked to a never one)

2) vacuum travers over partitions instead of data entries, so, there possible 
some races like: reader checks an entry; updater removes this entry from 
partition; vacuum doesn't see the entry and clean TxLog -> reader cannot check 
the entry state with TxLog and gets an exception. This race prevents an 
optimization when all entries, older than last successful vacuum version, are 
considered as COMMITTED (see previous suggestion)

3) all entry versions are placed in BTrees, so, we cannot do updates like PG - 
just adding a new version and linking the old one to it. Having only one 
unversioned item per row in all indexes making possible fast invoke operations 
on such indexes in MVCC mode. Also it let us not to update all indexes on each 
update operation (partition index isn't updated at all, only SQL indexes, built 
over changed fields need to be updated) - this dramatically reduces write 
operations, hence it reduces amount of pages to be "checkpointed" and reduces 
checkpoint mark phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10730) MVCC TX: Batch WAL datarecords

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-10730:
-

 Summary: MVCC TX: Batch WAL datarecords
 Key: IGNITE-10730
 URL: https://issues.apache.org/jira/browse/IGNITE-10730
 Project: Ignite
  Issue Type: Improvement
  Components: mvcc
Reporter: Igor Seliverstov


on MVCC updates we make changes one by one and WAL records are written straight 
after successful update under the same checkpoint lock. This produces 
contention on wal flush operations and harm overall performance. But we need 
only one guarantee - all the records have to be written into the log before 
prepare start. That means we free to batch such writes (for example 10 rows in 
one data record) it shouldn't cause memory issues but will resolve contention.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Case sensitive indexes question.

[Vladimir Ozerov]

Hi Zhenya,

Yes, will do in the nearest time.

On Sat, Dec 15, 2018 at 8:25 PM Zhenya Stanilovsky
 wrote:

> i fill the ticket  https://issues.apache.org/jira/browse/IGNITE-10654 ,
> Vladimir can u check it?
>
> thanks.
>
>
> >Суббота, 15 декабря 2018, 20:11 +03:00 от stanilovsky evgeny <
> estanilovs...@gridgain.com>:
> >
> >
> >
> >--- Forwarded message ---
> >From: "Vladimir Ozerov" < voze...@gridgain.com >
> >To: dev < dev@ignite.apache.org >
> >Cc:  arzamas...@mail.ru
> >Subject: Re: Case sensitive indexes question.
> >Date: Tue, 04 Dec 2018 15:27:02 +0300
> >
> >I think that this is not an exceptional case, as nothing is broken.
> >Throwing exception may make migration from other systems harder. Warning
> >should be enough.
> >Also remember that all SQL caches already have 1-2 automatic indexes out
> of
> >the box, and we haven't seen much performance complaints about that.
> >Additional duplicate index will not lead to severe performance degradation
> >or system stall.
> >
> >On Fri, Nov 30, 2018 at 3:52 PM Yakov Zhdanov < yzhda...@apache.org >
> wrote:
> >
> >> Zhenya,
> >>
> >> Vladimir suggested not to restrict anything. However, my opinion is to
> >> throw exception on duplicating indexes. We should better add ability to
> >> rename index if it can be useful for anyone. Having same field set
> >> indexed
> >> with same index type is pretty strange and adds a lot of risk for
> >> performance of the system. If this is hard to support in 2.x then let's
> >> do
> >> it in 3.0. Vladimir, what do you think?
> >>
> >> -- Yakov
>
>
> --
> Zhenya Stanilovsky
>

[jira] [Created] (IGNITE-10731) ZookeeperDiscoverySpiTestSuite4: IgniteCacheReplicatedQuerySelfTest.testNodeLeft fails

[Vitaliy Biryukov (JIRA)]

Vitaliy Biryukov created IGNITE-10731:
-

 Summary: ZookeeperDiscoverySpiTestSuite4: 
IgniteCacheReplicatedQuerySelfTest.testNodeLeft fails
 Key: IGNITE-10731
 URL: https://issues.apache.org/jira/browse/IGNITE-10731
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.7
 Environment: {noformat}
junit.framework.AssertionFailedError: 
Expected :0
Actual   :312
 


at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at 
org.apache.ignite.internal.processors.cache.distributed.replicated.IgniteCacheReplicatedQuerySelfTest.testNodeLeft(IgniteCacheReplicatedQuerySelfTest.java:348)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:151)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2102)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2117)
at java.lang.Thread.run(Thread.java:745)

{noformat}
Reporter: Vitaliy Biryukov
Assignee: Vitaliy Biryukov
 Fix For: 2.8






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Suggestion to improve deadlock detection

[Павлухин Иван]

Hi folks,

During implementation of edge-chasing deadlock detection algorithm in
scope of [1] it has been realized that basically we have 2 options for
"chasing" strategy. I will use terms Near when GridNearTxLocal is
assumed and Dht when GridDhtTxLocal (tx which updates primary
partition). So, here is 2 mentioned strategies:
1. Send initial probe when Dht tx blocks waiting for another tx to
release a key. Initial probe is sent from waiting Dht tx to Near tx
holding a lock. If receiving Near tx is blocked as well then it relays
the probe to Dht tx it awaits response from. So, the probe is
traveling like Dht -> Near -> Dht -> Near -> ... Let's call the
approach "forward-only".
2. Send probes only between Near transactions. This approach requires
additional request-response call which Near tx issues to Dht node to
check if tx sending a request is waiting for another tx. The call
returns identifier of a Near tx blocking tx sending a request (if
sending tx is in fact blocked). Then waiting Near tx relays a probe to
blocked Near tx. Let's call that approach "lock-checking".

I would like to define several points to consider while comparing
approaches (please point out more if I miss something):
1. Correctness. Especially regarding reporting false deadlocks.
2. Extensibility. Rollback to savepoint and generalization for classic
transactions should be considered.
3. Messaging overhead.
4. Simplicity of implementation.

You can find an implementation of "lock-checking" approach in PR
attached to the ticket [1]. Currently it works correctly only for
transactions obeying strict two-phase locking (2PL). It is fairly easy
to adopt PR to implement "forward-only" approach. Which will work
correct in conditions of 2PL. "lock-checking" algorithm uses 1.5 times
more messages than "forward-only". Implementation of "forward-only"
algorithm is simpler in a sense that it does not require introducing
lock-wait-check messages and future. But as for me it does not look as
big deal. I think that implementation efforts for both approaches are
almost equal so far.

But corner stone here is an extensibility. Let's imagine that we are
going to implement rollback to savepoint. One suggested approach to
extend "forward-only" approach for that case is invalidating sent
probe. Basically, a tx initiated deadlock check assigns unique id to a
probe. And this probe id is invalidated when the tx wakes up from
wait. If that tx receives back a probe with invalidated id it simply
discards it. If id is not invalidated it means a deadlock. But false
deadlock still can be detected. I will provide an example when it does
not work correctly.

Please note, that our transactions can request multiple locks
simultaneously (so-called AND model). Also rollback to savepoint
releases locks obtained after creating a savepoint. Let's use
following notation. "*" means savepoint. "^" means rollback to
previous savepoint. "!" means acquiring a lock. "?" means waiting for
a lock. "&" means requesting multiple locks simultaneously. See the
following transaction execution scenario for 3 transactions and 3
keys:
TA| TB |   TC
|  |   *
|  1!  |  2!
1? & 3! |   |
|  2? |
|  |  ^
|  |  3!

We can notice that suggested scenario does not introduce deadlock
because locks are requested in consistent order among all
transactions. But in different moments there exist waiting relations
TA w TB, TB w TC, TC w TA. So, we suspect a false deadlock could be
detected here. Also one can notice that a relation TC w TA is
established after TB w TC is destroyed. Messages on a timeline diagram
demonstrates how "forward-only with probe invalidation" approach can
detect a false deadlock here [2]. On the picture L means "lock", U
"unlock", W "wait". Red cross means reporting a false deadlock.
Intuition here is that while probe is in flight one wait-for edge can
be destroyed (TB w TC) and another one can be created (TC w TA). So,
the approach can report false deadlocks. Also, note that a false
deadlock can be found even when locking order is consistent!

After a while I will describe how "lock-checking" can be adopted to
support rollback to savepoint. Disclaimer here is that it will require
even more messages.


I think that we can already start a discussion.
Actually, while false deadlocks can be surprising we perhaps can
tolerate it because our transaction implementation assumes that
transactions can be rolled back by system. "forward-only" approach
requires lesser messages but in big-O terms both approaches have the
same complexity. Both correctness and complexity impact unfortunately
have hardly predictable impact for real workflows. In ideal world
thorough testing could give the answers.

Please, share your vision.

[1] https://issues.apache.org/jira/browse/IGNITE-9322
[2] 
https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/7d1aef9abcd014ec9fd

Re: Set 'TcpDiscoveryVmIpFinder' as default IP finder for tests instead of 'TcpDiscoveryMulticastIpFinder'

[Anton Vinogradov]

Folks,

Now I see 5-10% speedup at all TC suites right after the merge.
Great fix!


On Mon, Dec 17, 2018 at 2:39 PM Vyacheslav Daradur 
wrote:

> Andrey Mashenkov, at first sight, I have not seen any problems with
> .NET platform.
>
> I believe we need carefully configure ports in platform's examples,
> additional actions should not be required.
>
> On Mon, Dec 17, 2018 at 2:35 PM Vyacheslav Daradur 
> wrote:
> >
> > The task [1] is done.
> >
> > 'TcpDiscoveryVmIpFinder' is default IP finder in tests now.
> >
> > The IP finder is initialized on per tests class level to avoid hidden
> > affecting among tests, it means the test methods in the common test
> > class will use the same IP finder.
> >
> > You don't need to set up 'TcpDiscoveryVmIpFinder' in your tests
> > explicitly anymore, also task [2] has been filled to remove related
> > boilerplate if nobody minds.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10555
> > [2] https://issues.apache.org/jira/browse/IGNITE-10715
> >
> >
> > On Wed, Dec 5, 2018 at 7:17 PM Dmitriy Pavlov 
> wrote:
> > >
> > > ++1
> > >
> > > ср, 5 дек. 2018 г. в 18:38, Denis Mekhanikov :
> > >
> > > > Andrey,
> > > >
> > > > Multi-JVM tests may also use a static IP finder, but it should use
> some
> > > > specific port range instead of being shared.
> > > > Something like 127.0.0.1:48500..48509 would do.
> > > >
> > > > Denis
> > > >
> > > > ср, 5 дек. 2018 г. в 18:34, Vyacheslav Daradur  >:
> > > >
> > > > > I filled a task [1].
> > > > >
> > > > > >> Slava, do you think Platforms tests can be fixed as well or one
> more
> > > > > ticket
> > > > > should be created?
> > > > >
> > > > > I'll try to fix them within one ticket, it should be investigated
> a bit
> > > > > deeper.
> > > > >
> > > > > I'll inform about the task's progress in this thread later.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10555
> > > > > On Wed, Dec 5, 2018 at 6:28 PM Andrey Mashenkov
> > > > >  wrote:
> > > > > >
> > > > > > Slava,
> > > > > > +1 for your proposal.
> > > > > > Is there any ticket for this?
> > > > > >
> > > > > > Denis,
> > > > > > I've just read in nabble thread you suggest to allow multicast
> finder
> > > > for
> > > > > > multiJVM tests
> > > > > > and I'd think we shouldn't use multicast in test at all (excepts
> > > > > multicast
> > > > > > Ip finder self tests of course),
> > > > > > but e.g. add an assertion to force user to create ipfinder
> properly.
> > > > > >
> > > > > >
> > > > > > Also, we have a ticket for similar issue in 'examples' module.
> > > > > > Seems, there are some issues with Platforms module integration.
> > > > > > Slava, do you think Platforms tests can be fixed as well or one
> more
> > > > > ticket
> > > > > > should be created?
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6826
> > > > > >
> > > > > > On Wed, Dec 5, 2018 at 5:55 PM Denis Mekhanikov <
> dmekhani...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Slava,
> > > > > > >
> > > > > > > These are exactly my thoughts, so I fully support you here.
> > > > > > > I already wrote about it:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> http://apache-ignite-developers.2346864.n4.nabble.com/IP-finder-in-tests-td33322.html
> > > > > > > But I kind of abandoned this activity. Feel free to take over
> it.
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > ср, 5 дек. 2018 г. в 17:22, Vladimir Ozerov <
> voze...@gridgain.com>:
> > > > > > >
> > > > > > > > Huge +1
> > > > > > > >
> > > > > > > > On Wed, Dec 5, 2018 at 5:09 PM Vyacheslav Daradur <
> > > > > daradu...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > I've found that the project's test framework uses
> > > > > > > > > 'TcpDiscoveryMulticastIpFinder' as default IP finder for
> tests
> > > > and
> > > > > > > > > there are a lot of tests written by Ignite's experts that
> > > > override
> > > > > it
> > > > > > > > > to 'TcpDiscoveryVmIpFinder'.
> > > > > > > > >
> > > > > > > > > Most of our tests starting Ignite nodes in the same JVM,
> that
> > > > > allows
> > > > > > > > > us using shared 'TcpDiscoveryVmIpFinder'.
> > > > > > > > >
> > > > > > > > > I think that using of 'TcpDiscoveryMulticastIpFinder' may
> be
> > > > useful
> > > > > > > > > only in platforms tests, BTW multi-JVM tests use the tuned
> > > > > > > > > 'TcpDiscoveryVmIpFinder'.
> > > > > > > > >
> > > > > > > > > I see the following main advantages of using
> > > > > 'TcpDiscoveryVmIpFinder':
> > > > > > > > > * reducing possible conflicts in the development
> environment,
> > > > when
> > > > > > > > > nodes from different clusters may find each other;
> > > > > > > > > * speedup of nodes initial discovery, especially on
> Windows;
> > > > > > > > > * avoiding of overwriting 'getConfiguration' and copypasta
> only
> > > > to
> > > > > set
> > > > > > > > > up static IP finder in tests;
> > 

Re: Suggestion to improve deadlock detection

[Seliverstov Igor]

Ivan,

I would prefer forward-only implementation even knowing it allows false
positive check results.

Why I think so:

1) From my experience, when we have any future is waiting for reply, we
have to take a failover into consideration.
Usually failover implementations are way more complex than an initial
algorithm itself.
Forward-only version doesn't need any failover implementation as it expects
failed probes (the probes didn't face a deadlock ).

2) Simple lightweight feature implementation is a really good point to
start - it may be extended if needed but really often such extending
doesn't need at all.

3) Any implementation allow false positive result - we are working with
distributed system, there are a bunch of processes happens at the same time
and,
for example, concurrently happened rollback on timeout may solve a deadlock
but probe is finished at this time, so- there is a rollback on deadlock
also as a result.

4) The described case (when false positive result is probable) isn't usual
but, potentially, extremely rare, I don't think we should solve it since it
doesn't break the grid.

Regards,
Igor

вт, 18 дек. 2018 г. в 17:57, Павлухин Иван :

> Hi folks,
>
> During implementation of edge-chasing deadlock detection algorithm in
> scope of [1] it has been realized that basically we have 2 options for
> "chasing" strategy. I will use terms Near when GridNearTxLocal is
> assumed and Dht when GridDhtTxLocal (tx which updates primary
> partition). So, here is 2 mentioned strategies:
> 1. Send initial probe when Dht tx blocks waiting for another tx to
> release a key. Initial probe is sent from waiting Dht tx to Near tx
> holding a lock. If receiving Near tx is blocked as well then it relays
> the probe to Dht tx it awaits response from. So, the probe is
> traveling like Dht -> Near -> Dht -> Near -> ... Let's call the
> approach "forward-only".
> 2. Send probes only between Near transactions. This approach requires
> additional request-response call which Near tx issues to Dht node to
> check if tx sending a request is waiting for another tx. The call
> returns identifier of a Near tx blocking tx sending a request (if
> sending tx is in fact blocked). Then waiting Near tx relays a probe to
> blocked Near tx. Let's call that approach "lock-checking".
>
> I would like to define several points to consider while comparing
> approaches (please point out more if I miss something):
> 1. Correctness. Especially regarding reporting false deadlocks.
> 2. Extensibility. Rollback to savepoint and generalization for classic
> transactions should be considered.
> 3. Messaging overhead.
> 4. Simplicity of implementation.
>
> You can find an implementation of "lock-checking" approach in PR
> attached to the ticket [1]. Currently it works correctly only for
> transactions obeying strict two-phase locking (2PL). It is fairly easy
> to adopt PR to implement "forward-only" approach. Which will work
> correct in conditions of 2PL. "lock-checking" algorithm uses 1.5 times
> more messages than "forward-only". Implementation of "forward-only"
> algorithm is simpler in a sense that it does not require introducing
> lock-wait-check messages and future. But as for me it does not look as
> big deal. I think that implementation efforts for both approaches are
> almost equal so far.
>
> But corner stone here is an extensibility. Let's imagine that we are
> going to implement rollback to savepoint. One suggested approach to
> extend "forward-only" approach for that case is invalidating sent
> probe. Basically, a tx initiated deadlock check assigns unique id to a
> probe. And this probe id is invalidated when the tx wakes up from
> wait. If that tx receives back a probe with invalidated id it simply
> discards it. If id is not invalidated it means a deadlock. But false
> deadlock still can be detected. I will provide an example when it does
> not work correctly.
>
> Please note, that our transactions can request multiple locks
> simultaneously (so-called AND model). Also rollback to savepoint
> releases locks obtained after creating a savepoint. Let's use
> following notation. "*" means savepoint. "^" means rollback to
> previous savepoint. "!" means acquiring a lock. "?" means waiting for
> a lock. "&" means requesting multiple locks simultaneously. See the
> following transaction execution scenario for 3 transactions and 3
> keys:
> TA| TB |   TC
> |  |   *
> |  1!  |  2!
> 1? & 3! |   |
> |  2? |
> |  |  ^
> |  |  3!
>
> We can notice that suggested scenario does not introduce deadlock
> because locks are requested in consistent order among all
> transactions. But in different moments there exist waiting relations
> TA w TB, TB w TC, TC w TA. So, we suspect a false deadlock could be
> detected here. Also one can notice that a relation TC w TA is
> established after TB w TC is destroyed. Messages on a timeline diagram
>

[jira] [Created] (IGNITE-10732) Incorrect file.encoding leads to inconsistent SqlFieldsQuery results between nodes

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-10732:


 Summary: Incorrect file.encoding leads to inconsistent 
SqlFieldsQuery results between nodes
 Key: IGNITE-10732
 URL: https://issues.apache.org/jira/browse/IGNITE-10732
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev


When doing 
{code}
cache.query(new SqlFieldsQuery("SELECT _key FROM Cache"))
{code}
resulting Unicode values may be different when coming from Windows or Linux 
node.
Linux nodes will mostly use UTF-8 but Windows nodes will use local Cp 
encoding to encode query results, as bizzare as it may sound.
Windows <-> Windows and Linux <-> Linux will get correct result but Windows <-> 
Linux will get broken strings.
Note that if cluster has Windows and Linux nodes and cache is REPLICATED, 
results will be different for subsequent queries!

There is a workaround for this: set -Dfile.encoding=UTF-8 JVM arg on Windows.
There is probably an underlying problem in H2 but since non-UTF-8 file.encoding 
is dangerous (it affects String.getBytes()) I think we should peg it to UTF-8.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Code inspection

[Dmitriy Pavlov]

Both patches were applied. Maxim, thank you!

What about 1. An `Unexpected error during build messages processing in
TeamCity`, what can we do as the next step to fix it?

Sincerely,
Dmitriy Pavlov

пн, 17 дек. 2018 г. в 18:31, Andrey Mashenkov :

> Maxim,
>
> Looks ok. Let's apply IGNITE-10682.
>
> All,
>
> Also, I'd like to publish idea.logs into artefacts by default.
> This will give us more details for investigation in future if any failure
> will occurs.
> It will costs 1-10 kB.
>
> On Mon, Dec 17, 2018 at 3:21 PM Maxim Muzafarov 
> wrote:
>
> > Dmitry,
> >
> > It seems to me that we have two independent issues here.
> > 1. An `Unexpected error during build messages processing in TeamCity`
> > error message which is related to TC agent configuration. Suppose,
> > server.log will provide us more details about it. I have to access
> > there.
> > 2. A new set of inspection rules was introduced in 2018+ IntelliJ IDEA
> > and they should be disabled in our ignite_inspections_teamcity.xml
> > configuration file. They are not fixed in the Apache Ignite project
> > code yet. I've prepared the issue [1] for it. Please, take a look.
> >
> >
> > Andrey,
> >
> > I've fixed disabled plugins file according to your suggestions. The
> > issue [2] is ready. I've re-run `Excluded [Inspections] Core Debug`
> > suite and the log details show me that now only 3 plugins are enabled:
> > IDEA CORE, Maven Integration, Properties Support. It seems to me that
> > it's correct.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10709
> > [2] https://issues.apache.org/jira/browse/IGNITE-10682
> >
> > On Sat, 15 Dec 2018 at 15:22, Dmitriy Pavlov  wrote:
> > >
> > > Folks,
> > >
> > > There is a strange error on TC
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=2556875&buildTypeId=IgniteTests24Java8_InspectionsCore
> > >
> > > It appeared after TC update to the latest version.
> > >
> > > Sincerely,
> > > Dmitry Pavlov
> > >
> > > пт, 14 дек. 2018 г. в 16:09, Andrey Mashenkov <
> > andrey.mashen...@gmail.com>:
> > >
> > > > Maxim,
> > > >
> > > > PR is incomplete. Some plugins should be disabled with different
> > id\name.
> > > > Maven plugin shouldn't be disabled as Idea Inspector use it to use
> > Ignite
> > > > project pom file.
> > > >
> > > > Please, find details in ticket.
> > > >
> > > >
> > > > On Fri, Dec 14, 2018 at 12:00 PM Andrey Mashenkov <
> > > > andrey.mashen...@gmail.com> wrote:
> > > >
> > > > > Maxim,
> > > > >
> > > > > Thanks, I'll check PR and let you know about results.
> > > > >
> > > > > For now, Inspections task execution time looks much better (15-22
> > min),
> > > > > but fluctuation is still noticeable.
> > > > >
> > > > > On Fri, Dec 14, 2018 at 11:13 AM Maxim Muzafarov <
> maxmu...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Andrey,
> > > > >>
> > > > >> Thanks! I've consulted with the IntelliJ IDEA source code and
> found
> > > > >> how this disabled plugins file should look like. I've created a
> new
> > > > >> issue [1] and prepared PR [2] with the set of disabled plugins
> > (maybe
> > > > >> not complete set). I don't have access to change corresponding
> > > > >> `~Excluded [Inspections] Core Debug` test suite properties.
> > > > >> Can we test this PR?
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/IGNITE-10682
> > > > >> [2] https://github.com/apache/ignite/pull/5666
> > > > >> On Thu, 13 Dec 2018 at 17:35, Andrey Mashenkov
> > > > >>  wrote:
> > > > >> >
> > > > >> > Maxim,
> > > > >> >
> > > > >> > Idea has a file in config directory
> ./config/disabled_plugins.txt
> > ,
> > > > you
> > > > >> can easily find it at you local machine.
> > > > >> > Teamcity Inspections runner has an option "Disabled plugins"
> where
> > > > >> disabled_plugins.txt file content can be set.
> > > > >> >
> > > > >> > So, looks like we can disable useless plugins.
> > > > >> > But I'm not expert and can't suggest changes we can safely
> apply.
> > > > >> >
> > > > >> > On Thu, Dec 13, 2018 at 4:59 PM Maxim Muzafarov <
> > maxmu...@gmail.com>
> > > > >> wrote:
> > > > >> >>
> > > > >> >> Andrey,
> > > > >> >>
> > > > >> >> Thank you for solving this issue with GC pauses! I've checked
> the
> > > > >> >> given report. The inspections configuration is correct, but it
> > seems
> > > > >> >> to me that we have enabled by default rules of included plugins
> > (for
> > > > >> >> instance, KotlinInternalInJava in the report is enabled).
> > > > >> >>
> > > > >> >> Can you share more details about `disable plugin` option you
> > found?
> > > > >> >>
> > > > >> >> I see that idea instance starts with the default
> > -Didea.plugins.path
> > > > >> >> system property, can we change it so the plugins will be not
> > loaded
> > > > by
> > > > >> >> default?
> > > > >> >> On Thu, 13 Dec 2018 at 15:45, Andrey Mashenkov
> > > > >> >>  wrote:
> > > > >> >> >
> > > > >> >> > Maxim,
> > > > >> >> >
> > > > >> >> > It looks like we can't make logs more verbose due to possible
> > bug

Re: Async debugging in IDEA

[Anton Vinogradov]

Folks,

That's extremely awesome!
Now I see what stacktrace cause future creation and who will gain the
result and how.

Are there any chances to keep variable's values at original stacktrace?

Will this work correctly in case of concurrent calls?
for example, when more that one stacktrace will cause same async op.
Will this feature guarantee that I see exactly that what caused this async,
not one of possible?

Cons.
Is there any chances now to see the whole stacktrace starting from worker
got a message from the queue?

On Fri, Dec 14, 2018 at 6:53 PM Dmitriy Pavlov  wrote:

> Looks cool, I will try
>
> пт, 14 дек. 2018 г. в 16:46, Павлухин Иван :
>
> > Hi Igniters,
> >
> > I would like to share with you that now it is possible to use IDEA
> > async debugger for debugging Ignite futures. And we should thank Ivan
> > Bessonov for that awesome contribution [1].
> >
> > Most of your might be already familiar with async debugger in IDEA.
> > For the rest I prepared two screenshots [2], [3]. We can see regular
> > stacktrace on breakpoint hit in future completion callback which is
> > extended with an additional stacktrace showing a path where the
> > callback was installed.
> >
> > Thank you Ivan!
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10475
> > [2]
> >
> https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/bfe32b8c11a67ccc3bcbbb8b398e39a8a818cad6/adebug1.png
> > [3]
> >
> https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/bfe32b8c11a67ccc3bcbbb8b398e39a8a818cad6/adebug2.png
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

[jira] [Created] (IGNITE-10733) CassandraStore in write behind mode loses data when back-pressure is active

[Dmitry Konstantinov (JIRA)]

Dmitry Konstantinov created IGNITE-10733:


 Summary: CassandraStore in write behind mode loses data when 
back-pressure is active
 Key: IGNITE-10733
 URL: https://issues.apache.org/jira/browse/IGNITE-10733
 Project: Ignite
  Issue Type: Bug
  Components: cache, cassandra
Affects Versions: 2.7, 2.6, 2.5
Reporter: Dmitry Konstantinov


Whe



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Is it time to move forward to JUnit4 (5)?

[oignatenko]

Hi Ivan,

To answer your last question, yes, all the tests are to be marked with @Test
annotations, and those that are meant to be ignored are going to be marked
with @Ignore annotations. There is no exceptions to that, and these
annotations will work just as well on tests using our home brewed
beforeTestsStarted, beforeTest, afterTest, afterTestsStarted.

For the sake of completeness, developers will also be able to use Before* /
After* annotations on their own methods in tests. The only exception
(clarified in respective javadocs) is that these annotations shouldn't be
used on overrides of our home brewed methods - and these methods, in
addition, will be recommended (not mandated) to avoid wia deprecation
notices.

-

As for accessing tests which have problems running under junit4, the way how
I search for these in current master is regex search in IDEA for
"addTestSuite.*class", that is I search in testsuites for entries that are
added using method "addTestSuite" with parameter class.

Probably worth noting that some of the problems that were previously
blocking addition of particular tests have been resolved in the course of
working on IGNITE-10177 (https://github.com/apache/ignite/pull/5615). One
riddle that currently looks particularly difficult to crack is Teamcity
failures in "Queries 1", I even haven't yet figured how to reproduce these
locally. 

regards, Oleg


Павлухин Иван wrote
> Hi Oleg,
> 
> Now concerns about junit3 elimination are clear for me. And I agree
> that it is worth to do it. By the way is it possible to access tests
> which have problems running under junit4? I would like to take a look.
> 
> Also a clarifying bit regarding next migration steps is needed. Sorry
> if it was described but I missed it. Currently we have tons of tests
> which rely on our home brewed beforeTestsStarted, beforeTest,
> afterTest, afterTestsStarted. Are you going to mark them all with
> corresponding junit4 annotations?
> пн, 17 дек. 2018 г. в 19:13, oignatenko <

> oignatenko@

> >:
>>
>> Hi Ivan,
>>
>> Per my cursory check of the code currently in master, we have 239 test
>> classes that couldn't migrate (that's probably about 500 or something
>> test
>> cases). For comparison, over 1600 classes migrated without problems. This
>> is
>> to answer your questions about how many tests are affected by change and
>> how many tests were not migrated yet.
>>
>> -
>>
>> I think we can say that only small percent of tests turned out sensitive
>> to
>> JUnit framework change.
>>
>> In the same time, due to very large overall amount of our tests we have
>> quite a big number as an absolute value. I point this because if amount
>> of
>> troublesome tests was smaller (say, order of magnitude smaller) I would
>> consider delaying migration until all tests reworked to blend totally
>> seamlessly into new version JUnit. This is doable - as sort of "pilot
>> exercise" me and Ed adjusted one test case this way - but the sheer
>> amount
>> of work on 200+ classes raises a question, whether it is worth it (I
>> think
>> it isn't).
>>
>> With regards to question 2, changes to TestCounters are farly trivial
>> (and
>> necessary) if you look at the code diff. Prior code counted amount of
>> test
>> cases in the class by JUnit 3 convention: it picked public void methods
>> without parameters starting with "test". Changed code counts test cases
>> as
>> JUnit 4 does: by checking if method is annotated with @Test. Change is
>> necessary because it will allow test developers fully use JUnit 4
>> features
>> by adding test cases that don't follow old naming requirement. I
>> experimented with it a little and as far as I could see the old
>> TestCounters
>> indeed break when there are methods following new convention, it
>> triggered
>> afterTestsStopped too early.
>>
>> The answer to your question 3 lies in javadocs I added to runSerializer
>> declaration, or, more precisely, in TestCounters javadoc referred from
>> there. As you can see, current plan is to get rid of TestCounters fairly
>> soon (per IGNITE-10179) and this will also get rid of runSerializer so
>> this
>> is merely a temporary band aid to be removed soon.
>>
>> For the sake of completeness, my initial plan was to thoroughly
>> investigate
>> and test whether change from JUnit 3 to JUnit 4 would keep accessing
>> counters safe or not and only introduce runSerializer if it really does
>> but
>> after realising that this whole thing is likely to go away in a few days
>> from now I decided that it isn't worth wasting time and just temporarily
>> made it the way that is waterproof guaranteed to be safe.
>>
>> -
>>
>> Now, to answer your question whether it is an option for us to keep part
>> of
>> tests under JUnit 3, my answer is most definitely no.
>>
>> Main reason is that having part of tests under JUnit 3 will deprive us
>> ability to consistently use Ignore annotation and force us fallback to
>> old
>> way to fail the tests we want to ignore. This wou

[jira] [Created] (IGNITE-10734) Add documentation for the list of operations that should be retried in case of cluster topology changes

[Evgenii Zhuravlev (JIRA)]

Evgenii Zhuravlev created IGNITE-10734:
--

 Summary: Add documentation for the list of operations that should 
be retried in case of cluster topology changes
 Key: IGNITE-10734
 URL: https://issues.apache.org/jira/browse/IGNITE-10734
 Project: Ignite
  Issue Type: Improvement
Reporter: Evgenii Zhuravlev
Assignee: Artem Budnikov


Some of the operations, like get or getAll would throw ClusterTopologyException 
if primary node left topology, while other operations not. So, some operations 
should be re-tried from user code, while some operation will do it internally. 
We should prepare documentation for the list of these operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10735) Web console: use rxjs EMPTY instead of empty

[Ilya Borisov (JIRA)]

Ilya Borisov created IGNITE-10735:
-

 Summary: Web console: use rxjs EMPTY instead of empty
 Key: IGNITE-10735
 URL: https://issues.apache.org/jira/browse/IGNITE-10735
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Ilya Borisov
Assignee: Ilya Borisov


RxJS 6 [has 
deprecated|https://rxjs-dev.firebaseapp.com/api/index/function/empty] empty, 
EMPTY constant should be used instead, so let's do that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10736) Update tabs in

[Alexander Kalinin (JIRA)]

Alexander Kalinin created IGNITE-10736:
--

 Summary: Update tabs in
 Key: IGNITE-10736
 URL: https://issues.apache.org/jira/browse/IGNITE-10736
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Alexander Kalinin
Assignee: Alexander Kalinin


Now tabs in page-queries home is hardcoded in template, but one should be set 
dinamically in controller and iterated in template.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)