Please read job instead task
2017-07-25 15:20 GMT+03:00 Alexei Scherbakov <alexey.scherbak...@gmail.com>:
> Main point of the issue is to provide clean API for working with
> computations requiring data collocation
>
> affinityCall/Run provide the ability to run closure near data, but
> map/reduce API is a way reacher: continuous mapping, task session, etc.
>
> As for proposed API, I do not understand fully how it solves the problem.
>
> Maxim, please provide detailed javadoc for each method and each argument
> for presented API, and the answers to the following questions:
>
> 1. How would task know the partition it is running over ?
>
> 2. How can I assign task for each cache partition ?
>
> 3. How can I enforce partition reservation if task works with multiple
> caches at once ?
>
>
>
>
>
> 2017-07-25 12:30 GMT+03:00 Anton Vinogradov <avinogra...@gridgain.com>:
>
>> Val,
>>
>> Sure, we can, but we'd like to use map/reduce without fearing that
>> topology
>> can change.
>>
>> On Mon, Jul 24, 2017 at 11:17 PM, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>
>> > Anton,
>> >
>> > You can call affinityCallAsync multiple times and then reduce locally.
>> >
>> > -Val
>> >
>> > On Mon, Jul 24, 2017 at 3:05 AM, Anton Vinogradov <
>> > avinogra...@gridgain.com>
>> > wrote:
>> >
>> > > Val,
>> > >
>> > > > What is the use case for which current affinityRun/Call API doesn't
>> > work?
>> > > It does not work for map/reduce.
>> > >
>> > > On Fri, Jul 21, 2017 at 11:42 PM, Valentin Kulichenko <
>> > > valentin.kuliche...@gmail.com> wrote:
>> > >
>> > > > Maxim,
>> > > >
>> > > > The issue is that it's currently assumed to support job mapping,
>> but it
>> > > > actually doesn't. However, I agree that AffinityKeyMapped annotation
>> > > > doesn't fit the use case well. Let's fix documentation and JavaDoc
>> > then.
>> > > >
>> > > > As for the proposed API, it's overcomplicated, took me 15 minutes to
>> > > > understand what it does :)
>> > > >
>> > > > What is the use case for which current affinityRun/Call API doesn't
>> > work?
>> > > >
>> > > > -Val
>> > > >
>> > > > On Fri, Jul 21, 2017 at 5:57 AM, Kozlov Maxim <dreamx....@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > Valentin,
>> > > > >
>> > > > > The author of tiket wants to see to provide some API allows to map
>> > > > > ComputeJobs to partitions or keys. If we use @AffinityKeyMapped
>> then
>> > > you
>> > > > > need to enter the cache name parameter, I think this is not
>> > convenient
>> > > > for
>> > > > > the user. Therefore, I propose to extend the existing API.
>> > > > > Having consulted with Anton V. decided to make a separate
>> interface
>> > > > > ReducibleTask, which will allow us to have different map logic at
>> > each
>> > > > > inheritor.
>> > > > >
>> > > > > Old method, allows to map to node
>> > > > > public interface ComputeTask<T, R> extends ReducibleTask<R> {
>> > > > > @Nullable public Map<? extends ComputeJob, ClusterNode>
>> > > > > map(List<ClusterNode> subgrid, @Nullable T arg) throws
>> > IgniteException;
>> > > > > }
>> > > > >
>> > > > > Brand new method with mapping to partitions, which solves topology
>> > > change
>> > > > > issues.
>> > > > > public interface AffinityComputeTask<T, R> extends
>> ReducibleTask<R> {
>> > > > > @Nullable public Map<? extends ComputeJob, Integer>
>> > > > map(@NotnullString
>> > > > > cacheName, List<Integer> partIds, @Nullable T arg) throws
>> > > > IgniteException;
>> > > > > }
>> > > > >
>> > > > > public interface ReducibleTask<R> extends Serializable {
>> > > > > public ComputeJobResultPolicy result(ComputeJobResult res,
>> > > > > List<ComputeJobResult> rcvd) throws IgniteException;
>> > > > >
>> > > > > @Nullable public R reduce(List<ComputeJobResult> results)
>> throws
>> > > > > IgniteException;
>> > > > > }
>> > > > >
>> > > > > We also need to implement AffinityComputeTaskAdapter and
>> > > > > AffinityComputeTaskSplitAdapter, for implementation by default.
>> It
>> > is
>> > > > > right?
>> > > > >
>> > > > > In the IgniteCompute add:
>> > > > >
>> > > > > @IgniteAsyncSupported
>> > > > > public <T, R> R affinityExecute(Class<? extends
>> > AffinityComputeTask<T,
>> > > > R>>
>> > > > > taskCls, List<Integer> partIds, @Nullable T arg) throws
>> > > IgniteException;
>> > > > > @IgniteAsyncSupported
>> > > > > public <T, R> R affinityExecute(AffinityComputeTask<T, R> task,
>> > > > > List<Integer> partIds, @Nullable T arg) throws IgniteException;
>> > > > >
>> > > > > public <T, R> ComputeTaskFuture<R> affinityExecuteAsync(Class<?
>> > extends
>> > > > > AffinityComputeTask<T, R>> taskCls, List<Integer> partIds,
>> @Nullable
>> > T
>> > > > arg)
>> > > > > throws IgniteException;
>> > > > > public <T, R> ComputeTaskFuture<R> affinityExecuteAsync(
>> > > > AffinityComputeTask<T,
>> > > > > R> task, List<Integer> partIds, @Nullable T arg) throws
>> > > IgniteException;
>> > > > >
>> > > > >
>> > > > > How do you like this idea or do you insist that you need to use
>> > > > > @AffinityKeyMapped to solve the problem?
>> > > > >
>> > > > >
>> > > > > > 13 июля 2017 г., в 6:36, Valentin Kulichenko <
>> > > > > valentin.kuliche...@gmail.com> написал(а):
>> > > > > >
>> > > > > > Hi Max,
>> > > > > >
>> > > > > > This ticket doesn't assume any API changes, it's about broken
>> > > > > > functionality. I would start with checking what tests we have
>> > > > > > for @AffinityKeyMapped and creating missing one. From what I
>> > > understand
>> > > > > > functionality is broken completely or almost completely, so I
>> guess
>> > > > > testing
>> > > > > > coverage is very weak there.
>> > > > > >
>> > > > > > -Val
>> > > > > >
>> > > > > > On Wed, Jul 12, 2017 at 4:27 PM, Kozlov Maxim <
>> > dreamx....@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > >> Igniters,
>> > > > > >>
>> > > > > >> jira: https://issues.apache.org/jira/browse/IGNITE-5037 <
>> > > > > >> https://issues.apache.org/jira/browse/IGNITE-5037>
>> > > > > >> How do you look to solve this ticket by adding two methods to
>> the
>> > > > public
>> > > > > >> IgniteCompute API?
>> > > > > >>
>> > > > > >> @IgniteAsyncSupported
>> > > > > >> public void affinityRun(@NotNull Collection<String> cacheNames,
>> > > > > >> Collection<Object> keys, IgniteRunnable job)
>> > > > > >> throws IgniteException;
>> > > > > >>
>> > > > > >> @IgniteAsyncSupported
>> > > > > >> public <R> R affinityCall(@NotNull Collection<String>
>> cacheNames,
>> > > > > >> Collection<Object> keys, IgniteCallable<R> job)
>> > > > > >> throws IgniteException;
>> > > > > >>
>> > > > > >> There is also a question of how to act when changing the
>> topology
>> > > > during
>> > > > > >> the execution of the job.
>> > > > > >> 1) complete with an exception;
>> > > > > >> 2) stop execution and wait until the topology is rebuilt and
>> > > continue
>> > > > > >> execution;
>> > > > > >>
>> > > > > >> I think the second way, do you think?
>> > > > > >>
>> > > > > >> --
>> > > > > >> Best Regards,
>> > > > > >> Max K.
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > > --
>> > > > > Best Regards,
>> > > > > Max K.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
--
Best regards,
Alexei Scherbakov
