[
https://issues.apache.org/jira/browse/SPARK-20180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952828#comment-15952828
]
Cyril de Vogelaere commented on SPARK-20180:
--------------------------------------------
I'm not arguing for no max at all. Just for a special value (0) which allow a
user to find all pattern of any length.
Let's give a practical example, let's say i'm not the brightest user. I have a
really big dataset I need to analyse, but because I'm not too bright, I don't
run any analysis on that dataset, to know what the max possible length of a
sequence would be. I look at the parameters and I tell myself that a maxlength
of like 50 will be enough. I code quickly and launch it using spark's algo.
I wait a few day for the result, since it was a really big dataset, and I see
that I have a few solution pattern of length 50. And there is the problem, now
that I'm there, do I need to re-run everything because there was a larger
pattern that I wanted to find, or was 50 really the limit ?
This may waste a lot of time for some people if that happens to them. So I want
to create that special value (0) for maxpattern length. So that when the
algorithm ends, you get all the patterns of any length, no doubt possible.
Now honestly, while I want to say that this change may be usefull, a carefull
user can always set the maxpattern length at the max Integer value and, given
the time the algorithm needs to run on a small dataset, he will never have any
problem since it would take him months to get a pattern longer than that, even
with the biggest dataset we can imagine and lot's of computing power. So this
is not a mandatory feature, simply something I feel would be nice to have.
Also, in case there really was someone crazy enough to run this algorithm with
for goal to find pattern longer than the max for integer, that special value
would allow him to find them for sure.
I also would like to advocate for setting that special value (0) as the default
value, since the first time I ran a test was on a dataset that returned long
pattern (kosarak or protein, I don't remember which of the two) and thanks to
the default pattern length it finished super quickly and I tought I had a
performance improvement, in comparison to another algorithm that allowed
unllimited pattern length as default.
Took me two day to realise the default parameter was set to ten (guess I was
the dumb user there. Not my proudest moment, I will admit). So I want to
advocate for that 0 value as default, but I get for backward compatibility it
may not be best to change the default behavior. So I want a senior's opinion,
to know if changing that would be ok.
Again, we use pull requests to propose changes.
=> I know, (I read the contributor guide), but I'm waiting for the test to
finish. I tested my code and had some error, which I'm pretty sure were
unrelated to the few line of code I added. I'm relaunching it though, on the
code without my changes, if the error are the same (And you feel like the
changes I'm proposing are worth it) I will make the pull request. That way, I'm
sure I don't waste anybody's time on reviewing code nobody (but the little
inexperienced newbie I am) would feel usefull.
So there we go, I hope that answer your questions.
If not drop me another message, I will answer as best as I can. :)
> Unlimited max pattern length in Prefix span
> -------------------------------------------
>
> Key: SPARK-20180
> URL: https://issues.apache.org/jira/browse/SPARK-20180
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.1.0
> Reporter: Cyril de Vogelaere
> Priority: Minor
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> Right now, we need to use .setMaxPatternLength() method to
> specify is the maximum pattern length of a sequence. Any pattern longer than
> that won't be outputted.
> The current default maxPatternlength value being 10.
> This should be changed so that with input 0, all pattern of any length would
> be outputted. Additionally, the default value should be changed to 0, so that
> a new user could find all patterns in his dataset without looking at this
> parameter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
