Hi Debasish Das,
I found that there are lots of much useful information for me in your kind
reply.However, I am sorry that still I could not exactly catch each words you
said.
I just know spark mllib will use breeze as its underlying package, however, I
did not practise and do not know how to directly apply breeze in spark
program.What is and where is nnls in mllib optimization, and what is
spark-tfocs package , I do not find them in spark official website...Where is
to use breeze nonlinearminimizer... Next I will also meet with optimized
function with nonlinear constraint functions.
I am sorry that I do not know whether it is convenient for you to comment some
more for my above issues.
Thank you in advance.
Zhiliang
On Wednesday, November 4, 2015 2:25 AM, Debasish Das
<[email protected]> wrote:
Spark has nnls in mllib optimization. I have refactored nnls to breeze as well
but we could not move out nnls from mllib due to some runtime issues from
breeze.Issue in spark or breeze nnls is that it takes dense gram matrix which
does not scale if rank is high but it has been working fine for nnmf till 400
rank.I agree with Sean that you need to see if really simplex is needed. Many
constraints can be formulated as proximal operator and then you can use breeze
nonlinearminimizer or spark-tfocs package if it is stable.On Nov 2, 2015 10:13
AM, "Sean Owen" <[email protected]> wrote:
I might be steering this a bit off topic: does this need the simplex
method? this is just an instance of nonnegative least squares. I don't
think it relates to LDA either.
Spark doesn't have any particular support for NNLS (right?) or simplex though.
On Mon, Nov 2, 2015 at 6:03 PM, Debasish Das <[email protected]> wrote:
> Use breeze simplex which inturn uses apache maths simplex...if you want to
> use interior point method you can use ecos
> https://github.com/embotech/ecos-java-scala ...spark summit 2014 talk on
> quadratic solver in matrix factorization will show you example integration
> with spark. ecos runs as jni process in every executor.
>
> On Nov 1, 2015 9:52 AM, "Zhiliang Zhu" <[email protected]> wrote:
>>
>> Hi Ted Yu,
>>
>> Thanks very much for your kind reply.
>> Do you just mean that in spark there is no specific package for simplex
>> method?
>>
>> Then I may try to fix it by myself, do not decide whether it is convenient
>> to finish by spark, before finally fix it.
>>
>> Thank you,
>> Zhiliang
>>
>>
>>
>>
>> On Monday, November 2, 2015 1:43 AM, Ted Yu <[email protected]> wrote:
>>
>>
>> A brief search in code base shows the following:
>>
>> TODO: Add simplex constraints to allow alpha in (0,1).
>> ./mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala
>>
>> I guess the answer to your question is no.
>>
>> FYI
>>
>> On Sun, Nov 1, 2015 at 9:37 AM, Zhiliang Zhu <[email protected]>
>> wrote:
>>
>> Dear All,
>>
>> As I am facing some typical linear programming issue, and I know simplex
>> method is specific in solving LP question,
>> I am very sorry that whether there is already some mature package in spark
>> about simplex method...
>>
>> Thank you very much~
>> Best Wishes!
>> Zhiliang
>>
>>
>>
>>
>>
>
