Using Spark like a search engine

2015-05-24 Thread Сергей Мелехин
HI! We are developing scoring system for recruitment. Recruiter enters vacancy requirements, and we score tens of thousands of CVs to this requirements, and return e.g. top 10 matches. We do not use fulltext search and sometimes even dont filter input CVs prior to scoring (some vacancies do not hav

Re: Using Spark like a search engine

2015-05-25 Thread Сергей Мелехин
many CVs, and return top 10 matches. С Уважением, Сергей Мелехин. 2015-05-25 16:28 GMT+10:00 ankur chauhan : > Hi, > > I am sure you can use spark for this but it seems like a problem that > should be delegated to a text based indexing technology like elastic search > or somethin

Re: Using Spark like a search engine

2015-05-25 Thread Сергей Мелехин
Thanks I'll give it a try! С Уважением, Сергей Мелехин. 2015-05-26 12:56 GMT+10:00 Alex Chavez : > Сергей, > A simple implementation would be to create a DataFrame of CVs by issuing a > Spark SQL query against your Postgres database, persist it in memory, and > then to map F