how do you create your session? do you reuse it across threads? how do you create/close session manager? look for the problem in session creation, probably something deadlocked, as far as I remember hib.session should be created per thread
On 6 September 2015 at 07:11, Zoran Jeremic <[email protected]> wrote: > Hi, > > I'm developing long running process that should find RSS feeds that all > users in the system have registered to follow, parse these RSS feeds, > extract new entries and store it back to the database as Hibernate > entities, so user can retrieve it. I want to use Apache Spark to enable > parallel processing, since this process might take several hours depending > on the number of users. > > The approach I thought should work was to use > *useridsRDD.foreachPartition*, so I can have separate hibernate session > for each partition. I created Database session manager that is initialized > for each partition which keeps hibernate session alive until the process is > over. > > Once all RSS feeds from one source are parsed and Feed entities are > created, I'm sending the whole list to Database Manager method that saves > the whole list in batch: > >> public <T extends BaseEntity> void saveInBatch(List<T> entities) { >> try{ >> boolean isActive = session.getTransaction().isActive(); >> if ( !isActive) { >> session.beginTransaction(); >> } >> for(Object entity:entities){ >> session.save(entity); >> } >> session.getTransaction().commit(); >> }catch(Exception ex){ >> if(session.getTransaction()!=null) { >> session.getTransaction().rollback(); >> ex.printStackTrace(); >> } >> } >> >> However, this works only if I have one Spark partition. If there are two > or more partitions, the whole process is blocked once I try to save the > first entity. In order to make the things simpler, I tried to simplify Feed > entity, so it doesn't refer and is not referred from any other entity. It > also doesn't have any collection. > > I hope that some of you have already tried something similar and could > give me idea how to solve this problem > > Thanks, > Zoran > >
