@Shahab One of the thing I tried is 1) I set auto commit to True 2) reduced the default batch size to 1 and change condition from > to >= for executing it (count>=batchsize)
and this help me in this way as I set batch size to 1 each query is executed instead of pile up it into the batch and as auto-commit is true it will commit it immediately after execution. And this work for me.... *Dont know the pros and cons of it..... * Now will try for no_multiquery option. On Tue, May 28, 2013 at 7:33 AM, Shahab Yunus <[email protected]>wrote: > @Hardik, > > Try to run your run script with 'no_multiquery ' option: > > pig -no_multiquery myscript.pig > > Regards, > Shahab > > > > On Mon, May 27, 2013 at 8:41 AM, Hardik Shah <[email protected]> > wrote: > > > DBStorage is not working with other storage in pig script. means > DBStorage > > is not working with multiple storage statement. > > > > What I was trying for: 1) I was trying to Store one output using > DBStorage > > And was trying to store same or different output using simple Store to > file > > system 2) I also tried to store using DBStorage and using my custom store > > function > > > > But in both cases it not storing the data to database. If I comment out > > another storage than DBStorage is working properly. > > > > Even its not throwing any exception or error on reducer's machine.. > > > > Can anyone point out the problem? > > > > DBStorage is not working with Simple Store to file system. Its only > working > > if I put only DBStorage no other Store Statement.. > > > > pv_by_industry = GROUP profile_view by viewee_industry_id > > > > pv_avg_by_industry = FOREACH pv_by_industry GENERATE > > group as viewee_industry_id, AVG(profie_view) AS average_pv; > > > > STORE pv_avg_by_industry INTO '/tmp/hardik'; > > > > STORE pv_avg_by_industry into /tmp/hardik/db' INTO > > DBStorage('com.mysql.jdbc.Driver', > > 'dbc:mysql://hostname/dbname', 'user', > > 'pass', > > 'INSERT INTO table (viewee_industry_id,average_pv) VALUES(?,?)'); > > > > > > > > > > Few things are came into picture when I was debugging it. > > > > DBStorage is setting Auto commit to False. > > So when the batch is executed it will not be auto committed. > > > > After executing batch OutputCommiter's method commitTask in DBStorage > > (inline class' method) was called in which commit is written > > > > if (ps != null) { > > try {System.out.println("Executing Batch in commitTask"); > > ps.executeBatch(); > > con.commit(); > > ps.close(); > > con.close(); > > ps = null; > > con = null; > > } catch (SQLException e) {System.out.println("Exception in > > commitTask"); > > log.error("ps.close", e); > > throw new IOException("JDBC Error", e); > > } > > > > and this method is called by PigOutputCommiter > > > > public void commitTask(TaskAttemptContext context) throws IOException { > > if(HadoopShims.isMap(context.getTaskAttemptID())) { > > for (Pair<OutputCommitter, POStore> mapCommitter : > > mapOutputCommitters) { > > if (mapCommitter.first!=null) { > > TaskAttemptContext updatedContext = > > setUpContext(context, > > mapCommitter.second); > > mapCommitter.first.commitTask(updatedContext); > > } > > } > > } else { > > for (Pair<OutputCommitter, POStore> reduceCommitter : > > reduceOutputCommitters) { > > if (reduceCommitter.first!=null) { > > TaskAttemptContext updatedContext = > > setUpContext(context, > > reduceCommitter.second); > > reduceCommitter.first.commitTask(updatedContext); > > } > > } > > } > > } > > > > > > But when this commitTask is called its connection and preparedStatment > > Objects become null... so it ll not commit so data is not available in > > Database..... > > > > > > > > But if you write only DBStorage without any other Store statement in > script > > it will work properly.. > > > > > > Any clues??? > > > > > > > > > > > > > > > > -- > > Regards, > > Hardik Shah. > > > -- Regards, Hardik Shah.
