I'd recommend to try Sqoop for RDBMS-related tasks.
On Mon, May 27, 2013 at 4:41 PM, Hardik Shah <[email protected]> wrote: > DBStorage is not working with other storage in pig script. means DBStorage > is not working with multiple storage statement. > > What I was trying for: 1) I was trying to Store one output using DBStorage > And was trying to store same or different output using simple Store to file > system 2) I also tried to store using DBStorage and using my custom store > function > > But in both cases it not storing the data to database. If I comment out > another storage than DBStorage is working properly. > > Even its not throwing any exception or error on reducer's machine.. > > Can anyone point out the problem? > > DBStorage is not working with Simple Store to file system. Its only working > if I put only DBStorage no other Store Statement.. > > pv_by_industry = GROUP profile_view by viewee_industry_id > > pv_avg_by_industry = FOREACH pv_by_industry GENERATE > group as viewee_industry_id, AVG(profie_view) AS average_pv; > > STORE pv_avg_by_industry INTO '/tmp/hardik'; > > STORE pv_avg_by_industry into /tmp/hardik/db' INTO > DBStorage('com.mysql.jdbc.Driver', > 'dbc:mysql://hostname/dbname', 'user', > 'pass', > 'INSERT INTO table (viewee_industry_id,average_pv) VALUES(?,?)'); > > > > > Few things are came into picture when I was debugging it. > > DBStorage is setting Auto commit to False. > So when the batch is executed it will not be auto committed. > > After executing batch OutputCommiter's method commitTask in DBStorage > (inline class' method) was called in which commit is written > > if (ps != null) { > try {System.out.println("Executing Batch in commitTask"); > ps.executeBatch(); > con.commit(); > ps.close(); > con.close(); > ps = null; > con = null; > } catch (SQLException e) {System.out.println("Exception in > commitTask"); > log.error("ps.close", e); > throw new IOException("JDBC Error", e); > } > > and this method is called by PigOutputCommiter > > public void commitTask(TaskAttemptContext context) throws IOException { > if(HadoopShims.isMap(context.getTaskAttemptID())) { > for (Pair<OutputCommitter, POStore> mapCommitter : > mapOutputCommitters) { > if (mapCommitter.first!=null) { > TaskAttemptContext updatedContext = > setUpContext(context, > mapCommitter.second); > mapCommitter.first.commitTask(updatedContext); > } > } > } else { > for (Pair<OutputCommitter, POStore> reduceCommitter : > reduceOutputCommitters) { > if (reduceCommitter.first!=null) { > TaskAttemptContext updatedContext = > setUpContext(context, > reduceCommitter.second); > reduceCommitter.first.commitTask(updatedContext); > } > } > } > } > > > But when this commitTask is called its connection and preparedStatment > Objects become null... so it ll not commit so data is not available in > Database..... > > > > But if you write only DBStorage without any other Store statement in script > it will work properly.. > > > Any clues??? > > > > > > > > -- > Regards, > Hardik Shah. >
