Hi Jilani, Information of a JIRA creation/modification are sent to dev@ so you don't need to share the JIRA here again. Also, if you have set "Sqoop" at the ReviewBoard as a group to review people in that group will receive a notification about it.
I can see that you have created a JIRA <https://issues.apache.org/jira/browse/SQOOP-3149> but I don't see the patch file attached nor a link to the review board. Could you please add them as I have described it in one of my previous email? After you have attached the patch file and filed a review request (linked to the JIRA so that people can find your change to review) the community will review it which will take a while. Thanks, Bogi On Fri, Mar 10, 2017 at 7:08 PM, Jilani Shaik <jilani2...@gmail.com> wrote: > Hi Bogi, > > Do I need to share the JIRA and review ticket etc here again > or > Once I create a JIRA and review ticket and submit the detail, workflow > will follows from there like validating, comment/suggestion etc. > > Thanks, > Jilani > > On Fri, Mar 10, 2017 at 7:57 AM, Jilani Shaik <jilani2...@gmail.com> > wrote: > >> Yes you are correct, I am running from eclipse. Will run from command >> line. >> >> Sent from my iPhone >> >> On Mar 10, 2017, at 6:48 AM, Boglarka Egyed <b...@cloudera.com> wrote: >> >> Hi Jilani, >> >> Please try to execute "ant compile" and then "ant test" from command >> line, it will run unit tests. If I understood you well, you tried run tests >> from Eclipse which won't work. >> >> Thanks, >> Bogi >> >> On Fri, Mar 10, 2017 at 6:10 AM, Jilani Shaik <jilani2...@gmail.com> >> wrote: >> >>> Hi Bogi, >>> >>> Thanks for the providing direction. >>> >>> As you suggested I explored further and resolved the issue and able to >>> test >>> the fix on trunk based code changes in my hadoop cluster. >>> >>> Root cause for my issue: >>> 1.4.6 code base using the same avro version which is there in my hadoop >>> cluster so there is no issue for that jar component, whereas trunk code >>> base using the avro-1.8.1 jar files, which is not available in my hadoop >>> cluster. >>> >>> Can you suggest how to do unit test etc for this component. >>> >>> I tried with "test" target, I am getting all as failed as below. >>> >>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.415 sec >>> [junit] Running com.cloudera.sqoop.TestDirectImport >>> [junit] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time >>> elapsed: >>> 13.705 sec >>> [junit] Test com.cloudera.sqoop.TestDirectImport FAILED >>> [junit] Running com.cloudera.sqoop.TestExport >>> [junit] Tests run: 17, Failures: 0, Errors: 17, Skipped: 0, Time >>> elapsed: 22.564 sec >>> [junit] Test com.cloudera.sqoop.TestExport FAILED >>> [junit] Running com.cloudera.sqoop.TestExportUpdate >>> >>> Do I need to do any changes? I am running from eclipse with "test" >>> target. >>> >>> Thanks, >>> Jilani >>> >>> >>> On Thu, Mar 9, 2017 at 9:42 PM, Jilani Shaik <jilani2...@gmail.com> >>> wrote: >>> >>> > Hi Bogi, >>> > >>> > - Prepared jar using trunk with "jar-all" target >>> > >>> > - Copied the jar to /opt/mapr/sqoop/sqoop-1.4.6/ >>> > >>> > - Moved out existing jar to some other location >>> > >>> > - then execute the below command to do import >>> > sqoop import --connect jdbc:mysql://10.0.0.300/database123 --verbose >>> > --username test --password test123$ --table payment -m 2 --hbase-table >>> > /database/demoapp/hbase/payment --column-family pay --hbase-row-key >>> > payment_id --incremental lastmodified --merge-key payment_id >>> --check-column >>> > last_update --last-value '2017-01-08 08:02:05.0' >>> > >>> > >>> > The same steps I followed for both the jar from trunk code vs 1.4.6 >>> branch >>> > code. >>> > >>> > Where are you suggesting the multiple avro jars, is it at the time of >>> jar >>> > preparation or running the command using the jar. >>> > >>> > >>> > Thanks, >>> > Jilani >>> > >>> > On Thu, Mar 9, 2017 at 9:21 AM, Boglarka Egyed <b...@cloudera.com> >>> wrote: >>> > >>> >> Hi Jilani, >>> >> >>> >> I suspect that you have an old version of Avro or even multiple Avro >>> >> versions on your classpath and thus Sqoop uses an older one. >>> >> >>> >> Could you please provide a list of the exact commands you have >>> performed >>> >> so that I can reproduce the issue? >>> >> >>> >> Thanks, >>> >> Bogi >>> >> >>> >> On Thu, Mar 9, 2017 at 2:51 AM, Jilani Shaik <jilani2...@gmail.com> >>> >> wrote: >>> >> >>> >>> Can some one provide me the pointers what am I missing with trunk vs >>> >>> 1.4.6 >>> >>> builds, which is giving some error as mentioned in below mail chain. >>> >>> >>> >>> I did followed the same ant target to prepare jar for both branches, >>> but >>> >>> even though 1.4.6 jar is different to 1.4.7 which is created form >>> trunk. >>> >>> >>> >>> Thanks, >>> >>> Jilani >>> >>> >>> >>> >>> >>> On Wed, Mar 8, 2017 at 3:29 AM, Jilani Shaik <jilani2...@gmail.com> >>> >>> wrote: >>> >>> >>> >>> > Hi Bogi, >>> >>> > >>> >>> > I am getting below error, when I have prepared jar from trunk and >>> try >>> >>> to >>> >>> > do sqoop import with mysql database table and got below exception, >>> >>> where as >>> >>> > similar changes are working with branch 1.4.6. >>> >>> > >>> >>> > >>> >>> > 17/03/08 01:06:25 INFO sqoop.Sqoop: Running Sqoop version: >>> >>> 1.4.7-SNAPSHOT >>> >>> > 17/03/08 01:06:25 DEBUG tool.BaseSqoopTool: Enabled debug logging. >>> >>> > 17/03/08 01:06:25 WARN tool.BaseSqoopTool: Setting your password >>> on the >>> >>> > command-line is insecure. Consider using -P instead. >>> >>> > 17/03/08 01:06:25 DEBUG sqoop.ConnFactory: Loaded manager factory: >>> >>> > org.apache.sqoop.manager.oracle.OraOopManagerFactory >>> >>> > 17/03/08 01:06:25 DEBUG sqoop.ConnFactory: Loaded manager factory: >>> >>> > com.cloudera.sqoop.manager.DefaultManagerFactory >>> >>> > 17/03/08 01:06:25 DEBUG sqoop.ConnFactory: Trying ManagerFactory: >>> >>> > org.apache.sqoop.manager.oracle.OraOopManagerFactory >>> >>> > 17/03/08 01:06:25 DEBUG oracle.OraOopManagerFactory: Data >>> Connector for >>> >>> > Oracle and Hadoop can be called by Sqoop! >>> >>> > 17/03/08 01:06:25 DEBUG sqoop.ConnFactory: Trying ManagerFactory: >>> >>> > com.cloudera.sqoop.manager.DefaultManagerFactory >>> >>> > 17/03/08 01:06:25 DEBUG manager.DefaultManagerFactory: Trying with >>> >>> scheme: >>> >>> > jdbc:mysql: >>> >>> > Exception in thread "main" java.lang.NoClassDefFoundError: >>> >>> > org/apache/avro/LogicalType >>> >>> > at org.apache.sqoop.manager.DefaultManagerFactory.accept( >>> >>> > DefaultManagerFactory.java:67) >>> >>> > at org.apache.sqoop.ConnFactory.g >>> >>> etManager(ConnFactory.java:184) >>> >>> > at org.apache.sqoop.tool.BaseSqoopTool.init( >>> >>> > BaseSqoopTool.java:270) >>> >>> > at org.apache.sqoop.tool.ImportTo >>> ol.init(ImportTool.java:97) >>> >>> > at org.apache.sqoop.tool.ImportTo >>> ol.run(ImportTool.java:617) >>> >>> > at org.apache.sqoop.Sqoop.run(Sqoop.java:147) >>> >>> > at org.apache.hadoop.util.ToolRun >>> ner.run(ToolRunner.java:70) >>> >>> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) >>> >>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) >>> >>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) >>> >>> > at org.apache.sqoop.Sqoop.main(Sqoop.java:252) >>> >>> > Caused by: java.lang.ClassNotFoundException: >>> >>> org.apache.avro.LogicalType >>> >>> > at java.net.URLClassLoader.findCl >>> ass(URLClassLoader.java:381) >>> >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> >>> > at sun.misc.Launcher$AppClassLoad >>> >>> er.loadClass(Launcher.java:331) >>> >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> >>> > ... 11 more >>> >>> > >>> >>> > Please let me know what is missing and how to resolve this >>> exception, >>> >>> Let >>> >>> > me know if you need further details. >>> >>> > >>> >>> > Thanks, >>> >>> > Jilani >>> >>> > >>> >>> > On Wed, Mar 1, 2017 at 4:38 AM, Boglarka Egyed <b...@cloudera.com> >>> >>> wrote: >>> >>> > >>> >>> >> Hi Jilani, >>> >>> >> >>> >>> >> This is an example: SQOOP-3053 >>> >>> >> <https://issues.apache.org/jira/browse/SQOOP-3053> with the >>> review >>> >>> >> <https://reviews.apache.org/r/54206/> linked. Please make your >>> >>> changes on >>> >>> >> trunk as it will be used to cut the future release so your patch >>> >>> >> definitely >>> >>> >> needs to be be able to apply on it. >>> >>> >> >>> >>> >> Thanks, >>> >>> >> Bogi >>> >>> >> >>> >>> >> On Wed, Mar 1, 2017 at 3:46 AM, Jilani Shaik < >>> jilani2...@gmail.com> >>> >>> >> wrote: >>> >>> >> >>> >>> >> > Hi Bogi, >>> >>> >> > >>> >>> >> > Can you provide me sample Jira tickets and Review requests >>> similar >>> >>> to >>> >>> >> > this, to proceed further. >>> >>> >> > >>> >>> >> > I applied the code changes from sqoop git from this branch >>> >>> >> > "sqoop-release-1.4.6-rc0", If you suggest right branch I will >>> take >>> >>> the >>> >>> >> code >>> >>> >> > from there and apply the changes before submit review for >>> request. >>> >>> >> > >>> >>> >> > Thanks, >>> >>> >> > Jilani >>> >>> >> > >>> >>> >> > On Mon, Feb 27, 2017 at 3:05 AM, Boglarka Egyed < >>> b...@cloudera.com> >>> >>> >> wrote: >>> >>> >> > >>> >>> >> >> Hi Jilani, >>> >>> >> >> >>> >>> >> >> To get your change committed please do the following: >>> >>> >> >> * Open a JIRA ticket for your change in Apache's JIRA system >>> >>> >> >> <https://issues.apache.org/jira/browse/SQOOP/> for project >>> Sqoop >>> >>> >> >> * Create a review request at Apache's review board >>> >>> >> >> <https://reviews.apache.org/r/> for project Sqoop and link it >>> to >>> >>> the >>> >>> >> JIRA >>> >>> >> >> >>> >>> >> >> ticket >>> >>> >> >> >>> >>> >> >> Please consider the guidelines below: >>> >>> >> >> >>> >>> >> >> Review board >>> >>> >> >> * Summary: generate your summary using the issue's jira key + >>> jira >>> >>> >> title >>> >>> >> >> * Groups: add the relevant group so everyone on the project >>> will >>> >>> know >>> >>> >> >> about >>> >>> >> >> your patch (Sqoop) >>> >>> >> >> * Bugs: add the issue's jira key so it's easy to navigate to >>> the >>> >>> jira >>> >>> >> side >>> >>> >> >> * Repository: sqoop-trunk for Sqoop1 or sqoop-sqoop2 for Sqoop2 >>> >>> >> >> * And as soon as the patch gets committed, it's very useful >>> for the >>> >>> >> >> community if you close the review and mark it as "Submitted" >>> at the >>> >>> >> Review >>> >>> >> >> board. The button to do this is top right at your own tickets, >>> >>> right >>> >>> >> next >>> >>> >> >> to the Download Diff button. >>> >>> >> >> >>> >>> >> >> Jira >>> >>> >> >> * Link: please add the link of the review as an external/web >>> link >>> >>> so >>> >>> >> it's >>> >>> >> >> easy to navigate to the reviews side >>> >>> >> >> * Status: mark it as "patch available" >>> >>> >> >> >>> >>> >> >> Sqoop community will receive emails about your new ticket and >>> >>> review >>> >>> >> >> request and will review your change. >>> >>> >> >> >>> >>> >> >> Thanks, >>> >>> >> >> Bogi >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> On Sat, Feb 25, 2017 at 2:14 AM, Jilani Shaik < >>> >>> jilani2...@gmail.com> >>> >>> >> >> wrote: >>> >>> >> >> >>> >>> >> >> > Do we have any update? >>> >>> >> >> > >>> >>> >> >> > I did checkout of the 1.4.6 code and done code changes to >>> achieve >>> >>> >> this >>> >>> >> >> and >>> >>> >> >> > tested in cluster and it is working as expected. Is there a >>> way >>> >>> I can >>> >>> >> >> > contribute this as a patch and then the committers can >>> validate >>> >>> >> further >>> >>> >> >> and >>> >>> >> >> > suggest if any changes required to move further. Please >>> suggest >>> >>> the >>> >>> >> >> > approach. >>> >>> >> >> > >>> >>> >> >> > Thanks, >>> >>> >> >> > Jilani >>> >>> >> >> > >>> >>> >> >> > On Sun, Feb 5, 2017 at 10:41 PM, Jilani Shaik < >>> >>> jilani2...@gmail.com> >>> >>> >> >> > wrote: >>> >>> >> >> > >>> >>> >> >> > > Hi Liz, >>> >>> >> >> > > >>> >>> >> >> > > lets say we inserted data in a table with initial import, >>> that >>> >>> >> looks >>> >>> >> >> like >>> >>> >> >> > > this in hbase shell >>> >>> >> >> > > >>> >>> >> >> > > 1 column=pay:amount, >>> >>> >> >> > > timestamp=1485129654025, value=4.99 >>> >>> >> >> > > 1 >>> column=pay:customer_id, >>> >>> >> >> > > timestamp=1485129654025, value=1 >>> >>> >> >> > > 1 >>> column=pay:last_update, >>> >>> >> >> > > timestamp=1485129654025, value=2017-01-23 05:29:09.0 >>> >>> >> >> > > 1 >>> column=pay:payment_date, >>> >>> >> >> > > timestamp=1485129654025, value=2005-05-25 11:30:37.0 >>> >>> >> >> > > 1 >>> column=pay:rental_id, >>> >>> >> >> > > timestamp=1485129654025, value=573 >>> >>> >> >> > > 1 column=pay:staff_id, >>> >>> >> >> > > timestamp=1485129654025, value=1 >>> >>> >> >> > > 10 column=pay:amount, >>> >>> >> >> > > timestamp=1485129504390, value=5.99 >>> >>> >> >> > > 10 >>> column=pay:customer_id, >>> >>> >> >> > > timestamp=1485129504390, value=1 >>> >>> >> >> > > 10 >>> column=pay:last_update, >>> >>> >> >> > > timestamp=1485129504390, value=2006-02-15 22:12:30.0 >>> >>> >> >> > > 10 >>> column=pay:payment_date, >>> >>> >> >> > > timestamp=1485129504390, value=2005-07-08 03:17:05.0 >>> >>> >> >> > > 10 >>> column=pay:rental_id, >>> >>> >> >> > > timestamp=1485129504390, value=4526 >>> >>> >> >> > > 10 column=pay:staff_id, >>> >>> >> >> > > timestamp=1485129504390, value=2 >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > now assume that in source rental_id becomes NULL for rowkey >>> >>> "1", >>> >>> >> and >>> >>> >> >> then >>> >>> >> >> > > we are doing incremental import into HBase. With current >>> >>> import the >>> >>> >> >> final >>> >>> >> >> > > HBase data after incremental import will look like this. >>> >>> >> >> > > >>> >>> >> >> > > 1 column=pay:amount, >>> >>> >> >> > > timestamp=1485129654025, value=4.99 >>> >>> >> >> > > 1 >>> column=pay:customer_id, >>> >>> >> >> > > timestamp=1485129654025, value=1 >>> >>> >> >> > > 1 >>> column=pay:last_update, >>> >>> >> >> > > timestamp=1485129654025, value=2017-02-05 05:29:09.0 >>> >>> >> >> > > 1 >>> column=pay:payment_date, >>> >>> >> >> > > timestamp=1485129654025, value=2005-05-25 11:30:37.0 >>> >>> >> >> > > 1 >>> column=pay:rental_id, >>> >>> >> >> > > timestamp=1485129654025, value=573 >>> >>> >> >> > > 1 column=pay:staff_id, >>> >>> >> >> > > timestamp=1485129654025, value=1 >>> >>> >> >> > > 10 column=pay:amount, >>> >>> >> >> > > timestamp=1485129504390, value=5.99 >>> >>> >> >> > > 10 >>> column=pay:customer_id, >>> >>> >> >> > > timestamp=1485129504390, value=1 >>> >>> >> >> > > 10 >>> column=pay:last_update, >>> >>> >> >> > > timestamp=1485129504390, value=2017-02-05 05:12:30.0 >>> >>> >> >> > > 10 >>> column=pay:payment_date, >>> >>> >> >> > > timestamp=1485129504390, value=2005-07-08 03:17:05.0 >>> >>> >> >> > > 10 >>> column=pay:rental_id, >>> >>> >> >> > > timestamp=1485129504390, value=126 >>> >>> >> >> > > 10 column=pay:staff_id, >>> >>> >> >> > > timestamp=1485129504390, value=2 >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > As source column "rental_id" becomes NULL for rowkey "1", >>> the >>> >>> final >>> >>> >> >> HBase >>> >>> >> >> > > should not have the "rental_id" for this rowkey "1". I am >>> >>> expecting >>> >>> >> >> below >>> >>> >> >> > > data for these rowkeys. >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > 1 column=pay:amount, >>> >>> >> >> > > timestamp=1485129654025, value=4.99 >>> >>> >> >> > > 1 >>> column=pay:customer_id, >>> >>> >> >> > > timestamp=1485129654025, value=1 >>> >>> >> >> > > 1 >>> column=pay:last_update, >>> >>> >> >> > > timestamp=1485129654025, value=2017-02-05 05:29:09.0 >>> >>> >> >> > > 1 >>> column=pay:payment_date, >>> >>> >> >> > > timestamp=1485129654025, value=2005-05-25 11:30:37.0 >>> >>> >> >> > > 1 column=pay:staff_id, >>> >>> >> >> > > timestamp=1485129654025, value=1 >>> >>> >> >> > > 10 column=pay:amount, >>> >>> >> >> > > timestamp=1485129504390, value=5.99 >>> >>> >> >> > > 10 >>> column=pay:customer_id, >>> >>> >> >> > > timestamp=1485129504390, value=1 >>> >>> >> >> > > 10 >>> column=pay:last_update, >>> >>> >> >> > > timestamp=1485129504390, value=2017-02-05 05:12:30.0 >>> >>> >> >> > > 10 >>> column=pay:payment_date, >>> >>> >> >> > > timestamp=1485129504390, value=2005-07-08 03:17:05.0 >>> >>> >> >> > > 10 >>> column=pay:rental_id, >>> >>> >> >> > > timestamp=1485129504390, value=126 >>> >>> >> >> > > 10 column=pay:staff_id, >>> >>> >> >> > > timestamp=1485129504390, value=2 >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > Please let me know if anything required further. >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > Thanks, >>> >>> >> >> > > Jilani >>> >>> >> >> > > >>> >>> >> >> > > On Tue, Jan 31, 2017 at 3:38 AM, Erzsebet Szilagyi < >>> >>> >> >> > > liz.szila...@cloudera.com> wrote: >>> >>> >> >> > > >>> >>> >> >> > >> Hi Jilani, >>> >>> >> >> > >> I'm not sure I completely understand what you are trying >>> to >>> >>> do. >>> >>> >> Could >>> >>> >> >> > you >>> >>> >> >> > >> give us some examples with e.g. 4 columns and 2 rows of >>> >>> example >>> >>> >> data >>> >>> >> >> > >> showing the changes that happen compared to the changes >>> you'd >>> >>> >> like to >>> >>> >> >> > see? >>> >>> >> >> > >> Thanks, >>> >>> >> >> > >> Liz >>> >>> >> >> > >> >>> >>> >> >> > >> On Tue, Jan 31, 2017 at 5:18 AM, Jilani Shaik < >>> >>> >> jilani2...@gmail.com> >>> >>> >> >> > >> wrote: >>> >>> >> >> > >> >>> >>> >> >> > >> > >>> >>> >> >> > >> > Please help in resolving the issue, I am going through >>> >>> source >>> >>> >> code >>> >>> >> >> > some >>> >>> >> >> > >> > how the required nature is missing, But not sure is it >>> for >>> >>> some >>> >>> >> >> reason >>> >>> >> >> > >> we >>> >>> >> >> > >> > avoided this nature. >>> >>> >> >> > >> > >>> >>> >> >> > >> > Provide me some suggestions how to go with this >>> scenario. >>> >>> >> >> > >> > >>> >>> >> >> > >> > Thanks, >>> >>> >> >> > >> > Jilani >>> >>> >> >> > >> > >>> >>> >> >> > >> > On Sun, Jan 22, 2017 at 6:45 PM, Jilani Shaik < >>> >>> >> >> jilani2...@gmail.com> >>> >>> >> >> > >> > wrote: >>> >>> >> >> > >> > >>> >>> >> >> > >> >> Hi, >>> >>> >> >> > >> >> >>> >>> >> >> > >> >> We have a scenario where we are importing data into >>> HBase >>> >>> with >>> >>> >> >> sqoop >>> >>> >> >> > >> >> incremental import. >>> >>> >> >> > >> >> >>> >>> >> >> > >> >> Lets say we imported a table and later source table got >>> >>> updated >>> >>> >> >> for >>> >>> >> >> > >> some >>> >>> >> >> > >> >> columns as null values for some rows. Then while doing >>> >>> >> incremental >>> >>> >> >> > >> import >>> >>> >> >> > >> >> as per HBase these columns should not be there in HBase >>> >>> table. >>> >>> >> But >>> >>> >> >> > >> right >>> >>> >> >> > >> >> now these columns will be as it is available with >>> previous >>> >>> >> values. >>> >>> >> >> > >> >> >>> >>> >> >> > >> >> Is there any fix to overcome this issue? >>> >>> >> >> > >> >> >>> >>> >> >> > >> >> >>> >>> >> >> > >> >> Thanks, >>> >>> >> >> > >> >> Jilani >>> >>> >> >> > >> >> >>> >>> >> >> > >> > >>> >>> >> >> > >> > >>> >>> >> >> > >> >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > >>> >>> >> >> >>> >>> >> > >>> >>> >> > >>> >>> >> >>> >>> > >>> >>> > >>> >>> >>> >> >>> >> >>> > >>> >> >> >
