> On May 24, 2018, 1:07 p.m., Szabolcs Vasas wrote: > > src/test/org/apache/sqoop/TestOrcImport.java > > Lines 50 (patched) > > <https://reviews.apache.org/r/66548/diff/8/?file=2019922#file2019922line50> > > > > It seems that this test case only covers the HDFS import, can we add > > test cases which cover the Hive import too? > > daniel voros wrote: > 95% of this patch is making the HDFS import work, since the Hive import > part is only issueing a LOAD DATA statement. That part is tested in > `TestTableDefWriter`. I believe, that adding tests similar to the ones in > `TestHiveImport` wouldn't make much sense, since they're only checking if > CREATE TABLE/LOAD DATA statements are as expected or not. > > Please let me know if you can think of a better way to test the Hive > import!
Yeah, I agree that TestHiveImport is not an example of an ideal test case... TestTableDefWriter checks if 'STORED AS ORC' is in the CREATE TABLE statement but it does not check if the statement is syntactically correct. I think it would make sense to parameterize org.apache.sqoop.hive.TestHiveServer2TextImport to test all the supported file formats. It would be a really small change to the test but it would make sure that the CREATE TABLE statement is correct. Once https://issues.apache.org/jira/browse/SQOOP-3328 is done I could add the Parquet case to this test as well. - Szabolcs ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66548/#review203762 ----------------------------------------------------------- On May 2, 2018, 12:12 p.m., daniel voros wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66548/ > ----------------------------------------------------------- > > (Updated May 2, 2018, 12:12 p.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-3311 > https://issues.apache.org/jira/browse/SQOOP-3311 > > > Repository: sqoop-trunk > > > Description > ------- > > Hive 3 will introduce a switch (HIVE-18294) to create eligible tables as ACID > by default. This will probably result in increased usage of ACID tables and > the need to support importing into ACID tables with Sqoop. > > Currently the only table format supporting full ACID tables is ORC. > > The easiest and most effective way to support importing into these tables > would be to write out files as ORC and keep using LOAD DATA as we do for all > other Hive tables (supported since HIVE-17361). > > Workaround could be to create table as textfile (as before) and then CTAS > from that. This would push the responsibility of creating ORC format to Hive. > However it would result in writing every record twice; in text format and in > ORC. > > Note that ORC is only necessary for full ACID tables. Insert-only (aka. > micromanaged) ACID tables can use arbitrary file format. > > Supporting full ACID tables would also be the first step in making > "lastmodified" incremental imports work with Hive. > > > Diffs > ----- > > ivy.xml 1f587f3e > ivy/libraries.properties 565a8bf5 > src/docs/man/import-common-args.txt 22e3448e > src/docs/man/sqoop-import-all-tables.txt 6db38ad8 > src/docs/user/export-purpose.txt def6ead3 > src/docs/user/hcatalog.txt 2ae1d54d > src/docs/user/help.txt 8a0d1477 > src/docs/user/import-all-tables.txt fbad47b2 > src/docs/user/import-purpose.txt c7eca606 > src/docs/user/import.txt 2d074f49 > src/java/org/apache/sqoop/SqoopOptions.java d9984af3 > src/java/org/apache/sqoop/hive/TableDefWriter.java 27d988c5 > src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java a5962ba4 > src/java/org/apache/sqoop/mapreduce/OrcImportMapper.java PRE-CREATION > src/java/org/apache/sqoop/tool/BaseSqoopTool.java 783651a4 > src/java/org/apache/sqoop/tool/ExportTool.java 060f2c07 > src/java/org/apache/sqoop/tool/ImportTool.java ee79d8b7 > src/java/org/apache/sqoop/util/OrcConversionContext.java PRE-CREATION > src/java/org/apache/sqoop/util/OrcUtil.java PRE-CREATION > src/test/org/apache/sqoop/TestAllTables.java 16933a82 > src/test/org/apache/sqoop/TestOrcImport.java PRE-CREATION > src/test/org/apache/sqoop/hive/TestTableDefWriter.java 3ea61f64 > src/test/org/apache/sqoop/util/TestOrcConversionContext.java PRE-CREATION > src/test/org/apache/sqoop/util/TestOrcUtil.java PRE-CREATION > > > Diff: https://reviews.apache.org/r/66548/diff/10/ > > > Testing > ------- > > - added some unit tests > - tested basic Hive import scenarios on a cluster > > > Thanks, > > daniel voros > >
