----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36025/#review89835 -----------------------------------------------------------
Discussed with Jimmy about the general idea, looks good, just some style comments to make it easier to read. ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java (line 149) <https://reviews.apache.org/r/36025/#comment142885> Will this happen? Can we ensure that it's always filled out? ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/LineageCtx.java (line 92) <https://reviews.apache.org/r/36025/#comment142906> We should add javadocs to new methods in this class. ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/LineageCtx.java (line 152) <https://reviews.apache.org/r/36025/#comment142662> This can call the other method. ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java (line 129) <https://reviews.apache.org/r/36025/#comment142664> The code is a bit hard to read here, can it be simplified? (This and following line) ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java (line 215) <https://reviews.apache.org/r/36025/#comment142666> Getter method should not do a side-effect.. we should make it clearer what is going on. ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java (line 152) <https://reviews.apache.org/r/36025/#comment142907> We can add a javadoc, with some example of what this means. - Szehon Ho On June 29, 2015, 9:55 p.m., Jimmy Xiang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36025/ > ----------------------------------------------------------- > > (Updated June 29, 2015, 9:55 p.m.) > > > Review request for hive and Xuefu Zhang. > > > Repository: hive-git > > > Description > ------- > > Extended HIVE-1131 to log more lineage info. A post exec hook, LineageLogger, > is added to log the lineage in JSON format. > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/Driver.java 669e6be > ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java a0d61f5 > ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 0c6a938 > ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java f98b38b > ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/log/NoDeleteRollingFileAppender.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/ExprProcFactory.java > c930b80 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/Generator.java > 51bef04 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/LineageCtx.java > cef24e3 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java > 5957ac0 > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f41668b > ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 3a1a4af > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 3a4ea2f > ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java e716ed2 > > ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java > e1cab79 > ql/src/test/queries/clientpositive/lineage2.q PRE-CREATION > ql/src/test/queries/clientpositive/lineage3.q PRE-CREATION > ql/src/test/results/clientpositive/alter_partition_change_col.q.out 0d97b7a > ql/src/test/results/clientpositive/alter_table_cascade.q.out 0139466 > ql/src/test/results/clientpositive/combine2.q.out 2400c96 > ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 34cd1ff > ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 0d631ce > ql/src/test/results/clientpositive/index_auto_mult_tables.q.out c3c1fc8 > ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out > e3dfcb7 > ql/src/test/results/clientpositive/index_auto_partitioned.q.out f3ae876 > ql/src/test/results/clientpositive/index_auto_update.q.out 52509be > ql/src/test/results/clientpositive/index_bitmap.q.out 596312d > ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out > 8d4774d > ql/src/test/results/clientpositive/index_bitmap_rc.q.out 45fe339 > ql/src/test/results/clientpositive/index_compact.q.out a33f82a > ql/src/test/results/clientpositive/index_compact_2.q.out fd4cdf9 > ql/src/test/results/clientpositive/join34.q.out 48c3b74 > ql/src/test/results/clientpositive/join35.q.out c0372a7 > ql/src/test/results/clientpositive/lineage1.q.out d9f1ce3 > ql/src/test/results/clientpositive/lineage2.q.out PRE-CREATION > ql/src/test/results/clientpositive/lineage3.q.out PRE-CREATION > ql/src/test/results/clientpositive/load_dyn_part13.q.out 1776e12 > ql/src/test/results/clientpositive/multiMapJoin1.q.out 08d2bc1 > ql/src/test/results/clientpositive/multi_insert.q.out ea2e554 > > ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out > 0fbad49 > ql/src/test/results/clientpositive/orc_dictionary_threshold.q.out 10ed4ac > ql/src/test/results/clientpositive/ptf.q.out 6bd1747 > ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 6c0f4a5 > ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out > 8248a70 > ql/src/test/results/clientpositive/spark/join34.q.out 09b8b6b > ql/src/test/results/clientpositive/spark/join35.q.out e84c860 > ql/src/test/results/clientpositive/spark/load_dyn_part13.q.out 4480310 > ql/src/test/results/clientpositive/spark/multi_insert.q.out c77eb05 > > ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out > e3ef39e > ql/src/test/results/clientpositive/spark/ptf.q.out 1ca6951 > ql/src/test/results/clientpositive/spark/union22.q.out fdb4d47 > ql/src/test/results/clientpositive/spark/union28.q.out 98582df > ql/src/test/results/clientpositive/spark/union29.q.out 1776b4d > ql/src/test/results/clientpositive/spark/union30.q.out 3409623 > ql/src/test/results/clientpositive/spark/union33.q.out 0e6b1aa > ql/src/test/results/clientpositive/spark/union_date_trim.q.out 86b96ac > ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 > ql/src/test/results/clientpositive/spark/union_remove_10.q.out 2718775 > ql/src/test/results/clientpositive/spark/union_remove_11.q.out be65741 > ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab > ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 > ql/src/test/results/clientpositive/spark/union_remove_17.q.out 74a5b23 > ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 > ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 > ql/src/test/results/clientpositive/spark/union_remove_2.q.out 26c4eff > ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 > ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 > ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 > ql/src/test/results/clientpositive/spark/union_remove_23.q.out d9875f3 > ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 > ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 > ql/src/test/results/clientpositive/spark/union_remove_3.q.out a14a8b4 > ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 > ql/src/test/results/clientpositive/spark/union_remove_5.q.out 55baa8b > ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 > ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 > ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 > ql/src/test/results/clientpositive/spark/union_remove_8.q.out f580bd8 > ql/src/test/results/clientpositive/spark/union_remove_9.q.out 0931d11 > ql/src/test/results/clientpositive/spark/union_top_level.q.out 9b10a46 > ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 8447c04 > ql/src/test/results/clientpositive/spark/windowing.q.out c8117d5 > ql/src/test/results/clientpositive/temp_table_windowing_expressions.q.out > 417f595 > ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out 3d04de9 > ql/src/test/results/clientpositive/union22.q.out f14c512 > ql/src/test/results/clientpositive/union28.q.out ab3186e > ql/src/test/results/clientpositive/union29.q.out 3a60d0a > ql/src/test/results/clientpositive/union30.q.out 4529074 > ql/src/test/results/clientpositive/union33.q.out b44e7b2 > ql/src/test/results/clientpositive/unionDistinct_1.q.out 207caf2 > ql/src/test/results/clientpositive/union_date_trim.q.out 86b96ac > ql/src/test/results/clientpositive/union_remove_1.q.out 5fa8caf > ql/src/test/results/clientpositive/union_remove_10.q.out 228ff9e > ql/src/test/results/clientpositive/union_remove_11.q.out bf38f0a > ql/src/test/results/clientpositive/union_remove_15.q.out 67bdb1e > ql/src/test/results/clientpositive/union_remove_16.q.out 2444337 > ql/src/test/results/clientpositive/union_remove_17.q.out a1b9474 > ql/src/test/results/clientpositive/union_remove_18.q.out 26e52e0 > ql/src/test/results/clientpositive/union_remove_19.q.out d8314f8 > ql/src/test/results/clientpositive/union_remove_2.q.out d3522f2 > ql/src/test/results/clientpositive/union_remove_20.q.out 4503726 > ql/src/test/results/clientpositive/union_remove_21.q.out 98e0166 > ql/src/test/results/clientpositive/union_remove_22.q.out 4f086f3 > ql/src/test/results/clientpositive/union_remove_23.q.out d0e21ba > ql/src/test/results/clientpositive/union_remove_24.q.out f2f6bba > ql/src/test/results/clientpositive/union_remove_25.q.out 3858bdc > ql/src/test/results/clientpositive/union_remove_3.q.out 40e554a > ql/src/test/results/clientpositive/union_remove_4.q.out 10a8e89 > ql/src/test/results/clientpositive/union_remove_5.q.out b59d6da > ql/src/test/results/clientpositive/union_remove_6.q.out 67ba74c > ql/src/test/results/clientpositive/union_remove_6_subq.q.out 0d1a436 > ql/src/test/results/clientpositive/union_remove_7.q.out 76c678f > ql/src/test/results/clientpositive/union_remove_8.q.out fe67448 > ql/src/test/results/clientpositive/union_remove_9.q.out 684091b > ql/src/test/results/clientpositive/union_top_level.q.out 1db196f > ql/src/test/results/clientpositive/vectorized_ptf.q.out 194261f > ql/src/test/results/clientpositive/windowing.q.out c8117d5 > ql/src/test/results/clientpositive/windowing_expressions.q.out 5c411ef > > Diff: https://reviews.apache.org/r/36025/diff/ > > > Testing > ------- > > Unit test, Hive on Spark/MR. > > > Thanks, > > Jimmy Xiang > >
