----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64193/#review193155 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java Line 201 (original), 208 (patched) <https://reviews.apache.org/r/64193/#comment271675> Can we use `parseContext.getQueryState()`? ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java Lines 164 (patched) <https://reviews.apache.org/r/64193/#comment271674> Think u can just use `parseContext.getQueryState()` ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java Lines 351 (patched) <https://reviews.apache.org/r/64193/#comment271673> I think `pCtx#getQueryState` could be used for all the changes to the `TaskCompiler`s? - Sahil Takiar On Dec. 7, 2017, 7:12 p.m., Andrew Sherman wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64193/ > ----------------------------------------------------------- > > (Updated Dec. 7, 2017, 7:12 p.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > ------- > > A Hive Session can contain multiple concurrent sql Operations. > Lineage is currently tracked in SessionState and is cleared when a query > completes. This results in Lineage for other running queries being lost. > > To fix this, move LineageState from SessionState to QueryState. > In MoveTask/MoveWork use the LineageState from the MoveTask's QueryState > rather than trying to use it from MoveWork. > Add a test which runs multiple jdbc queries in a thread pool > against the same connection and show that Vertices are not lost from Lineage. > As part of this test, add ReadableHook, an ExecuteWithHookContext that stores > HookContexts in memory and makes them available for reading. > Make LineageLogger methods static so they can be used elsewhere. > > Sometimes a running query (originating in a Driver) will instantiate > another Driver to run or compile another query. Because these Drivers > shared a Session, the child Driver would accumulate Lineage information > along with that of the parent Driver. For consistency a LineageState is > passed to these child Drivers and stored in the new Driver's QueryState. > > > Diffs > ----- > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/ReadableHook.java > PRE-CREATION > > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java > 70bd29c5178456c683652cf2377206059b735514 > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > d3df015288fe1963d2b548e32db53cfc2310af21 > ql/src/java/org/apache/hadoop/hive/ql/QueryState.java > f3a46dbcaf151706521c735654f377a2f2f76a81 > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java > 55ef8de9a5c7144931d0a6ff13224765ee737fea > ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java > f5a5e713bb0e081591a53a30caf56f97750c3f8e > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java > 1f0487f4f72ab18bcf876f45ad5758d83a7f001b > > ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java > 262225fc202d4627652acfd77350e44b0284b3da > > ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java > 545b7a8b7e9f1370b767fc777cb10fa59bd81917 > ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java > 7b617309f6b0d8a7ce0dea80ab1f790c2651b147 > ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java > 2f764f8a29a9d41a7db013a949ffe3a8a9417d32 > ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java > 68709b4d3baf15d78e60e948ccdef3df84f28cec > ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java > 1e577da82343a1b7361467fb662661f9c6642ec0 > ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java > 29886ae7f97f8dae7116f4fc9a2417ab8f9dac0a > ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java > 7b067a0d45e33bc3347c43b050af933c296a9227 > > ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java > 504b0623142a6fa6cdb45a26b49f146e12ec2d7a > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java > d7a83f775abca39b219f71aff88173a14ffaee9f > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java > 4387c4297fee48d4c03e95d5a2fcb822ab480eeb > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > bdaf105697fd2c2074885fa3a35548043167c7e7 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java > 338c1856672f09bb7da35d2336ebb5b6f3fdc5a6 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/Generator.java > e6c07713b24df719315d804f006151106eea9aed > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > a09b7961c2dbc26b4d2fa912d0be7037885f63e4 > ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java > 065c7e50986872cd35386feee712f3452597d643 > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java > 0c160acf46eb1eb07c5f04091099c1024e166638 > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java > b6f1139fe1a78283277bf4d0c5224ab1d718c634 > ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java > 83d53bc157f35b4b57fc37bb24b6c400ac58d8ca > ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java > f31775ed942160da73344c4dca707da7b8c658a6 > ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java > cc956da57567114aa29ee0552566ca62c68f6be7 > ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java > d7a56e5846d5754dec5070d8c44443543a3695e4 > > ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java > 498b6741c3f40b92ce3fb218e91e7809a17383f0 > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java > 28e3621d3264f4f704da0d775b396f7b7764fdb6 > ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java > 7b2937032ab8dd57f8923e0a9e7aab4a92de55ee > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > be33f380030ea8b416a4549c3947d767bba66356 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java > 4d2bcfa285dc08811106f3c234346efff22afd99 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java > 604c8aee151a45cf942852a3644b5e79f779f353 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 965044d9253585eeaeef50d7fe4fc4d818042df8 > ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java > 28a33740b30b7be0057ce91de55a0407dd2f2cbf > ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java > 056d6141d6239816699ed5f730cbd14e48d8d9bb > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java > bb6ddc6fa4667ac0e30994d0f9ee8b969542383c > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java > 340689255c738ea497bcd269463b8b8bc38cf34c > ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java > 2c28c398ca49ba661df460c9f3e6d578c785d3ce > > > Diff: https://reviews.apache.org/r/64193/diff/2/ > > > Testing > ------- > > > Thanks, > > Andrew Sherman > >
