----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68474/#review209130 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java Line 60 (original), 69 (patched) <https://reviews.apache.org/r/68474/#comment293420> keep the explicit cache method and call it in `MapJoinOperator#closeOp`. This way when a task finishes, we still keep the small table around for at least 30 seconds, which gives any tasks scheduled in the future a chance to re-use the small table. ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java Lines 75 (patched) <https://reviews.apache.org/r/68474/#comment293419> can u add some javadocs to this class explaining what it is doing ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java Lines 82 (patched) <https://reviews.apache.org/r/68474/#comment293416> rename to something like `cleanupService` ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java Lines 90 (patched) <https://reviews.apache.org/r/68474/#comment293417> nit: make `INTEGER_ONE` a static import ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java Lines 91 (patched) <https://reviews.apache.org/r/68474/#comment293415> "SmallTableCache maintenance thread" -> "SmallTableCache Cleanup Thread" ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java Lines 117 (patched) <https://reviews.apache.org/r/68474/#comment293418> replace with `cacheL1.get(key, valueLoader)` where `valueLoader` loads from `cacheL2` - Sahil Takiar On Sept. 19, 2018, 11:14 p.m., Antal Sinkovits wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68474/ > ----------------------------------------------------------- > > (Updated Sept. 19, 2018, 11:14 p.m.) > > > Review request for hive, Naveen Gangam, Sahil Takiar, Adam Szita, and Xuefu > Zhang. > > > Repository: hive-git > > > Description > ------- > > I've modified the SmallTableCache to use guava cache, with soft references. > By using a value loader, I've also eliminated the synchronization on the > intern-ed string of the path. > > > Diffs > ----- > > ql/pom.xml d73deba440702ec39fc5610df28e0fe54baef025 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java > cf27e92bafdc63096ec0fa8c3106657bab52f370 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java > 3293100af96dc60408c53065fa89143ead98f818 > ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSmallTableCache.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/68474/diff/2/ > > > Testing > ------- > > > Thanks, > > Antal Sinkovits > >
