[jira] [Commented] (HIVE-28650) Upgrade Apache ORC version to 2.0.3

2024-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-28650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902254#comment-17902254 ] Steve Loughran commented on HIVE-28650: --- Those slides were done by [~mthakur], I ju

[jira] [Commented] (HIVE-27884) LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon

2024-09-16 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882031#comment-17882031 ] Steve Loughran commented on HIVE-27884: --- thanks. this will speed up s3a, abfs and g

[jira] [Commented] (HIVE-28335) Review deleteOnExitUsage

2024-08-16 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-28335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874217#comment-17874217 ] Steve Loughran commented on HIVE-28335: --- this used in production? it tends to be a

[jira] [Commented] (HIVE-27884) LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon

2024-06-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855990#comment-17855990 ] Steve Loughran commented on HIVE-27884: --- deleteOnExit() is really for test cleanup;

[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

2022-12-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648183#comment-17648183 ] Steve Loughran commented on HIVE-26699: --- in the builder pattern we use in hadoop. .

[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

2022-12-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647671#comment-17647671 ] Steve Loughran commented on HIVE-26699: --- the api itself went in to hadoop earlier,

[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

2022-11-12 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632676#comment-17632676 ] Steve Loughran commented on HIVE-26699: --- you should be using the openFile() api cal

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2022-10-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622163#comment-17622163 ] Steve Loughran commented on HIVE-16983: --- its fixed in hadoop-3.0+ with a moved to s

[jira] [Commented] (HIVE-26063) Upgrade Apache parent POM to version 25

2022-10-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617189#comment-17617189 ] Steve Loughran commented on HIVE-26063: --- apparently this or an explicit update to t

[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1 And Tez to 0.10.2

2022-08-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17574635#comment-17574635 ] Steve Loughran commented on HIVE-24484: --- nice! > Upgrade Hadoop to 3.3.1 And Tez t

[jira] [Commented] (HIVE-25827) Parquet file footer is read multiple times, when multiple splits are created in same file

2022-06-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17554217#comment-17554217 ] Steve Loughran commented on HIVE-25827: --- thanks. next question: do have one or more

[jira] [Commented] (HIVE-25980) Reduce fs calls in HiveMetaStoreChecker.checkTable

2022-06-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553577#comment-17553577 ] Steve Loughran commented on HIVE-25980: --- ok. I'd still recommend the method {{listS

[jira] [Commented] (HIVE-25827) Parquet file footer is read multiple times, when multiple splits are created in same file

2022-04-08 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519777#comment-17519777 ] Steve Loughran commented on HIVE-25827: --- is this per input stream, or are separate

[jira] [Commented] (HIVE-25980) Reduce fs calls in HiveMetaStoreChecker.checkTable

2022-03-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513422#comment-17513422 ] Steve Loughran commented on HIVE-25980: --- use listStatusIterator for incremental lis

[jira] [Updated] (HIVE-25912) Drop external table at root of s3 bucket throws NPE

2022-02-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-25912: -- Summary: Drop external table at root of s3 bucket throws NPE (was: Drop external table throw N

[jira] [Commented] (HIVE-24852) Add support for Snapshots during external table replication

2021-11-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436757#comment-17436757 ] Steve Loughran commented on HIVE-24852: --- # Does this downgrade properly when the de

[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1

2021-09-09 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412556#comment-17412556 ] Steve Loughran commented on HIVE-24484: --- HADOOP-17313 actually went in to deal with

[jira] [Commented] (HIVE-24546) Avoid unwanted cloud storage call during dynamic partition load

2021-07-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380090#comment-17380090 ] Steve Loughran commented on HIVE-24546: --- I'd recommend * skip the dest path check *

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372145#comment-17372145 ] Steve Loughran commented on HIVE-24849: --- Something like this * existence check

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371535#comment-17371535 ] Steve Loughran commented on HIVE-24849: --- How does tbl.isEmpty() work? Does it do a

[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1

2021-06-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368073#comment-17368073 ] Steve Loughran commented on HIVE-24484: --- bq. Would be great if folks could work on

[jira] [Commented] (HIVE-24916) EXPORT TABLE command to ADLS Gen2/s3 failing

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367447#comment-17367447 ] Steve Loughran commented on HIVE-24916: --- If the hadoop version is recent, then call

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367439#comment-17367439 ] Steve Loughran commented on HIVE-24849: --- [~glapark] bq. Now, HiveServer2 does not

[jira] [Commented] (HIVE-24849) Create external table socket timeout when location has large number of files

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367438#comment-17367438 ] Steve Loughran commented on HIVE-24849: --- is hive doing its own recursive treewalk o

[jira] [Commented] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo

2021-06-22 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367301#comment-17367301 ] Steve Loughran commented on HIVE-17133: --- Is this ready to go in? even without a new

[jira] [Commented] (HIVE-24717) Migrate to listStatusIterator in moving files

2021-02-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277866#comment-17277866 ] Steve Loughran commented on HIVE-24717: --- happy to review a hadoop PR with the relev

[jira] [Updated] (HIVE-23492) Remove unnecessary FileSystem#exists calls from ql module

2020-07-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-23492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-23492: -- Description: Wherever there is an exists() call before open() or delete(), remove it and infer

[jira] [Commented] (HIVE-22819) Refactor Hive::listFilesCreatedByQuery to make it faster for object stores

2020-02-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044471#comment-17044471 ] Steve Loughran commented on HIVE-22819: --- LGTM -this saves two round trips to HDFS,

[jira] [Commented] (HIVE-14165) Remove Hive file listing during split computation

2020-02-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033607#comment-17033607 ] Steve Loughran commented on HIVE-14165: --- What is the current status of this? Is it

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2020-01-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007496#comment-17007496 ] Steve Loughran commented on HIVE-16295: --- yeah, where are we with this? Is anyone ac

[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location

2019-12-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989710#comment-16989710 ] Steve Loughran commented on HIVE-22548: --- OK. BTW, if you call toString on the S3A

[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location

2019-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987093#comment-16987093 ] Steve Loughran commented on HIVE-22548: --- do you need that return code from removeEm

[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location

2019-11-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983416#comment-16983416 ] Steve Loughran commented on HIVE-22548: --- Also L1644 it calls path.exists() before t

[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts

2019-10-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964170#comment-16964170 ] Steve Loughran commented on HIVE-22411: --- patch looks functional to me at a glance

[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts

2019-10-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962280#comment-16962280 ] Steve Loughran commented on HIVE-22411: --- FYI [~gabor.bota][~rajesh.balamohan] > Pe

[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts

2019-10-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962271#comment-16962271 ] Steve Loughran commented on HIVE-22411: --- Why do you need to list every single file

[jira] [Commented] (HIVE-22054) Avoid recursive listing to check if a directory is empty

2019-07-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896009#comment-16896009 ] Steve Loughran commented on HIVE-22054: --- you are correct, the getContentSummary cal

[jira] [Resolved] (HIVE-19580) Hive 2.3.2 with ORC files & stored on S3 are case sensitive on EMR

2019-02-20 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HIVE-19580. --- Resolution: Not A Problem OK. closing. Trying hard to think of the best way to classify, e.g

[jira] [Updated] (HIVE-19580) Hive 2.3.2 with ORC files & stored on S3 are case sensitive on EMR

2019-02-20 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-19580: -- Summary: Hive 2.3.2 with ORC files & stored on S3 are case sensitive on EMR (was: Hive 2.3.2 w

[jira] [Comment Edited] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2019-02-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772255#comment-16772255 ] Steve Loughran edited comment on HIVE-19580 at 2/19/19 7:21 PM: ---

[jira] [Commented] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2019-02-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772255#comment-16772255 ] Steve Loughran commented on HIVE-19580: --- If this is EMR then AWS are the only perso

[jira] [Updated] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2019-02-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-19580: -- Environment: EMR s3:// connector Spark 2.3 but also true for lower versions Hive 2.3.2 was

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2018-11-02 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672912#comment-16672912 ] Steve Loughran commented on HIVE-16913: --- DTs aren't sufficient here as Hive uses it

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-07-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554709#comment-16554709 ] Steve Loughran commented on HIVE-16295: --- w.r.t maven dependencies, if you are build

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506017#comment-16506017 ] Steve Loughran commented on HIVE-16391: --- I'm pleased to see the kryo version stuff

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-06-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503216#comment-16503216 ] Steve Loughran commented on HIVE-16295: --- * PathOutputCommitterFactory; you can ask

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502129#comment-16502129 ] Steve Loughran commented on HIVE-16391: --- bq. The problem with that is that it chang

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501534#comment-16501534 ] Steve Loughran commented on HIVE-16391: --- Generally uses .patch files attached to th

[jira] [Commented] (HIVE-19580) Hive 2.3.2 with ORC files stored on S3 are case sensitive

2018-05-30 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495447#comment-16495447 ] Steve Loughran commented on HIVE-19580: --- Don't see why this should be s3-related. *

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452645#comment-16452645 ] Steve Loughran commented on HIVE-16295: --- bq. is there a reason PathOutputCommitterFa

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452112#comment-16452112 ] Steve Loughran commented on HIVE-16295: --- One other comment: you can rely on _SUCCESS

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

2018-04-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450480#comment-16450480 ] Steve Loughran commented on HIVE-16295: --- Impressive. I'm not knowledgeable about hiv

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392751#comment-16392751 ] Steve Loughran commented on HIVE-18861: --- thx for your help nurturing this in. > dru

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-08 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391555#comment-16391555 ] Steve Loughran commented on HIVE-18861: --- I don't see these tests being related' ther

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, cr

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390366#comment-16390366 ] Steve Loughran commented on HIVE-18861: --- Not seeing any updates after 9h. Cancelling

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Open (was: Patch Available) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) got it; cut the -version marker. You must be using a differ

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Open (was: Patch Available) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-07 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, cr

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Open (was: Patch Available) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861-001.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387697#comment-16387697 ] Steve Loughran commented on HIVE-18861: --- [~ashutoshc]: I dont see jira running tests

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386345#comment-16386345 ] Steve Loughran commented on HIVE-18861: --- thanks! If this goes it in, it will be firs

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Status: Patch Available (was: Open) > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386228#comment-16386228 ] Steve Loughran commented on HIVE-18861: --- Dependencies before the patch when built ag

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386231#comment-16386231 ] Steve Loughran commented on HIVE-18861: --- And after {code} [INFO] | +- io.druid.exte

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Attachment: HIVE-18861-001.patch > druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386225#comment-16386225 ] Steve Loughran commented on HIVE-18861: --- Patch 001: pulls the hadoop JAR and the aws

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Summary: druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath probl

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Description: druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which crea

[jira] [Updated] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HIVE-18861: -- Summary: druid-hdfs-storage is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoo

[jira] [Assigned] (HIVE-18861) druid-server is pulling in hadoop-aws-2.7.2, creating classpath problems on hadoop 3.x

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HIVE-18861: - > druid-server is pulling in hadoop-aws-2.7.2, creating classpath problems on > hadoop 3.x >

[jira] [Commented] (HIVE-1620) Patch to write directly to S3 from Hive

2018-03-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386192#comment-16386192 ] Steve Loughran commented on HIVE-1620: -- This is the wrong way to handle variations in

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-11 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082117#comment-16082117 ] Steve Loughran commented on HIVE-16983: --- * The joda time update will be mandatory fo

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080062#comment-16080062 ] Steve Loughran commented on HIVE-16983: --- Patch itself LGTM from an S3a perspective

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072760#comment-16072760 ] Steve Loughran commented on HIVE-16983: --- good point Everyone: look at the S3A troub

[jira] [Commented] (HIVE-16983) getFileStatus on accessible s3a://[bucket-name]/folder: throws com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error

2017-07-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072239#comment-16072239 ] Steve Loughran commented on HIVE-16983: --- Clearly, somehow, your credentials aren't g

[jira] [Commented] (HIVE-9012) Not able to move and populate the data fully on to the table when the scratch directory is on S3

2017-07-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072238#comment-16072238 ] Steve Loughran commented on HIVE-9012: -- This is just rename() being emulated in S3 wit

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2017-06-29 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068127#comment-16068127 ] Steve Loughran commented on HIVE-16913: --- You are going to need a multi-tenant Hive s

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2017-06-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066255#comment-16066255 ] Steve Loughran commented on HIVE-16913: --- Note that if you try and be clever about ke

[jira] [Commented] (HIVE-16913) Support per-session S3 credentials

2017-06-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066253#comment-16066253 ] Steve Loughran commented on HIVE-16913: --- # credentials on Hadoop 2.7+ can go in JCEK

[jira] [Commented] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting t

2017-05-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017265#comment-16017265 ] Steve Loughran commented on HIVE-16446: --- # try with s3a URS and the fs.s3a secret an

[jira] [Commented] (HIVE-16446) org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting t

2017-04-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979869#comment-15979869 ] Steve Loughran commented on HIVE-16446: --- you should switch to using s3a:// URLs in t

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's OutputCommitter

2017-04-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965660#comment-15965660 ] Steve Loughran commented on HIVE-16295: --- Thanks for starting this 1. We're making c

[jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask when src is a directory

2017-03-04 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895818#comment-15895818 ] Steve Loughran commented on HIVE-14864: --- {{FileSystem.getContentSummary()}} does a r

[jira] [Commented] (HIVE-15502) CTAS on S3 is broken with credentials exception

2017-03-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890221#comment-15890221 ] Steve Loughran commented on HIVE-15502: --- probably comes down to the ordering of the

[jira] [Commented] (HIVE-15368) consider optimizing Utilities::handleMmTableFinalPath

2017-03-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890216#comment-15890216 ] Steve Loughran commented on HIVE-15368: --- If you can use {{FileSystem.listFiles(path,

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-12-29 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15785633#comment-15785633 ] Steve Loughran commented on HIVE-15016: --- don't think Hadoop is making much use of co

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-12-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756877#comment-15756877 ] Steve Loughran commented on HIVE-15016: --- if you check out hadoop trunk, all you need

[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT

2016-12-02 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714928#comment-15714928 ] Steve Loughran commented on HIVE-15326: --- HIVE-15016 includes a fix for that, simply

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-12-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712671#comment-15712671 ] Steve Loughran commented on HIVE-15016: --- What's the issue with the codahale JAR? Inc

[jira] [Commented] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT

2016-12-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711687#comment-15711687 ] Steve Loughran commented on HIVE-15326: --- Test is easy; attempt to instantiate a Hive

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687108#comment-15687108 ] Steve Loughran commented on HIVE-15199: --- I do think I'd rather fix this in s3, becau

[jira] [Comment Edited] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15684283#comment-15684283 ] Steve Loughran edited comment on HIVE-15199 at 11/22/16 3:19 PM: ---

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-21 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15684283#comment-15684283 ] Steve Loughran commented on HIVE-15199: --- you are right, I am wrong: serves me right

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-19 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679190#comment-15679190 ] Steve Loughran commented on HIVE-15199: --- if you do listStatus(path, recursive=true)

[jira] [Commented] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

2016-11-17 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673317#comment-15673317 ] Steve Loughran commented on HIVE-15199: --- # as sahil notes, blobstore copy calls must

  1   2   >