Hi Anusha,
1. Well, not quite. What my solution gives you is only a way to move your
data from 's3://some-bucket/pageviews/dt=20120311/key=ACME1234/site=
example.com/Output-file-1' to 's3://some-bucket/pageviews/20120311/ACME1234/
example.com/Output-file-1'. You could actually do this via the linux
terminal by writing a small bash script that uses the same regex logic to
move/copy data from existing folders to a new folder. You could also set up
a cron job to execute this bash script every X hours.
2. Unfortunately, I don't know of a way to do that. I've not used Hive
partitions much, but I believe at the back end, to retrieve the correct
location of the data, Hive too uses a regex to get the correct file path.
So internally, I'm thinking Hive needs the file path to be like "
s3://some-bucket/pageviews/dt=20120311/key=ACME1234/site=
example.com/Output-file-1" to fetch you the correct partition to match your
WHERE clause(s).
Personally, I think you're better off writing that cron job that runs a
batch script to mv/cp the data every X hours :)
Best Regards,
Nishant Kelkar
On Tue, Sep 9, 2014 at 10:02 AM, anusha Mangina <anusha.mang...@gmail.com>
wrote:
> Thanks Nishanth.. I got thousands of records inserted into dynamically
> partitioned Tables.
>
> 1)Do you think this is ideal solution to CONVERT the path for every record
> or didnt i understand your answer.?
>
> 2) Is there anyway we can set up so the initial path formed as we
> need(only with Column values and column names excluded)
>
> On Tue, Sep 9, 2014 at 11:58 AM, anusha Mangina <anusha.mang...@gmail.com>
> wrote:
>
>> Thanks Nishanth.. I got thousands of records inserted into dynamically
>> partitioned Tables.
>>
>> 1)Do you think this is ideal solution to CONVERT the path for every
>> record or didnt i understand your answer.?
>>
>> 2) Is there anyway we can set up so the initial path formed as we
>> need(only with Column values and column names excluded)
>>
>> On Tue, Sep 9, 2014 at 11:43 AM, Nishant Kelkar <nishant....@gmail.com>
>> wrote:
>>
>>> You can use a regex to solve this. If you're using this file path in
>>> Java, you could try something like the following:
>>>
>>> String s =
>>> "s3://some-bucket/pageviews/dt=20120311/key=ACME1234/site=
>>> example.com/Output-file-1";
>>> System.out.println(s.replaceAll("*[a-z]{2,4}=*", ""));
>>>
>>> If you'd like to do this in Hive, there's the following function offered
>>> out of the box:
>>>
>>> regexp_replace(string INITIAL_STRING, string PATTERN, string REPLACEMENT)
>>>
>>> So if you're field is called class_path, then you could do:
>>>
>>> regexp_replace(class_path, "*[a-z]{2,4}=*", "");
>>>
>>> Hope that helps!
>>>
>>> Best Regards,
>>> Nishant Kelkar
>>>
>>>
>>> On Tue, Sep 9, 2014 at 8:27 AM, anusha Mangina <anusha.mang...@gmail.com
>>> > wrote:
>>>
>>>> My Table has Dynamic Partitions and creates the File Path as
>>>>
>>>> s3://some-bucket/pageviews/dt=20120311/key=ACME1234/site=
>>>> example.com/Output-file-1
>>>>
>>>>
>>>> Is there something i can do so i can have the path always as
>>>>
>>>> s3://some-bucket/pageviews/20120311/ACME1234/example.com/Output-file-1
>>>>
>>>> Please help me out guys
>>>>
>>>>
>>>
>>
>
