ashb commented on a change in pull request #4091: [AIRFLOW-2524] Update
SageMaker hook, operator and sensor for training, tuning and transform
URL: https://github.com/apache/incubator-airflow/pull/4091#discussion_r228727399
##########
File path: airflow/hooks/S3_hook.py
##########
@@ -401,6 +418,41 @@ def load_bytes(self,
client = self.get_conn()
client.upload_fileobj(filelike_buffer, bucket_name, key,
ExtraArgs=extra_args)
+ def load_file_obj(self,
+ file_obj,
+ key,
+ bucket_name=None,
+ replace=False,
+ encrypt=False):
+ """
+ Loads file object to S3
+
+ :param file_obj: file-like object to set as content for the key.
+ :type file_obj: file-like object
+ :param key: S3 key that will point to the file
+ :type key: str
+ :param bucket_name: Name of the bucket in which to store the file
+ :type bucket_name: str
+ :param replace: A flag to decide whether or not to overwrite the key
+ if it already exists
+ :type replace: bool
+ :param encrypt: If True, the file will be encrypted on the server-side
+ by S3 and will be stored in an encrypted form while at rest in S3.
+ :type encrypt: bool
+ """
+ if not bucket_name:
+ (bucket_name, key) = self.parse_s3_url(key)
+
+ if not replace and self.check_for_key(key, bucket_name):
+ raise ValueError("The key {key} already exists.".format(key=key))
+
+ extra_args = {}
+ if encrypt:
+ extra_args['ServerSideEncryption'] = "AES256"
Review comment:
Should we add `extra_args` as a dict? - this way people could use this with
other things like ACLs for BucketOwnerFullControl or a to encrypt using a KMS
key.
It should default to `None`, not `{}` though
https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
