# s3transfer - pydoc - phpman Help on package s3transfer: ## NAME s3transfer - Abstractions over S3's upload/download operations. ## DESCRIPTION This module provides high level abstractions for efficient uploads/downloads. It handles several things for the user: * Automatically switching to multipart transfers when a file is over a specific size threshold * Uploading/downloading a file in parallel * Throttling based on max bandwidth * Progress callbacks to monitor transfers * Retries. While botocore handles retries for streaming uploads, it is not possible for it to handle retries for streaming downloads. This module handles retries for both cases so you don't need to implement any retry logic yourself. This module has a reasonable set of defaults. It also allows you to configure many aspects of the transfer process including: * Multipart threshold size * Max parallel downloads * Max bandwidth * Socket timeouts * Retry amounts There is no support for s3->s3 multipart copies at this time. .. _ref_s3transfer_usage: Usage ===== The simplest way to use this module is: .. code-block:: python client = boto3.client('s3', 'us-west-2') transfer = S3Transfer(client) # Upload /tmp/myfile to s3://bucket/key transfer.upload_file('/tmp/myfile', 'bucket', 'key') # Download s3://bucket/key to /tmp/myfile transfer.download_file('bucket', 'key', '/tmp/myfile') The ``upload_file`` and ``download_file`` methods also accept ``**kwargs``, which will be forwarded through to the corresponding client operation. Here are a few examples using ``upload_file``:: # Making the object public transfer.upload_file('/tmp/myfile', 'bucket', 'key', extra_args={'ACL': 'public-read'}) # Setting metadata transfer.upload_file('/tmp/myfile', 'bucket', 'key', extra_args={'Metadata': {'a': 'b', 'c': 'd'}}) # Setting content type transfer.upload_file('/tmp/myfile.json', 'bucket', 'key', extra_args={'ContentType': "application/json"}) The ``S3Transfer`` clas also supports progress callbacks so you can provide transfer progress to users. Both the ``upload_file`` and ``download_file`` methods take an optional ``callback`` parameter. Here's an example of how to print a simple progress percentage to the user: .. code-block:: python class ProgressPercentage(object): def __init__(self, filename): self._filename = filename self._size = float(os.path.getsize(filename)) self._seen_so_far = 0 self._lock = threading.Lock() def __call__(self, bytes_amount): # To simplify we'll assume this is hooked up # to a single filename. with self._lock: self._seen_so_far += bytes_amount percentage = (self._seen_so_far / self._size) * 100 sys.stdout.write( " %s %s / %s (%.2f%%)" % (self._filename, self._seen_so_far, self._size, percentage)) sys.stdout.flush() transfer = S3Transfer(boto3.client('s3', 'us-west-2')) # Upload /tmp/myfile to s3://bucket/key and print upload progress. transfer.upload_file('/tmp/myfile', 'bucket', 'key', callback=ProgressPercentage('/tmp/myfile')) You can also provide a TransferConfig object to the S3Transfer object that gives you more fine grained control over the transfer. For example: .. code-block:: python client = boto3.client('s3', 'us-west-2') config = TransferConfig( multipart_threshold=8 * 1024 * 1024, max_concurrency=10, num_download_attempts=10, ) transfer = S3Transfer(client, config) transfer.upload_file('/tmp/foo', 'bucket', 'key') ## PACKAGE CONTENTS bandwidth compat constants copies crt delete download exceptions futures manager processpool subscribers tasks upload utils ## CLASSES builtins.Exception(builtins.BaseException) QueueShutdownError builtins.object MultipartDownloader MultipartUploader OSUtils ReadFileChunk S3Transfer StreamReaderProgress TransferConfig logging.Handler(logging.Filterer) NullHandler queue.Queue(builtins.object) ShutdownQueue ### class MultipartDownloader | MultipartDownloader(client, config, osutil, executor_cls=) | | Methods defined here: | | __init__(self, client, config, osutil, executor_cls=) | Initialize self. See help(type(self)) for accurate signature. | | download_file(self, bucket, key, filename, object_size, extra_args, callback=None) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ### class MultipartUploader | MultipartUploader(client, config, osutil, executor_cls=) | | Methods defined here: | | __init__(self, client, config, osutil, executor_cls=) | Initialize self. See help(type(self)) for accurate signature. | | upload_file(self, filename, bucket, key, callback, extra_args) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | UPLOAD_PART_ARGS = ['SSECustomerKey', 'SSECustomerAlgorithm', 'SSECust... ### class NullHandler | NullHandler(level=0) | | Method resolution order: | NullHandler | logging.Handler | logging.Filterer | builtins.object | | Methods defined here: | | emit(self, record) | Do whatever it takes to actually log the specified logging record. | | This version is intended to be implemented by subclasses and so | raises a NotImplementedError. | | ---------------------------------------------------------------------- | Methods inherited from logging.Handler: | | __init__(self, level=0) | Initializes the instance - basically setting the formatter to None | and the filter list to empty. | | __repr__(self) | Return repr(self). | | acquire(self) | Acquire the I/O thread lock. | | close(self) | Tidy up any resources used by the handler. | | This version removes the handler from an internal map of handlers, | _handlers, which is used for handler lookup by name. Subclasses | should ensure that this gets called from overridden close() | methods. | | createLock(self) | Acquire a thread lock for serializing access to the underlying I/O. | | flush(self) | Ensure all logging output has been flushed. | | This version does nothing and is intended to be implemented by | subclasses. | | format(self, record) | Format the specified record. | | If a formatter is set, use it. Otherwise, use the default formatter | for the module. | | get_name(self) | | handle(self, record) | Conditionally emit the specified logging record. | | Emission depends on filters which may have been added to the handler. | Wrap the actual emission of the record with acquisition/release of | the I/O thread lock. Returns whether the filter passed the record for | emission. | | handleError(self, record) | Handle errors which occur during an emit() call. | | This method should be called from handlers when an exception is | encountered during an emit() call. If raiseExceptions is false, | exceptions get silently ignored. This is what is mostly wanted | for a logging system - most users will not care about errors in | the logging system, they are more interested in application errors. | You could, however, replace this with a custom handler if you wish. | The record which was being processed is passed in to this method. | | release(self) | Release the I/O thread lock. | | setFormatter(self, fmt) | Set the formatter for this handler. | | setLevel(self, level) | Set the logging level of this handler. level must be an int or a str. | | set_name(self, name) | | ---------------------------------------------------------------------- | Data descriptors inherited from logging.Handler: | | name | | ---------------------------------------------------------------------- | Methods inherited from logging.Filterer: | | addFilter(self, filter) | Add the specified filter to this handler. | | filter(self, record) | Determine if a record is loggable by consulting all the filters. | | The default is to allow the record to be logged; any filter can veto | this and the record is then dropped. Returns a zero value if a record | is to be dropped, else non-zero. | | .. versionchanged:: 3.2 | | Allow filters to be just callables. | | removeFilter(self, filter) | Remove the specified filter from this handler. | | ---------------------------------------------------------------------- | Data descriptors inherited from logging.Filterer: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ### class OSUtils | Methods defined here: | | get_file_size(self, filename) | | open(self, filename, mode) | | open_file_chunk_reader(self, filename, start_byte, size, callback) | | remove_file(self, filename) | Remove a file, noop if file does not exist. | | rename_file(self, current_filename, new_filename) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ### class QueueShutdownError | Method resolution order: | QueueShutdownError | builtins.Exception | builtins.BaseException | builtins.object | | Data descriptors defined here: | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Methods inherited from builtins.Exception: | | __init__(self, /, *args, **kwargs) | Initialize self. See help(type(self)) for accurate signature. | | ---------------------------------------------------------------------- | Static methods inherited from builtins.Exception: | | __new__(*args, **kwargs) from builtins.type | Create and return a new object. See help(type) for accurate signature. | | ---------------------------------------------------------------------- | Methods inherited from builtins.BaseException: | | __delattr__(self, name, /) | Implement delattr(self, name). | | __getattribute__(self, name, /) | Return getattr(self, name). | | __reduce__(...) | Helper for pickle. | | __repr__(self, /) | Return repr(self). | | __setattr__(self, name, value, /) | Implement setattr(self, name, value). | | __setstate__(...) | | __str__(self, /) | Return str(self). | | with_traceback(...) | Exception.with_traceback(tb) -- | set self.__traceback__ to tb and return self. | | ---------------------------------------------------------------------- | Data descriptors inherited from builtins.BaseException: | | __cause__ | exception cause | | __context__ | exception context | | __dict__ | | __suppress_context__ | | __traceback__ | | args ### class ReadFileChunk | ReadFileChunk(fileobj, start_byte, chunk_size, full_file_size, callback=None, enable_callback=True) | | Methods defined here: | | __enter__(self) | | __exit__(self, *args, **kwargs) | | __init__(self, fileobj, start_byte, chunk_size, full_file_size, callback=None, enable_callback=True) | Given a file object shown below: | | |___________________________________________________| | 0 | | full_file_size | |----chunk_size---| | start_byte | | :type fileobj: file | :param fileobj: File like object | | :type start_byte: int | :param start_byte: The first byte from which to start reading. | | :type chunk_size: int | :param chunk_size: The max chunk size to read. Trying to read | pass the end of the chunk size will behave like you've | reached the end of the file. | | :type full_file_size: int | :param full_file_size: The entire content length associated | with ``fileobj``. | | :type callback: function(amount_read) | :param callback: Called whenever data is read from this object. | | __iter__(self) | | __len__(self) | | close(self) | | disable_callback(self) | | enable_callback(self) | | read(self, amount=None) | | seek(self, where) | | tell(self) | | ---------------------------------------------------------------------- | Class methods defined here: | | from_filename(filename, start_byte, chunk_size, callback=None, enable_callback=True) from builtins.type | Convenience factory function to create from a filename. | | :type start_byte: int | :param start_byte: The first byte from which to start reading. | | :type chunk_size: int | :param chunk_size: The max chunk size to read. Trying to read | pass the end of the chunk size will behave like you've | reached the end of the file. | | :type full_file_size: int | :param full_file_size: The entire content length associated | with ``fileobj``. | | :type callback: function(amount_read) | :param callback: Called whenever data is read from this object. | | :type enable_callback: bool | :param enable_callback: Indicate whether to invoke callback | during read() calls. | | :rtype: ``ReadFileChunk`` | :return: A new instance of ``ReadFileChunk`` | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ### class S3Transfer | S3Transfer(client, config=None, osutil=None) | | Methods defined here: | | __init__(self, client, config=None, osutil=None) | Initialize self. See help(type(self)) for accurate signature. | | download_file(self, bucket, key, filename, extra_args=None, callback=None) | Download an S3 object to a file. | | Variants have also been injected into S3 client, Bucket and Object. | You don't have to use S3Transfer.download_file() directly. | | upload_file(self, filename, bucket, key, callback=None, extra_args=None) | Upload a file to an S3 object. | | Variants have also been injected into S3 client, Bucket and Object. | You don't have to use S3Transfer.upload_file() directly. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | ALLOWED_DOWNLOAD_ARGS = ['VersionId', 'SSECustomerAlgorithm', 'SSECust... | | ALLOWED_UPLOAD_ARGS = ['ACL', 'CacheControl', 'ContentDisposition', 'C... ### class ShutdownQueue | ShutdownQueue(maxsize=0) | | A queue implementation that can be shutdown. | | Shutting down a queue means that this class adds a | trigger_shutdown method that will trigger all subsequent | calls to put() to fail with a ``QueueShutdownError``. | | It purposefully deviates from queue.Queue, and is *not* meant | to be a drop in replacement for ``queue.Queue``. | | Method resolution order: | ShutdownQueue | queue.Queue | builtins.object | | Methods defined here: | | put(self, item) | Put an item into the queue. | | If optional args 'block' is true and 'timeout' is None (the default), | block if necessary until a free slot is available. If 'timeout' is | a non-negative number, it blocks at most 'timeout' seconds and raises | the Full exception if no free slot was available within that time. | Otherwise ('block' is false), put an item on the queue if a free slot | is immediately available, else raise the Full exception ('timeout' | is ignored in that case). | | trigger_shutdown(self) | | ---------------------------------------------------------------------- | Methods inherited from queue.Queue: | | __init__(self, maxsize=0) | Initialize self. See help(type(self)) for accurate signature. | | empty(self) | Return True if the queue is empty, False otherwise (not reliable!). | | This method is likely to be removed at some point. Use qsize() == 0 | as a direct substitute, but be aware that either approach risks a race | condition where a queue can grow before the result of empty() or | qsize() can be used. | | To create code that needs to wait for all queued tasks to be | completed, the preferred technique is to use the join() method. | | full(self) | Return True if the queue is full, False otherwise (not reliable!). | | This method is likely to be removed at some point. Use qsize() >= n | as a direct substitute, but be aware that either approach risks a race | condition where a queue can shrink before the result of full() or | qsize() can be used. | | get(self, block=True, timeout=None) | Remove and return an item from the queue. | | If optional args 'block' is true and 'timeout' is None (the default), | block if necessary until an item is available. If 'timeout' is | a non-negative number, it blocks at most 'timeout' seconds and raises | the Empty exception if no item was available within that time. | Otherwise ('block' is false), return an item if one is immediately | available, else raise the Empty exception ('timeout' is ignored | in that case). | | get_nowait(self) | Remove and return an item from the queue without blocking. | | Only get an item if one is immediately available. Otherwise | raise the Empty exception. | | join(self) | Blocks until all items in the Queue have been gotten and processed. | | The count of unfinished tasks goes up whenever an item is added to the | queue. The count goes down whenever a consumer thread calls task_done() | to indicate the item was retrieved and all work on it is complete. | | When the count of unfinished tasks drops to zero, join() unblocks. | | put_nowait(self, item) | Put an item into the queue without blocking. | | Only enqueue the item if a free slot is immediately available. | Otherwise raise the Full exception. | | qsize(self) | Return the approximate size of the queue (not reliable!). | | task_done(self) | Indicate that a formerly enqueued task is complete. | | Used by Queue consumer threads. For each get() used to fetch a task, | a subsequent call to task_done() tells the queue that the processing | on the task is complete. | | If a join() is currently blocking, it will resume when all items | have been processed (meaning that a task_done() call was received | for every item that had been put() into the queue). | | Raises a ValueError if called more times than there were items | placed in the queue. | | ---------------------------------------------------------------------- | Class methods inherited from queue.Queue: | | __class_getitem__ = GenericAlias(...) from builtins.type | Represent a PEP 585 generic type | | E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,). | | ---------------------------------------------------------------------- | Data descriptors inherited from queue.Queue: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ### class StreamReaderProgress | StreamReaderProgress(stream, callback=None) | | Wrapper for a read only stream that adds progress callbacks. | | Methods defined here: | | __init__(self, stream, callback=None) | Initialize self. See help(type(self)) for accurate signature. | | read(self, *args, **kwargs) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ### class TransferConfig | TransferConfig(multipart_threshold=8388608, max_concurrency=10, multipart_chunksize=8388608, num_download_attempts=5, max_io_queue=100) | | Methods defined here: | | __init__(self, multipart_threshold=8388608, max_concurrency=10, multipart_chunksize=8388608, num_download_attempts=5, max_io_queue=100) | Initialize self. See help(type(self)) for accurate signature. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ## FUNCTIONS ### disable_upload_callbacks ### enable_upload_callbacks ### random_file_extension ## DATA MB = 1048576 SHUTDOWN_SENTINEL = logger = ## VERSION 0.5.0 ## AUTHOR Amazon Web Services ## FILE /usr/lib/python3/dist-packages/s3transfer/__init__.py