senaduka/old-parkingkoncept
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9bfb1e1bc355c94cc3db07bd11c18f2cbe6c8a9c
old-parkingkoncept/parkingkonceptvenv/lib/python3.7/site-packages/s3transfer/__pycache__/upload.cpython-37.pyc

256 lines
23 KiB
Plaintext
Raw Normal View History

Initial commit
2019-11-17 12:44:16 +01:00
B
<00>[<5B>]vs<00>@sddlZddlmZddlmZmZddlmZddlm Z ddlm
Z
ddlm Z ddlm Z dd l mZdd
l mZdd l mZmZGd d <0A>d e<12>ZGdd<0F>de<12>ZGdd<11>de<12>ZGdd<13>de<15>ZGdd<15>de<16>ZGdd<17>de<15>ZGdd<19>de
<EFBFBD>ZGdd<1B>de <09>ZGdd<1D>de <09>ZdS)<1E>N)<01>six)<02>seekable<6C>readable)<01>IN_MEMORY_UPLOAD_TAG)<01>Task)<01>SubmissionTask)<01>CreateMultipartUploadTask)<01>CompleteMultipartUploadTask)<01> get_callbacks)<01>get_filtered_dict)<02>DeferredOpenFile<6C>ChunksizeAdjusterc@s.eZdZd dd<03>Zdd<05>Zdd<07>Zdd <09>Zd
S) <0C>AggregatedProgressCallback<63>cCs||_||_d|_dS)a<>Aggregates progress updates for every provided progress callback
:type callbacks: A list of functions that accepts bytes_transferred
as a single argument
:param callbacks: The callbacks to invoke when threshold is reached
:type threshold: int
:param threshold: The progress threshold in which to take the
aggregated progress and invoke the progress callback with that
aggregated progress total
rN)<03>
_callbacks<EFBFBD>
_threshold<EFBFBD> _bytes_seen)<03>self<6C> callbacks<6B> threshold<6C>r<00>9/tmp/pip-install-u_6_z2pz/s3transfer/s3transfer/upload.py<70>__init__s z#AggregatedProgressCallback.__init__cCs&|j|7_|j|jkr"|<00><02>dS)N)rr<00>_trigger_callbacks)r<00>bytes_transferredrrr<00>__call__-s z#AggregatedProgressCallback.__call__cCs|jdkr|<00><01>dS)z@Flushes out any progress that has not been sent to its callbacksrN)rr)rrrr<00>flush2s
z AggregatedProgressCallback.flushcCs&x|jD]}||jd<01>qWd|_dS)N)rr)rr)r<00>callbackrrrr7s z-AggregatedProgressCallback._trigger_callbacksN)r)<07>__name__<5F>
__module__<EFBFBD> __qualname__rrrrrrrrrs
rc@sJeZdZdZdd<03>Zddd<06>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Zd d<0E>Z dd<10>Z
dS)<12>InterruptReadera<72>Wrapper that can interrupt reading using an error
It uses a transfer coordinator to propagate an error if it notices
that a read is being made while the file is being read from.
:type fileobj: file-like obj
:param fileobj: The file-like object to read from
:type transfer_coordinator: s3transfer.futures.TransferCoordinator
:param transfer_coordinator: The transfer coordinator to use if the
reader needs to be interrupted.
cCs||_||_dS)N)<02>_fileobj<62>_transfer_coordinator)r<00>fileobj<62>transfer_coordinatorrrrrJszInterruptReader.__init__NcCs|jjr|jj<01>|j<02>|<01>S)N)r#<00> exceptionr"<00>read)r<00>amountrrrr'NszInterruptReader.readcCs|j<00>|<01>dS)N)r"<00>seek)r<00>whererrrr)XszInterruptReader.seekcCs
|j<00><01>S)N)r"<00>tell)rrrrr+[szInterruptReader.tellcCs|j<00><01>dS)N)r"<00>close)rrrrr,^szInterruptReader.closecCs|S)Nr)rrrr<00> __enter__aszInterruptReader.__enter__cOs |<00><00>dS)N)r,)r<00>args<67>kwargsrrr<00>__exit__dszInterruptReader.__exit__)N) rrr <00>__doc__rr'r)r+r,r-r0rrrrr!=s 
r!c@sfeZdZdZddd<04>Zedd<06><00>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Z d d<0E>Z
dd<10>Z dd<12>Z dd<14>Z dd<16>ZdS)<18>UploadInputManageraJBase manager class for handling various types of files for uploads
This class is typically used for the UploadSubmissionTask class to help
determine the following:
* How to determine the size of the file
* How to determine if a multipart upload is required
* How to retrieve the body for a PutObject
* How to retrieve the bodies for a set of UploadParts
The answers/implementations differ for the various types of file inputs
that may be accepted. All implementations must subclass and override
public methods from this class.
NcCs||_||_||_dS)N)<03>_osutilr#<00>_bandwidth_limiter)r<00>osutilr%<00>bandwidth_limiterrrrrwszUploadInputManager.__init__cCs td<01><01>dS)aDetermines if the source for the upload is compatible with manager
:param upload_source: The source for which the upload will pull data
from.
:returns: True if the manager can handle the type of source specified
otherwise returns False.
zmust implement _is_compatible()N)<01>NotImplementedError)<02>cls<6C> upload_sourcerrr<00> is_compatible|s
z UploadInputManager.is_compatiblecCs td<01><01>dS)a<>Whether the body it provides are stored in-memory
:type operation_name: str
:param operation_name: The name of the client operation that the body
is being used for. Valid operation_names are ``put_object`` and
``upload_part``.
:rtype: boolean
:returns: True if the body returned by the manager will be stored in
memory. False if the manager will not directly store the body in
memory.
z%must implement store_body_in_memory()N)<01>NotImplemented)r<00>operation_namerrr<00>stores_body_in_memory<72>s z(UploadInputManager.stores_body_in_memorycCs td<01><01>dS)z<>Provides the transfer size of an upload
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
z&must implement provide_transfer_size()N)r7)r<00>transfer_futurerrr<00>provide_transfer_size<7A>sz(UploadInputManager.provide_transfer_sizecCs td<01><01>dS)a<>Determines where a multipart upload is required
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: boolean
:returns: True, if the upload should be multipart based on
configuartion and size. False, otherwise.
z*must implement requires_multipart_upload()N)r7)rr><00>configrrr<00>requires_multipart_upload<61>s z,UploadInputManager.requires_multipart_uploadcCs td<01><01>dS)a<>Returns the body to use for PutObject
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: s3transfer.utils.ReadFileChunk
:returns: A ReadFileChunk including all progress callbacks
associated with the transfer future.
z$must implement get_put_object_body()N)r7)rr>rrr<00>get_put_object_body<64>s z&UploadInputManager.get_put_object_bodycCs td<01><01>dS)aYields the part number and body to use for each UploadPart
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type chunksize: int
:param chunksize: The chunksize to use for this upload.
:rtype: int, s3transfer.utils.ReadFileChunk
:returns: Yields the part number and the ReadFileChunk including all
progress callbacks associated with the transfer future for that
specific yielded part.
z)must implement yield_upload_part_bodies()N)r7)rr><00> chunksizerrr<00>yield_upload_part_bodies<65>sz+UploadInputManager.yield_upload_part_bodiescCs*t||j<01>}|jr&|jj||jdd<02>}|S)NF)Zenabled)r!r#r4Zget_bandwith_limited_stream)rr$rrr<00> _wrap_fileobj<62>s
 z UploadInputManager._wrap_fileobjcCst|d<01>}|rt|<02>gSgS)N<>progress)r
r)rr>rrrr<00>_get_progress_callbacks<6B>s

z*UploadInputManager._get_progress_callbackscCsdd<02>|D<00>S)NcSsg|]
}|j<00>qSr)r)<02>.0rrrr<00>
<listcomp><3E>sz;UploadInputManager._get_close_callbacks.<locals>.<listcomp>r)rZaggregated_progress_callbacksrrr<00>_get_close_callbacks<6B>sz'UploadInputManager._get_close_callbacks)N)rrr r1r<00> classmethodr:r=r?rArBrDrErGrJrrrrr2hs
  r2c@sdeZdZdZedd<03><00>Zdd<05>Zdd<07>Zdd <09>Zd
d <0B>Z d d <0A>Z
dd<0F>Z dd<11>Z dd<13>Z dd<15>ZdS)<17>UploadFilenameInputManagerzUpload utility for filenamescCs t|tj<02>S)N)<03>
isinstancer<00> string_types)r8r9rrrr:<00>sz(UploadFilenameInputManager.is_compatiblecCsdS)NFr)rr<rrrr=<00>sz0UploadFilenameInputManager.stores_body_in_memorycCs|j<00>|j<02>|jjj<05><01>dS)N)<06>metar?r3Z get_file_size<7A> call_argsr$)rr>rrrr?<00>sz0UploadFilenameInputManager.provide_transfer_sizecCs|jj|jkS)N)rO<00>size<7A>multipart_threshold)rr>r@rrrrA<00>sz4UploadFilenameInputManager.requires_multipart_uploadcCsJ|<00>|<01>\}}|<00>|<02>}|<00>|<01>}|<00>|<04>}|jj}|jj|||||d<01>S)N)r$<00>
chunk_size<EFBFBD>full_file_sizer<00>close_callbacks)<08>&_get_put_object_fileobj_with_full_sizerErGrJrOrQr3<00>#open_file_chunk_reader_from_fileobj)rr>r$<00> full_sizerrUrQrrrrB<00>s



z.UploadFilenameInputManager.get_put_object_bodyc cs<>|jj}|<00>||<02>}x|td|d<00>D]j}|<00>|<01>}|<00>|<06>}||d}|j|jjj|||d<02>\} }
|<00> | <09>} |j
j | ||
||d<03>} || fVq$WdS)N<>)<03>
start_byte<EFBFBD> part_sizerT)r$rSrTrrU) rOrQ<00>_get_num_parts<74>rangerGrJ<00>'_get_upload_part_fileobj_with_full_sizerPr$rEr3rW) rr>rCrTZ num_parts<74> part_numberrrUrZr$rXZread_file_chunkrrrrDs 

 

z3UploadFilenameInputManager.yield_upload_part_bodiescCst|||jjd<01>}|S)N)Z open_function)r r3<00>open)rr$rZrrr<00>_get_deferred_open_filesz2UploadFilenameInputManager._get_deferred_open_filecCs"|jjj}|jj}|<00>|d<01>|fS)Nr)rOrPr$rQra)rr>r$rQrrrrV#s
zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizecKs |d}|d}|<00>||<03>|fS)NrZrT)ra)rr$r/rZrXrrrr^(szBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizecCstt<01>|jjt|<02><00><01>S)N)<06>int<6E>math<74>ceilrOrQ<00>float)rr>r[rrrr\-sz)UploadFilenameInputManager._get_num_partsN)rrr r1rKr:r=r?rArBrDrarVr^r\rrrrrL<00>s rLc@s<eZdZdZedd<03><00>Zdd<05>Zdd<07>Zdd <09>Zd
d <0B>Z d S) <0A>UploadSeekableInputManagerz&Upload utility for an open file objectcCst|<01>ot|<01>S)N)rr)r8r9rrrr:4sz(UploadSeekableInputManager.is_compatiblecCs|dkr dSdSdS)N<>
put_objectFTr)rr<rrrr=8sz0UploadSeekableInputManager.stores_body_in_memorycCsD|jjj}|<02><03>}|<02>dd<02>|<02><03>}|<02>|<03>|j<00>||<00>dS)Nr<00>)rOrPr$r+r)r?)rr>r$Zstart_positionZ end_positionrrrr?>s
 
z0UploadSeekableInputManager.provide_transfer_sizecKs |<01>|d<00>}t<01>|<03>t|<03>fS)Nr[)r'r<00>BytesIO<49>len)rr$r/<00>datarrrr^JszBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizecCs"|jjj}|<02><03>|jj}||fS)N)rOrPr$r+rQ)rr>r$rQrrrrVYs
zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN)
rrr r1rKr:r=r?r^rVrrrrrf2s   rfcsheZdZdZd<16>fdd<04> Zedd<06><00>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Z d d<0E>Z
dd<10>Z ddd<13>Z dd<15>Z <0A>ZS)<18>UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Ncstt|<00><02>|||<03>d|_dS)N<>)<04>superrlr<00> _initial_data)rr5r%r6)<01> __class__rrrcs

z&UploadNonSeekableInputManager.__init__cCst|<01>S)N)r)r8r9rrrr:hsz+UploadNonSeekableInputManager.is_compatiblecCsdS)NTr)rr<rrrr=lsz3UploadNonSeekableInputManager.stores_body_in_memorycCsdS)Nr)rr>rrrr?osz3UploadNonSeekableInputManager.provide_transfer_sizecCsT|jjdk r|jj|jkS|jjj}|j}|<00>||d<01>|_t|j<06>|krLdSdSdS)NFT)rOrQrRrPr$<00>_readrorj)rr>r@r$rrrrrAts 
z7UploadNonSeekableInputManager.requires_multipart_uploadcCs@|<00>|<01>}|<00>|<02>}|jjj}|<00>|j|<04><07>||<03>}d|_|S)N)rGrJrOrPr$<00>
_wrap_dataror')rr>rrUr$<00>bodyrrrrB<00>s


z1UploadNonSeekableInputManager.get_put_object_bodyc csb|jjj}d}xN|<00>|<01>}|<00>|<05>}|d7}|<00>||<02>}|s>P|<00>|||<06>}d}||fVqWdS)NrrY)rOrPr$rGrJrqrr) rr>rCZ file_objectr_rrUZ part_contentZ part_objectrrrrD<00>s


 
z6UploadNonSeekableInputManager.yield_upload_part_bodiesTcCsxt|j<01>dkr|<01>|<02>S|t|j<01>krL|jd|<02>}|rH|j|d<02>|_|S|t|j<01>}|j|<01>|<05>}|rtd|_|S)a=
Reads a specific amount of data from a stream and returns it. If there
is any data in initial_data, that will be popped out first.
:type fileobj: A file-like object that implements read
:param fileobj: The stream to read from.
:type amount: int
:param amount: The number of bytes to read from the stream.
:type truncate: bool
:param truncate: Whether or not to truncate initial_data after
reading from it.
:return: Generator which generates part bodies from the initial data.
rNrm)rjror')rr$r(<00>truncaterkZamount_to_readrrrrq<00>s
z#UploadNonSeekableInputManager._readcCs.|<00>t<01>|<01><01>}|jj|t|<01>t|<01>||d<01>S)a<>
Wraps data with the interrupt reader and the file chunk reader.
:type data: bytes
:param data: The data to wrap.
:type callbacks: list
:param callbacks: The callbacks associated with the transfer future.
:type close_callbacks: list
:param close_callbacks: The callbacks to be called when closing the
wrapper for the data.
:return: Fully wrapped data.
)r$rSrTrrU)rErrir3rWrj)rrkrrUr$rrrrr<00>sz(UploadNonSeekableInputManager._wrap_data)N)T)rrr r1rrKr:r=r?rArBrDrqrr<00> __classcell__rr)rprrlas  
*rlc@s\eZdZdZddddgZdgZdd<07>Zdd d
<EFBFBD>Zd d <0C>Zd d<0E>Z dd<10>Z
dd<12>Z dd<14>Z dS)<16>UploadSubmissionTaskz.Task for submitting tasks to execute an uploadZSSECustomerKeyZSSECustomerAlgorithmZSSECustomerKeyMD5Z RequestPayercCsHtttg}|jjj}x|D]}|<04>|<03>r|SqWtd|t|<03>f<00><01>dS)aoRetrieves a class for managing input for an upload based on file type
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future for the request
:rtype: class of UploadInputManager
:returns: The appropriate class to use for managing a specific type of
input for uploads.
z&Input %s of type: %s is not supported.N) rLrfrlrOrPr$r:<00> RuntimeError<6F>type)rr>Zupload_manager_resolver_chainr$Zupload_manager_clsrrr<00>_get_upload_input_manager_cls<6C>s 


z2UploadSubmissionTask._get_upload_input_manager_clsNcCsd|<00>|<05>||j|<06>}|jjdkr*|<07>|<05>|<07>||<02>sL|<00>||||||<07>n|<00>||||||<07>dS)a<>
:param client: The client associated with the transfer manager
:type config: s3transfer.manager.TransferConfig
:param config: The transfer config associated with the transfer
manager
:type osutil: s3transfer.utils.OSUtil
:param osutil: The os utility associated to the transfer manager
:type request_executor: s3transfer.futures.BoundedExecutor
:param request_executor: The request executor associated with the
transfer manager
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future associated with the
transfer request that tasks are being submitted for
N)ryr#rOrQr?rA<00>_submit_upload_request<73>_submit_multipart_request)r<00>clientr@r5<00>request_executorr>r6<00>upload_input_managerrrr<00>_submit s  


zUploadSubmissionTask._submitc
CsN|jj}|<00>|d<01>}|jj|t|j||<06>|<05>|j|j|j d<02>dd<04>|d<05>dS)Nrg)r|r$<00>bucket<65>key<65>
extra_argsT)r%<00> main_kwargs<67>is_final)<01>tag)
rOrP<00>_get_upload_task_tagr#<00>submit<69> PutObjectTaskrBr<>r<>r<>) rr|r@r5r}r>r~rPZput_object_tagrrrrz2sz+UploadSubmissionTask._submit_upload_requestcCs|jj}|j<02>|t|j||j|j|jd<01>d<02><02>}g} |<00>|j<07>}
|<00> |d<03>} |jj
} t <0B>} | <0A> |j | <0C>}|<06>||<0E>}xJ|D]B\}}| <09>|jj|t|j|||j|j||
d<04>d|id<06>| d<07><03>qzW|<00>|j<07>}|j<02>|t|j||j|j|d<01>|| d<08>d d
<EFBFBD><04>dS) N)r|r<>r<>r<>)r%r<><00> upload_part)r|r$r<>r<>r_r<><00> upload_id)r%r<><00>pending_main_kwargs)r<>)r<><00>partsT)r%r<>r<>r<>)rOrPr#r<>rr<>r<>r<><00>_extra_upload_part_argsr<73>rQr Zadjust_chunksizeZmultipart_chunksizerD<00>append<6E>UploadPartTask<73>_extra_complete_multipart_argsr )rr|r@r5r}r>r~rPZcreate_multipart_futureZ part_futuresZextra_part_argsZupload_part_tagrQZadjusterrCZ part_iteratorr_r$Zcomplete_multipart_extra_argsrrrr{LsZ 
z.UploadSubmissionTask._submit_multipart_requestcCs t||j<01>S)N)r <00>UPLOAD_PART_ARGS)rr<>rrrr<><00>sz,UploadSubmissionTask._extra_upload_part_argscCs t||j<01>S)N)r <00>COMPLETE_MULTIPART_ARGS)rr<>rrrr<><00>sz3UploadSubmissionTask._extra_complete_multipart_argscCsd}|<01>|<02>rt}|S)N)r=r)rr~r<r<>rrrr<><00>s
z)UploadSubmissionTask._get_upload_task_tag)N) rrr r1r<>r<>ryrrzr{r<>r<>r<>rrrrrv<00>s
&Mrvc@seZdZdZdd<03>ZdS)r<>z Task to do a nonmultipart uploadc Cs,|<02>}|jf|||d<01>|<05><02>WdQRXdS)aP
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
)<03>Bucket<65>Key<65>BodyN)rg)rr|r$r<>r<>r<>rsrrr<00>_main<69>s zPutObjectTask._mainN)rrr r1r<>rrrrr<><00>sr<>c@seZdZdZdd<03>ZdS)r<>z+Task to upload a part in a multipart uploadc Cs>|<02>"}|jf|||||d<01>|<07><02>} WdQRX| d}
|
|d<04>S)a<>
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param upload_id: The id of the upload
:param part_number: The number representing the part of the multipart
upload
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
:rtype: dict
:returns: A dictionary representing a part::
{'Etag': etag_value, 'PartNumber': part_number}
This value can be appended to a list to be used to complete
the multipart upload.
)r<>r<>ZUploadId<49>
PartNumberr<EFBFBD>N<>ETag)r<>r<>)r<>) rr|r$r<>r<>r<>r_r<>rs<00>response<73>etagrrrr<><00>szUploadPartTask._mainN)rrr r1r<>rrrrr<><00>sr<>)rcZbotocore.compatrZs3transfer.compatrrZs3transfer.futuresrZs3transfer.tasksrrrr Zs3transfer.utilsr
r r r <00>objectrr!r2rLrfrlrvr<>r<>rrrr<00><module> s*        !+yQ/D
Reference in New Issue Copy Permalink