-=[ Mr. Bumblebee ]=-
_Indonesia_
ó
ô¤áQc @@ s d Z d d l m Z d d l m Z d d l Z d d l m Z m Z m Z m Z d d l
Z
d d l Z d d l Z d d l
m Z d d g Z d „ Z d
„ Z e j e j e j e j e j e j d „ Z e j e j e j e j e j e j d „ Z d e j f d
„ ƒ YZ d S( sI Bzrlib specific gzip tunings. We plan to feed these to the upstream gzip.i ( t absolute_import( t StringION( t FEXTRAt FCOMMENTt FNAMEt FHCRC( t symbol_versioningt GzipFilet
bytes_to_gzipc C@ s | d k r | d 7} n | S( s‹ Return i as an unsigned integer, assuming it fits in 32 bits.
If it's >= 2GB when viewed as a 32-bit unsigned int, return a long.
i l i l ( ( t i( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt U32% s
c C@ s | d @S( s> Return the low-order 32 bits of an int, as a non-negative int.l ÿÿ ( ( R ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt LOWU32/ s c C@ s
t | g ƒ S( s; Create a gzip file containing bytes and return its content.( t chunks_to_gzip( t bytest factoryt levelt methodt widtht memt crc32( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR 4 s c
C@ sÊ d g } | | | | | d ƒ } d } d }
xR | D]J } | | | ƒ } |
t | ƒ 7}
| j | ƒ } | r4 | j | ƒ q4 q4 W| j | j ƒ ƒ | j t j d t | ƒ t |
ƒ ƒ ƒ d j | ƒ S( sœ Create a gzip file containing chunks and return its content.
:param chunks: An iterable of strings. Each string can have arbitrary
layout.
s
‹ ÿi s <LLt ( t lent compresst appendt flusht structt packR t join(
t chunksR R R R R R t resultR t crct total_lent chunkt zbytes( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR < s
(c B@ sÈ e Z d Z d „ Z e j d k r0 d „ Z n d „ Z d „ Z e j d k ri d d „ Z d
„ Z n d d „ Z d d
„ Z d d „ Z
d d „ Z d „ Z d „ Z e j d k rÆ d „ Z n RS( s» Knit tuned version of GzipFile.
This is based on the following lsprof stats:
python 2.4 stock GzipFile write:
58971 0 5644.3090 2721.4730 gzip:193(write)
+58971 0 1159.5530 1159.5530 +<built-in method compress>
+176913 0 987.0320 987.0320 +<len>
+58971 0 423.1450 423.1450 +<zlib.crc32>
+58971 0 353.1060 353.1060 +<method 'write' of 'cStringIO.
StringO' objects>
tuned GzipFile write:
58971 0 4477.2590 2103.1120 bzrlib.knit:1250(write)
+58971 0 1297.7620 1297.7620 +<built-in method compress>
+58971 0 406.2160 406.2160 +<zlib.crc32>
+58971 0 341.9020 341.9020 +<method 'write' of 'cStringIO.
StringO' objects>
+58971 0 328.2670 328.2670 +<len>
Yes, its only 1.6 seconds, but they add up.
c O@ s= t j t j d ƒ d t d d ƒt j j | | | Ž d S( Ni i i s bzrlib.tuned_gzip.GzipFilet
stacklevel( i i i ( R t warnt
deprecated_int DeprecationWarningt gzipR t __init__( t selft argst kwargs( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR' { s
i i i c C@ s| t | ƒ } t j | | j ƒ d @| _ | j | j } | j | | | _ | j | | _ | j | _ | j | | _ d S( Nl ÿÿ ( R t zlibR R t offsett
extrastartt extrabuft extrasizet size( R( t datat len_dataR, ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt _add_read_dataƒ s c C@ sU t | ƒ } t j | | j ƒ | _ | j | 7_ | j | 7_ | j | 7_ d S( N( R R+ R R R. R/ R0 ( R( R1 R2 ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR3 s
c C@ s | j j d ƒ d S( s. A tuned version of gzip._write_gzip_header
We have some extra constrains that plain Gzip does not.
1) We want to write the whole blob at once. rather than multiple
calls to fileobj.write().
2) We never have a filename
3) We don't care about the time
s
‹ ÿN( t fileobjt write( R( ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt _write_gzip_header™ s i c C@ s | j d k r t d ‚ n | j r‡ | j j d ƒ } | d k rN t d ‚ n | j ƒ | j | ƒ t j t j ƒ | _
t | _ n | j j | ƒ } | d k r| j | j
j
ƒ ƒ t | j
j ƒ d k râ t d ƒ ‚ n | j
j d d !| _ | j ƒ t d ‚ n | j | j
j
| ƒ ƒ | j
j d k rýt | j
j ƒ d } | d k rˆ| j j | d ƒ | j
j d d !| _ n_ | d k rØd | k s¯t d ƒ ‚ n | j j | ƒ } | j
j | | _ n | j
j | _ | j ƒ t | _ n d S(
Ns Reached EOFi
R i s what does flush do?i i iøÿÿÿs too great a seek( R4 t Nonet EOFErrort _new_membert readt
_init_readt _read_gzip_headerR+ t
decompressobjt MAX_WBITSt
decompresst FalseR3 R R t unused_datat AssertionErrort
_gzip_tailt _read_eoft seekt True( R( R0 t next_header_bytest buft seek_length( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt _read² s@
c C@ s˜ t | j ƒ d k s$ t d ƒ ‚ n t j d | j ƒ \ } } | t | j ƒ k rs t d | t | j ƒ f ‚ n! | t | j ƒ k r” t d ‚ n d S( sð tuned to reduce function calls and eliminate file seeking:
pass 1:
reduces lsprof count from 800 to 288
4168 in 296
avoid U32 call by using struct format L
4168 in 200
i s! gzip trailer is incorrect length.s <LLs CRC check failed %d %ds! Incorrect length of data producedN(
R RC RB R t unpackR
R t IOErrorR R0 ( R( R t isize( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyRD ÷ s c C@ s€ | d
k r! | j j d ƒ } n | d d !} | d k rF t d ‚ n t | d d !ƒ } | d k rq t d ‚ n t | d d !ƒ } | t @rÙ t | j j d
ƒ ƒ } | d t | j j d
ƒ ƒ } | j j | ƒ n | t @r x6 t r| j j d
ƒ } | s| d k ræ Pqæ qæ Wn | t @r_x6 t r[| j j d
ƒ } | sT| d k r)Pq)q)Wn | t @r|| j j d ƒ n d
S( sq Supply bytes if the minimum header size is already read.
:param bytes: 10 bytes of header data.
i
i i s ‹s Not a gzipped filei i s Unknown compression methodi i i t N(
R7 R4 R: RL t ordR R RF R R ( R( R
t magicR t flagt xlent s( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR< s2
iÿÿÿÿc C@ sq | d k r t j } n g } t d | ƒ } x=t rl| d k rO d j | ƒ S| j | ƒ } t | ƒ } | j d ƒ } | d k rÆ | d k rª | | k rª | d } qÆ | | k rÆ | d } qÆ n | d k sÞ | d k r?| j | | d ƒ | | d | j
| _
| | | j | _ | j | | 8_ d j | ƒ S| j | ƒ | | } t | | d ƒ } q0 Wd S( sõ Tuned to remove buffer length calls in _unread and...
also removes multiple len(c) calls, inlines _unread,
total savings - lsprof 5800 to 5300
phase 2:
4168 calls in 2233
8176 calls to read() in 1684
changing the min chunk size to 200 halved all the cache misses
leading to a drop to:
4168 calls in 1977
4168 call to read() in 1646
- i.e. just reduced the function call overhead. May be worth
keeping.
i iÈ R s
iÿÿÿÿi i N(
t syst maxintt minRF R R: R t findR7 R R. R/ R, ( R( R0 t bufst readsizet ct len_cR ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt readline9 s0
i c C@ s t | j d ƒ ƒ } | j ƒ S( Niÿÿÿÿ( R R: t readlines( R( t sizehintt content( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR] n s c C@ sN | d k r t | ƒ } n | | j | _ | | j | _ | j | 8_ d S( sq tuned to remove unneeded len calls.
because this is such an inner routine in readline, and readline is
in many inner loops, this has been inlined into readline().
The len_buf parameter combined with the reduction in len calls dropped
the lsprof ms count for this routine on my test data from 800 to 200 -
a 75% saving.
N( R7 R R. R/ R, ( R( RH t len_buf( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt _unread€ s
c C@ sÀ | j t j k r3 d d l } t | j d ƒ ‚ n | j d k rN t d ‚ n t | ƒ } | d k r¼ | j
| | _
t j | | j
ƒ | _
| j j | j j | ƒ ƒ | j | 7_ n d S( Ni s$ write() on read-only GzipFile objects! write() on closed GzipFile object( t modeR& t WRITEt errnoRL t EBADFR4 R7 t
ValueErrorR R0 R+ R R R5 R R, ( R( R1 Rd t data_len( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR5 s c C@ s | j d j | ƒ ƒ d S( NR ( R5 R ( R( t lines( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt
writelinesž s c C@ sK | j d k r d S| j t j k r7 | j d M_ n t j j | ƒ d S( Nl ÿÿ ( R4 R7 Rb R& Rc R R t close( R( ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyRj § s
( i i i ( i i i N( i i ( t __name__t
__module__t __doc__R' RT t version_infoR3 R6 RJ RD R7 R< R\ R] Ra R5 Ri Rj ( ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyR d s"
E + 5 ( Rm t
__future__R t cStringIOR R& R R R R RT R R+ t bzrlibR t __all__R
R t compressobjt Z_DEFAULT_COMPRESSIONt DEFLATEDR> t
DEF_MEM_LEVELR R R R ( ( ( s5 /usr/lib/python2.7/dist-packages/bzrlib/tuned_gzip.pyt <module> s( "
%
Copyright © 2017 || Recoded By Mr.Bumblebee