���ZB � @ s d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m
Z
d d l m Z m
Z
d d l m Z d d l m Z m Z d d l m Z m Z m Z m Z m Z m Z m Z m Z m Z d d
l m Z d d l m Z m! Z! m" Z" d d l m# Z# m$ Z$ m% Z% m& Z& d d
l' m( Z( d d l) m* Z* y d d l+ Z+ Wn e, k
r�d Z+ Yn Xd d d f Z- Gd d � d � Z. Gd d � d � Z/ Gd d � d � Z0 Gd d � d � Z1 Gd d � d e0 � Z2 Gd d � d e0 � Z3 d S)� N)� defaultdict)�suppress)� SimpleCookie)�cycle�islice)� monotonic� )�hdrs�helpers) �ClientConnectionError� ClientConnectorCertificateError�ClientConnectorError�ClientConnectorSSLError�ClientHttpProxyError�ClientProxyConnectionError�ServerFingerprintMismatch�certificate_errors�
ssl_errors)�ResponseHandler)�
ClientRequest� Fingerprint�_merge_ssl_params)�PY_36�
is_ip_address�noop�sentinel)�EventResultOrError)�DefaultResolver�
BaseConnector� TCPConnector�
UnixConnectorc @ s� e Z d Z d Z d Z d d � Z d d � Z e d d � Z e d d � � Z
e d
d � � Z e d d
� � Z e d d � � Z
d d � Z d d � Z d d � Z d d � Z d d � Z e d d � � Z d S) �
ConnectionNc C sX | | _ | | _ | | _ | | _ g | _ | j � rT t j t j d � � | _
d S)Nr ) �_key�
_connector�_loop� _protocol�
_callbacks� get_debug� traceback�
extract_stack�sys� _getframe�_source_traceback)�self� connector�key�protocol�loop� r2 �
/connector.py�__init__) s zConnection.__init__c C s d j | j � S)NzConnection<{}>)�formatr" )r- r2 r2 r3 �__repr__3 s zConnection.__repr__c C s� | j d k r� t r$ d | i } n i } | j d j | � t | � | j j � rY d S| j j | j | j d d �d | d d i } | j
d k r� | j
| d <| j j | � d S) N�sourcezUnclosed connection {!r}� should_closeTZclient_connection�messagezUnclosed connection�source_traceback) r% r �warnr5 �ResourceWarningr$ � is_closedr# �_releaser" r, �call_exception_handler)r- � _warnings�kwargs�contextr2 r2 r3 �__del__6 s
zConnection.__del__c C s | j S)N)r$ )r- r2 r2 r3 r1 K s zConnection.loopc C s
| j j S)N)r% � transport)r- r2 r2 r3 rD O s zConnection.transportc C s | j S)N)r% )r- r2 r2 r3 r0 S s zConnection.protocolc C s
| j j S)N)r% �writer)r- r2 r2 r3 rE W s zConnection.writerc C s | d k r | j j | � d S)N)r& �append)r- �callbackr2 r2 r3 � add_callback[ s zConnection.add_callbackc
C sM | j d d � g } | _ x) | D]! } t t � � | � Wd QRXq$ Wd S)N)r& r � Exception)r- Z callbacks�cbr2 r2 r3 �_notify_release_ s
zConnection._notify_releasec C sE | j � | j d k rA | j j | j | j d d �d | _ d S)Nr8 T)rK r% r# r> r" )r- r2 r2 r3 �closef s
zConnection.closec C sK | j � | j d k rG | j j | j | j d | j j �d | _ d S)Nr8 )rK r% r# r> r" r8 )r- r2 r2 r3 �releasen s
zConnection.releasec C s9 | j � | j d k r, | j j | j � d | _ d S)N)rK r% r# �_release_acquired)r- r2 r2 r3 �detachw s
zConnection.detachc C s | j d k p | j j � S)N)r% � is_connected)r- r2 r2 r3 �closed~ s zConnection.closed)�__name__�
__module__� __qualname__r, �
_transportr4 r6 �warningsrC �propertyr1 rD r0 rE rH rK rL rM rO rQ r2 r2 r2 r3 r! $ s
r! c @ s" e Z d Z d Z d d � Z d S)�_TransportPlaceholderz0 placeholder for BaseConnector.connect function c C s d S)Nr2 )r- r2 r2 r3 rL � s z_TransportPlaceholder.closeN)rR rS rT �__doc__rL r2 r2 r2 r3 rX � s rX c @ sK e Z d Z d Z d Z d Z d Z d e d d d d d
d d d d
d d d � Z e d d � Z
d d � Z d d � Z e
d d � � Z e
d d � � Z e
d d � � Z d d � Z d d � Z d d! � Z d" d# � Z e
d$ d% � � Z d d& d' � Z d( d) � Z d* d+ � Z d, d- � Z d. d d/ d0 � Z d d1 d2 � Z d S)3r a� Base connector class.
keepalive_timeout - (optional) Keep-alive timeout.
force_close - Set to True to force close and do reconnect
after each request (and between redirects).
limit - The total number of simultaneous connections.
limit_per_host - Number of simultaneous connections to one host.
enable_cleanup_closed - Enables clean-up closed ssl transports.
Disabled by default.
loop - Optional event loop.
TNg @�keepalive_timeout� force_closeF�limit�d �limit_per_hostr �enable_cleanup_closedr1 c C s> | r- | d k r? | t k r? t d � � n | t k r? d } | d k rW t j � } d | _ | j � r� t j t j d � � | _
i | _ | | _ | | _
t � | _ t t � | _ | | _ | | _ t t � | _ | | _ t j t d | �| _ t � | _ d | _ d | _ | | _ g | _ | j! � d S)Nz6keepalive_timeout cannot be set if force_close is Trueg .@Fr r1 )"r �
ValueError�asyncioZget_event_loop�_closedr' r( r) r* r+ r, �_conns�_limit�_limit_per_host�set� _acquiredr �_acquired_per_host�_keepalive_timeout� _force_close�list�_waitersr$ � functools�partialr �_factoryr Zcookies�_cleanup_handle�_cleanup_closed_handle�_cleanup_closed_disabled�_cleanup_closed_transports�_cleanup_closed)r- rZ r[ r\ r^ r_ r1 r2 r2 r3 r4 � s6
zBaseConnector.__init__c C s� | j r
d S| j s d Sd d � | j j � D� } | j � t rU d | i } n i } | j d j | � t | � d | d | d d i } | j d k r� | j | d <| j j
| � d S)
Nc S s g | ] } t | � � q Sr2 )�repr)�.0�cr2 r2 r3 �
<listcomp>� s z)BaseConnector.__del__.<locals>.<listcomp>r7 zUnclosed connector {!r}r. � connectionsr9 zUnclosed connectorr: ) rb rc �valuesrL r r; r5 r< r, r$ r? )r- r@ �connsrA rB r2 r2 r3 rC � s$
zBaseConnector.__del__c C s | S)Nr2 )r- r2 r2 r3 � __enter__� s zBaseConnector.__enter__c G s | j � d S)N)rL )r- �excr2 r2 r3 �__exit__� s zBaseConnector.__exit__c C s | j S)z1Ultimately close connection on releasing if True.)rj )r- r2 r2 r3 r[ � s zBaseConnector.force_closec C s | j S)z�The total number for simultaneous connections.
If limit is 0 the connector has no limit.
The default limit size is 100.
)rd )r- r2 r2 r3 r\ � s zBaseConnector.limitc C s | j S)z�The limit_per_host for simultaneous connections
to the same endpoint.
Endpoints are the same if they are have equal
(host, port, is_ssl) triple.
)re )r- r2 r2 r3 r^ � s z BaseConnector.limit_per_hostc C s* | j r | j j � | j j � } | j } | j r� i } | | } x� | j j � D]� \ } } g } xv | D]n \ } } | j � rp | | d k r� | j � }
| d r� | j r� | j
j |
� qp | j | | f � qp W| rW | | | <qW W| | _ | j r&t j
| d | | j � | _ d S)zCleanup unused transports.r r �_cleanupN���)rp �cancelr$ �timeri rc �itemsrP rL rr rs rF r
�weakref_handle) r- Znow�timeoutry �deadliner/ r{ �alive�protoZuse_timerD r2 r2 r3 r � s.
zBaseConnector._cleanupc C sD | j } | | k r d S| | } | j | � | s@ | j | =d S)N)rh �remove)r- r/ �valZacquired_per_hostr{ r2 r2 r3 �_drop_acquired_per_host s
z%BaseConnector._drop_acquired_per_hostc C sw | j r | j j � x' | j D] } | d k r | j � q Wg | _ | j ss t j | d | j | j � | _ d S)z}Double confirmation for transport close.
Some broken ssl servers may leave socket open without proper close.
Nrt ) rq r� rs �abortrr r
r� �_cleanup_closed_periodr$ )r- rD r2 r2 r3 rt ( s
z BaseConnector._cleanup_closedc C s- | j r
d Sd | _ z� | j j � r/ t � S| j rE | j j � | j r[ | j j � x8 | j j � D]' } x | D] \ } } | j � qx Wqk Wx | j
D] } | j � q� Wx' | j D] } | d k r� | j � q� WWd | j j
� | j
j
� | j j
� d | _ | j j
� d | _ Xd S)z Close all opened transports.NT)rb r$ r= r rp r� rq rc rz rL rg rs r� �clearrl )r- �datar� �t0rD r2 r2 r3 rL : s0
zBaseConnector.closec C s | j S)z;Is connector closed.
A readonly property.
)rb )r- r2 r2 r3 rQ ` s zBaseConnector.closedc � s | j } | j rn | j t | j � } | j r� | d k r� | | j k r� | j t | j j | � � } n@ | j r� | | j k r� | j t | j j | � � } n d } | d k r]| j j � } | j | } | j
| � | r x | D] } | j � Id Hq� Wz
| Id HWd | j | � | s6| j | =X| r]x | D] } | j
� Id HqDW| j | � } | d k r�t � } | j j | � | j | j | � | r�x | D] } | j � Id Hq�Wz� y= | j | d | �Id H} | j r
| j � t d � � WnV t k
rc| | j k r\x/ | j | D] }
|
j � s8|
j d � Pq8W� Yn XWd | j s�| j j | � | j | | � X| r�xF | D] } | j � Id Hq�Wn& | r�x | D] } | j � Id Hq�W| j j | � | j | j | � t | | | | j � S)z'Get from pool or create new connection.r r N�traceszConnector is closed.) �connection_keyrd �lenrg re rh �getr$ Z
create_futurerl rF Z send_connection_queued_startr� Zsend_connection_queued_end�_getrX �addZ send_connection_create_start�_create_connectionrb rL r �
BaseException�done�
set_resultr� Zsend_connection_create_endZsend_connection_reuseconnr! ) r- �reqr� r/ Z availableZfut�waiters�tracer� Z placeholder�waiterr2 r2 r3 �connecth sx
zBaseConnector.connectc C s� y | j | } Wn t k
r) d SYn X| j j � } x� | r� | j � \ } } | j � r< | | | j k r� | j � } | d r� | j r� | j j
| � q< | s� | j | =| Sq< W| j | =d S)Nr r� ) rc �KeyErrorr$ r� �poprP ri rL rr rs rF )r- r/ r{ �t1r� r� rD r2 r2 r3 r� � s"
zBaseConnector._getc C s� | j rp | j t | j � d k r� x� | j j � D]4 \ } } | r5 | d j � sh | d j d � Pq5 WnQ | j r� xE | j j � D]4 \ } } | r� | d j � s� | d j d � Pq� Wd S)Nr )rd r� rg rl r� r� r� re )r- r/ r� r2 r2 r3 �_release_waiter� s z BaseConnector._release_waiterc C sT | j r
d Sy$ | j j | � | j | | � Wn t k
rE Yn X| j � d S)N)rb rg r� r� r� r� )r- r/ r� r2 r2 r3 rN � s
z BaseConnector._release_acquiredr8 c C s� | j r
d S| j | | � | j r, d } | s; | j rn | j � } | d r� | j r� | j j | � n{ | j j | � } | d k r� g } | j | <| j | | j
j � f � | j d k r� t
j | d | j | j
� | _ d S)NTr r r� )rb rN rj r8 rL rr rs rF rc r� r$ r� rp r
r� ri )r- r/ r0 r8 rD r{ r2 r2 r3 r> � s zBaseConnector._releasec � s
t � � d S)N)�NotImplementedError)r- r� r� r2 r2 r3 r� s z BaseConnector._create_connection) rR rS rT rY rb r, r� r r4 rV rC r| r~ rW r[ r\ r^ r r� rt rL rQ r� r� r� rN r> r� r2 r2 r2 r3 r � s0 ) &X c @ sg e Z d Z d d d � Z d d � Z d d � Z d d � Z d
d � Z d d
� Z d d � Z d S)�_DNSCacheTableNc C s i | _ i | _ | | _ d S)N)� _addrs_rr� _timestamps�_ttl)r- �ttlr2 r2 r3 r4 s z_DNSCacheTable.__init__c C s
| | j k S)N)r� )r- �hostr2 r2 r3 � __contains__! s z_DNSCacheTable.__contains__c C s< t | � t | � f | j | <| j r8 t � | j | <d S)N)r r� r� r� r r� )r- r� �addrsr2 r2 r3 r� $ s z_DNSCacheTable.addc C s3 | j j | d � | j r/ | j j | d � d S)N)r� r� r� r� )r- r� r2 r2 r3 r� * s z_DNSCacheTable.removec C s | j j � | j j � d S)N)r� r� r� )r- r2 r2 r3 r� 0 s
z_DNSCacheTable.clearc C s6 | j | \ } } t t | | � � } t | � | S)N)r� rk r �next)r- r� r1 �lengthr� r2 r2 r3 �
next_addrs4 s
z_DNSCacheTable.next_addrsc C s. | j d k r d S| j | | j t � k S)NF)r� r� r )r- r� r2 r2 r3 �expired; s z_DNSCacheTable.expired)
rR rS rT r4 r� r� r� r� r� r� r2 r2 r2 r3 r� s r� c ! sZ e Z d Z d Z d d d d d d d d d d
d d d d d
d d d d e d d d d d d
d d d d � f d d � Z � f d d � Z e d d � � Z e d d � � Z d d d d � Z
d d! d" � Z d d# d$ � Z e
e j d � d% d& � � � Z d' d( � Z d) d* � Z d+ e d, d- � Z d+ e d. d d/ d0 � Z d d1 d2 � Z � S)3r a/ TCP connector.
verify_ssl - Set to True to check ssl certifications.
fingerprint - Pass the binary sha256
digest of the expected certificate in DER format to verify
that the certificate the server presents matches. See also
https://en.wikipedia.org/wiki/Transport_Layer_Security#Certificate_pinning
resolver - Enable DNS lookups and use this
resolver
use_dns_cache - Use memory cache for DNS lookups.
ttl_dns_cache - Max seconds having cached a DNS entry, None forever.
family - socket address family
local_addr - local tuple of (host, port) to bind socket to
keepalive_timeout - (optional) Keep-alive timeout.
force_close - Set to True to force close and do reconnect
after each request (and between redirects).
limit - The total number of simultaneous connections.
limit_per_host - Number of simultaneous connections to one host.
enable_cleanup_closed - Enables clean-up closed ssl transports.
Disabled by default.
loop - Optional event loop.
�
verify_sslT� fingerprintN�
use_dns_cache�
ttl_dns_cache�
�familyr � ssl_context�ssl�
local_addr�resolverrZ r[ Fr\ r] r^ r_ r1 c
s� t � j d |
d | d | d |
d | d | � t | | | | � | _ | d k rg t d | j � } | | _ | | _ t d | � | _ i | _
| | _ | | _ d S)NrZ r[ r\ r^ r_ r1 r� )
�superr4 r �_sslr r$ � _resolver�_use_dns_cacher� �
_cached_hosts�_throttle_dns_events�_family� _local_addr)r- r� r� r� r� r� r� r� r� r� rZ r[ r\ r^ r_ r1 )� __class__r2 r3 r4 [ s zTCPConnector.__init__c s5 x! | j j � D] } | j � q Wt � j � d S)z Close all ongoing DNS calls.N)r� rz r� r� rL )r- Zev)r� r2 r3 rL s s zTCPConnector.closec C s | j S)zSocket family like AF_INET.)r� )r- r2 r2 r3 r� z s zTCPConnector.familyc C s | j S)z%True if local DNS caching is enabled.)r� )r- r2 r2 r3 r� s zTCPConnector.use_dns_cachec C si | d k r1 | d k r1 | j j | | f � n4 | d k sI | d k rX t d � � n
| j j � d S)z8Remove specified host/port or clear all dns local cache.Nz5either both host and port or none of them are allowed)r� r� r` r� )r- r� �portr2 r2 r3 �clear_dns_cache� s
z TCPConnector.clear_dns_cachec � s� t | � r: d | d | d | d | j d d d d i g S| j s� | rl x | D] } | j | � Id HqP W| j j | | d | j �Id H} | r� x | D] } | j | � Id Hq� W| S| | f } | | j k r#| j j | � r#| rx | D] } | j | � Id Hq� W| j j
| � S| | j k rt| r[x | D] } | j | � Id Hq?W| j | j � Id HnH| r�x | D] } | j
| � Id Hq�Wt | j � | j | <z� y� | r�x | D] } | j | � Id Hq�Wt j | j j | | d | j �d | j �Id H} | r@x | D] } | j | � Id Hq$W| j j | | � | j | j � Wn? t k
r�} z | j | j d | � � WYd d } ~ Xn XWd | j j | � X| j j
| � S)
N�hostnamer� r� r� r� r �flagsr1 r} )r r� r� Zsend_dns_resolvehost_startr� ZresolveZsend_dns_resolvehost_endr� r� Zsend_dns_cache_hitr� r� �waitZsend_dns_cache_missr r$ ra Zshieldr� rf r� r� ) r- r� r� r� r� �resr/ r� �er2 r2 r3 �
_resolve_host� s^
zTCPConnector._resolve_hostc � sP | j r, | j | d d �Id H\ } } n | j | d d �Id H\ } } | S)zcCreate connection.
Has same keyword arguments as BaseEventLoop.create_connection.
r� N)�proxy�_create_proxy_connection�_create_direct_connection)r- r� r� �_r� r2 r2 r3 r� � s z TCPConnector._create_connectionc C sj | r t j � St j t j � } | j t j O_ | j t j O_ | j t j O_ | j � | Sd S)N) r� Zcreate_default_context�
SSLContextZPROTOCOL_SSLv23ZoptionsZ OP_NO_SSLv2Z OP_NO_SSLv3ZOP_NO_COMPRESSIONZset_default_verify_paths)Zverified�
sslcontextr2 r2 r3 �_make_ssl_context� s
z TCPConnector._make_ssl_contextc C s� | j � r� t d k r$ t d � � | j } t | t j � rC | S| d k r\ | j d � S| j } t | t j � r{ | S| d k r� | j d � S| j d � Sd Sd S)a* Logic to get the correct SSL context
0. if req.ssl is false, return None
1. if ssl_context is specified in req, use it
2. if _ssl_context is specified in self, use it
3. otherwise:
1. if verify_ssl is not specified in req, use self.ssl_context
(will generate a default context according to self.verify_ssl)
2. if verify_ssl is True in req, generate a default SSL context
3. if verify_ssl is False in req, generate a SSL context that
won't verify
NzSSL is not supported.FT)�is_sslr� � RuntimeError�
isinstancer� r� r� )r- r� r� r2 r2 r3 �_get_ssl_context� s
z TCPConnector._get_ssl_contextc C s< | j } t | t � r | S| j } t | t � r8 | Sd S)N)r� r� r r� )r- r� Zretr2 r2 r3 �_get_fingerprint s z TCPConnector._get_fingerprint� client_errorc � s� y | j j | | � Id HSWn� t k
rX } z t | j | � | � WYd d } ~ Xns t k
r� } z t | j | � | � WYd d } ~ Xn: t k
r� } z | | j | � | � WYd d } ~ Xn Xd S)N)r$ �create_connectionr r r� r r �OSError)r- r� r� �argsrA r} r2 r2 r3 �_wrap_create_connection s $'z$TCPConnector._wrap_create_connectionr� c - � s� | j | � } | j | � } y* | j | j j | j d | �Id H} Wn: t k
r� } z t | j | � | � WYd d } ~ Xn Xd } xM| D]?} | d }
| d } yv | j | j
|
| d | d | d d | d d | d d | r� | d n d d
| j d | d | �Id H\ } }
Wn. t k
rR} z | } w� WYd d } ~ Xn X| j � r�| r�y | j
| � WnQ t k
r�} z1 | j � | j s�| j j | � | } w� WYd d } ~ Xn X| |
f SW| � d S)
Nr� r� r� r� r� r� r� �server_hostnamer� r� r� r� )r� r� r� �urlZraw_hostr� r� r
r� r� ro r� r� �checkr rL rr rs rF )r- r� r� r� r� r� Zhostsr} �last_excZhinfor� r� Ztranspr� r2 r2 r3 r� ) sF (
z&TCPConnector._create_direct_connectionc � s� i } | j d k r | j } | j t j | t j <t t j | j d | d | j d | j d | j �} | j
| d t �Id H\ } } | j j t j
d � } | d k r� | j � s� | | j t j <n | | j t j <| j � r}| j | � } t j | _ | j | _ | j | j | j f } t | | | | j � }
| j |
� } z y | j |
d � Id H} Wn) t k
r�| j � |
j � � Yn� Xd |
_ d |
_ z| | j d k r�t | j | j d | j d | j! d | j �� | j" d
d d �}
|
d k rt# d � � |
j$ � }
Wd | j � X| j% | j& d | d
|
d | j d | �Id H\ } } Wd | j � X| | f S)N�headers�authr1 r� r� T�� �coder9 �socket�defaultz)Transport does not expose socket instance�sockr� r� )'Z
proxy_headersr� r ZHOSTr ZMETH_GETr� Z
proxy_authr$ r� r� r r� Z
AUTHORIZATIONr� ZPROXY_AUTHORIZATIONr� Z METH_CONNECT�methodr� r� r� r! �send�startr� rL r% rU �statusr Z request_info�history�reasonZget_extra_infor� �dupr� ro )r- r� r� r� Z proxy_reqrD r� r� r� r/ �connZ
proxy_respZrespZrawsockr2 r2 r3 r� Y sh
z%TCPConnector._create_proxy_connection)rR rS rT rY r r4 rL rW r� r� r� r� r� � staticmethodrm � lru_cacher� r� r� r
r� r� r� r2 r2 )r� r3 r B s.
E ! .c sX e Z d Z d Z d e d d d � f d d � Z e d d � � Z d d
d � Z � S) r a� Unix socket connector.
path - Unix socket path.
keepalive_timeout - (optional) Keep-alive timeout.
force_close - Set to True to force close and do reconnect
after each request (and between redirects).
limit - The total number of simultaneous connections.
limit_per_host - Number of simultaneous connections to one host.
loop - Optional event loop.
Fr] r Nc s8 t � j d | d | d | d | d | � | | _ d S)Nr[ rZ r\ r^ r1 )r� r4 �_path)r- �pathr[ rZ r\ r^ r1 )r� r2 r3 r4 � s zUnixConnector.__init__c C s | j S)zPath to unix socket.)r� )r- r2 r2 r3 r� � s zUnixConnector.pathc � sk y* | j j | j | j � Id H\ } } Wn: t k
rf } z t | j | � | � WYd d } ~ Xn X| S)N)r$ Zcreate_unix_connectionro r� r� r
r� )r- r� r� r� r� r} r2 r2 r3 r� � s !(z UnixConnector._create_connection) rR rS rT rY r r4 rW r� r� r2 r2 )r� r3 r � s
)4ra rm r* r( rV � collectionsr �
contextlibr Z http.cookiesr � itertoolsr r r� r � r r
Zclient_exceptionsr r r
r r r r r r Z client_protor Z
client_reqrepr r r r r r r Zlocksr r� r r� � ImportError�__all__r! rX r r� r r r2 r2 r2 r3 �<module> s: @ "
_� �(� d