R6�W5 � - @ s d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z y d d l
m Z Wn e k
r� d Z Yn Xy d d l
m
Z
Wn" e k
r� d d l m
Z
Yn Xy d d l m Z Wn e k
rd Z Yn Xd d l m Z d d l m Z y d d l m Z Wn> e k
r�d d l m Z d d l m Z d
d � Z Yn Xy d d l m Z Wn e k
r�d Z Yn Xd
Z d Z d Z d Z d Z d Z i Z! i Z" i Z# Gd d � d e$ � Z% e% � Z& e% � Z' d d � Z( d d � Z) d d � Z* d d � Z+ d d � Z, d d � Z- d! d" � Z. e* e � d d# d$ � � Z/ e* e � d% d& � � Z0 e* e � d' d( d) � � Z1 e* e � d* d+ � � Z2 e* e � d, d- � � Z3 e j4 d d. k r,e* e � d/ d0 d1 � � Z5 e* e � d2 d3 � � Z6 n3 e* e � d/ d4 d1 � � Z5 e* e � d5 d3 � � Z6 e* e � d6 d7 � � Z7 e* e � d d8 d9 � � Z8 e* e � d: d; � � Z9 e* e � d< d= � � Z: e* e � d> d? � � Z; e* e � d d d@ dA � � Z< e* e � dB dC � � Z= e* e � dD dE � � Z> e* e � dF dG � � Z? e* e � dH dI � � Z@ e* e � dJ dK � � ZA dL dM � ZB e* e � dN dO � � ZC e* e � dP dQ � � ZD e* e � dR dS � � ZE e* e � dT dU � � ZF e* e � dV dW � � ZG e* e � dX dY � � ZH GdZ d[ � d[ eI � ZJ Gd\ d] � d] eJ � ZK e j4 d d. � du k r`d_ d` � ZL n da db � ZL e( e � Gdc dd � dd eK � � ZM e( e � Gde df � df eK � � ZN e( e � Gdg dh � dh eI � � ZO e( e � Gdi dj � dj eI � � ZP e( e � Gdk dl � dl eJ � � ZQ e( e � Gdm dn � dn eJ � � ZR e( e � Gdo dp � dp eI � � ZS e d k r�e* e � e jT � ZT e* e � e jU � ZU e* e � e jV � ZV e( e � e jW � ZW e d k r
e) e � Gdq dr � dr e � � ZX e) e � Gds dt � dt e � � ZY d S)v� N)�Counter)�urlparse)�
TableFunction)�binary_construct)� unicode_type)�format_date_time_sqlite)�format_date_time)�SQLITE_DATETIME_FORMATSc C s
t | t � S)N)r r )Z
date_value� r
�/sqlite_udf.pyr ! s r )� _sqlite_udfZ control_flow�date�fileZhelpers�math�stringc sX e Z d Z � f d d � Z � f d d � Z � f d d � Z � f d d � Z � S) �synchronized_dictc s, t t | � j | | � t j � | _ d S)N)�superr �__init__� threading�Lock�_lock)�self�args�kwargs)� __class__r
r r 8 s zsynchronized_dict.__init__c
s+ | j � t t | � j | � SWd QRXd S)N)r r r � __getitem__)r �key)r r
r r < s
z synchronized_dict.__getitem__c
s. | j � t t | � j | | � SWd QRXd S)N)r r r � __setitem__)r r �value)r r
r r @ s
z synchronized_dict.__setitem__c
s+ | j � t t | � j | � SWd QRXd S)N)r r r � __delitem__)r r )r r
r r D s
z synchronized_dict.__delitem__)�__name__�
__module__� __qualname__r r r r r
r
)r r r 7 s r c s � f d d � } | S)Nc s6 x/ � D]' } t j | g � t | j | � q W| S)N)�AGGREGATE_COLLECTION�
setdefault�append)�klass�group)�groupsr
r � decoratorN s
z aggregate.<locals>.decoratorr
)r( r) r
)r( r � aggregateM s r* c s � f d d � } | S)Nc s6 x/ � D]' } t j | g � t | j | � q W| S)N)�TABLE_FUNCTION_COLLECTIONr$ r% )r& r' )r( r
r r) V s
z!table_function.<locals>.decoratorr
)r( r) r
)r( r �table_functionU s r, c s � f d d � } | S)Nc s6 x/ � D]' } t j | g � t | j | � q W| S)N)�UDF_COLLECTIONr$ r% )�fnr' )r( r
r r) ^ s
zudf.<locals>.decoratorr
)r( r) r
)r( r �udf] s r/ c G sz t � } xj | D]b } t | } xO | D]G } t | d | j � } | | k r' | j | � | j | d | � q' Wq Wd S)N�name� ���)�setr# �getattrr �addZcreate_aggregate)�connr( �seenr' �klassesr& r0 r
r
r �register_aggregate_groupsf s
r9 c G se t � } xU | D]M } t | } x: | D]2 } | j | k r' | j | j � | j | � q' Wq Wd S)N)r3 r+ r0 r5 �register)r6 r( r7 r' r8 r& r
r
r � register_table_function_groupsp s
r; c G sn t � } x^ | D]V } t | } xC | D]; } | j } | | k r' | j | � | j | d | � q' Wq Wd S)Nr1 r2 )r3 r- r r5 Zcreate_function)r6 r( r7 r' Z functions�functionr0 r
r
r �register_udf_groupsy s
r= c C s+ t | t � t | t � t | t � d S)N)r9 r# r; r+ r= r- )r6 r
r
r � register_all� s
r> c C s | r
| S| S)Nr
)ZcondZtruthyZfalseyr
r
r � if_then_else� s r? c C sh | j d d � } | j d � } | d k r; | d | � S| j d � } | d k rd | d | � S| S)N�T� �+r1 �-�
r2 )�replace�find)Zdate_strZtz_idx1Ztz_idx2r
r
r �strip_tz� s rG z, c C s� d } g } xa | D]Y \ } } t | | � \ } } | r | d
k rL d pO d } | j d | | | f � q W| sz d S| j | � S) N�Q im �year� �month� �week�day� �hour�< �minuter1 �second�s� z%s %s%sz 0 seconds�3��rV rI � �' �rX rK �: �rZ rM �rH rN �rO rP �rQ rR �r1 rS )rW rY r[ r\ r] r^ r_ )�divmodr% �join)ZnsecondsZglue�partsZaccum�offsetr0 �val�suffixr
r
r � human_delta� s rf c C s7 y t j j | � } Wn t k
r. d SYn X| d S)Nr1 )�os�path�splitext�
ValueError)�filename�resr
r
r �file_ext� s
rm c C s3 y$ t | � � } | j � SWd QRXWn Yn Xd S)N)�open�read)rk Zfhr
r
r � file_read� s
rp � � c C s t t j | | � � S)N)r �zlib�compress)�data� compressionr
r
r �gzip� s rw c C s
t j | � S)N)rs �
decompress)ru r
r
r �gunzip� s ry c C s t j t | � | � S)N)rs rt r )ru rv r
r
r rw � s c C s t j | � j d � S)Nzutf-8)rs rx �decode)ru r
r
r ry � s c C s t | � } | r | j Sd S)N)r Znetloc)ZurlZ parse_resultr
r
r �hostname� s r{ c C sA | j � } | d k r% | t | <n t j | � t | <} | S)N)�lower�STATE�get)r Zonr
r
r �toggle� s
r c G s> | s t j | � St | � d k r6 | d t | <n d Sd S)Nr1 r F)�SETTINGSr~ �len)r r r
r
r �setting� s
r� c C s t j � d S)N)r� �clearr
r
r
r �clear_settings� s r� c C s t j � d S)N)r} r� r
r
r
r �
clear_toggles� s r� c C sA | d k r d | } } n | d k r. d } t j | | | � S)Nr r1 )�random� randrange)�start�end�stepr
r
r � randomrange� s
r� c C s1 y t j | | � SWn t k
r, d SYn Xd S)N)r� �gaussrj )Zmean�sigmar
r
r �gauss_distribution� s
r� c C s. y t j | � SWn t k
r) d SYn Xd S)N)r �sqrtrj )�nr
r
r r� s
r� c C sD y t | � SWn/ t k
r? y t | � SWn d SYn XYn Xd S)N)�intrj �float)rT r
r
r �tonumber s
r� c C s | s | r d S| j | � S)Nr )�count)�haystackZneedler
r
r � substr_count s r� c C s t | � j | � S)N)r �strip)r� �charsr
r
r � strip_chars s r� c G s1 | � } x | D] } | j | � q W| j � S)N)�update� hexdigest)� constructorr Zhash_obj�argr
r
r �_hash s
r� c G s
t t j � S)N)r� �hashlib�md5)�valsr
r
r r� # s r� c G s
t t j � S)N)r� r� �sha1)r� r
r
r r� ' s r� c G s
t t j � S)N)r� r� �sha256)r� r
r
r r� + s r� c G s
t t j � S)N)r� r� �sha512)r� r
r
r r� / s r� c C s
t j | � S)N)rs �adler32)rT r
r
r r� 3 s r� c C s
t j | � S)N)rs �crc32)rT r
r
r r� 7 s r� c @ s4 e Z d Z d d � Z d d � Z d d � Z d S)� _heap_aggc C s g | _ d | _ d S)Nr )�heap�ct)r r
r
r r = s z_heap_agg.__init__c C s | S)Nr
)r r r
r
r �processA s z_heap_agg.processc C s/ | j d 7_ t j | j | j | � � d S)Nr1 )r� �heapq�heappushr� r� )r r r
r
r r� D s z_heap_agg.stepN)r r! r" r r� r� r
r
r
r r� < s r� c @ s e Z d Z d d � Z d S)�_datetime_heap_aggc C s
t | � S)N)r )r r r
r
r r� I s z_datetime_heap_agg.processN)r r! r" r� r
r
r
r r� H s r� � c C s | j | j d | j d S)Ni�Q g $@r� g ��.A)�secondsZdaysZ microseconds)�tdr
r
r �
total_secondsM s r� c C s
| j � S)N)r� )r� r
r
r �<lambda>R s r� c @ s e Z d Z d d � Z d S)�mintdiffc C s� d } } xz | j r� | d k rC | d k rC t j | j � } q
t j | j � } | | } | d k sw | | k r} | } | } q
W| d k r� t | � Sd S)N)r� r� �heappopr� )r �dtp�min_diff�dt�diffr
r
r �finalizeV s
zmintdiff.finalizeN)r r! r" r� r
r
r
r r� T s r� c @ s e Z d Z d d � Z d S)�avgtdiffc C s� | j d k r d S| j d k r& d Sd } } d } xv | j r� | d k ro | d k ro t j | j � } q9 t j | j � } | | } | d 7} | t | � 7} | } q9 Wt | � | S)Nr1 r )r� r� r� r� r� r� )r �totalr� r� r� r� r
r
r r� g s"
zavgtdiff.finalizeN)r r! r" r� r
r
r
r r� e s r� c @ s4 e Z d Z d d � Z d d � Z d d � Z d S)�durationc C s d | _ | _ d S)N)�_min�_max)r r
r
r r s zduration.__init__c C s^ t | � } | j d k s* | | j k r3 | | _ | j d k sQ | | j k rZ | | _ d S)N)r r� r� )r r r� r
r
r r� � s
z
duration.stepc C s0 | j r, | j r, | j | j } t | � Sd S)N)r� r� r� )r r� r
r
r r� � s
zduration.finalizeN)r r! r" r r� r� r
r
r
r r� } s r� c @ sa e Z d Z e r9 d d � Z d d � Z d d � Z n$ d d � Z d d � Z d d � Z d
S) �modec C s t � | _ d S)N)r �items)r r
r
r r � s z
mode.__init__c G s | j j | � d S)N)r� r� )r r r
r
r r� � s z mode.stepc C s% | j r! | j j d � d d Sd S)Nr1 r )r� � most_common)r r
r
r r� � s z
mode.finalizec C s
g | _ d S)N)r� )r r
r
r r � s c C s | j j | � d S)N)r� r% )r �itemr
r
r r� � s c C s, | j r( t t | j � d | j j �Sd S)Nr )r� �maxr3 r� )r r
r
r r� � s N)r r! r" r r r� r� r
r
r
r r� � s r� c @ s e Z d Z d d � Z d S)�minrangec C s� | j d k r d S| j d k r& d Sd } } xz | j r� | d k ri | d k ri t j | j � } q3 t j | j � } | | } | d k s� | | k r� | } | } q3 W| S)Nr r1 )r� r� r� r� )r �prevr� �currr� r
r
r r� � s
zminrange.finalizeN)r r! r" r� r
r
r
r r� � s r� c @ s e Z d Z d d � Z d S)�avgrangec C s� | j d k r d S| j d k r& d Sd } } d } xp | j r� | d k ro | d k ro t j | j � } q9 t j | j � } | | } | d 7} | | 7} | } q9 Wt | � | S)Nr r1 )r� r� r� r� r� )r r� r� r� r� r� r
r
r r� � s"
zavgrange.finalizeN)r r! r" r� r
r
r
r r� � s r� c @ s: e Z d Z d Z d d � Z d d � Z d d � Z d S) �_range�rangec C s d | _ | _ d S)N)r� r� )r r
r
r r � s z_range.__init__c C sR | j d k s | | j k r' | | _ | j d k sE | | j k rN | | _ d S)N)r� r� )r r r
r
r r� � s z _range.stepc C s0 | j d k r, | j d k r, | j | j Sd S)N)r� r� )r r
r
r r� � s z_range.finalizeN)r r! r" r0 r r� r� r
r
r
r r� � s r� c @ sI e Z d Z d d g Z d g Z d Z d d d d � Z d d � Z d S)
� RegexSearch�regex�
search_string�matchZ regex_searchNc C s t j | | � | _ d S)N)�re�finditer�_iter)r r� r� r
r
r �
initialize� s zRegexSearch.initializec C s t | j � j d � f S)Nr )�nextr� r' )r �idxr
r
r �iterate� s zRegexSearch.iterate)r r! r" �params�columnsr0 r� r� r
r
r
r r� � s
r� c @ sI e Z d Z d d d g Z d g Z d Z d d d � Z d d
� Z d S) �
DateSeriesr� �stop� step_secondsr
Z date_seriesi�Q c C s t | � | _ t | � | _ t | � } t j d | � | _ | j j d k r� | j j d k r� | j j d k r� | d k r� d | _
n� | j j d k r| j j d k r| j j
d k r| j j d k r| j j d k r| j j
d k r| d k rd | _
n d | _
d S) Nr� r i�Q z%Y-%m-%dil r1 z%H:%M:%Sz%Y-%m-%d %H:%M:%S)r r� r� r� �datetimeZ timedeltar� rP rR rS �formatrI rK rN )r r� r� r� r
r
r r� s$ zDateSeries.initializec C sF | j | j k r t � | j } | j | j 7_ | j | j � f S)N)r� r� �
StopIterationr� �strftimer� )r r� �currentr
r
r r� s
zDateSeries.iterateN)r r! r" r� r� r0 r� r� r
r
r
r r� � s
r� )rq r� )Zr� r� r� r rg r� r� �sysr rs � collectionsr � ImportErrorr Z urllib.parseZvtfuncr Zpeeweer r Zplayhouse._speedupsr r r Z playhouser Z
cython_udfZ CONTROL_FLOWZDATE�FILEZHELPERZMATH�STRINGr# r+ r- �dictr r} r� r* r, r/ r9 r; r= r> r? rG rf rm rp � version_inforw ry r{ r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� �objectr� r� r� r� r� r� r� r� r� r� Zdamerau_levenshtein_distZlevenshtein_distZstr_distZmedianr� r� r
r
r
r �<module> s�