python (3.12.0)
̑e$ d Z ddlZddlZddlZdgZ ej dd Z G d d Z G d d Z G d d
Z
y) a% robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
NRobotFileParser RequestRatezrequests secondsc Z e Zd ZdZddZd Zd Zd Zd Zd Z d Z
d Z d
Z d Z
d Zd
Zy)r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
c z g | _ g | _ d | _ d| _ d| _ | j | d| _ y )NFr )entriessitemaps
default_entry disallow_all allow_allset_url last_checkedselfurls F/BuggyBox/python/3.12.0/bootstrap/lib/python3.12/urllib/robotparser.py__init__zRobotFileParser.__init__ s;
! ! S c | j S )zReturns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r
r s r mtimezRobotFileParser.mtime% s r c 6 ddl }|j | _ y)zYSets the time the robots.txt file was last fetched to the
current time.
r N)timer
)r r s r modifiedzRobotFileParser.modified. s
IIKr c p || _ t j j | dd \ | _ | _ y)z,Sets the URL referring to a robots.txt file. N)r urllibparseurlparsehostpathr s r r zRobotFileParser.set_url6 s- %|| 4 4S 9!A > 49r c t j j | j }|j }| j |j
d j y# t j j $ rT}|j dv rd| _ n4|j dk\ r |j dk r d| _ Y d}~yY d}~yY d}~yY d}~yd}~ww xY w)z4Reads the robots.txt URL and feeds it to the parser.zutf-8)i i Ti i N)
r requesturlopenr readr decode
splitlineserror HTTPErrorcoder
r )r frawerrs r r% zRobotFileParser.read; s 9&&txx0A &&(C JJszz'*557 8 ||%% &xx:%$(!S SXX^!% &4 " &s )A* * C;CCc d|j v r| j || _ y y | j j | y N*)
useragentsr r append)r entrys r
_add_entryzRobotFileParser._add_entryH s= %"" "!!)%*" *
LL &r c d}t }| j |D ] }|s4|dk( r
t }d}n"|dk( r | j | t }d}|j d }|dk\ r|d| }|j }|sh|j dd }t
| dk( s|d j j |d< t j j |d j |d< |d dk( rB|dk( r| j | t }|j j |d d}*|d dk( r3|dk7 s9|j j t |d d d}e|d d
k( r3|dk7 st|j j t |d d d}|d d k( r?|dk7 s|d j j rt! |d |_ d}|d d
k( r|dk7 s|d j d }t
| dk( rk|d j j rJ|d j j r)t% t! |d t! |d |_ d}|d dk( s| j( j |d |dk( r| j | yy)zParse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r #N:z
user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryr r4 findstripsplitlenlowerr r unquoter1 r2 rulelinesRuleLineisdigitintdelayr req_rater )r linesstater3 lineinumberss r r zRobotFileParser.parseQ s
7 2DA: !GE EaZOOE* !GE E #AAvBQx::<D::c1%D4yA~ q'--///1Q ,,..tAw}}?Q7l* z . %$$++DG4 E!W
* z ..xQ/GH !!W' z ..xQ/FG !!W
- z 7==?224*-d1g,EK !!W. z"&q'--"4 LA -'!*2B2B2D2L2L2N '
0 0 2 : : <-8WQZ #gVWj/-ZEN !!W )
MM((a1o7 2p A: OOE " r c b | j ry| j ry| j syt j j t j j
| }t j j dd|j |j |j |j f }t j j | }|sd}| j D ]& }|j | s|j | c S | j r| j j | S y)z=using the parsed robots.txt decide if useragent can fetch urlFT r; )r
r r
r r r rC
urlunparser! paramsqueryfragmentquoter
applies_to allowancer )r useragentr
parsed_urlr3 s r can_fetchzRobotFileParser.can_fetch s >>
\\**6<<+?+?+DE
ll%%r"Z__ j . .
0C0C'E Fll %C\\ ,E * s++ , %%//4 4r c | j sy | j D ]! }|j | s|j c S | j r| j j S y N)r r rV rH r r rX r3 s r crawl_delayzRobotFileParser.crawl_delay sY zz|\\ #E * {{" # %%++ +r c | j sy | j D ]! }|j | s|j c S | j r| j j S y r\ )r r rV rI r r] s r request_ratez RobotFileParser.request_rate sY zz|\\ &E * ~~% & %%.. .r c 4 | j sy | j S r\ )r r s r site_mapszRobotFileParser.site_maps s }}}} r c | j }| j || j gz }dj t t | S )Nz
)r r joinmapstr)r r s r __str__zRobotFileParser.__str__ s@ ,, ) !3!3 44G{{3sG,--r N)rP )__name__
__module__ __qualname____doc__r r r r r% r4 r rZ r^ r` rb rg r r r r sE
!(?
9'G#R:
.r c " e Zd ZdZd Zd Zd Zy)rE zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c |dk( r|sd}t j j t j j | }t j j | | _ || _ y )NrP T)r r rQ r rU r! rW )r r! rW s r r zRuleLine.__init__ sP 2:i I||&&v||'<'<T'BCLL&&t, "r c Z | j dk( xs |j | j S r/ )r!
startswith)r filenames r rV zRuleLine.applies_to s% yyC A8#6#6tyy#AAr c B | j rdnddz | j z S )NAllowDisallowz: )rW r! r s r rg zRuleLine.__str__ s >>zTADIIMMr N)rh ri rj rk r rV rg rl r r rE rE s 1#BNr rE c ( e Zd ZdZd Zd Zd Zd Zy)r= z?An entry has one or more user-agents and zero or more rulelinesc < g | _ g | _ d | _ d | _ y r\ )r1 rD rH rI r s r r zEntry.__init__ s
r c g }| j D ] }|j d| | j |j d| j | j 7| j }|j d|j d|j
|j
t t | j dj | S )Nz User-agent: z
Crawl-delay: zRequest-rate: r;
) r1 r2 rH rI requestssecondsextendre rf rD rd )r retagentrates r rg z
Entry.__str__ s __ /E JJ eW- . / :: ! JJ tzzl3 4 == $==D JJ
a ~F G
3sDNN+,yy~ r c |j d d j }| j D ] }|dk( r y|j }||v s y y)z2check if this entry applies to the specified agentr; r r0 TF)r@ rB r1 )r rX r} s r rV zEntry.applies_to sX OOC(+113 __ E|KKME !
r c d | j D ]! }|j | s|j c S y)zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)rD rV rW )r rq rL s r rW zEntry.allowance
s2 NN &Dx(~~% & r N)rh ri rj rk r rg rV rW rl r r r= r= s I
r r= ) rk collections urllib.parser urllib.request__all__
namedtupler r rE r= rl r r <module>r sU
$k$$]4FG ~. ~.BN N$( (r 
Browse
Usage
Versions