= Lc @ s8 d Z d d k Z e i d j o" d GHd e i GHe i d n d d k Z d d k Z d d k Z d d k Z d d k Z d d k
Z
d d k Z d d k Z d d k
Z
d d k Z d d k Z d d k Z y
e Z [ Wn e j
o d Z d Z n Xd Z d Z d
Z d d d d
d d d d d d g
Z d d d g Z d Z d Z e
i d Z e e
i d d d d d d d g Z d! d" d# d$ d% d& d' g Z! d( Z" d) Z# d* Z$ d+ Z% d, Z& d- Z' d. Z( d/ d0 d1 h d2 d3 f g Z) d4 e* f d5 YZ+ d6 e+ f d7 YZ, d8 f d9 YZ- e- Z. d: f d; YZ/ e/ Z0 d< e1 f d= YZ2 d> f d? YZ3 d@ f dA YZ4 dB f dC YZ5 dD f dE YZ6 dF f dG YZ7 dH e i8 i9 i: f dI YZ; dJ f dK YZ< dL f dM YZ= dN e i8 i9 i: f dO YZ> dP Z? dQ Z@ dR ZA dS ZB dT ZC dU ZD eE dV j o eD e iF d ZG eG p eG iH dW p
eG iH dX o e0 iI e d nw eG iH dY ZJ eC eG dW eJ ZK eK o e0 iI dZ d n9 eK iL e0 iI d[ e0 iM d e0 iI d\ e0 iN d n d S(] sW A simple script to automatically produce sitemaps for a webserver,
in the Google Sitemap Protocol (GSP).
Usage: python sitemap_gen.py --config=config.xml [--help] [--testing]
--config=config.xml, specifies config file location
--help, displays usage message
--testing, specified when user is experimenting
iNi s) This script requires Python 2.2 or later.s Currently run with version: %si i t ASCIIs UTF-8t IDNAs US-ASCIIt USt IBM367t CP367s ISO646-USISO_646.IRV:1991s ISO-IR-6s ANSI_X3.4-1968s ANSI_X3.4-1986t CPASCIIs
ISO-8859-1s
ISO-8859-2s
ISO-8859-5iP s
_index.xmls5 .+\s+"([^\s]+)\s+([^\s]+)\s+HTTP/\d+\.\d+"\s+200\s+.*s
^\d\d\d\d$s ^\d\d\d\d-\d\d$s ^\d\d\d\d-\d\d-\d\d$s ^\d\d\d\d-\d\d-\d\dT\d\d:\d\dZ$s+ ^\d\d\d\d-\d\d-\d\dT\d\d:\d\d[+-]\d\d:\d\d$s, ^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?Z$s8 ^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?[+-]\d\d:\d\d$t alwayst hourlyt dailyt weeklyt monthlyt yearlyt nevers4
s
sM %(loc)s%(lastmod)s
s,
s
s
s
t https www.google.coms webmasters/sitemaps/pingt t sitemapt Errorc B s e Z d Z RS( s
Base exception class. In this module we tend not to use our own exception
types for very much, but they come in very handy on XML parsing with SAX.
( t __name__t
__module__t __doc__( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s t SchemaErrorc B s e Z d Z RS( s? Failure to process an XML file according to the schema we know.( R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s t Encoderc B s; e Z d Z d Z d Z d Z d Z d Z RS( s
Manages wide-character/narrow-character conversions for just about all
text that flows into or out of the script.
You should always use this class for string coercion, as opposed to
letting Python handle coercions automatically. Reason: Python
usually assumes ASCII (7-bit) as a default narrow character encoding,
which is not the kind of data we generally deal with.
General high-level methodologies used in sitemap_gen:
[PATHS]
File system paths may be wide or narrow, depending on platform.
This works fine, just be aware of it and be very careful to not
mix them. That is, if you have to pass several file path arguments
into a library call, make sure they are all narrow or all wide.
This class has MaybeNarrowPath() which should be called on every
file system path you deal with.
[URLS]
URL locations are stored in Narrow form, already escaped. This has the
benefit of keeping escaping and encoding as close as possible to the format
we read them in. The downside is we may end up with URLs that have
intermingled encodings -- the root path may be encoded in one way
while the filename is encoded in another. This is obviously wrong, but
it should hopefully be an issue hit by very few users. The workaround
from the user level (assuming they notice) is to specify a default_encoding
parameter in their config file.
[OTHER]
Other text, such as attributes of the URL class, configuration options,
etc, are generally stored in Unicode for simplicity.
c C s% d | _ g | _ t | _ y t i i | _ WnE t j
o9 y t i
t i j | _ Wqv t j
o qv Xn Xy: t i } | o# | i
t j o | g | _ n Wn t j
o n X| i p: t i } | o# | i
t j o | g | _ q
n | i p
t | _ n d S( N( t Nonet _usert _learnedt Falset
_widefilest ost patht supports_unicode_filenamest AttributeErrort syst getwindowsversiont VER_PLATFORM_WIN32_NTt getfilesystemencodingt uppert ENC_ASCII_LISTt getdefaultencodingt ENC_DEFAULT_LIST( t selft encoding( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt __init__ s,
c C s
| | _ d S( N( R ( R' R( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt SetUserEncoding s c C s| t | t i j o | S| ox y; | i | } | | i j o | i i | n | SWq t j
o q t j
o t i d | q Xn | i
oc y | i | i
SWqt j
o qt j
o* | i
} d | _
t i d | qXn x; | i o0 y | i | i d SWq
| i d =q
Xq
Wy | i t SWn t j
o n X| i t
d S( s" Narrow a piece of arbitrary text s Unknown encoding: %ss Unknown default_encoding: %si t ignoreN( t typet typest UnicodeTypet encodeR t appendt UnicodeErrort LookupErrort outputt WarnR R t ENC_UTF8t ENC_ASCII( R' t textR( t resultt temp( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt
NarrowText s@
c C s | i o | S| i | d S( s# Paths may be allowed to stay wide N( R R: R ( R' R7 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt MaybeNarrowPath' s
c
C s t | t i j o | S| ox y; t | | } | | i j o | i i | n | SWq t j
o q t j
o t i d | q Xn | i
oc y t | | i
SWqt j
o qt j
o* | i
} d | _
t i d | qXn x; | i o0 y t | | i d SWq
| i d =q
Xq
Wy t | t SWn t j
o n Xt i d | | i
p t i d n | i
t d S( s! Widen a piece of arbitrary text s Unknown encoding: %ss Unknown default_encoding: %si s! Unrecognized encoding in text: %ssB You may need to set a default_encoding in your configuration file.R+ N( R, R- t
StringTypet unicodeR R0 R1 R2 R3 R4 R R R5 t decodeR6 ( R' R7 R( R8 R9 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt WidenText. sF
( R R R R) R* R: R; R? ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s ! - t Outputc B sD e Z d Z d Z d Z d Z d Z d Z d Z RS( s
Exposes logging functionality, and tracks how many errors
we have thus output.
Logging levels should be used as thus:
Fatal -- extremely sparingly
Error -- config errors, entire blocks of user 'intention' lost
Warn -- individual URLs lost
Log(,0) -- Un-suppressable text that's not an error
Log(,1) -- touched files, major actions
Log(,2) -- parsing notes, filtered or duplicated URLs
Log(,3) -- each accepted URL
c C s1 d | _ d | _ h | _ h | _ d | _ d S( Ni ( t
num_errorst num_warnst
_errors_shownt _warns_shownt _verbose( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) r s
c C s: | o/ t i | d } | i | j o | GHq6 n d S( sC Output a blurb of diagnostic text, if the verbose level allows it N( t encoderR: R RE ( R' R7 t level( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Log{ s c C s | o| t i | d } t i | i } | i i | p d | i | %s
( RU t SITEURL_XML_PREFIXR R R, R- R. RF R: R R< t strt xmlt saxt saxutilst escapet SITEURL_XML_SUFFIXt write( R' t fileR R] R^ ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt WriteXMLp s
( s locs lastmods
changefreqs priority( R R R R R) RZ R_ Rh t staticmethodR[ R R RH R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyRT s
< 9
t Filterc B s e Z d Z d Z d Z RS( sQ
A filter on the stream of URLs we find. A filter is, in essence,
a wildcard applied to the stream. You can think of this as an
operator that returns a tri-state when given a URL:
True -- this URL is to be included in the sitemap
None -- this URL is undecided
False -- this URL is to be dropped from the sitemap
c C s d | _ d | _ t | _ t d | d p d St i } | i d } | i d d } | i d d } | o | i } n | o | i } n | p t i
d n_ | p | d j o | d j o t i
d n, | d
j o | d j o t i
d n | d j o
t | _ n | d
j o
t | _ n | d j o
| | _ nR | d j oD y t i
| | _ Wqt i j
o t i
d | qXn | t i j o! t i d
| | | f d n d S( Nt FILTERR R, t actiont wildcardt drops1 On a filter you must specify a "pattern" to matcht regexpsH On a filter you must specify either 'type="wildcard"' or 'type="regexp"'t passsT If you specify a filter action, it must be either 'action="pass"' or 'action="drop"'s Bad regular expression: %ss' Filter: %s any URL that matches %s "%s"i ( s patterns types action( R t _wildcardt _regexpR t _passt ValidateAttributesR3 RA t getR R Rb t ret compilet errorRH ( R' t
attributesRA R R, R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) sD
"
c C s | p | i o d S| i o& t i | i | i o | i Sd S| i o# | i i | i o | i Sd St p t d S( s Process the URL, as above. N(
RU R R t fnmatcht fnmatchcaseR R t searchR Rz ( R' t url( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Apply s
( R R R R) R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s 3t InputURLc B s e Z d Z d Z d Z RS( s
Each Input class knows how to yield a set of URLs from a data source.
This one handles a single URL, manually specified in the config file.
c C s d | _ t d | d
p d St } xM | i D]? } | d j o | i d | | q7 | i | | | q7 W| i p t i d d S| | _ t i d | i i d d S( NRT t hrefRV RW RX RU s( Url entries must have an href attribute.s Input: From URL "%s"i ( s hrefs lastmods
changefreqs priority(
R t _urlR RT t keysR_ RU R3 R RH ( R' R R t attr( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s
c C s" | i o | | i t n d S( sD Produces URLs from our data source, hands them in to the consumer. N( R Rb ( R' t consumer( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt ProduceURLs s
( R R R R) R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s t InputURLListc B s e Z d Z d Z d Z RS( s
Each Input class knows how to yield a set of URLs from a data source.
This one handles a text file with a list of URLs
c C s d | _ d | _ t d | d p d S| i d | _ | i d t | _ | i og t i | i | _ t i i
| i o t i d | i d q t i
d | i d | _ n t i
d d S( Nt URLLISTR R( s Input: From URLLIST "%s"i s Can not locate file: %ss- Urllist entries must have a "path" attribute.( s paths encoding( R t _patht _encodingR R R5 RF R; R R t isfileR3 RH R ( R' R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s
c
C s t | i d \ } } | p d Sd } xj| i D]\} | d } | i o t i | | i } n | i } | p | d d j o q7 n t } | i d } x1 t d t
| D] } | | i | |
( R R R R) R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s t InputDirectoryc B s e Z d Z d Z d Z RS( s
Each Input class knows how to yield a set of URLs from a data source.
This one handles a directory that acts as base for walking the filesystem.
c C s d | _ d | _ d | _ t d | d
p d S| i d } | p t i d d St i | } | i
t i p | t i } n t i
i | p t i d | d S| i d } | p t i d d St i | } | i
d p | d } n | i | pB t i | | } | i | p t i d | | f d Sn | i d } | o> t i | } t i | j o t i d | d } qn | | _ | | _ | | _ | o! t i d
| | | f d n t i d | | f d d S( Nt DIRECTORYR R t default_files< Directory entries must have both "path" and "url" attributess Can not locate directory: %sR s: The directory URL "%s" is not relative to the base_url: %ss7 The default_file "%s" can not include path information.s6 Input: From DIRECTORY "%s" (%s) with default file "%s"i s4 Input: From DIRECTORY "%s" (%s) with no default file( s paths urls default_file( R R R t
_default_fileR R R3 R RF R; R R t sepR t isdirRT R[ R| R` R{ RH ( R' R R R R R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) N sX
c s | i p d S| i | i | i f d f d } t i d | i d | i d t i i | i | d d S( sD Produces URLs from our data source, hands them in to the consumer. Nc
s t } t } y | o t i i | | } n | } t i i | } d } | oP oI t i i | } y t i | t i } Wq t j
o q Xn | p t i | t i } n t
| | _ Wn% t j
o n t j
o n X| t
} t i d j o | i t i d } n | o | d } n | o# | | } | o | d } qxn | i d t i | d | o( | j o | i d d d d d S | t d S( sn
Called once per file.
Note that 'name' will occasionally be None -- for a directory itself
R RU R s IGNORED (default file)RG i N( RT R R R t joinR R t statt ST_MTIMEt OSErrort TimestampISO8601RV RR Rq R t replaceR_ RF R? RH ( t dirpatht nameR R R t timeR t middle( t root_URLt root_filet root_pathR ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt PerFile sF
c sN | } | i p t i d | d Sx | D] } | | q3 Wd S( sT
Called once per directory with a list of all the contained files/dirs.
s8 Unable to decide what the root path is for directory: %sN( R| R3 R4 ( R+ R t namelistR ( R R ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt PerDirectory s s Walking DIRECTORY "%s"i ( R R R R3 RH R R R t walk( R' R R ( ( R R R R R sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
1( R R R R) R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR G s :t InputAccessLogc B sD e Z d Z d Z d Z d Z d Z d Z d Z RS( s-
Each Input class knows how to yield a set of URLs from a data source.
This one handles access logs. It's non-trivial in that we want to
auto-detect log files in the Common Logfile Format (as used by Apache,
for instance) and the Extended Log File Format (as used by IIS, for
instance).
c C s d | _ d | _ t | _ t | _ d | _ d | _ d | _ d | _ d | _
t d | d p d S| i d | _ | i d t
| _ | i og t i | i | _ t i i | i o t i d | i d qt i d | i d | _ n t i d d S(
Nit ACCESSLOGR R( s Input: From ACCESSLOG "%s"i s Can not locate file: %ss/ Accesslog entries must have a "path" attribute.( s paths encoding( R R R R t _is_elft _is_clft _elf_statust _elf_methodt _elf_urit
_elf_urifrag1t
_elf_urifrag2R R R5 RF R; R R R R3 RH R ( R' R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s(
c C s | i d p t S| i d } | d =x t d t | D] } | | i } | d j o
| | _ qA | d j o
| | _ qA | d j o
| | _ qA | d j o
| | _ qA | d j o
| | _
qA qA Wt i d d
t
S( s7 Recognize the Fields directive that heads an ELF file s #Fields:R i s sc-statuss cs-methods cs-uris cs-uri-stems cs-uri-querys, Recognized an Extended Log File Format file.i ( R| R Rp R Rq R R R R R R R3 RH Rb ( R' R t fieldsR t field( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt RecognizeELFLine s&
c C s | i d } t | } | i d j o8 | i | j o d S| | i i d j p d Sn | i d j o8 | i | j o d S| | i i d j o d Sn | i d j o> | i | j o d S| | i i } | d j o | Sn | i d j o | i | j p | i | j o d S| | i i } d } | i d j o | | i } n | o8 | d j o+ | o | d j o | d | } n | Sn d S(
s* Fetch the requested URL from an ELF line R i t 200t HEADt GETt -t ?N( R R ( Rp Rq R R R R R R R ( R' R R t countR t urlfrag1t urlfrag2( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt
GetELFLine s<
c C sJ t i | } | o | i d d j } | o t i d d n | S( sR Try to tokenize a logfile line according to CLF pattern and see if
it works. i R R s( Recognized a Common Logfile Format file.i ( s HEADs GET( t ACCESSLOG_CLF_PATTERNR~ t groupR3 RH ( R' R R~ t recognize( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt RecognizeCLFLine> s
c C sH t i | } | o. | i d } | d j o | i d Sn d S( s) Fetch the requested URL from a CLF line i R R i ( s HEADs GETN( R R~ R R ( R' R R~ t request( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt
GetCLFLineH s
c C s? t | i d \ } } | p d Sx | i D] } | i o t i | | i } n | i } | i o3 | i o( | i | | _ | i
| | _ n d } | i o | i | } n | i o | i
| } n | p q1 n t } | i d | | | t q1 W| i | o | i n d S( sD Produces URLs from our data source, hands them in to the consumer. R NRU ( R R R R RF R? R R R R R R R R RT R_ Rb R ( R' R R R R R~ R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR R s0
( R R R R) R R R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s +
t InputSitemapc B s e Z d Z d e f d YZ d e f d YZ d e f d YZ d e f d YZ d e f d
YZ d e f d YZ d
Z
d Z d Z d Z
d Z d Z d Z RS( s
Each Input class knows how to yield a set of URLs from a data source.
This one handles Sitemap files and Sitemap index files. For the sake
of simplicity in design (and simplicity in interfacing with the SAX
package), we do not handle these at the same time, recursively. Instead
we read an index file completely and make a list of Sitemap files, then
go back and process each Sitemap.
t _ContextBasec B sD e Z d Z d Z d Z d Z d Z d Z d Z RS( st Base class for context handlers in our SAX processing. A context
handler is a class that is responsible for understanding one level of
depth in the XML schema. The class knows what sub-tags are allowed,
and doing any processing specific for the tag we're in.
This base class is the API filled in by specific context handlers,
all defined below.
c C s | | _ d | _ d S( sU Initialize with a sequence of the sub-tags that would be valid in
this context.N( t
_allowed_tagsR t _last_tag( R' t subtags( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s c C s0 | | i j } | o
| | _ n
d | _ | S( s< Returns True iff opening a sub-tag is valid in this context.N( R R R ( R' t tagt valid( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt AcceptTag s
c C s t S( s: Returns True iff a blurb of text is valid in this context.( R ( R' R7 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt
AcceptText s c C s d S( s+ The context is opening. Do initialization.N( ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Open s c C s d S( s3 The context is closing. Return our result, if any.N( ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Close s c C s | o
t n d S( s
We're returning to this context after handling a sub-tag. This
method is called with the result data from the sub-tag that just
closed. Here in _ContextBase, if we ever see a result it means
the derived child class forgot to override this method.N( t NotImplementedError( R' R8 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Return s ( R R R R) R R R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
t _ContextUrlSetc B s e Z d Z d Z RS( s3 Context handler for the document node in a Sitemap.c C s t i i | d d S( NR ( s url( R R R) ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s ( R R R R) ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s t _ContextUrlc B s2 e Z d Z d Z d Z d Z d Z RS( s, Context handler for a URL node in a Sitemap.c C s, t i i | t i d | _ | | _ d S( sU Initialize this context handler with the callable consumer that
wants our URLs.N( R R R) RT R R R t _consumer( R' R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s c C s" | i p t t | _ d S( s Initialize the URL.N( R Rz RT ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s c C s1 | i p t | i | i t d | _ d S( s2 Pass the URL to the consumer and reset it to None.N( R Rz R
R R ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s c C s6 | i p t | o | i i | i | n d S( s7 A value context has closed, absorb the data it gave us.N( R Rz R_ R ( R' R8 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s ( R R R R) R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
t _ContextSitemapIndexc B s2 e Z d Z d Z d Z d Z d Z RS( s7 Context handler for the document node in an index file.c C s t i i | d g | _ d S( NR ( s sitemap( R R R) t _loclist( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s c C s | i p t d S( s Just a quick verify of state.N( R Rz ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s c C s% | i o | i } g | _ | Sd S( s$ Return our list of accumulated URLs.N( R ( R' R9 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
c C s | o | i i | n d S( s0 Getting a new loc URL, add it to the collection.N( R R0 ( R' R8 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s ( R R R R) R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
t _ContextSitemapc B s2 e Z d Z d Z d Z d Z d Z RS( s5 Context handler for a Sitemap entry in an index file.c C s t i i | d d | _ d S( NRU RV ( s locs lastmod( R R R) R t _loc( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s c C s | i p t d S( s Just a quick verify of state.N( R Rz ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s c C s2 | i o | i } d | _ | St i d d S( s Return our URL to our parent.s: In the Sitemap index file, a "sitemap" entry had no "loc".N( R R R3 R4 ( R' R9 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
c C s( | o | i d j o
| | _ n d S( s2 A value has closed. If it was a 'loc', absorb it.RU N( R R ( R' R8 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s ( R R R R) R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR
s
t
_ContextValuec B s2 e Z d Z d Z d Z d Z d Z RS( s Context handler for a single value. We return just the value. The
higher level context has to remember what tag led into us.c C s t i i | d d | _ d S( N( ( R R R) R t _text( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) % s c C s+ | i o | i | | _ n
| | _ t S( s( Allow all text, adding it to our buffer.( R Rb ( R' R7 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR * s
c C s
d | _ d S( s Initialize our buffer.N( R R ( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR 3 s c C s- | i } d | _ | o | i } n | S( s Return what's in our buffer.N( R R R ( R' R7 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR 8 s
( R R R R) R R R ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s
c C s t i i i i | d | _ d | _ d | _ d | _ d | _
t d | d g p d S| i d } | o[ t
i | } t i i | o$ t i d | d | g | _ q t i d | n t i d d S( sQ Initialize with a dictionary of attributes from our entry in the
config file.it SITEMAPR Ns Input: From SITEMAP "%s"i s Can not locate file "%s"s- Sitemap entries must have a "path" attribute.( R R t handlert ContentHandlerR) R t _pathlistt _currentt _contextst
_contexts_idxt
_contexts_stmR R RF R; R R R R3 RH R ( R' R R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) B s c C s t i t i t i g | _ t i t i | t i g | _ | i p t | i d } d | _ | i | | i | _ x" | i d D] } | i | q Wd S( s In general: Produces URLs from our data source, hand them to the
callable consumer.
In specific: Iterate over our list of paths and delegate the actual
processing to helper methods. This is a complexity no other data source
needs to suffer. We are unique in that we can have files that tell us
to bring in other files.
Note the decision to allow an index file or not is made in this method.
If we call our parser with (self._contexts == None) the parser will
grab whichever context stack can handle the file. IE: index is allowed.
If instead we set (self._contexts = ...) before parsing, the parser
will only use the stack we specify. IE: index not allowed.
i i N(
R R R
R R R R R R Rz R R t _ProcessFile( R' R R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR \ s
c C s | p t t | d \ } } | p d Sy d | _ t i i | | Wn t j
o t i d | nk t j
o t i d | nH t i i
i j
o2 } t i d | | i | i
| i f n X| i | o | i n d S( sB Do per-file reading/parsing/consuming for the file path passed in.R Nis8 An error in file "%s" made us abort reading the Sitemap.s Cannot read from file "%s"s3 XML error in the file "%s" (line %d, column %d): %s( Rz R R R R t parseR R3 R t IOErrort _exceptionst SAXParseExceptiont _linenumt _colnumt
getMessageR ( R' R R R t e( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR s$ %
c C sD | i p t | i d } t i i | } t i i | } t } t | t i j o
t
} n x | D] } t i | } t
i d | d t i | \ } } } } } t i i | }
t i |
}
| o t i |
}
n | o | t i |
}
n |
o( | i i |
t
i d |
d qo qo Wd S( sq Given a list of URLs, munge them into our self._pathlist property.
We do this by assuming all the files live in the same directory as
the first file in the existing pathlist. That is, we assume a
Sitemap index points to Sitemaps only in the same directory. This
is not true in general, but will be true for any output produced
by this script.
i s# Index points to Sitemap file at: %si s% Will attempt to read Sitemap file: %si N( R Rz R R t normpatht dirnameR R, R- R. Rb RT R[ R3 RH R` Ra t basenameRk t unquoteRF R? R R0 ( R' t urllistR t dirt wideR Rd Re Rf Rg R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt _MungeLocationListIntoFiles s*
c C s" | i d j o | i o| | i o
| i p t | d j o | i | _ q | d j o | i | _ t i d d q t i d t n | i d j o7 | i | i j o$ | d j o t i d t n | o d } x | i D]s } | i d j o> | i
d d j o q n | i
d
d j o q qLn | o | d } n | | } q W| o t i d | | f qn | i d j p | i | i i | oH | i d
| _ | i t
| i j p t | i | i i n t i d | t d S( s SAX processing, called per node in the config stream.
As long as the new tag is legal in our current context, this
becomes an Open call on one context deeper.
i t urlsett sitemapindexs File is a Sitemap index.i sA The document appears to be neither a Sitemap nor a Sitemap index.s7 A Sitemap index can not refer to another Sitemap index.R t xmlnst xsis , sM Did not expect any attributes on any tag, instead tag "%s" had attributes: %si s) Can not accept tag "%s" where it appears.N( R R R R Rz R3 RH R R R t findR4 R Rq R ( R' R R R7 R ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt startElement sD
#
* c C s | } | i d j p t | i | i i } | i d | _ | i d j o | i | i i | n, | o$ | i | i j o | i | n d S( s SAX processing, called per node in the config stream.
This becomes a call to Close on one context followed by a call
to Return on the previous.
i i N( R Rz R R R R R) ( R' R t retval( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt
endElement s c C s[ | i d j p | i | i i | o, | i o t i d | t qW n d S( s SAX processing, called when text values are read. Important to
note that one single text value may be split across multiple calls
of this method.
i s* Can not accept text "%s" where it appears.N( R R R R R3 R R ( R' R7 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt
characters s
( R R R t objectR R R R R
R R) R R R) R/ R1 R2 ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR z s
5 ! " $ 3 t FilePathGeneratorc B s; e Z d Z d Z d Z d Z d Z d Z RS( s^
This class generates filenames in a series, upon request.
You can request any iteration number at any time, you don't
have to go in order.
Example of iterations for '/path/foo.xml.gz':
0 --> /path/foo.xml.gz
1 --> /path/foo1.xml.gz
2 --> /path/foo2.xml.gz
_index.xml --> /path/foo_index.xml
c C s( t | _ d | _ d | _ d | _ d S( N( R t is_gzipR R t _prefixt _suffix( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) s c C s" t i | } t i i | } t i i | i } | p t i d | t St
| } d } d d d g } x/ | D]' } | i | o t
| } Pq~ q~ W| p t i d | t S| i d | _ t
| } | | | | _
| | | | | !| _ | | | | _ t S( s6 Splits up a path into forms ready for recombination. s Couldn't parse the file path: %si s .xmls .xml.gzs .gzs8 The path "%s" doesn't end in a supported file extension.( RF R; R R R" R$ R R3 R R Rq R R5 R R6 R7 Rb ( R' R t baset lenbaset lensuffixt compare_suffixt suffixt lenpath( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Preload s0 c C sV | i | i } t | t i j o( | o d | | | i f S| | i S| | S( s/ Generates the iterations, as described above. s %s%d%s( R R6 R, R- RP R7 ( R' t instanceR ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt GeneratePathB s c C sr | | i } d } t | t i j o2 | o d | | | i f } qe | | i } n | | } t i | S( s7 Generates iterations, but as a URL instead of a path. s %s%d%sN( R6 R R, R- RP R7 RT R[ ( R' R? t root_urlR R0 ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt GenerateURLL s
c C sH t i | | i } t i | | i } | t | } | d | S( s; Generates a wildcard that should match all our iterations t *( RT R[ R6 R7 Rq ( R' RA R R9 R< ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt GenerateWildURLZ s ( R R R R) R> R@ RB RD ( ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR4 s "
t PerURLStatisticsc B s) e Z d Z d Z d Z d Z RS( sD Keep track of some simple per-URL statistics, like file extension. c C s
h | _ d S( N( t _extensions( R' ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR) g s c C s | o| i ot i | i \ } } } } } | p d S| i d oA | i i d o | i d d | i d RD RS RB t SITEINDEX_SUFFIXRT R] R, R- R< R. R ( R' t all_good( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt ValidateBasicConfig sF
%
&c C s x! | i D] } | i | i q
Wt | i o | i n | i p t i d | i n | i d j o | i n | i
| i i d S( s1 Run over all the Inputs and ask them to Produce s0 No URLs were recorded, writing an empty sitemap.i N(
RO R t
ConsumeURLRq RQ t FlushSetRU R3 R4 t
WriteIndext NotifySearchRW RH ( R' t input( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt Generate s
c C s | p d S| i | i | p d Sd } x2 | i D]' } | i | } | d j o Pq7 q7 W| p
| d j p | i d d d d d St i | i | i p t i | i | i
o | i d d d d d S| i } | i i
| og | i | } | d j o8 | d } | | i | <| i | j o
| | _ qOn | i d d d Sd | i | <| i i | | i i | | i t | i t j o | i n d S(
s
All per-URL processing comes together here, regardless of Input.
Here we run filters, remove duplicates, spill to disk as needed, etc.
NR t FILTEREDRG i s IGNORED (output file)i i t DUPLICATE( R R[ R RN R RH R R RU RS RT R RP RL RV RQ R0 RW RJ Rq t MAXURLS_PER_SITEMAPRe ( R' R R t acceptt filterRM t dup( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyRd sB
c C s t i d d | i i xv | i D]k } | i } | i | } | d j o? d | i | <| i p$ d t | t | i | _ q q' q' W| i i
| i } | p t i d n | i d | _ t i d | t
| i f d d } d } y | i i oC t i i | } t | d } t i d | d
| d d } n t | d } | i t x | i D] } | i | q{W| i t | i | o | i n d } d } Wn$ t j
o t i d
| n Xt i | d g | _ d S( s
Flush the current set of URLs to the output. This is a little
slow because we like to sort them all and normalize the priorities
before dumping.
s' Sorting and normalizing collected URLs.i i is %.4fs. Unexpected: Couldn't generate output filename.s& Writing Sitemap file "%s" with %d URLst wbt fileobjt filenamet modet wts Couldn't write out to file: %si N( R3 RH RQ RK R RP RX R RV RR R@ RU RO Rq R R5 R R R$ t opent gzipt GzipFileR t SITEMAP_HEADERR t SITEMAP_FOOTERR R t chmod( R' R RM Ro Rr R R R$ ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyRe 6 sL
,
"
c C s+ | i i t } | p t i d n t i d | | i f d t t i } y t | d } | i
t xW t d | i D]C } | i i
| | i } h | d 6| d 6} | i
t | q W| i
t | i d
} Wn$ t j
o t i d | n Xt i | d d
S( s- Write the master index of all Sitemap files s4 Unexpected: Couldn't generate output index filename.s( Writing index file "%s" with %d Sitemapsi Rt i RU RV s Couldn't write out to file: %si N( RR R@ Ra R3 RO RH RU R R Ru R t SITEINDEX_HEADERR RB R[ t SITEINDEX_ENTRYt SITEINDEX_FOOTERR R R R Rz ( R' Rr RV t fdt mapnumbert mapurlt
mapattributes( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyRf o s(
c
C s | i o t i d d d St i d d d t i f d Y} t i } | t _ | i d j o | i i t | i
} n | i i d | i
} y t i | } | i Wn1 t
j
o% t i d | t i d n Xx t D] } | d
} | d } | | | ISO 8601 time string.s %Y-%m-%dT%H:%M:%SZ( R t strftimet gmtime( t t( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyR T s c C s t i } t | } y+ t i d | d t i i | | Wn t j
o t i d | nm t i i i
j
o/ } t i d | i | i | i
f n* t i i i j
o t i d n X| t i j o | Sd S( sF Sets up a new Sitemap object from the specified configuration file. s Reading configuration file: %si s" Cannot read configuration file: %ss5 XML error in the config file (line %d, column %d): %ss Some installs of Python 2.2 did not include complete support for XML.
Please try upgrading your version of Python and re-running the script.N( R3 RA RM RH R R R R R R R R R R t SAXReaderNotAvailableR ( t
configpathR^ RA R R! ( ( sk /home.restoring/home/endymion/informationwithoutborders.endymion.com/sitemap/sitemap_gen-1.4/sitemap_gen.pyt CreateSitemapFromFileY s !c C s h } d } d } d | d | d } t i | } x | D] } ye | i | i } | i d o | d | | d t j
o d SXq> W| S(
s
Parse command line flags per specified usage, pick off key, value pairs
All flags of type "--key=value" will be processed as __flags[key] = value,
"--option" will be processed as __flags[option] = option
s --(?P\S*)[=](?P\S*)s --(?P