pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity

crawler: increase size limit from 100KiB to 500 

I'm looking at you, worldbankgroup.csod.com/ats/careersite/search.aspx
master
pictuga 2020-05-12 19:34:16 +02:00
parent 83dd2925d3
commit 36e2a1c3fd
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
morss/crawler.py
View File

@ -93,7 +93,7 @@ def custom_handler(follow=None, delay=None, encoding=None):
# & HTTPSHandler # & HTTPSHandler
#handlers.append(DebugHandler()) #handlers.append(DebugHandler())
handlers.append(SizeLimitHandler(100*1024)) # 100KiB handlers.append(SizeLimitHandler(500*1024)) # 500KiB
handlers.append(HTTPCookieProcessor()) handlers.append(HTTPCookieProcessor())
handlers.append(GZIPHandler()) handlers.append(GZIPHandler())
handlers.append(HTTPEquivHandler()) handlers.append(HTTPEquivHandler())