pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity

crawler: better use of chardet 

Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
master
pictuga 2017-03-18 22:19:54 -10:00
parent 9ee6ff60e1
commit 65055290d4
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
morss/crawler.py
View File

@ -64,8 +64,8 @@ def detect_encoding(data, con=None):
if match: if match:
return match.groups()[0].lower().decode() return match.groups()[0].lower().decode()
enc = chardet.detect(data[:1000])['encoding'] enc = chardet.detect(data)['encoding']
if enc: if enc and enc != 'ascii':
return enc return enc
return 'utf-8' return 'utf-8'