pictuga
0df6409b0e
crawler: use `with con` to commit, journal WAL for perf
2017-10-28 01:28:47 +02:00
pictuga
7b85f692a0
crawler: fix encoding detection
2017-10-27 23:14:08 +02:00
pictuga
840842d246
crawler: limit download to 500KiB
...
More can only be linked to a fraudulent/incorrect use of the service
2017-10-27 23:12:40 +02:00
pictuga
fbe811384a
crawler: add (unused) DebugHandler to output headers sent/received
...
Saves a lot of time when debugging
2017-10-27 23:10:03 +02:00
pictuga
e0092387b1
Add "six" as a requirement in README
...
Actually a dependency of "chardet", but since people (including me) might just send file onto their server instead of installing the libs properly, it's a nice reminder (not in requirements.txt since that one is automatic)
2017-07-30 16:58:53 +02:00
pictuga
b8d8d9ea20
Fix www/.htaccess
...
Deny/allow issue
2017-07-29 17:12:11 +02:00
pictuga
64babd6713
morss: make readabilite links absolute
2017-07-29 14:37:37 +02:00
pictuga
bbca5dca6b
Create LICENSE
2017-07-28 10:28:17 +02:00
pictuga
10cbebb80c
Imrove provided .htaccess
2017-07-23 20:22:40 +02:00
pictuga
3bfad54add
readabilite: change cleaning & code structure
...
Kinda struggled to make some "nice" code
2017-07-17 00:27:41 +02:00
pictuga
386bafd391
readabilite: write_all use "node" instead of "item"
2017-07-17 00:13:15 +02:00
pictuga
a61b259792
readabilite: easy option to highlight the nodes
2017-07-17 00:11:49 +02:00
pictuga
c52b47616d
readabilite: always return common of 2 best nodes
...
Better results. Less is not more
2017-07-17 00:10:58 +02:00
pictuga
bfdda18b9c
readbilite: better explain lowest_common output
2017-07-17 00:08:00 +02:00
pictuga
2afea497a3
readabilite: br2p use "node" instead of "item"
...
Confusing with rss items otherwise
2017-07-17 00:06:39 +02:00
pictuga
843dc97fbf
readabilite: change scoring algorithm
...
Use 3 groups of keywords instead
2017-07-17 00:01:44 +02:00
pictuga
df22396838
Only use chardet on 2k letters
...
Takes forever otherwise
2017-07-16 23:59:06 +02:00
pictuga
6f0efd5802
crawler: add cookies support
...
Somehow got dropped when splitting the big handler
2017-03-25 19:51:42 -10:00
pictuga
d3bc2926fc
Remove :hungry
...
Mostly usless. If you need it, you might as well not need to use morss in the first place...
2017-03-25 13:52:58 -10:00
pictuga
505b02d70d
crawler: remove debugging print()
2017-03-25 13:45:12 -10:00
pictuga
3ca6ed5bb0
readabilite: add author/about to black list
2017-03-24 22:02:41 -10:00
pictuga
4aa25bf3d8
readabilite: clean_html before scoring
...
Surprisingly efficient
2017-03-24 21:50:46 -10:00
pictuga
bfefa8d599
readabilite: add tags to black list
2017-03-24 21:50:26 -10:00
pictuga
91da0f36dc
readabilite: comment the clean_html function
2017-03-24 21:50:01 -10:00
pictuga
67889a1d14
readabilite: drop useless tags
...
This extra cluster actually jams the algorithm
2017-03-24 21:49:14 -10:00
pictuga
6024728341
README: mention / > | in webserver
2017-03-20 21:05:40 -10:00
pictuga
c929ce6278
README: update about options.items
2017-03-20 21:02:25 -10:00
pictuga
167e3e4a15
feedify: accept xpath rules passed as parameters
2017-03-20 20:56:48 -10:00
pictuga
bf3ef586c2
feedify: remove unused downloader
2017-03-20 20:53:52 -10:00
pictuga
08f08ef704
improve morss url detection regex
2017-03-20 20:51:13 -10:00
pictuga
1b4341f741
accept query_string in morss cgi
2017-03-20 20:50:04 -10:00
pictuga
f965566054
feedify; make function use clearer
2017-03-20 20:19:08 -10:00
pictuga
d6882e0a6a
readabilite: (try to) emprove detection
...
Kinda hopeless
2017-03-19 02:00:31 -10:00
pictuga
79a8ada9f4
readabilite: add tags to score
2017-03-19 01:57:54 -10:00
pictuga
4a5150e030
readabilite: fix iter while iterating
2017-03-19 01:56:33 -10:00
pictuga
e65c88abf8
readabilite: fix re.match
2017-03-19 01:55:40 -10:00
pictuga
9c331300eb
crawler: move UAHandler to basic
...
Fuck u feedburner
2017-03-19 01:49:17 -10:00
pictuga
5e61686373
Only use full feed for articles & feedify
...
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
2017-03-18 23:43:28 -10:00
pictuga
0b6e553054
Move iTunes code to feedify.py
2017-03-18 23:41:37 -10:00
pictuga
d4937812a8
Remove HTTPError code
...
Only used to look nice but useless (inherits from IOError anyway)
2017-03-18 23:39:32 -10:00
pictuga
99f3c519f2
crawler: fix accept code
2017-03-18 23:37:51 -10:00
pictuga
67f5a21019
Move build_opener to crawler
...
Forgotten
2017-03-18 23:03:04 -10:00
pictuga
f7d570d4c8
crawler: add some broken as rss mimetype
...
Seen out there
2017-03-18 23:00:13 -10:00
pictuga
2003e2760b
Move custom_handler to crawler
...
Makes more sense. Easier to reuse. Also cleaned up a bit the code
2017-03-18 22:51:27 -10:00
pictuga
beec6469cc
README: link to todo-list
2017-03-18 22:38:50 -10:00
pictuga
10fef6be4e
README: "update" markdown format
...
Thanks GitHub for changing what you accept
2017-03-18 22:36:32 -10:00
pictuga
e1a13a623c
crawler: remove unefficient feedburner-specific code
2017-03-18 22:31:03 -10:00
pictuga
367f86987d
readabilite: spread score to all ancestors
...
Instead of just parents and grandparents
2017-03-18 22:24:38 -10:00
pictuga
e3ab3c6823
crawler: use less tertiary operator
...
Inherited from fork
2017-03-18 22:23:39 -10:00
pictuga
65055290d4
crawler: better use of chardet
...
Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
2017-03-18 22:19:54 -10:00
pictuga