 pictuga
|
df22396838
|
Only use chardet on 2k letters
Takes forever otherwise
|
2017-07-16 23:59:06 +02:00 |
|
pictuga
|
6f0efd5802
|
crawler: add cookies support
Somehow got dropped when splitting the big handler
|
2017-03-25 19:51:42 -10:00 |
|
pictuga
|
d3bc2926fc
|
Remove :hungry
Mostly usless. If you need it, you might as well not need to use morss in the first place...
|
2017-03-25 13:52:58 -10:00 |
|
pictuga
|
505b02d70d
|
crawler: remove debugging print()
|
2017-03-25 13:45:12 -10:00 |
|
pictuga
|
3ca6ed5bb0
|
readabilite: add author/about to black list
|
2017-03-24 22:02:41 -10:00 |
|
pictuga
|
4aa25bf3d8
|
readabilite: clean_html before scoring
Surprisingly efficient
|
2017-03-24 21:50:46 -10:00 |
|
pictuga
|
bfefa8d599
|
readabilite: add tags to black list
|
2017-03-24 21:50:26 -10:00 |
|
pictuga
|
91da0f36dc
|
readabilite: comment the clean_html function
|
2017-03-24 21:50:01 -10:00 |
|
pictuga
|
67889a1d14
|
readabilite: drop useless tags
This extra cluster actually jams the algorithm
|
2017-03-24 21:49:14 -10:00 |
|
pictuga
|
6024728341
|
README: mention / > | in webserver
|
2017-03-20 21:05:40 -10:00 |
|
pictuga
|
c929ce6278
|
README: update about options.items
|
2017-03-20 21:02:25 -10:00 |
|
pictuga
|
167e3e4a15
|
feedify: accept xpath rules passed as parameters
|
2017-03-20 20:56:48 -10:00 |
|
pictuga
|
bf3ef586c2
|
feedify: remove unused downloader
|
2017-03-20 20:53:52 -10:00 |
|
pictuga
|
08f08ef704
|
improve morss url detection regex
|
2017-03-20 20:51:13 -10:00 |
|
pictuga
|
1b4341f741
|
accept query_string in morss cgi
|
2017-03-20 20:50:04 -10:00 |
|
pictuga
|
f965566054
|
feedify; make function use clearer
|
2017-03-20 20:19:08 -10:00 |
|
pictuga
|
d6882e0a6a
|
readabilite: (try to) emprove detection
Kinda hopeless
|
2017-03-19 02:00:31 -10:00 |
|
pictuga
|
79a8ada9f4
|
readabilite: add tags to score
|
2017-03-19 01:57:54 -10:00 |
|
pictuga
|
4a5150e030
|
readabilite: fix iter while iterating
|
2017-03-19 01:56:33 -10:00 |
|
pictuga
|
e65c88abf8
|
readabilite: fix re.match
|
2017-03-19 01:55:40 -10:00 |
|
pictuga
|
9c331300eb
|
crawler: move UAHandler to basic
Fuck u feedburner
|
2017-03-19 01:49:17 -10:00 |
|
pictuga
|
5e61686373
|
Only use full feed for articles & feedify
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
|
2017-03-18 23:43:28 -10:00 |
|
pictuga
|
0b6e553054
|
Move iTunes code to feedify.py
|
2017-03-18 23:41:37 -10:00 |
|
pictuga
|
d4937812a8
|
Remove HTTPError code
Only used to look nice but useless (inherits from IOError anyway)
|
2017-03-18 23:39:32 -10:00 |
|
pictuga
|
99f3c519f2
|
crawler: fix accept code
|
2017-03-18 23:37:51 -10:00 |
|
pictuga
|
67f5a21019
|
Move build_opener to crawler
Forgotten
|
2017-03-18 23:03:04 -10:00 |
|
pictuga
|
f7d570d4c8
|
crawler: add some broken as rss mimetype
Seen out there
|
2017-03-18 23:00:13 -10:00 |
|
pictuga
|
2003e2760b
|
Move custom_handler to crawler
Makes more sense. Easier to reuse. Also cleaned up a bit the code
|
2017-03-18 22:51:27 -10:00 |
|
pictuga
|
beec6469cc
|
README: link to todo-list
|
2017-03-18 22:38:50 -10:00 |
|
pictuga
|
10fef6be4e
|
README: "update" markdown format
Thanks GitHub for changing what you accept
|
2017-03-18 22:36:32 -10:00 |
|
pictuga
|
e1a13a623c
|
crawler: remove unefficient feedburner-specific code
|
2017-03-18 22:31:03 -10:00 |
|
pictuga
|
367f86987d
|
readabilite: spread score to all ancestors
Instead of just parents and grandparents
|
2017-03-18 22:24:38 -10:00 |
|
pictuga
|
e3ab3c6823
|
crawler: use less tertiary operator
Inherited from fork
|
2017-03-18 22:23:39 -10:00 |
|
pictuga
|
65055290d4
|
crawler: better use of chardet
Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
|
2017-03-18 22:19:54 -10:00 |
|
pictuga
|
9ee6ff60e1
|
crawler: 301 http code doesn't respect headers
More or less according to the specs
|
2017-03-18 22:18:10 -10:00 |
|
pictuga
|
f4abc4e8a4
|
Detect encoding (using crawler) before readabilite
|
2017-03-11 02:30:57 -10:00 |
|
pictuga
|
c952b85d92
|
crawler: cache 301 HTTP code, for a week
|
2017-03-09 09:37:05 -10:00 |
|
pictuga
|
e8023e4336
|
crawler: remove unused NotInCache error-class
|
2017-03-09 09:35:40 -10:00 |
|
pictuga
|
3385198f4f
|
README: remove removed :force
|
2017-03-08 19:20:43 -10:00 |
|
pictuga
|
385f9eb39a
|
morss: use crawler strict accept for feed
|
2017-03-08 19:05:48 -10:00 |
Florian Muenchbach
|
993ac638a3
|
Added override for auto-detected character encoding of parsed pages.
|
2017-03-08 18:45:20 -10:00 |
|
pictuga
|
627163abff
|
Make cache settings in morss nicer
|
2017-03-08 18:09:24 -10:00 |
|
pictuga
|
e5f8e43659
|
Shifted the <link rel='alternate'/> redirect to crawler
Now using MIMETYPE var from crawler within morss.py
|
2017-03-08 18:03:34 -10:00 |
|
pictuga
|
fb8825b410
|
crawler: parse html to get http-equiv
For sure slower, but way cleaner (and probably more stable)
|
2017-03-08 17:50:57 -10:00 |
|
pictuga
|
92b4a5c57c
|
README: remove :smart doc as it was dropped
|
2017-03-08 17:38:00 -10:00 |
|
Florian Muenchbach
|
543ae89e22
|
Added new 'encoding' option to README.
|
2017-03-08 17:38:00 -10:00 |
|
pictuga
|
f4f6a86147
|
feeds: make wheezy.template mandatory
Cleaner code. Less confusing.
|
2017-03-08 15:38:59 -10:00 |
|
pictuga
|
795a0b281f
|
README: add wheezy.template as dep
|
2017-03-08 15:25:45 -10:00 |
|
pictuga
|
af66ab5980
|
README: simplify the sample file tree for mod_cgi
So that it never gets outdated again...
|
2017-03-08 15:21:26 -10:00 |
|
pictuga
|
fe3f6b7e33
|
README: Fetch->FeedFetch
|
2017-03-08 15:17:40 -10:00 |