Commit Graph

995 Commits (e90c3ca7476ebbcf719dfe9386a7ba607c75d3b1)
 

Author SHA1 Message Date
pictuga 6f0efd5802 crawler: add cookies support
Somehow got dropped when splitting the big handler
2017-03-25 19:51:42 -10:00
pictuga d3bc2926fc Remove :hungry
Mostly usless. If you need it, you might as well not need to use morss in the first place...
2017-03-25 13:52:58 -10:00
pictuga 505b02d70d crawler: remove debugging print() 2017-03-25 13:45:12 -10:00
pictuga 3ca6ed5bb0 readabilite: add author/about to black list 2017-03-24 22:02:41 -10:00
pictuga 4aa25bf3d8 readabilite: clean_html before scoring
Surprisingly efficient
2017-03-24 21:50:46 -10:00
pictuga bfefa8d599 readabilite: add tags to black list 2017-03-24 21:50:26 -10:00
pictuga 91da0f36dc readabilite: comment the clean_html function 2017-03-24 21:50:01 -10:00
pictuga 67889a1d14 readabilite: drop useless tags
This extra cluster actually jams the algorithm
2017-03-24 21:49:14 -10:00
pictuga 6024728341 README: mention / > | in webserver 2017-03-20 21:05:40 -10:00
pictuga c929ce6278 README: update about options.items 2017-03-20 21:02:25 -10:00
pictuga 167e3e4a15 feedify: accept xpath rules passed as parameters 2017-03-20 20:56:48 -10:00
pictuga bf3ef586c2 feedify: remove unused downloader 2017-03-20 20:53:52 -10:00
pictuga 08f08ef704 improve morss url detection regex 2017-03-20 20:51:13 -10:00
pictuga 1b4341f741 accept query_string in morss cgi 2017-03-20 20:50:04 -10:00
pictuga f965566054 feedify; make function use clearer 2017-03-20 20:19:08 -10:00
pictuga d6882e0a6a readabilite: (try to) emprove detection
Kinda hopeless
2017-03-19 02:00:31 -10:00
pictuga 79a8ada9f4 readabilite: add tags to score 2017-03-19 01:57:54 -10:00
pictuga 4a5150e030 readabilite: fix iter while iterating 2017-03-19 01:56:33 -10:00
pictuga e65c88abf8 readabilite: fix re.match 2017-03-19 01:55:40 -10:00
pictuga 9c331300eb crawler: move UAHandler to basic
Fuck u feedburner
2017-03-19 01:49:17 -10:00
pictuga 5e61686373 Only use full feed for articles & feedify
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
2017-03-18 23:43:28 -10:00
pictuga 0b6e553054 Move iTunes code to feedify.py 2017-03-18 23:41:37 -10:00
pictuga d4937812a8 Remove HTTPError code
Only used to look nice but useless (inherits from IOError anyway)
2017-03-18 23:39:32 -10:00
pictuga 99f3c519f2 crawler: fix accept code 2017-03-18 23:37:51 -10:00
pictuga 67f5a21019 Move build_opener to crawler
Forgotten
2017-03-18 23:03:04 -10:00
pictuga f7d570d4c8 crawler: add some broken as rss mimetype
Seen out there
2017-03-18 23:00:13 -10:00
pictuga 2003e2760b Move custom_handler to crawler
Makes more sense. Easier to reuse. Also cleaned up a bit the code
2017-03-18 22:51:27 -10:00
pictuga beec6469cc README: link to todo-list 2017-03-18 22:38:50 -10:00
pictuga 10fef6be4e README: "update" markdown format
Thanks GitHub for changing what you accept
2017-03-18 22:36:32 -10:00
pictuga e1a13a623c crawler: remove unefficient feedburner-specific code 2017-03-18 22:31:03 -10:00
pictuga 367f86987d readabilite: spread score to all ancestors
Instead of just parents and grandparents
2017-03-18 22:24:38 -10:00
pictuga e3ab3c6823 crawler: use less tertiary operator
Inherited from fork
2017-03-18 22:23:39 -10:00
pictuga 65055290d4 crawler: better use of chardet
Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
2017-03-18 22:19:54 -10:00
pictuga 9ee6ff60e1 crawler: 301 http code doesn't respect headers
More or less according to the specs
2017-03-18 22:18:10 -10:00
pictuga f4abc4e8a4 Detect encoding (using crawler) before readabilite 2017-03-11 02:30:57 -10:00
pictuga c952b85d92 crawler: cache 301 HTTP code, for a week 2017-03-09 09:37:05 -10:00
pictuga e8023e4336 crawler: remove unused NotInCache error-class 2017-03-09 09:35:40 -10:00
pictuga 3385198f4f README: remove removed :force 2017-03-08 19:20:43 -10:00
pictuga 385f9eb39a morss: use crawler strict accept for feed 2017-03-08 19:05:48 -10:00
Florian Muenchbach 993ac638a3 Added override for auto-detected character encoding of parsed pages. 2017-03-08 18:45:20 -10:00
pictuga 627163abff Make cache settings in morss nicer 2017-03-08 18:09:24 -10:00
pictuga e5f8e43659 Shifted the <link rel='alternate'/> redirect to crawler
Now using MIMETYPE var from crawler within morss.py
2017-03-08 18:03:34 -10:00
pictuga fb8825b410 crawler: parse html to get http-equiv
For sure slower, but way cleaner (and probably more stable)
2017-03-08 17:50:57 -10:00
pictuga 92b4a5c57c README: remove :smart doc as it was dropped 2017-03-08 17:38:00 -10:00
Florian Muenchbach 543ae89e22 Added new 'encoding' option to README. 2017-03-08 17:38:00 -10:00
pictuga f4f6a86147 feeds: make wheezy.template mandatory
Cleaner code. Less confusing.
2017-03-08 15:38:59 -10:00
pictuga 795a0b281f README: add wheezy.template as dep 2017-03-08 15:25:45 -10:00
pictuga af66ab5980 README: simplify the sample file tree for mod_cgi
So that it never gets outdated again...
2017-03-08 15:21:26 -10:00
pictuga fe3f6b7e33 README: Fetch->FeedFetch 2017-03-08 15:17:40 -10:00
pictuga ff9690666d Add chardet to requirements.txt 2017-03-08 11:46:11 -10:00