pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity
977 Commits
1 Branch
2 Tags
3.5 MiB
Commit Graph

977 Commits (7cdcbd23e1d80c45b979875b3f7db90a1616b58b)
 

Author SHA1 Message Date
pictuga e65c88abf8 readabilite: fix re.match 2017-03-19 01:55:40 -10:00
pictuga 9c331300eb crawler: move UAHandler to basic 
Fuck u feedburner
 2017-03-19 01:49:17 -10:00
pictuga 5e61686373 Only use full feed for articles & feedify 
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
 2017-03-18 23:43:28 -10:00
pictuga 0b6e553054 Move iTunes code to feedify.py 2017-03-18 23:41:37 -10:00
pictuga d4937812a8 Remove HTTPError code 
Only used to look nice but useless (inherits from IOError anyway)
 2017-03-18 23:39:32 -10:00
pictuga 99f3c519f2 crawler: fix accept code 2017-03-18 23:37:51 -10:00
pictuga 67f5a21019 Move build_opener to crawler 
Forgotten
 2017-03-18 23:03:04 -10:00
pictuga f7d570d4c8 crawler: add some broken as rss mimetype 
Seen out there
 2017-03-18 23:00:13 -10:00
pictuga 2003e2760b Move custom_handler to crawler 
Makes more sense. Easier to reuse. Also cleaned up a bit the code
 2017-03-18 22:51:27 -10:00
pictuga beec6469cc README: link to todo-list 2017-03-18 22:38:50 -10:00
pictuga 10fef6be4e README: "update" markdown format 
Thanks GitHub for changing what you accept
 2017-03-18 22:36:32 -10:00
pictuga e1a13a623c crawler: remove unefficient feedburner-specific code 2017-03-18 22:31:03 -10:00
pictuga 367f86987d readabilite: spread score to all ancestors 
Instead of just parents and grandparents
 2017-03-18 22:24:38 -10:00
pictuga e3ab3c6823 crawler: use less tertiary operator 
Inherited from fork
 2017-03-18 22:23:39 -10:00
pictuga 65055290d4 crawler: better use of chardet 
Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
 2017-03-18 22:19:54 -10:00
pictuga 9ee6ff60e1 crawler: 301 http code doesn't respect headers 
More or less according to the specs
 2017-03-18 22:18:10 -10:00
pictuga f4abc4e8a4 Detect encoding (using crawler) before readabilite 2017-03-11 02:30:57 -10:00
pictuga c952b85d92 crawler: cache 301 HTTP code, for a week 2017-03-09 09:37:05 -10:00
pictuga e8023e4336 crawler: remove unused NotInCache error-class 2017-03-09 09:35:40 -10:00
pictuga 3385198f4f README: remove removed :force 2017-03-08 19:20:43 -10:00
pictuga 385f9eb39a morss: use crawler strict accept for feed 2017-03-08 19:05:48 -10:00
Florian Muenchbach 993ac638a3 Added override for auto-detected character encoding of parsed pages. 2017-03-08 18:45:20 -10:00
pictuga 627163abff Make cache settings in morss nicer 2017-03-08 18:09:24 -10:00
pictuga e5f8e43659 Shifted the <link rel='alternate'/> redirect to crawler 
Now using MIMETYPE var from crawler within morss.py
 2017-03-08 18:03:34 -10:00
pictuga fb8825b410 crawler: parse html to get http-equiv 
For sure slower, but way cleaner (and probably more stable)
 2017-03-08 17:50:57 -10:00
pictuga 92b4a5c57c README: remove :smart doc as it was dropped 2017-03-08 17:38:00 -10:00
Florian Muenchbach 543ae89e22 Added new 'encoding' option to README. 2017-03-08 17:38:00 -10:00
pictuga f4f6a86147 feeds: make wheezy.template mandatory 
Cleaner code. Less confusing.
 2017-03-08 15:38:59 -10:00
pictuga 795a0b281f README: add wheezy.template as dep 2017-03-08 15:25:45 -10:00
pictuga af66ab5980 README: simplify the sample file tree for mod_cgi 
So that it never gets outdated again...
 2017-03-08 15:21:26 -10:00
pictuga fe3f6b7e33 README: Fetch->FeedFetch 2017-03-08 15:17:40 -10:00
pictuga ff9690666d Add chardet to requirements.txt 2017-03-08 11:46:11 -10:00
pictuga 2b4269e1ac README: add chardet 2017-03-08 11:45:13 -10:00
pictuga ad9bf946ec crawler: use chardet again 
Always nice in case no encoding is specified. Somehow got dropped with commit 245ba99. Most probably by accident
 2017-03-08 11:37:12 -10:00
pictuga 4b8e3d1b8b Improve provided .htaccess 
Should be less unsafe...
 2017-03-04 19:30:30 -10:00
pictuga 3fc89d5359 readabilite: improve score for <p> 
Helps a lot with bbc, le monde. Might backfire on other websites tho...
 2017-03-01 18:02:45 -10:00
pictuga a8ac2ed1ca Turn FeedBefore/After into ItemBefore/After 
To reduce the number of loops
 2017-02-28 23:24:32 -10:00
pictuga fcc5e8a076 Add "Feed/Item" in functions name 
To make it instantly clearer what they work on
 2017-02-28 23:23:15 -10:00
pictuga 60e3311e97 Use readabilite properly 
Not thru some weird wrapper anymore
 2017-02-28 22:45:26 -10:00
pictuga 52661985d1 Remove one more readability fork reference 2017-02-28 19:25:59 -10:00
pictuga 47d33099b2 Remove readability fork from deps 
Since we now use our own, faster, implementation
 2017-02-28 19:21:51 -10:00
pictuga dc8423550f Support xml starting with \s 2017-02-25 19:04:32 -10:00
pictuga e0f533ca31 readabilite: test to replace <br/> with div 2017-02-25 18:16:15 -10:00
pictuga c6c113b8a8 readabilite: function to clean up the html code 2017-02-25 18:15:33 -10:00
pictuga 58d9f65735 readabilite: explain the use of .tail 2017-02-25 18:14:13 -10:00
pictuga a5aec8c7a6 readability: more keywords to the filter list 
Also fixed indentation
 2017-02-25 18:13:15 -10:00
pictuga 026903ce73 crawler: change http header after uncompressing 
Change content-encoding to "identity"
 2017-02-25 18:10:43 -10:00
pictuga e71fc967ce readabilite: shift "good" tags to a var (list) 
So that this list can later be re-used
 2017-02-25 18:07:28 -10:00
pictuga b14381f575 Use internal readability fork 
Much simpler, doesn't clean the html, probably less efficient, but much faster
 2016-05-31 02:50:03 +02:00
pictuga 2b9bfb47e5 Remove :smart and etag headers 
Dirty code, not very useful. Use simple cache-control instead.
 2016-05-31 02:47:49 +02:00