pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity
612 Commits
1 Branch
2 Tags
3.5 MiB
Commit Graph

324 Commits (2704e91a3d400101fe51719320ad2e14ab05ef69)

Author SHA1 Message Date
pictuga 94372af868 feeds: transitional code for json/csv/html export 2018-11-09 22:04:46 +01:00
pictuga 6d28323e3a feeds: add XML support for merger 2018-11-09 22:04:08 +01:00
pictuga 5a4a86d622 feeds: add base classes for merger 2018-11-09 22:02:44 +01:00
pictuga d321550166 feeds: prepare feedify.ini for merger 2018-11-09 21:53:19 +01:00
pictuga d1aab99b80 feeds: replacement code for descriptors 2018-10-31 22:15:34 +01:00
pictuga 16f3ffa96e feeds: remove further Descriptor code 2018-10-31 22:15:15 +01:00
pictuga 02b7e07097 feeds: fix typo 2018-10-31 22:07:49 +01:00
pictuga 8487a43c6c feeds: remove FeedList(Descriptor) 2018-10-31 22:07:16 +01:00
pictuga 081d560bc4 feeds: create obj to keep FeedItems unique 2018-10-31 21:47:19 +01:00
pictuga cfd758b6b5 feeds: shift easy ones to @property 2018-10-26 19:48:39 +02:00
pictuga 4e144487db Test for feedify support first 
Otherwise might never be called if the content-type is also supported
 2018-10-25 01:17:24 +02:00
pictuga d13362c4ac feeds: drop .iterchildren 
Redundant
 2018-10-25 01:16:28 +02:00
pictuga 17856929fe feeds: pretty_print was made a default 2018-10-25 01:16:07 +02:00
pictuga 90110a4661 crawler: reduce max file size 2018-10-25 01:15:09 +02:00
pictuga 91a084e5ed crawler: make py2/3 code distinction clearer 2018-10-25 01:14:46 +02:00
pictuga 5d93d68f62 readabilite: add some function descriptions 2018-10-25 01:12:42 +02:00
pictuga 8d7e1811fd readabilite: update lists 
Some code was also meant to be committed earlier
 2018-10-25 01:12:08 +02:00
pictuga 72d03f21fe readabilite: forgot count_content 
Was meant to be in an earlier commit
 2018-10-25 01:11:29 +02:00
pictuga 1d6d0b8ff1 readabilite: move br2p in the cleaning code 2018-10-25 01:09:15 +02:00
pictuga 7d005e9a65 readabilite: run the new cleaning code 2018-10-25 01:08:25 +02:00
pictuga 58fe5243af readabilite: improve cleaning code 2018-10-25 01:07:25 +02:00
pictuga f044c242ef readabilite: simplify scoring loop 
For perfomance
 2018-10-25 00:59:39 +02:00
pictuga a6befad136 readabilite: change scoring 2018-10-25 00:57:43 +02:00
pictuga 9e71de8d40 readabilite: improve output 2018-10-24 23:49:16 +02:00
pictuga 787d90fac0 readabilite: some technical improvements for score 
Linear, removed misplaced debugging code
 2018-10-24 23:47:37 +02:00
pictuga 040d2cb889 readabilite: improve word count 2018-10-23 00:09:34 +02:00
pictuga 9fcef826f5 reader: everything in one file 
Including css & js. Should now works by itself
 2018-10-22 23:55:14 +02:00
pictuga e72ca3f984 morss: improved output type 2018-09-30 22:02:29 +02:00
pictuga 2ccf36617a morss: improve http parameter parsing 2018-09-30 22:01:19 +02:00
pictuga 945e0dceab crawler: typo in comment 2018-09-30 21:59:50 +02:00
pictuga 5111d40011 feedify: update rules 
They obviously no longer worked after so long without updating them...
 2018-09-30 21:54:10 +02:00
pictuga f9217102f3 crawler: fix sqlite/binary issue 2017-11-25 19:58:14 +01:00
pictuga 21480f90de Move from gzip to zlib to decompress data 
Faster on incomplete files
 2017-11-25 19:57:41 +01:00
pictuga d091e74d56 crawler: add MySQL backend 
With extra dependency
 2017-11-04 14:51:41 +01:00
pictuga f29a107a09 crawler: make SQLiteCache inherit from BaseCache 
Saves some time for other cache backends
 2017-11-04 14:48:00 +01:00
pictuga 2d5bf7b38b Fix xml detection regex 
Also (dirtily) fixes #18 for now
 2017-11-04 14:21:05 +01:00
pictuga b7db78f631 crawler: use BLOB in sqlite and drop "buffer" 
Can't really remember why "buffer" was introduced in the first place
 2017-11-04 13:54:40 +01:00
pictuga 203ba10dbd Explain __init__.py and __main__.py use 2017-11-04 13:17:12 +01:00
pictuga 194465544a crawler: separate CacheHander and actual caching 
Default cache is now just an in-memory {}
 2017-11-04 12:41:56 +01:00
pictuga 523b250907 crawler: SQL request in CAPS for readability 2017-11-04 12:36:58 +01:00
pictuga 2d7d0fcdca morss: fix cgi in python 3 
Needs explicit [] in py3
 2017-11-04 12:27:47 +01:00
pictuga a8c2df7f41 crawler: fix truncated gzip reader 
For python 3
 2017-11-04 12:07:08 +01:00
pictuga d39d0f4cae crawler: properly define default sqlite file 2017-11-02 22:50:40 +01:00
pictuga f563040809 readabilite: threshold to detect if it contains an article 
Useful for videos/images-based images
 2017-10-28 01:30:21 +02:00
pictuga 0df6409b0e crawler: use `with con` to commit, journal WAL for perf 2017-10-28 01:28:47 +02:00
pictuga 7b85f692a0 crawler: fix encoding detection 2017-10-27 23:14:08 +02:00
pictuga 840842d246 crawler: limit download to 500KiB 
More can only be linked to a fraudulent/incorrect use of the service
 2017-10-27 23:12:40 +02:00
pictuga fbe811384a crawler: add (unused) DebugHandler to output headers sent/received 
Saves a lot of time when debugging
 2017-10-27 23:10:03 +02:00
pictuga 64babd6713 morss: make readabilite links absolute 2017-07-29 14:37:37 +02:00
pictuga 3bfad54add readabilite: change cleaning & code structure 
Kinda struggled to make some "nice" code
 2017-07-17 00:27:41 +02:00
pictuga 386bafd391 readabilite: write_all use "node" instead of "item" 2017-07-17 00:13:15 +02:00
pictuga a61b259792 readabilite: easy option to highlight the nodes 2017-07-17 00:11:49 +02:00
pictuga c52b47616d readabilite: always return common of 2 best nodes 
Better results. Less is not more
 2017-07-17 00:10:58 +02:00
pictuga bfdda18b9c readbilite: better explain lowest_common output 2017-07-17 00:08:00 +02:00
pictuga 2afea497a3 readabilite: br2p use "node" instead of "item" 
Confusing with rss items otherwise
 2017-07-17 00:06:39 +02:00
pictuga 843dc97fbf readabilite: change scoring algorithm 
Use 3 groups of keywords instead
 2017-07-17 00:01:44 +02:00
pictuga df22396838 Only use chardet on 2k letters 
Takes forever otherwise
 2017-07-16 23:59:06 +02:00
pictuga 6f0efd5802 crawler: add cookies support 
Somehow got dropped when splitting the big handler
 2017-03-25 19:51:42 -10:00
pictuga d3bc2926fc Remove :hungry 
Mostly usless. If you need it, you might as well not need to use morss in the first place...
 2017-03-25 13:52:58 -10:00
pictuga 505b02d70d crawler: remove debugging print() 2017-03-25 13:45:12 -10:00
pictuga 3ca6ed5bb0 readabilite: add author/about to black list 2017-03-24 22:02:41 -10:00
pictuga 4aa25bf3d8 readabilite: clean_html before scoring 
Surprisingly efficient
 2017-03-24 21:50:46 -10:00
pictuga bfefa8d599 readabilite: add tags to black list 2017-03-24 21:50:26 -10:00
pictuga 91da0f36dc readabilite: comment the clean_html function 2017-03-24 21:50:01 -10:00
pictuga 67889a1d14 readabilite: drop useless tags 
This extra cluster actually jams the algorithm
 2017-03-24 21:49:14 -10:00
pictuga 167e3e4a15 feedify: accept xpath rules passed as parameters 2017-03-20 20:56:48 -10:00
pictuga bf3ef586c2 feedify: remove unused downloader 2017-03-20 20:53:52 -10:00
pictuga 08f08ef704 improve morss url detection regex 2017-03-20 20:51:13 -10:00
pictuga 1b4341f741 accept query_string in morss cgi 2017-03-20 20:50:04 -10:00
pictuga f965566054 feedify; make function use clearer 2017-03-20 20:19:08 -10:00
pictuga d6882e0a6a readabilite: (try to) emprove detection 
Kinda hopeless
 2017-03-19 02:00:31 -10:00
pictuga 79a8ada9f4 readabilite: add tags to score 2017-03-19 01:57:54 -10:00
pictuga 4a5150e030 readabilite: fix iter while iterating 2017-03-19 01:56:33 -10:00
pictuga e65c88abf8 readabilite: fix re.match 2017-03-19 01:55:40 -10:00
pictuga 9c331300eb crawler: move UAHandler to basic 
Fuck u feedburner
 2017-03-19 01:49:17 -10:00
pictuga 5e61686373 Only use full feed for articles & feedify 
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
 2017-03-18 23:43:28 -10:00
pictuga 0b6e553054 Move iTunes code to feedify.py 2017-03-18 23:41:37 -10:00
pictuga d4937812a8 Remove HTTPError code 
Only used to look nice but useless (inherits from IOError anyway)
 2017-03-18 23:39:32 -10:00
pictuga 99f3c519f2 crawler: fix accept code 2017-03-18 23:37:51 -10:00
pictuga 67f5a21019 Move build_opener to crawler 
Forgotten
 2017-03-18 23:03:04 -10:00
pictuga f7d570d4c8 crawler: add some broken as rss mimetype 
Seen out there
 2017-03-18 23:00:13 -10:00
pictuga 2003e2760b Move custom_handler to crawler 
Makes more sense. Easier to reuse. Also cleaned up a bit the code
 2017-03-18 22:51:27 -10:00
pictuga e1a13a623c crawler: remove unefficient feedburner-specific code 2017-03-18 22:31:03 -10:00
pictuga 367f86987d readabilite: spread score to all ancestors 
Instead of just parents and grandparents
 2017-03-18 22:24:38 -10:00
pictuga e3ab3c6823 crawler: use less tertiary operator 
Inherited from fork
 2017-03-18 22:23:39 -10:00
pictuga 65055290d4 crawler: better use of chardet 
Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
 2017-03-18 22:19:54 -10:00
pictuga 9ee6ff60e1 crawler: 301 http code doesn't respect headers 
More or less according to the specs
 2017-03-18 22:18:10 -10:00
pictuga f4abc4e8a4 Detect encoding (using crawler) before readabilite 2017-03-11 02:30:57 -10:00
pictuga c952b85d92 crawler: cache 301 HTTP code, for a week 2017-03-09 09:37:05 -10:00
pictuga e8023e4336 crawler: remove unused NotInCache error-class 2017-03-09 09:35:40 -10:00
pictuga 385f9eb39a morss: use crawler strict accept for feed 2017-03-08 19:05:48 -10:00
Florian Muenchbach 993ac638a3 Added override for auto-detected character encoding of parsed pages. 2017-03-08 18:45:20 -10:00
pictuga 627163abff Make cache settings in morss nicer 2017-03-08 18:09:24 -10:00
pictuga e5f8e43659 Shifted the <link rel='alternate'/> redirect to crawler 
Now using MIMETYPE var from crawler within morss.py
 2017-03-08 18:03:34 -10:00
pictuga fb8825b410 crawler: parse html to get http-equiv 
For sure slower, but way cleaner (and probably more stable)
 2017-03-08 17:50:57 -10:00
pictuga f4f6a86147 feeds: make wheezy.template mandatory 
Cleaner code. Less confusing.
 2017-03-08 15:38:59 -10:00
pictuga ad9bf946ec crawler: use chardet again 
Always nice in case no encoding is specified. Somehow got dropped with commit 245ba99. Most probably by accident
 2017-03-08 11:37:12 -10:00
pictuga 3fc89d5359 readabilite: improve score for <p> 
Helps a lot with bbc, le monde. Might backfire on other websites tho...
 2017-03-01 18:02:45 -10:00
pictuga a8ac2ed1ca Turn FeedBefore/After into ItemBefore/After 
To reduce the number of loops
 2017-02-28 23:24:32 -10:00
pictuga fcc5e8a076 Add "Feed/Item" in functions name 
To make it instantly clearer what they work on
 2017-02-28 23:23:15 -10:00