Commit Graph

626 Commits (f8c09af563e1d74771f0456dabf2f7852991967d)
 

Author SHA1 Message Date
pictuga 5111d40011 feedify: update rules
They obviously no longer worked after so long without updating them...
2018-09-30 21:54:10 +02:00
pictuga f9217102f3 crawler: fix sqlite/binary issue 2017-11-25 19:58:14 +01:00
pictuga 21480f90de Move from gzip to zlib to decompress data
Faster on incomplete files
2017-11-25 19:57:41 +01:00
pictuga d091e74d56 crawler: add MySQL backend
With extra dependency
2017-11-04 14:51:41 +01:00
pictuga f29a107a09 crawler: make SQLiteCache inherit from BaseCache
Saves some time for other cache backends
2017-11-04 14:48:00 +01:00
pictuga 2d5bf7b38b Fix xml detection regex
Also (dirtily) fixes #18 for now
2017-11-04 14:21:05 +01:00
pictuga b7db78f631 crawler: use BLOB in sqlite and drop "buffer"
Can't really remember why "buffer" was introduced in the first place
2017-11-04 13:54:40 +01:00
pictuga 203ba10dbd Explain __init__.py and __main__.py use 2017-11-04 13:17:12 +01:00
pictuga 194465544a crawler: separate CacheHander and actual caching
Default cache is now just an in-memory {}
2017-11-04 12:41:56 +01:00
pictuga 523b250907 crawler: SQL request in CAPS for readability 2017-11-04 12:36:58 +01:00
pictuga 2d7d0fcdca morss: fix cgi in python 3
Needs explicit [] in py3
2017-11-04 12:27:47 +01:00
pictuga a8c2df7f41 crawler: fix truncated gzip reader
For python 3
2017-11-04 12:07:08 +01:00
pictuga d39d0f4cae crawler: properly define default sqlite file 2017-11-02 22:50:40 +01:00
pictuga f563040809 readabilite: threshold to detect if it contains an article
Useful for videos/images-based images
2017-10-28 01:30:21 +02:00
pictuga 0df6409b0e crawler: use `with con` to commit, journal WAL for perf 2017-10-28 01:28:47 +02:00
pictuga 7b85f692a0 crawler: fix encoding detection 2017-10-27 23:14:08 +02:00
pictuga 840842d246 crawler: limit download to 500KiB
More can only be linked to a fraudulent/incorrect use of the service
2017-10-27 23:12:40 +02:00
pictuga fbe811384a crawler: add (unused) DebugHandler to output headers sent/received
Saves a lot of time when debugging
2017-10-27 23:10:03 +02:00
pictuga e0092387b1 Add "six" as a requirement in README
Actually a dependency of "chardet", but since people (including me) might just send file onto their server instead of installing the libs properly, it's a nice reminder (not in requirements.txt since that one is automatic)
2017-07-30 16:58:53 +02:00
pictuga b8d8d9ea20 Fix www/.htaccess
Deny/allow issue
2017-07-29 17:12:11 +02:00
pictuga 64babd6713 morss: make readabilite links absolute 2017-07-29 14:37:37 +02:00
pictuga bbca5dca6b Create LICENSE 2017-07-28 10:28:17 +02:00
pictuga 10cbebb80c Imrove provided .htaccess 2017-07-23 20:22:40 +02:00
pictuga 3bfad54add readabilite: change cleaning & code structure
Kinda struggled to make some "nice" code
2017-07-17 00:27:41 +02:00
pictuga 386bafd391 readabilite: write_all use "node" instead of "item" 2017-07-17 00:13:15 +02:00
pictuga a61b259792 readabilite: easy option to highlight the nodes 2017-07-17 00:11:49 +02:00
pictuga c52b47616d readabilite: always return common of 2 best nodes
Better results. Less is not more
2017-07-17 00:10:58 +02:00
pictuga bfdda18b9c readbilite: better explain lowest_common output 2017-07-17 00:08:00 +02:00
pictuga 2afea497a3 readabilite: br2p use "node" instead of "item"
Confusing with rss items otherwise
2017-07-17 00:06:39 +02:00
pictuga 843dc97fbf readabilite: change scoring algorithm
Use 3 groups of keywords instead
2017-07-17 00:01:44 +02:00
pictuga df22396838 Only use chardet on 2k letters
Takes forever otherwise
2017-07-16 23:59:06 +02:00
pictuga 6f0efd5802 crawler: add cookies support
Somehow got dropped when splitting the big handler
2017-03-25 19:51:42 -10:00
pictuga d3bc2926fc Remove :hungry
Mostly usless. If you need it, you might as well not need to use morss in the first place...
2017-03-25 13:52:58 -10:00
pictuga 505b02d70d crawler: remove debugging print() 2017-03-25 13:45:12 -10:00
pictuga 3ca6ed5bb0 readabilite: add author/about to black list 2017-03-24 22:02:41 -10:00
pictuga 4aa25bf3d8 readabilite: clean_html before scoring
Surprisingly efficient
2017-03-24 21:50:46 -10:00
pictuga bfefa8d599 readabilite: add tags to black list 2017-03-24 21:50:26 -10:00
pictuga 91da0f36dc readabilite: comment the clean_html function 2017-03-24 21:50:01 -10:00
pictuga 67889a1d14 readabilite: drop useless tags
This extra cluster actually jams the algorithm
2017-03-24 21:49:14 -10:00
pictuga 6024728341 README: mention / > | in webserver 2017-03-20 21:05:40 -10:00
pictuga c929ce6278 README: update about options.items 2017-03-20 21:02:25 -10:00
pictuga 167e3e4a15 feedify: accept xpath rules passed as parameters 2017-03-20 20:56:48 -10:00
pictuga bf3ef586c2 feedify: remove unused downloader 2017-03-20 20:53:52 -10:00
pictuga 08f08ef704 improve morss url detection regex 2017-03-20 20:51:13 -10:00
pictuga 1b4341f741 accept query_string in morss cgi 2017-03-20 20:50:04 -10:00
pictuga f965566054 feedify; make function use clearer 2017-03-20 20:19:08 -10:00
pictuga d6882e0a6a readabilite: (try to) emprove detection
Kinda hopeless
2017-03-19 02:00:31 -10:00
pictuga 79a8ada9f4 readabilite: add tags to score 2017-03-19 01:57:54 -10:00
pictuga 4a5150e030 readabilite: fix iter while iterating 2017-03-19 01:56:33 -10:00
pictuga e65c88abf8 readabilite: fix re.match 2017-03-19 01:55:40 -10:00