 pictuga
|
2ccf36617a
|
morss: improve http parameter parsing
|
2018-09-30 22:01:19 +02:00 |
|
pictuga
|
945e0dceab
|
crawler: typo in comment
|
2018-09-30 21:59:50 +02:00 |
|
pictuga
|
5111d40011
|
feedify: update rules
They obviously no longer worked after so long without updating them...
|
2018-09-30 21:54:10 +02:00 |
|
pictuga
|
f9217102f3
|
crawler: fix sqlite/binary issue
|
2017-11-25 19:58:14 +01:00 |
|
pictuga
|
21480f90de
|
Move from gzip to zlib to decompress data
Faster on incomplete files
|
2017-11-25 19:57:41 +01:00 |
|
pictuga
|
d091e74d56
|
crawler: add MySQL backend
With extra dependency
|
2017-11-04 14:51:41 +01:00 |
|
pictuga
|
f29a107a09
|
crawler: make SQLiteCache inherit from BaseCache
Saves some time for other cache backends
|
2017-11-04 14:48:00 +01:00 |
|
pictuga
|
2d5bf7b38b
|
Fix xml detection regex
Also (dirtily) fixes #18 for now
|
2017-11-04 14:21:05 +01:00 |
|
pictuga
|
b7db78f631
|
crawler: use BLOB in sqlite and drop "buffer"
Can't really remember why "buffer" was introduced in the first place
|
2017-11-04 13:54:40 +01:00 |
|
pictuga
|
203ba10dbd
|
Explain __init__.py and __main__.py use
|
2017-11-04 13:17:12 +01:00 |
|
pictuga
|
194465544a
|
crawler: separate CacheHander and actual caching
Default cache is now just an in-memory {}
|
2017-11-04 12:41:56 +01:00 |
|
pictuga
|
523b250907
|
crawler: SQL request in CAPS for readability
|
2017-11-04 12:36:58 +01:00 |
|
pictuga
|
2d7d0fcdca
|
morss: fix cgi in python 3
Needs explicit [] in py3
|
2017-11-04 12:27:47 +01:00 |
|
pictuga
|
a8c2df7f41
|
crawler: fix truncated gzip reader
For python 3
|
2017-11-04 12:07:08 +01:00 |
|
pictuga
|
d39d0f4cae
|
crawler: properly define default sqlite file
|
2017-11-02 22:50:40 +01:00 |
|
pictuga
|
f563040809
|
readabilite: threshold to detect if it contains an article
Useful for videos/images-based images
|
2017-10-28 01:30:21 +02:00 |
|
pictuga
|
0df6409b0e
|
crawler: use `with con` to commit, journal WAL for perf
|
2017-10-28 01:28:47 +02:00 |
|
pictuga
|
7b85f692a0
|
crawler: fix encoding detection
|
2017-10-27 23:14:08 +02:00 |
|
pictuga
|
840842d246
|
crawler: limit download to 500KiB
More can only be linked to a fraudulent/incorrect use of the service
|
2017-10-27 23:12:40 +02:00 |
|
pictuga
|
fbe811384a
|
crawler: add (unused) DebugHandler to output headers sent/received
Saves a lot of time when debugging
|
2017-10-27 23:10:03 +02:00 |
|
pictuga
|
64babd6713
|
morss: make readabilite links absolute
|
2017-07-29 14:37:37 +02:00 |
|
pictuga
|
3bfad54add
|
readabilite: change cleaning & code structure
Kinda struggled to make some "nice" code
|
2017-07-17 00:27:41 +02:00 |
|
pictuga
|
386bafd391
|
readabilite: write_all use "node" instead of "item"
|
2017-07-17 00:13:15 +02:00 |
|
pictuga
|
a61b259792
|
readabilite: easy option to highlight the nodes
|
2017-07-17 00:11:49 +02:00 |
|
pictuga
|
c52b47616d
|
readabilite: always return common of 2 best nodes
Better results. Less is not more
|
2017-07-17 00:10:58 +02:00 |
|
pictuga
|
bfdda18b9c
|
readbilite: better explain lowest_common output
|
2017-07-17 00:08:00 +02:00 |
|
pictuga
|
2afea497a3
|
readabilite: br2p use "node" instead of "item"
Confusing with rss items otherwise
|
2017-07-17 00:06:39 +02:00 |
|
pictuga
|
843dc97fbf
|
readabilite: change scoring algorithm
Use 3 groups of keywords instead
|
2017-07-17 00:01:44 +02:00 |
|
pictuga
|
df22396838
|
Only use chardet on 2k letters
Takes forever otherwise
|
2017-07-16 23:59:06 +02:00 |
|
pictuga
|
6f0efd5802
|
crawler: add cookies support
Somehow got dropped when splitting the big handler
|
2017-03-25 19:51:42 -10:00 |
|
pictuga
|
d3bc2926fc
|
Remove :hungry
Mostly usless. If you need it, you might as well not need to use morss in the first place...
|
2017-03-25 13:52:58 -10:00 |
|
pictuga
|
505b02d70d
|
crawler: remove debugging print()
|
2017-03-25 13:45:12 -10:00 |
|
pictuga
|
3ca6ed5bb0
|
readabilite: add author/about to black list
|
2017-03-24 22:02:41 -10:00 |
|
pictuga
|
4aa25bf3d8
|
readabilite: clean_html before scoring
Surprisingly efficient
|
2017-03-24 21:50:46 -10:00 |
|
pictuga
|
bfefa8d599
|
readabilite: add tags to black list
|
2017-03-24 21:50:26 -10:00 |
|
pictuga
|
91da0f36dc
|
readabilite: comment the clean_html function
|
2017-03-24 21:50:01 -10:00 |
|
pictuga
|
67889a1d14
|
readabilite: drop useless tags
This extra cluster actually jams the algorithm
|
2017-03-24 21:49:14 -10:00 |
|
pictuga
|
167e3e4a15
|
feedify: accept xpath rules passed as parameters
|
2017-03-20 20:56:48 -10:00 |
|
pictuga
|
bf3ef586c2
|
feedify: remove unused downloader
|
2017-03-20 20:53:52 -10:00 |
|
pictuga
|
08f08ef704
|
improve morss url detection regex
|
2017-03-20 20:51:13 -10:00 |
|
pictuga
|
1b4341f741
|
accept query_string in morss cgi
|
2017-03-20 20:50:04 -10:00 |
|
pictuga
|
f965566054
|
feedify; make function use clearer
|
2017-03-20 20:19:08 -10:00 |
|
pictuga
|
d6882e0a6a
|
readabilite: (try to) emprove detection
Kinda hopeless
|
2017-03-19 02:00:31 -10:00 |
|
pictuga
|
79a8ada9f4
|
readabilite: add tags to score
|
2017-03-19 01:57:54 -10:00 |
|
pictuga
|
4a5150e030
|
readabilite: fix iter while iterating
|
2017-03-19 01:56:33 -10:00 |
|
pictuga
|
e65c88abf8
|
readabilite: fix re.match
|
2017-03-19 01:55:40 -10:00 |
|
pictuga
|
9c331300eb
|
crawler: move UAHandler to basic
Fuck u feedburner
|
2017-03-19 01:49:17 -10:00 |
|
pictuga
|
5e61686373
|
Only use full feed for articles & feedify
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
|
2017-03-18 23:43:28 -10:00 |
|
pictuga
|
0b6e553054
|
Move iTunes code to feedify.py
|
2017-03-18 23:41:37 -10:00 |
|
pictuga
|
d4937812a8
|
Remove HTTPError code
Only used to look nice but useless (inherits from IOError anyway)
|
2017-03-18 23:39:32 -10:00 |