pictuga
72d03f21fe
readabilite: forgot count_content
...
Was meant to be in an earlier commit
2018-10-25 01:11:29 +02:00
pictuga
1d6d0b8ff1
readabilite: move br2p in the cleaning code
2018-10-25 01:09:15 +02:00
pictuga
7d005e9a65
readabilite: run the new cleaning code
2018-10-25 01:08:25 +02:00
pictuga
58fe5243af
readabilite: improve cleaning code
2018-10-25 01:07:25 +02:00
pictuga
f044c242ef
readabilite: simplify scoring loop
...
For perfomance
2018-10-25 00:59:39 +02:00
pictuga
a6befad136
readabilite: change scoring
2018-10-25 00:57:43 +02:00
pictuga
9e71de8d40
readabilite: improve output
2018-10-24 23:49:16 +02:00
pictuga
787d90fac0
readabilite: some technical improvements for score
...
Linear, removed misplaced debugging code
2018-10-24 23:47:37 +02:00
pictuga
040d2cb889
readabilite: improve word count
2018-10-23 00:09:34 +02:00
pictuga
9fcef826f5
reader: everything in one file
...
Including css & js. Should now works by itself
2018-10-22 23:55:14 +02:00
pictuga
e72ca3f984
morss: improved output type
2018-09-30 22:02:29 +02:00
pictuga
2ccf36617a
morss: improve http parameter parsing
2018-09-30 22:01:19 +02:00
pictuga
945e0dceab
crawler: typo in comment
2018-09-30 21:59:50 +02:00
pictuga
3c09dbdf31
README: list some non-obvious features
2018-09-30 21:56:30 +02:00
pictuga
ba673392d7
README: test website no longer online
2018-09-30 21:56:03 +02:00
pictuga
5111d40011
feedify: update rules
...
They obviously no longer worked after so long without updating them...
2018-09-30 21:54:10 +02:00
pictuga
f9217102f3
crawler: fix sqlite/binary issue
2017-11-25 19:58:14 +01:00
pictuga
21480f90de
Move from gzip to zlib to decompress data
...
Faster on incomplete files
2017-11-25 19:57:41 +01:00
pictuga
d091e74d56
crawler: add MySQL backend
...
With extra dependency
2017-11-04 14:51:41 +01:00
pictuga
f29a107a09
crawler: make SQLiteCache inherit from BaseCache
...
Saves some time for other cache backends
2017-11-04 14:48:00 +01:00
pictuga
2d5bf7b38b
Fix xml detection regex
...
Also (dirtily) fixes #18 for now
2017-11-04 14:21:05 +01:00
pictuga
b7db78f631
crawler: use BLOB in sqlite and drop "buffer"
...
Can't really remember why "buffer" was introduced in the first place
2017-11-04 13:54:40 +01:00
pictuga
203ba10dbd
Explain __init__.py and __main__.py use
2017-11-04 13:17:12 +01:00
pictuga
194465544a
crawler: separate CacheHander and actual caching
...
Default cache is now just an in-memory {}
2017-11-04 12:41:56 +01:00
pictuga
523b250907
crawler: SQL request in CAPS for readability
2017-11-04 12:36:58 +01:00
pictuga
2d7d0fcdca
morss: fix cgi in python 3
...
Needs explicit [] in py3
2017-11-04 12:27:47 +01:00
pictuga
a8c2df7f41
crawler: fix truncated gzip reader
...
For python 3
2017-11-04 12:07:08 +01:00
pictuga
d39d0f4cae
crawler: properly define default sqlite file
2017-11-02 22:50:40 +01:00
pictuga
f563040809
readabilite: threshold to detect if it contains an article
...
Useful for videos/images-based images
2017-10-28 01:30:21 +02:00
pictuga
0df6409b0e
crawler: use `with con` to commit, journal WAL for perf
2017-10-28 01:28:47 +02:00
pictuga
7b85f692a0
crawler: fix encoding detection
2017-10-27 23:14:08 +02:00
pictuga
840842d246
crawler: limit download to 500KiB
...
More can only be linked to a fraudulent/incorrect use of the service
2017-10-27 23:12:40 +02:00
pictuga
fbe811384a
crawler: add (unused) DebugHandler to output headers sent/received
...
Saves a lot of time when debugging
2017-10-27 23:10:03 +02:00
pictuga
e0092387b1
Add "six" as a requirement in README
...
Actually a dependency of "chardet", but since people (including me) might just send file onto their server instead of installing the libs properly, it's a nice reminder (not in requirements.txt since that one is automatic)
2017-07-30 16:58:53 +02:00
pictuga
b8d8d9ea20
Fix www/.htaccess
...
Deny/allow issue
2017-07-29 17:12:11 +02:00
pictuga
64babd6713
morss: make readabilite links absolute
2017-07-29 14:37:37 +02:00
pictuga
bbca5dca6b
Create LICENSE
2017-07-28 10:28:17 +02:00
pictuga
10cbebb80c
Imrove provided .htaccess
2017-07-23 20:22:40 +02:00
pictuga
3bfad54add
readabilite: change cleaning & code structure
...
Kinda struggled to make some "nice" code
2017-07-17 00:27:41 +02:00
pictuga
386bafd391
readabilite: write_all use "node" instead of "item"
2017-07-17 00:13:15 +02:00
pictuga
a61b259792
readabilite: easy option to highlight the nodes
2017-07-17 00:11:49 +02:00
pictuga
c52b47616d
readabilite: always return common of 2 best nodes
...
Better results. Less is not more
2017-07-17 00:10:58 +02:00
pictuga
bfdda18b9c
readbilite: better explain lowest_common output
2017-07-17 00:08:00 +02:00
pictuga
2afea497a3
readabilite: br2p use "node" instead of "item"
...
Confusing with rss items otherwise
2017-07-17 00:06:39 +02:00
pictuga
843dc97fbf
readabilite: change scoring algorithm
...
Use 3 groups of keywords instead
2017-07-17 00:01:44 +02:00
pictuga
df22396838
Only use chardet on 2k letters
...
Takes forever otherwise
2017-07-16 23:59:06 +02:00
pictuga
6f0efd5802
crawler: add cookies support
...
Somehow got dropped when splitting the big handler
2017-03-25 19:51:42 -10:00
pictuga
d3bc2926fc
Remove :hungry
...
Mostly usless. If you need it, you might as well not need to use morss in the first place...
2017-03-25 13:52:58 -10:00
pictuga
505b02d70d
crawler: remove debugging print()
2017-03-25 13:45:12 -10:00
pictuga
3ca6ed5bb0
readabilite: add author/about to black list
2017-03-24 22:02:41 -10:00
pictuga