pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity
671 Commits
1 Branch
2 Tags
3.5 MiB
Commit Graph

370 Commits (3617f86e9d899021281c98295e50a7399dc18f80)

Author SHA1 Message Date
pictuga 449bc3c695 feeds: fix handling of html code 2020-03-19 09:48:53 +01:00
pictuga 13ea52ef80 feeds: add .torss() 2020-03-19 09:47:58 +01:00
pictuga aa2b56b266 feeds: small code cleanups 2020-03-19 09:47:17 +01:00
pictuga 296e9a6c13 feedifiy.ini: clean up following feeds.py's html intro 2020-03-18 16:52:28 +01:00
pictuga 9dbe061fd6 Remove markdown-related code 
Time to clean up the code and stop with those non-core features
They just make the code harder to maintain
 2020-03-18 16:47:00 +01:00
pictuga 4a70aa9dfa feeds: auto-parse() 2020-03-18 16:34:40 +01:00
pictuga c2f85da94a feeds: add html support, adapt .tohtml() 2020-03-18 16:33:10 +01:00
pictuga e3528a8f36 feeds: use FeddJSON for .tojson() 
Clean up related code
 2020-03-18 16:31:36 +01:00
pictuga 2dd9ae202d feeds: add 'mode' to Parsers 2020-03-18 16:24:08 +01:00
pictuga 8e3d32f24c feds: rename 'ruleset' into 'default_ruleset' 
Better reflects its use
 2020-03-18 16:22:03 +01:00
pictuga 6ce616106b feeds: disable 'multi' ruleset 
RSS ruleset has been cut into 4 rulesets
 2020-03-18 16:20:42 +01:00
pictuga 186fa2b408 feedify: remove empty 'path' lines 2020-03-18 16:18:22 +01:00
pictuga e9d46cb6a9 feeds: move mimetypes into .py from .ini 2020-03-18 16:08:42 +01:00
pictuga 7644c550ec feedify: remove id, is_permalink as well 2020-03-17 16:48:36 +01:00
pictuga e460fdc8f4 feeds: remove little-used properties 
id, is_permalink
Also a good opportunity to trash the unreliable bool-parsing code
 2020-03-17 16:46:42 +01:00
pictuga 1e714ab34b feeds: add ability to convert to another typo of feed 2020-03-17 14:02:24 +01:00
pictuga 4ba4d73ce6 feeds: add json support 2020-03-17 14:02:01 +01:00
pictuga 6d5aa8c222 feeds: clean up .append() 2020-03-17 13:59:51 +01:00
pictuga f10727f94a feeds: small code cleanup's 2020-03-17 12:26:34 +01:00
pictuga d42e19a165 feeds: beter rules handling 
"Dynamic" rule set picking, better handling of non-multi rules
 2020-03-17 12:23:36 +01:00
pictuga fe46c6c522 feeds: pass parent Feed to Items 2020-03-17 12:22:14 +01:00
pictuga 9c557ea02c feeds: fix function def 2020-03-17 11:08:40 +01:00
pictuga 8a4f86210c feedify.ini: remove utf-8 declaration 
Screws up with the parser as it is read as unicode (and xml parser expects bytes)
 2020-03-17 11:06:59 +01:00
pictuga ce30952fa2 feeds: make "rule" split clearer 
"rrule" var name to tell appart the original "rule" from the parsed one
 2020-03-16 17:46:04 +01:00
pictuga 3fb6ff891c feeds: share more code, add comments 
Should reduce redundancy
 2020-03-16 17:45:08 +01:00
pictuga f5acd2c14c feeds: use RawConfigParser 
This one does not try to replace non-std characters (e.g. %)
 2020-03-16 17:43:03 +01:00
pictuga 7cb3b29ef2 feeds: remove unused import 2020-03-16 17:38:48 +01:00
pictuga 9cb2d5bb86 feeds: centralize time format/parse 
At the same code _should_ apply to most, if not all, parsers
 2018-11-18 16:03:02 +01:00
pictuga e606c5eefb feeds: various small cleanup/fixes 2018-11-18 15:14:38 +01:00
pictuga 24c8a0ecd0 feeds: fix typo 2018-11-13 21:23:24 +01:00
pictuga 9a62e6ae75 feeds: remove old code 2018-11-13 21:22:50 +01:00
pictuga adbaed9e54 feeds: put code tgt 2018-11-11 17:24:56 +01:00
pictuga 3581f34db7 Various feeds.py related fixes 2018-11-11 16:46:23 +01:00
pictuga 966559bdd3 feeds: fix remove function in case of no match 2018-11-11 16:33:36 +01:00
pictuga 4fb98bc2ed feeds: fix append content 2018-11-11 16:33:18 +01:00
pictuga 679628c7fa Small code clean up 2018-11-11 16:11:00 +01:00
pictuga 399e867c94 morss: add py2 indication 2018-11-11 16:07:25 +01:00
pictuga c5d8b064ae feeds: fix an error when no match 2018-11-11 15:31:46 +01:00
pictuga c2a6ea7cfe feeds: give example of regex 2018-11-11 15:26:46 +01:00
pictuga 221e1f85ad feeds: fix implementation in morss 2018-11-11 15:26:09 +01:00
pictuga 857bb9c366 feeds: fix remove() unclear function naming 2018-11-11 15:25:03 +01:00
pictuga 75f691b009 feeds: fix multi rules parsing 2018-11-11 15:21:43 +01:00
pictuga 401dfbc1ff feeds: fix atom xhtml handling 2018-11-11 15:21:06 +01:00
pictuga 8aceda4957 feeds: fix feedify.ini 2018-11-11 15:19:41 +01:00
pictuga 024466733c feeds: remove old code 2018-11-09 22:09:59 +01:00
pictuga 92b06bea6d feeds: fix Uniq for merger 2018-11-09 22:05:13 +01:00
pictuga 94372af868 feeds: transitional code for json/csv/html export 2018-11-09 22:04:46 +01:00
pictuga 6d28323e3a feeds: add XML support for merger 2018-11-09 22:04:08 +01:00
pictuga 5a4a86d622 feeds: add base classes for merger 2018-11-09 22:02:44 +01:00
pictuga d321550166 feeds: prepare feedify.ini for merger 2018-11-09 21:53:19 +01:00
pictuga d1aab99b80 feeds: replacement code for descriptors 2018-10-31 22:15:34 +01:00
pictuga 16f3ffa96e feeds: remove further Descriptor code 2018-10-31 22:15:15 +01:00
pictuga 02b7e07097 feeds: fix typo 2018-10-31 22:07:49 +01:00
pictuga 8487a43c6c feeds: remove FeedList(Descriptor) 2018-10-31 22:07:16 +01:00
pictuga 081d560bc4 feeds: create obj to keep FeedItems unique 2018-10-31 21:47:19 +01:00
pictuga cfd758b6b5 feeds: shift easy ones to @property 2018-10-26 19:48:39 +02:00
pictuga 4e144487db Test for feedify support first 
Otherwise might never be called if the content-type is also supported
 2018-10-25 01:17:24 +02:00
pictuga d13362c4ac feeds: drop .iterchildren 
Redundant
 2018-10-25 01:16:28 +02:00
pictuga 17856929fe feeds: pretty_print was made a default 2018-10-25 01:16:07 +02:00
pictuga 90110a4661 crawler: reduce max file size 2018-10-25 01:15:09 +02:00
pictuga 91a084e5ed crawler: make py2/3 code distinction clearer 2018-10-25 01:14:46 +02:00
pictuga 5d93d68f62 readabilite: add some function descriptions 2018-10-25 01:12:42 +02:00
pictuga 8d7e1811fd readabilite: update lists 
Some code was also meant to be committed earlier
 2018-10-25 01:12:08 +02:00
pictuga 72d03f21fe readabilite: forgot count_content 
Was meant to be in an earlier commit
 2018-10-25 01:11:29 +02:00
pictuga 1d6d0b8ff1 readabilite: move br2p in the cleaning code 2018-10-25 01:09:15 +02:00
pictuga 7d005e9a65 readabilite: run the new cleaning code 2018-10-25 01:08:25 +02:00
pictuga 58fe5243af readabilite: improve cleaning code 2018-10-25 01:07:25 +02:00
pictuga f044c242ef readabilite: simplify scoring loop 
For perfomance
 2018-10-25 00:59:39 +02:00
pictuga a6befad136 readabilite: change scoring 2018-10-25 00:57:43 +02:00
pictuga 9e71de8d40 readabilite: improve output 2018-10-24 23:49:16 +02:00
pictuga 787d90fac0 readabilite: some technical improvements for score 
Linear, removed misplaced debugging code
 2018-10-24 23:47:37 +02:00
pictuga 040d2cb889 readabilite: improve word count 2018-10-23 00:09:34 +02:00
pictuga 9fcef826f5 reader: everything in one file 
Including css & js. Should now works by itself
 2018-10-22 23:55:14 +02:00
pictuga e72ca3f984 morss: improved output type 2018-09-30 22:02:29 +02:00
pictuga 2ccf36617a morss: improve http parameter parsing 2018-09-30 22:01:19 +02:00
pictuga 945e0dceab crawler: typo in comment 2018-09-30 21:59:50 +02:00
pictuga 5111d40011 feedify: update rules 
They obviously no longer worked after so long without updating them...
 2018-09-30 21:54:10 +02:00
pictuga f9217102f3 crawler: fix sqlite/binary issue 2017-11-25 19:58:14 +01:00
pictuga 21480f90de Move from gzip to zlib to decompress data 
Faster on incomplete files
 2017-11-25 19:57:41 +01:00
pictuga d091e74d56 crawler: add MySQL backend 
With extra dependency
 2017-11-04 14:51:41 +01:00
pictuga f29a107a09 crawler: make SQLiteCache inherit from BaseCache 
Saves some time for other cache backends
 2017-11-04 14:48:00 +01:00
pictuga 2d5bf7b38b Fix xml detection regex 
Also (dirtily) fixes #18 for now
 2017-11-04 14:21:05 +01:00
pictuga b7db78f631 crawler: use BLOB in sqlite and drop "buffer" 
Can't really remember why "buffer" was introduced in the first place
 2017-11-04 13:54:40 +01:00
pictuga 203ba10dbd Explain __init__.py and __main__.py use 2017-11-04 13:17:12 +01:00
pictuga 194465544a crawler: separate CacheHander and actual caching 
Default cache is now just an in-memory {}
 2017-11-04 12:41:56 +01:00
pictuga 523b250907 crawler: SQL request in CAPS for readability 2017-11-04 12:36:58 +01:00
pictuga 2d7d0fcdca morss: fix cgi in python 3 
Needs explicit [] in py3
 2017-11-04 12:27:47 +01:00
pictuga a8c2df7f41 crawler: fix truncated gzip reader 
For python 3
 2017-11-04 12:07:08 +01:00
pictuga d39d0f4cae crawler: properly define default sqlite file 2017-11-02 22:50:40 +01:00
pictuga f563040809 readabilite: threshold to detect if it contains an article 
Useful for videos/images-based images
 2017-10-28 01:30:21 +02:00
pictuga 0df6409b0e crawler: use `with con` to commit, journal WAL for perf 2017-10-28 01:28:47 +02:00
pictuga 7b85f692a0 crawler: fix encoding detection 2017-10-27 23:14:08 +02:00
pictuga 840842d246 crawler: limit download to 500KiB 
More can only be linked to a fraudulent/incorrect use of the service
 2017-10-27 23:12:40 +02:00
pictuga fbe811384a crawler: add (unused) DebugHandler to output headers sent/received 
Saves a lot of time when debugging
 2017-10-27 23:10:03 +02:00
pictuga 64babd6713 morss: make readabilite links absolute 2017-07-29 14:37:37 +02:00
pictuga 3bfad54add readabilite: change cleaning & code structure 
Kinda struggled to make some "nice" code
 2017-07-17 00:27:41 +02:00
pictuga 386bafd391 readabilite: write_all use "node" instead of "item" 2017-07-17 00:13:15 +02:00
pictuga a61b259792 readabilite: easy option to highlight the nodes 2017-07-17 00:11:49 +02:00
pictuga c52b47616d readabilite: always return common of 2 best nodes 
Better results. Less is not more
 2017-07-17 00:10:58 +02:00
pictuga bfdda18b9c readbilite: better explain lowest_common output 2017-07-17 00:08:00 +02:00