pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity
1,046 Commits
1 Branch
2 Tags
3.5 MiB
Commit Graph

581 Commits (master)

Author SHA1 Message Date
pictuga a09831415f feeds: fix bug when mimetype matches nothing 2020-04-06 18:53:07 +02:00
pictuga bfad6b7a4a readabilite: clean before counting 
To remove links which are not kept anyway
 2020-04-06 16:55:39 +02:00
pictuga 6b8c3e51e7 readabilite: fix threshold feature 
Awkward typo...
 2020-04-06 16:52:06 +02:00
pictuga dc9e425247 readabilite: don't clean-out the top 10% nodes 
Loosen up the code once again to limit over-kill
 2020-04-06 14:26:28 +02:00
pictuga 2f48e18bb1 readabilite: put scores directly in html node 
Probably slower but makes code somewhat cleaner...
 2020-04-06 14:21:41 +02:00
pictuga a82ec96eb7 Delete feedify.py leftover code 
iTunes integration untested, unreliable and not working...
 2020-04-05 22:16:52 +02:00
pictuga aad2398e69 feeds: turns out lxml.etree doesn't have drop_tag 2020-04-05 21:50:38 +02:00
pictuga eeac630855 crawler: add more "realistic" headers 2020-04-05 21:11:57 +02:00
pictuga e136b0feb2 readabilite: loosen the slayer 
Previous impl. lead to too many empty results
 2020-04-05 20:47:30 +02:00
pictuga 6cf32af6c0 readabilite: also use BS 2020-04-05 20:46:42 +02:00
pictuga 568e7d7dd2 feeds: make BS's output bytes for lxml's sake 2020-04-05 20:46:04 +02:00
pictuga 3617f86e9d morss: make cgi_encore more robust 2020-04-05 16:43:11 +02:00
pictuga d90756b337 morss: drop 'keep' option 
Because the Firefox behaviour it is working around is no longer in use
 2020-04-05 16:37:27 +02:00
pictuga 40c69f17d2 feeds: parse html with BS 
More robust & to make it consistent with :getpage
 2020-04-05 16:12:41 +02:00
pictuga 99461ea185 crawler: fix var name issues (private_cache) 2020-04-05 16:11:36 +02:00
pictuga bf86c1e962 crawler: make AutoUA match http(s) type 2020-04-05 16:07:51 +02:00
pictuga d20f6237bd crawler: replace ContentNegoHandler with AlternateHandler 
More basic. Sends the same headers no matter what. Make requests more "replicable".
Also, drop "text/xml" from RSS contenttype, too broad, matches garbage
 2020-04-05 16:05:59 +02:00
pictuga 8a4d68d72c crawler: drop 'basic' toggle 
Can't even remember the use case
 2020-04-05 16:03:06 +02:00
pictuga e6811138fd morss: use redirected url in :getpage 
Still have to find how to do the same thing with feeds...
 2020-04-04 20:04:57 +02:00
pictuga 35b702fffd morss: default values for feed creation 2020-04-04 19:39:32 +02:00
pictuga 4a88886767 morss: get_page to act as a basic proxy (for iframes) 2020-04-04 16:37:15 +02:00
pictuga 1653394cf7 morss: cgi_dispatcher to be able to create extra functions 2020-04-04 16:35:16 +02:00
pictuga a8a90cf414 morss: move url/options parsing to own function 
For future re-use
 2020-04-04 16:33:52 +02:00
pictuga bdbaf0f8a7 morss/cgi: fix handling of special chars in url 2020-04-04 16:21:37 +02:00
pictuga d0e447a2a6 ItemFix: clean up Pocket links 2020-04-04 16:20:39 +02:00
pictuga 7c3091d64c morss: code spacing 
One of those commits that make me feel useful
 2020-03-21 23:41:46 +01:00
pictuga 37b4e144a9 morss: small fixes 
Includes dropping off ftp support
 2020-03-21 23:30:18 +01:00
pictuga bd4b7b5bb2 morss: convert HTML feeds to XML ones for completeness 2020-03-21 23:27:42 +01:00
pictuga 68d920d4b5 morss: make FeedFormat more flexible with encoding 2020-03-21 23:26:35 +01:00
pictuga 758ff404a8 morss: fix cgi_app silent output 
*Must* return sth
 2020-03-21 23:25:25 +01:00
pictuga 463530f02c morss: middleware to enforce encoding 
bytes are always expected
 2020-03-21 23:23:50 +01:00
pictuga ec0a28a91d morss: use middleware for wsgi apps 2020-03-21 23:23:21 +01:00
pictuga 421acb439d morss: make errors more readable over http 2020-03-21 23:08:29 +01:00
pictuga 42c5d09ccb morss: split "options" var into "raw_options" & "options" 
To make it clearer who-is-what
 2020-03-21 23:07:07 +01:00
pictuga 056de12484 morss: add sheet.xsl to file handled by http server 2020-03-21 23:06:28 +01:00
pictuga 961a31141f morss: fix url fixing 2020-03-21 17:28:00 +01:00
pictuga a7b01ee85e readabilite: further html processing instructions fix 2020-03-21 17:23:50 +01:00
pictuga 4d785820d9 feeds: ignore provided stylesheets and add ours 
Provided sheets usually create errors. Ours is (hopefully) more informative for users not familiar with RSS feeds
 2020-03-20 15:32:44 +01:00
pictuga 6a01fc439e feeds: better handle "empty" datetime 2020-03-20 12:30:42 +01:00
pictuga d24734110a morss: convert all feeds to RSS 
As html feeds might not contain some feeds, leading to data loss
 2020-03-20 12:26:34 +01:00
pictuga a41c2a3a62 morss: fix twitter link detection 2020-03-20 12:26:19 +01:00
pictuga dd2651061f feeds & morss: clean up comments/empty lines 2020-03-20 12:25:48 +01:00
pictuga 912c323c40 feeds: make function output more consistent 
e.g. setters return nothing, getters return sth relevant or None (i.e. no empty strings)
 2020-03-20 12:23:15 +01:00
pictuga 5705a0be17 feeds: fix delete/rmv code 2020-03-20 12:22:07 +01:00
pictuga 4735ffba45 feeds: fix .convert auto-convert 
To fix inheritance loophole
 2020-03-20 12:20:41 +01:00
pictuga 08e39f5631 feeds: give simpler name to helper functions 2020-03-20 12:20:15 +01:00
pictuga 765a43511e feeds: remove unused import 2020-03-20 12:19:08 +01:00
pictuga 5865af64f9 Fix indent output for html/xml 2020-03-20 12:18:13 +01:00
pictuga b3b90c067a morss.py: remove "useless" functions 
Have to keep the code clean
 2020-03-20 11:19:06 +01:00
pictuga 5288cc8796 Clean up unused import's 2020-03-19 15:09:53 +01:00
pictuga d12d44a500 Remove unused hash-bangs 
Leftovers from debugging
 2020-03-19 15:06:28 +01:00
pictuga ee8c57c1fc feeds: avoid convert to self 2020-03-19 12:54:04 +01:00
pictuga bda51b0fc7 feeds & morss: many encoding/tostring fixes 2020-03-19 12:53:25 +01:00
pictuga c09b457168 feeds: fix .dic code 
Bug introduced recently…
 2020-03-19 11:36:20 +01:00
pictuga b47e40246c feeds: clean up html code handling 2020-03-19 11:35:51 +01:00
pictuga 9cf933723f feeds: clean up time handling 
Includes a shameful fix on @property
 2020-03-19 11:35:02 +01:00
pictuga d26795dce8 morss: from feedify to feeds 
Also scrap obsolete feedify code
 2020-03-19 10:27:44 +01:00
pictuga 2704e91a3d readabilite: handle another weird html stuff 2020-03-19 10:24:09 +01:00
pictuga f48961a7e4 feeds: small code cleanups 2020-03-19 10:13:22 +01:00
pictuga fd51f74eb5 feeds: drop facebook-related code 
No longer maintained and of little use anyway
 2020-03-19 10:03:59 +01:00
pictuga d3fe51cea5 feeds: remove duplicated code 2020-03-19 09:49:52 +01:00
pictuga 449bc3c695 feeds: fix handling of html code 2020-03-19 09:48:53 +01:00
pictuga 13ea52ef80 feeds: add .torss() 2020-03-19 09:47:58 +01:00
pictuga aa2b56b266 feeds: small code cleanups 2020-03-19 09:47:17 +01:00
pictuga 296e9a6c13 feedifiy.ini: clean up following feeds.py's html intro 2020-03-18 16:52:28 +01:00
pictuga 9dbe061fd6 Remove markdown-related code 
Time to clean up the code and stop with those non-core features
They just make the code harder to maintain
 2020-03-18 16:47:00 +01:00
pictuga 4a70aa9dfa feeds: auto-parse() 2020-03-18 16:34:40 +01:00
pictuga c2f85da94a feeds: add html support, adapt .tohtml() 2020-03-18 16:33:10 +01:00
pictuga e3528a8f36 feeds: use FeddJSON for .tojson() 
Clean up related code
 2020-03-18 16:31:36 +01:00
pictuga 2dd9ae202d feeds: add 'mode' to Parsers 2020-03-18 16:24:08 +01:00
pictuga 8e3d32f24c feds: rename 'ruleset' into 'default_ruleset' 
Better reflects its use
 2020-03-18 16:22:03 +01:00
pictuga 6ce616106b feeds: disable 'multi' ruleset 
RSS ruleset has been cut into 4 rulesets
 2020-03-18 16:20:42 +01:00
pictuga 186fa2b408 feedify: remove empty 'path' lines 2020-03-18 16:18:22 +01:00
pictuga e9d46cb6a9 feeds: move mimetypes into .py from .ini 2020-03-18 16:08:42 +01:00
pictuga 7644c550ec feedify: remove id, is_permalink as well 2020-03-17 16:48:36 +01:00
pictuga e460fdc8f4 feeds: remove little-used properties 
id, is_permalink
Also a good opportunity to trash the unreliable bool-parsing code
 2020-03-17 16:46:42 +01:00
pictuga 1e714ab34b feeds: add ability to convert to another typo of feed 2020-03-17 14:02:24 +01:00
pictuga 4ba4d73ce6 feeds: add json support 2020-03-17 14:02:01 +01:00
pictuga 6d5aa8c222 feeds: clean up .append() 2020-03-17 13:59:51 +01:00
pictuga f10727f94a feeds: small code cleanup's 2020-03-17 12:26:34 +01:00
pictuga d42e19a165 feeds: beter rules handling 
"Dynamic" rule set picking, better handling of non-multi rules
 2020-03-17 12:23:36 +01:00
pictuga fe46c6c522 feeds: pass parent Feed to Items 2020-03-17 12:22:14 +01:00
pictuga 9c557ea02c feeds: fix function def 2020-03-17 11:08:40 +01:00
pictuga 8a4f86210c feedify.ini: remove utf-8 declaration 
Screws up with the parser as it is read as unicode (and xml parser expects bytes)
 2020-03-17 11:06:59 +01:00
pictuga ce30952fa2 feeds: make "rule" split clearer 
"rrule" var name to tell appart the original "rule" from the parsed one
 2020-03-16 17:46:04 +01:00
pictuga 3fb6ff891c feeds: share more code, add comments 
Should reduce redundancy
 2020-03-16 17:45:08 +01:00
pictuga f5acd2c14c feeds: use RawConfigParser 
This one does not try to replace non-std characters (e.g. %)
 2020-03-16 17:43:03 +01:00
pictuga 7cb3b29ef2 feeds: remove unused import 2020-03-16 17:38:48 +01:00
pictuga 9cb2d5bb86 feeds: centralize time format/parse 
At the same code _should_ apply to most, if not all, parsers
 2018-11-18 16:03:02 +01:00
pictuga e606c5eefb feeds: various small cleanup/fixes 2018-11-18 15:14:38 +01:00
pictuga 24c8a0ecd0 feeds: fix typo 2018-11-13 21:23:24 +01:00
pictuga 9a62e6ae75 feeds: remove old code 2018-11-13 21:22:50 +01:00
pictuga adbaed9e54 feeds: put code tgt 2018-11-11 17:24:56 +01:00
pictuga 3581f34db7 Various feeds.py related fixes 2018-11-11 16:46:23 +01:00
pictuga 966559bdd3 feeds: fix remove function in case of no match 2018-11-11 16:33:36 +01:00
pictuga 4fb98bc2ed feeds: fix append content 2018-11-11 16:33:18 +01:00
pictuga 679628c7fa Small code clean up 2018-11-11 16:11:00 +01:00
pictuga 399e867c94 morss: add py2 indication 2018-11-11 16:07:25 +01:00
pictuga c5d8b064ae feeds: fix an error when no match 2018-11-11 15:31:46 +01:00
pictuga c2a6ea7cfe feeds: give example of regex 2018-11-11 15:26:46 +01:00
pictuga 221e1f85ad feeds: fix implementation in morss 2018-11-11 15:26:09 +01:00
pictuga 857bb9c366 feeds: fix remove() unclear function naming 2018-11-11 15:25:03 +01:00
pictuga 75f691b009 feeds: fix multi rules parsing 2018-11-11 15:21:43 +01:00
pictuga 401dfbc1ff feeds: fix atom xhtml handling 2018-11-11 15:21:06 +01:00
pictuga 8aceda4957 feeds: fix feedify.ini 2018-11-11 15:19:41 +01:00
pictuga 024466733c feeds: remove old code 2018-11-09 22:09:59 +01:00
pictuga 92b06bea6d feeds: fix Uniq for merger 2018-11-09 22:05:13 +01:00
pictuga 94372af868 feeds: transitional code for json/csv/html export 2018-11-09 22:04:46 +01:00
pictuga 6d28323e3a feeds: add XML support for merger 2018-11-09 22:04:08 +01:00
pictuga 5a4a86d622 feeds: add base classes for merger 2018-11-09 22:02:44 +01:00
pictuga d321550166 feeds: prepare feedify.ini for merger 2018-11-09 21:53:19 +01:00
pictuga d1aab99b80 feeds: replacement code for descriptors 2018-10-31 22:15:34 +01:00
pictuga 16f3ffa96e feeds: remove further Descriptor code 2018-10-31 22:15:15 +01:00
pictuga 02b7e07097 feeds: fix typo 2018-10-31 22:07:49 +01:00
pictuga 8487a43c6c feeds: remove FeedList(Descriptor) 2018-10-31 22:07:16 +01:00
pictuga 081d560bc4 feeds: create obj to keep FeedItems unique 2018-10-31 21:47:19 +01:00
pictuga cfd758b6b5 feeds: shift easy ones to @property 2018-10-26 19:48:39 +02:00
pictuga 4e144487db Test for feedify support first 
Otherwise might never be called if the content-type is also supported
 2018-10-25 01:17:24 +02:00
pictuga d13362c4ac feeds: drop .iterchildren 
Redundant
 2018-10-25 01:16:28 +02:00
pictuga 17856929fe feeds: pretty_print was made a default 2018-10-25 01:16:07 +02:00
pictuga 90110a4661 crawler: reduce max file size 2018-10-25 01:15:09 +02:00
pictuga 91a084e5ed crawler: make py2/3 code distinction clearer 2018-10-25 01:14:46 +02:00
pictuga 5d93d68f62 readabilite: add some function descriptions 2018-10-25 01:12:42 +02:00
pictuga 8d7e1811fd readabilite: update lists 
Some code was also meant to be committed earlier
 2018-10-25 01:12:08 +02:00
pictuga 72d03f21fe readabilite: forgot count_content 
Was meant to be in an earlier commit
 2018-10-25 01:11:29 +02:00
pictuga 1d6d0b8ff1 readabilite: move br2p in the cleaning code 2018-10-25 01:09:15 +02:00
pictuga 7d005e9a65 readabilite: run the new cleaning code 2018-10-25 01:08:25 +02:00
pictuga 58fe5243af readabilite: improve cleaning code 2018-10-25 01:07:25 +02:00
pictuga f044c242ef readabilite: simplify scoring loop 
For perfomance
 2018-10-25 00:59:39 +02:00
pictuga a6befad136 readabilite: change scoring 2018-10-25 00:57:43 +02:00
pictuga 9e71de8d40 readabilite: improve output 2018-10-24 23:49:16 +02:00
pictuga 787d90fac0 readabilite: some technical improvements for score 
Linear, removed misplaced debugging code
 2018-10-24 23:47:37 +02:00
pictuga 040d2cb889 readabilite: improve word count 2018-10-23 00:09:34 +02:00
pictuga 9fcef826f5 reader: everything in one file 
Including css & js. Should now works by itself
 2018-10-22 23:55:14 +02:00
pictuga e72ca3f984 morss: improved output type 2018-09-30 22:02:29 +02:00
pictuga 2ccf36617a morss: improve http parameter parsing 2018-09-30 22:01:19 +02:00
pictuga 945e0dceab crawler: typo in comment 2018-09-30 21:59:50 +02:00
pictuga 5111d40011 feedify: update rules 
They obviously no longer worked after so long without updating them...
 2018-09-30 21:54:10 +02:00
pictuga f9217102f3 crawler: fix sqlite/binary issue 2017-11-25 19:58:14 +01:00
pictuga 21480f90de Move from gzip to zlib to decompress data 
Faster on incomplete files
 2017-11-25 19:57:41 +01:00
pictuga d091e74d56 crawler: add MySQL backend 
With extra dependency
 2017-11-04 14:51:41 +01:00
pictuga f29a107a09 crawler: make SQLiteCache inherit from BaseCache 
Saves some time for other cache backends
 2017-11-04 14:48:00 +01:00
pictuga 2d5bf7b38b Fix xml detection regex 
Also (dirtily) fixes #18 for now
 2017-11-04 14:21:05 +01:00
pictuga b7db78f631 crawler: use BLOB in sqlite and drop "buffer" 
Can't really remember why "buffer" was introduced in the first place
 2017-11-04 13:54:40 +01:00
pictuga 203ba10dbd Explain __init__.py and __main__.py use 2017-11-04 13:17:12 +01:00
pictuga 194465544a crawler: separate CacheHander and actual caching 
Default cache is now just an in-memory {}
 2017-11-04 12:41:56 +01:00
pictuga 523b250907 crawler: SQL request in CAPS for readability 2017-11-04 12:36:58 +01:00
pictuga 2d7d0fcdca morss: fix cgi in python 3 
Needs explicit [] in py3
 2017-11-04 12:27:47 +01:00
pictuga a8c2df7f41 crawler: fix truncated gzip reader 
For python 3
 2017-11-04 12:07:08 +01:00
pictuga d39d0f4cae crawler: properly define default sqlite file 2017-11-02 22:50:40 +01:00
pictuga f563040809 readabilite: threshold to detect if it contains an article 
Useful for videos/images-based images
 2017-10-28 01:30:21 +02:00
pictuga 0df6409b0e crawler: use `with con` to commit, journal WAL for perf 2017-10-28 01:28:47 +02:00
pictuga 7b85f692a0 crawler: fix encoding detection 2017-10-27 23:14:08 +02:00
pictuga 840842d246 crawler: limit download to 500KiB 
More can only be linked to a fraudulent/incorrect use of the service
 2017-10-27 23:12:40 +02:00
pictuga fbe811384a crawler: add (unused) DebugHandler to output headers sent/received 
Saves a lot of time when debugging
 2017-10-27 23:10:03 +02:00
pictuga 64babd6713 morss: make readabilite links absolute 2017-07-29 14:37:37 +02:00
pictuga 3bfad54add readabilite: change cleaning & code structure 
Kinda struggled to make some "nice" code
 2017-07-17 00:27:41 +02:00
pictuga 386bafd391 readabilite: write_all use "node" instead of "item" 2017-07-17 00:13:15 +02:00
pictuga a61b259792 readabilite: easy option to highlight the nodes 2017-07-17 00:11:49 +02:00
pictuga c52b47616d readabilite: always return common of 2 best nodes 
Better results. Less is not more
 2017-07-17 00:10:58 +02:00
pictuga bfdda18b9c readbilite: better explain lowest_common output 2017-07-17 00:08:00 +02:00
pictuga 2afea497a3 readabilite: br2p use "node" instead of "item" 
Confusing with rss items otherwise
 2017-07-17 00:06:39 +02:00
pictuga 843dc97fbf readabilite: change scoring algorithm 
Use 3 groups of keywords instead
 2017-07-17 00:01:44 +02:00
pictuga df22396838 Only use chardet on 2k letters 
Takes forever otherwise
 2017-07-16 23:59:06 +02:00
pictuga 6f0efd5802 crawler: add cookies support 
Somehow got dropped when splitting the big handler
 2017-03-25 19:51:42 -10:00
pictuga d3bc2926fc Remove :hungry 
Mostly usless. If you need it, you might as well not need to use morss in the first place...
 2017-03-25 13:52:58 -10:00
pictuga 505b02d70d crawler: remove debugging print() 2017-03-25 13:45:12 -10:00
pictuga 3ca6ed5bb0 readabilite: add author/about to black list 2017-03-24 22:02:41 -10:00
pictuga 4aa25bf3d8 readabilite: clean_html before scoring 
Surprisingly efficient
 2017-03-24 21:50:46 -10:00
pictuga bfefa8d599 readabilite: add tags to black list 2017-03-24 21:50:26 -10:00
pictuga 91da0f36dc readabilite: comment the clean_html function 2017-03-24 21:50:01 -10:00
pictuga 67889a1d14 readabilite: drop useless tags 
This extra cluster actually jams the algorithm
 2017-03-24 21:49:14 -10:00
pictuga 167e3e4a15 feedify: accept xpath rules passed as parameters 2017-03-20 20:56:48 -10:00
pictuga bf3ef586c2 feedify: remove unused downloader 2017-03-20 20:53:52 -10:00
pictuga 08f08ef704 improve morss url detection regex 2017-03-20 20:51:13 -10:00
pictuga 1b4341f741 accept query_string in morss cgi 2017-03-20 20:50:04 -10:00
pictuga f965566054 feedify; make function use clearer 2017-03-20 20:19:08 -10:00
pictuga d6882e0a6a readabilite: (try to) emprove detection 
Kinda hopeless
 2017-03-19 02:00:31 -10:00
pictuga 79a8ada9f4 readabilite: add tags to score 2017-03-19 01:57:54 -10:00
pictuga 4a5150e030 readabilite: fix iter while iterating 2017-03-19 01:56:33 -10:00
pictuga e65c88abf8 readabilite: fix re.match 2017-03-19 01:55:40 -10:00
pictuga 9c331300eb crawler: move UAHandler to basic 
Fuck u feedburner
 2017-03-19 01:49:17 -10:00
pictuga 5e61686373 Only use full feed for articles & feedify 
Sometimes using referrer and/or useragent makes some dumb websites return diferent content (hello feedburner)
 2017-03-18 23:43:28 -10:00
pictuga 0b6e553054 Move iTunes code to feedify.py 2017-03-18 23:41:37 -10:00
pictuga d4937812a8 Remove HTTPError code 
Only used to look nice but useless (inherits from IOError anyway)
 2017-03-18 23:39:32 -10:00
pictuga 99f3c519f2 crawler: fix accept code 2017-03-18 23:37:51 -10:00
pictuga 67f5a21019 Move build_opener to crawler 
Forgotten
 2017-03-18 23:03:04 -10:00
pictuga f7d570d4c8 crawler: add some broken as rss mimetype 
Seen out there
 2017-03-18 23:00:13 -10:00
pictuga 2003e2760b Move custom_handler to crawler 
Makes more sense. Easier to reuse. Also cleaned up a bit the code
 2017-03-18 22:51:27 -10:00
pictuga e1a13a623c crawler: remove unefficient feedburner-specific code 2017-03-18 22:31:03 -10:00
pictuga 367f86987d readabilite: spread score to all ancestors 
Instead of just parents and grandparents
 2017-03-18 22:24:38 -10:00
pictuga e3ab3c6823 crawler: use less tertiary operator 
Inherited from fork
 2017-03-18 22:23:39 -10:00
pictuga 65055290d4 crawler: better use of chardet 
Scan whole doc since beginning of html pages tends to be too regular. Ignore ASCII detection for the same reason.
 2017-03-18 22:19:54 -10:00
pictuga 9ee6ff60e1 crawler: 301 http code doesn't respect headers 
More or less according to the specs
 2017-03-18 22:18:10 -10:00
pictuga f4abc4e8a4 Detect encoding (using crawler) before readabilite 2017-03-11 02:30:57 -10:00
pictuga c952b85d92 crawler: cache 301 HTTP code, for a week 2017-03-09 09:37:05 -10:00
pictuga e8023e4336 crawler: remove unused NotInCache error-class 2017-03-09 09:35:40 -10:00
pictuga 385f9eb39a morss: use crawler strict accept for feed 2017-03-08 19:05:48 -10:00
Florian Muenchbach 993ac638a3 Added override for auto-detected character encoding of parsed pages. 2017-03-08 18:45:20 -10:00
pictuga 627163abff Make cache settings in morss nicer 2017-03-08 18:09:24 -10:00
pictuga e5f8e43659 Shifted the <link rel='alternate'/> redirect to crawler 
Now using MIMETYPE var from crawler within morss.py
 2017-03-08 18:03:34 -10:00
pictuga fb8825b410 crawler: parse html to get http-equiv 
For sure slower, but way cleaner (and probably more stable)
 2017-03-08 17:50:57 -10:00
pictuga f4f6a86147 feeds: make wheezy.template mandatory 
Cleaner code. Less confusing.
 2017-03-08 15:38:59 -10:00
pictuga ad9bf946ec crawler: use chardet again 
Always nice in case no encoding is specified. Somehow got dropped with commit 245ba99. Most probably by accident
 2017-03-08 11:37:12 -10:00
pictuga 3fc89d5359 readabilite: improve score for <p> 
Helps a lot with bbc, le monde. Might backfire on other websites tho...
 2017-03-01 18:02:45 -10:00
pictuga a8ac2ed1ca Turn FeedBefore/After into ItemBefore/After 
To reduce the number of loops
 2017-02-28 23:24:32 -10:00
pictuga fcc5e8a076 Add "Feed/Item" in functions name 
To make it instantly clearer what they work on
 2017-02-28 23:23:15 -10:00
pictuga 60e3311e97 Use readabilite properly 
Not thru some weird wrapper anymore
 2017-02-28 22:45:26 -10:00
pictuga dc8423550f Support xml starting with \s 2017-02-25 19:04:32 -10:00
pictuga e0f533ca31 readabilite: test to replace <br/> with div 2017-02-25 18:16:15 -10:00
pictuga c6c113b8a8 readabilite: function to clean up the html code 2017-02-25 18:15:33 -10:00
pictuga 58d9f65735 readabilite: explain the use of .tail 2017-02-25 18:14:13 -10:00
pictuga a5aec8c7a6 readability: more keywords to the filter list 
Also fixed indentation
 2017-02-25 18:13:15 -10:00
pictuga 026903ce73 crawler: change http header after uncompressing 
Change content-encoding to "identity"
 2017-02-25 18:10:43 -10:00
pictuga e71fc967ce readabilite: shift "good" tags to a var (list) 
So that this list can later be re-used
 2017-02-25 18:07:28 -10:00
pictuga b14381f575 Use internal readability fork 
Much simpler, doesn't clean the html, probably less efficient, but much faster
 2016-05-31 02:50:03 +02:00
pictuga 2b9bfb47e5 Remove :smart and etag headers 
Dirty code, not very useful. Use simple cache-control instead.
 2016-05-31 02:47:49 +02:00
pictuga 4ff80cec86 Check argv length before using it 2016-05-31 02:46:28 +02:00
pictuga 466d8e47d6 Also make buriy's readability port compatible 
Should be faster, and it now supports py3
 2015-08-29 18:33:12 +02:00
pictuga 95d9d847e9 :proxy implies :keep 2015-08-29 17:48:07 +02:00
pictuga 8a1c00abf0 Typo in python version check 2015-08-28 19:29:09 +02:00
pictuga 624fa47f4f Allow CLI change of the www/ path 2015-08-28 19:22:55 +02:00
pictuga 31fc939d52 Allow CLI change of the http server port 2015-08-28 19:22:23 +02:00
pictuga 4f9000beed Comment code of launching modes 2015-08-28 19:18:09 +02:00
pictuga 5e87b56a03 Return error code in plain text in file server 2015-08-28 19:16:15 +02:00
pictuga ffda3fac7e Improve file detection in web server 2015-08-28 19:15:40 +02:00
pictuga 6741a408dd Remove now-useless ca-cert file path 2015-08-28 19:13:54 +02:00
Massimo Vannucci 8656e53b84 Correct Python version check 2015-08-05 23:36:11 +02:00
Massimo Vannucci 098a306c91 Fixed typo 2015-08-05 23:24:44 +02:00
pictuga 5c2151ffd6 Improve widely feedsportal url decoder 2015-06-14 20:32:47 +08:00
pictuga 8418212475 Use good path for html template access 2015-05-04 22:26:31 +08:00
pictuga 931fd53da6 Fix 304-cache handling 
To make sure that the cached request also gets processed (by GZip and stuff)
 2015-05-04 22:25:26 +08:00
pictuga ae062ebe90 Remove deprecated https error catch 2015-04-07 18:59:37 +08:00
pictuga 7a3b257328 Make :mono use basic loop 
Makes profiling easier
 2015-04-07 18:16:08 +08:00
pictuga 2f86a2a44b Remove useless obscure cgi code 2015-04-07 09:49:44 +08:00
pictuga 131ba09207 Change :cache mode behavior 
Makes underlying code way cleaner
 2015-04-07 09:38:22 +08:00
pictuga cafb87d561 Fix sqlite relative path in cgi 2015-04-07 09:37:25 +08:00
pictuga decb3f15f6 Move the mod_cgi files to /cgi/ 2015-04-07 09:36:00 +08:00
pictuga b267791199 Remove hashbang from __init__.py 2015-04-07 09:34:22 +08:00
pictuga acae47dc79 2to3: fix cli_app string print 2015-04-06 23:27:15 +08:00
pictuga 32aa96afa7 Cache HTTP content using a custom Handler 
Much much cleaner. Nothing comparable
 2015-04-06 23:26:12 +08:00
pictuga 006478d451 2to3: fix feeds.py string handling 
Use bytes strings
 2015-04-06 23:13:46 +08:00
pictuga a35225a234 2to3: fix feedify string handling 2015-04-06 23:12:50 +08:00
pictuga 1b4fc88ad0 Replace MetaRedirect handler with two cleaner ones 
One for <meta http-equiv> and one for HTTP 'refresh' header
 2015-04-06 23:03:17 +08:00
pictuga f2fe4fc364 Drop HTTPS SSL certificate verification 
Breaks everything with python 3. Now built-in in recent python 2.7.9 and python 3.4-ish
 2015-04-06 22:54:59 +08:00
pictuga 88af80e817 feeds: no need to decode xml strings 
It event makes python3 lxml get angry
 2015-04-06 22:37:33 +08:00
pictuga 1335b3fdda feedify: use better relative path for the .ini 2015-04-06 22:19:13 +08:00
pictuga c41c0761b6 feedify: don't insert useless url when none is found 2015-04-06 22:15:59 +08:00
pictuga dbc92068f0 feedify: explanation of methods' purpose 
Kinda messy when reading code after a year
 2015-04-06 22:11:31 +08:00
pictuga 9d64c31947 Feeds: use crawler.py encoding detection 2015-03-24 23:23:40 +08:00