Commit Graph

885 Commits (04afa28fe7f8795a3ed1bda49daed28c2dfc1c5e)
 

Author SHA1 Message Date
pictuga 0ae0dbc175 README: mention csv output 2020-04-07 09:24:32 +02:00
pictuga f1d0431e68 morss: drop :html, replaced with :reader
README updated accordingly
2020-04-07 09:23:29 +02:00
pictuga a09831415f feeds: fix bug when mimetype matches nothing 2020-04-06 18:53:07 +02:00
pictuga bfad6b7a4a readabilite: clean before counting
To remove links which are not kept anyway
2020-04-06 16:55:39 +02:00
pictuga 6b8c3e51e7 readabilite: fix threshold feature
Awkward typo...
2020-04-06 16:52:06 +02:00
pictuga dc9e425247 readabilite: don't clean-out the top 10% nodes
Loosen up the code once again to limit over-kill
2020-04-06 14:26:28 +02:00
pictuga 2f48e18bb1 readabilite: put scores directly in html node
Probably slower but makes code somewhat cleaner...
2020-04-06 14:21:41 +02:00
pictuga 31cac921c7 README: remove ref to iTunes 2020-04-05 22:20:33 +02:00
pictuga a82ec96eb7 Delete feedify.py leftover code
iTunes integration untested, unreliable and not working...
2020-04-05 22:16:52 +02:00
pictuga aad2398e69 feeds: turns out lxml.etree doesn't have drop_tag 2020-04-05 21:50:38 +02:00
pictuga eeac630855 crawler: add more "realistic" headers 2020-04-05 21:11:57 +02:00
pictuga e136b0feb2 readabilite: loosen the slayer
Previous impl. lead to too many empty results
2020-04-05 20:47:30 +02:00
pictuga 6cf32af6c0 readabilite: also use BS 2020-04-05 20:46:42 +02:00
pictuga 568e7d7dd2 feeds: make BS's output bytes for lxml's sake 2020-04-05 20:46:04 +02:00
pictuga 3617f86e9d morss: make cgi_encore more robust 2020-04-05 16:43:11 +02:00
pictuga d90756b337 morss: drop 'keep' option
Because the Firefox behaviour it is working around is no longer in use
2020-04-05 16:37:27 +02:00
pictuga 40c69f17d2 feeds: parse html with BS
More robust & to make it consistent with :getpage
2020-04-05 16:12:41 +02:00
pictuga 99461ea185 crawler: fix var name issues (private_cache) 2020-04-05 16:11:36 +02:00
pictuga bf86c1e962 crawler: make AutoUA match http(s) type 2020-04-05 16:07:51 +02:00
pictuga d20f6237bd crawler: replace ContentNegoHandler with AlternateHandler
More basic. Sends the same headers no matter what. Make requests more "replicable".
Also, drop "text/xml" from RSS contenttype, too broad, matches garbage
2020-04-05 16:05:59 +02:00
pictuga 8a4d68d72c crawler: drop 'basic' toggle
Can't even remember the use case
2020-04-05 16:03:06 +02:00
pictuga e6811138fd morss: use redirected url in :getpage
Still have to find how to do the same thing with feeds...
2020-04-04 20:04:57 +02:00
pictuga 35b702fffd morss: default values for feed creation 2020-04-04 19:39:32 +02:00
pictuga 4a88886767 morss: get_page to act as a basic proxy (for iframes) 2020-04-04 16:37:15 +02:00
pictuga 1653394cf7 morss: cgi_dispatcher to be able to create extra functions 2020-04-04 16:35:16 +02:00
pictuga a8a90cf414 morss: move url/options parsing to own function
For future re-use
2020-04-04 16:33:52 +02:00
pictuga bdbaf0f8a7 morss/cgi: fix handling of special chars in url 2020-04-04 16:21:37 +02:00
pictuga d0e447a2a6 ItemFix: clean up Pocket links 2020-04-04 16:20:39 +02:00
pictuga e6817e01b4 sheet.xsl: set font to "sans"
Browsers don't all have the same default font. Overriding for consistency
2020-04-03 17:47:19 +02:00
pictuga 7c3091d64c morss: code spacing
One of those commits that make me feel useful
2020-03-21 23:41:46 +01:00
pictuga 37b4e144a9 morss: small fixes
Includes dropping off ftp support
2020-03-21 23:30:18 +01:00
pictuga bd4b7b5bb2 morss: convert HTML feeds to XML ones for completeness 2020-03-21 23:27:42 +01:00
pictuga 68d920d4b5 morss: make FeedFormat more flexible with encoding 2020-03-21 23:26:35 +01:00
pictuga 758ff404a8 morss: fix cgi_app silent output
*Must* return sth
2020-03-21 23:25:25 +01:00
pictuga 463530f02c morss: middleware to enforce encoding
bytes are always expected
2020-03-21 23:23:50 +01:00
pictuga ec0a28a91d morss: use middleware for wsgi apps 2020-03-21 23:23:21 +01:00
pictuga 421acb439d morss: make errors more readable over http 2020-03-21 23:08:29 +01:00
pictuga 42c5d09ccb morss: split "options" var into "raw_options" & "options"
To make it clearer who-is-what
2020-03-21 23:07:07 +01:00
pictuga 056de12484 morss: add sheet.xsl to file handled by http server 2020-03-21 23:06:28 +01:00
pictuga 961a31141f morss: fix url fixing 2020-03-21 17:28:00 +01:00
pictuga a7b01ee85e readabilite: further html processing instructions fix 2020-03-21 17:23:50 +01:00
pictuga fbcb23cf88 index.Html: fix meta tag 2020-03-20 20:21:08 +01:00
pictuga a0e8e84a67 sheet.xsl: mobile-friendly view & url fix 2020-03-20 19:47:43 +01:00
pictuga a90fd682db sheet.xsl: show feed url (js-based) 2020-03-20 19:46:20 +01:00
pictuga 2c245f9506 sheet.xsl: improve output formatting
Include tags, better CSS
2020-03-20 16:44:49 +01:00
pictuga 3d45451fef sheet.xsl: improve element content output 2020-03-20 15:44:25 +01:00
pictuga 4d785820d9 feeds: ignore provided stylesheets and add ours
Provided sheets usually create errors. Ours is (hopefully) more informative for users not familiar with RSS feeds
2020-03-20 15:32:44 +01:00
pictuga 6a01fc439e feeds: better handle "empty" datetime 2020-03-20 12:30:42 +01:00
pictuga d24734110a morss: convert all feeds to RSS
As html feeds might not contain some feeds, leading to data loss
2020-03-20 12:26:34 +01:00
pictuga a41c2a3a62 morss: fix twitter link detection 2020-03-20 12:26:19 +01:00