Commit Graph

977 Commits (7cdcbd23e1d80c45b979875b3f7db90a1616b58b)
 

Author SHA1 Message Date
pictuga 6a0531ca03 crawler: randomize user agent 2020-04-24 11:28:39 +02:00
pictuga 8187876a06 crawler: stop at first alternative link
Should save a few ms and the first one is usually (?) the most relevant/generic
2020-04-23 11:23:45 +02:00
pictuga 325a373e3e feeds: add SyntaxError catch 2020-04-20 16:15:15 +02:00
pictuga 2719bd6776 crawler: fix chinese encoding 2020-04-20 16:14:55 +02:00
pictuga 285e1e5f42 docker: pip install local 2020-04-19 13:25:53 +02:00
pictuga 41a63900c2 README: improve docker instructions 2020-04-19 13:01:08 +02:00
pictuga ec8edb02f1 Various small bug fixes 2020-04-19 12:54:02 +02:00
pictuga d01b943597 Remove leftover threading var 2020-04-19 12:51:11 +02:00
pictuga b361aa2867 Add timeout to :get 2020-04-19 12:50:26 +02:00
pictuga 4ce3c7cb32 Small code clean ups 2020-04-19 12:50:05 +02:00
pictuga 7e45b2611d Disable multi-threading
Impact was mostly negative due to locks
2020-04-19 12:29:52 +02:00
pictuga 036e5190f1 crawler: remove unused code 2020-04-18 21:40:02 +02:00
pictuga e99c5b3b71 morss: more sensible default MAX/LIM values 2020-04-18 17:21:45 +02:00
pictuga 4f44df8d63 Make all ports default to 8080 2020-04-18 17:15:59 +02:00
pictuga 497c14db81 Add dockerfile & how to in README 2020-04-18 17:04:44 +02:00
pictuga a4e1dba8b7 sheet.xsl: improve url display 2020-04-16 10:33:36 +02:00
pictuga 7375adce33 sheet.xsl: fix & improve 2020-04-15 23:34:28 +02:00
pictuga 663212de0a sheet.xsl: various cosmetic improvements 2020-04-15 23:22:45 +02:00
pictuga 4a2ea1bce9 README: add gunicorn instructions 2020-04-15 22:31:21 +02:00
pictuga fe82b19c91 Merge .xsl & html template
Turns out they somehow serve a similar purpose
2020-04-15 22:30:45 +02:00
pictuga 0b31e97492 morss: remove debug code in http file handler 2020-04-14 23:20:03 +02:00
pictuga b0ad7c259d Add README & LICENSE to data_files 2020-04-14 19:34:12 +02:00
pictuga bffb23f884 README: how to use cli 2020-04-14 18:21:32 +02:00
pictuga 59139272fd Auto-detect the location of www/
Either ../www or /usr/share/morss
Adapted README accordingly
2020-04-14 18:07:19 +02:00
pictuga 39b0a1d7cc setup.py: fix deps & files 2020-04-14 17:36:42 +02:00
pictuga 65803b328d New git url and updated date in provided index.html 2020-04-13 15:30:32 +02:00
pictuga e6b7c0eb33 Fix app definition for uwsgi 2020-04-13 15:30:09 +02:00
pictuga 67c096ad5b feeds: add fake path to default html parser
Without it, some websites were accidentally matching it (false positives)
2020-04-12 13:00:56 +02:00
pictuga f018437544 crawler: make mysql backend thread safe 2020-04-12 12:53:05 +02:00
pictuga 8e5e8d24a4 Timezone fixes 2020-04-10 20:33:59 +02:00
pictuga ee78a7875a morss: focus on the most recent feed items 2020-04-10 16:08:13 +02:00
pictuga 9e7b9d95ee feeds: properly use html template 2020-04-09 20:00:51 +02:00
pictuga 987a719c4e feeds: try all parsers regardless of contenttype
Turns out some websites send the wrong contenttype (json for html, html for xml, etc.)
2020-04-09 19:17:51 +02:00
pictuga 47b33f4baa morss: specify server output encoding 2020-04-09 19:10:45 +02:00
pictuga 3c7f512583 feeds: handle several errors 2020-04-09 19:09:10 +02:00
pictuga a32f5a8536 readabilite: add debug option (also used by :get) 2020-04-09 19:08:13 +02:00
pictuga 63a06524b7 morss: various encoding fixes 2020-04-09 19:06:51 +02:00
pictuga b0f80c6d3c morss: fix csv output encoding 2020-04-09 19:05:50 +02:00
pictuga 78cea10ead morss: replace :getpage with :get
Also provides readabilite debugging
2020-04-09 18:43:20 +02:00
pictuga e5a82ff1f4 crawler: drop auto-referer
Was solving some issues. But creating even more issues.
2020-04-07 10:39:21 +02:00
pictuga f3d1f92b39 Detect encoding everytime 2020-04-07 10:38:36 +02:00
pictuga 7691df5257 Use wrapper for http calls 2020-04-07 10:30:17 +02:00
pictuga 0ae0dbc175 README: mention csv output 2020-04-07 09:24:32 +02:00
pictuga f1d0431e68 morss: drop :html, replaced with :reader
README updated accordingly
2020-04-07 09:23:29 +02:00
pictuga a09831415f feeds: fix bug when mimetype matches nothing 2020-04-06 18:53:07 +02:00
pictuga bfad6b7a4a readabilite: clean before counting
To remove links which are not kept anyway
2020-04-06 16:55:39 +02:00
pictuga 6b8c3e51e7 readabilite: fix threshold feature
Awkward typo...
2020-04-06 16:52:06 +02:00
pictuga dc9e425247 readabilite: don't clean-out the top 10% nodes
Loosen up the code once again to limit over-kill
2020-04-06 14:26:28 +02:00
pictuga 2f48e18bb1 readabilite: put scores directly in html node
Probably slower but makes code somewhat cleaner...
2020-04-06 14:21:41 +02:00
pictuga 31cac921c7 README: remove ref to iTunes 2020-04-05 22:20:33 +02:00