Commit Graph

1037 Commits (438c32a312fea2555e8070f572e0907418f9394c)
 

Author SHA1 Message Date
pictuga 438c32a312 Remove sqlite & mysql cache backends
continuous-integration/drone/push Build is failing Details
Obsoleted since the introduction of diskcache & redis
2022-12-13 22:40:13 +01:00
pictuga 8b26797e93 README: add recommended install way
continuous-integration/drone/push Build was killed Details
continuous-integration/drone Build is passing Details
Part of discussions on #94
2022-12-13 22:07:21 +01:00
pictuga e1ed33f320 crawler: improve html iter code
continuous-integration/drone/push Build is passing Details
Ignores tags without attributes. Avoids bug with unclosed tags.
2022-02-09 15:57:12 +01:00
pictuga b65272daab crawler: accept more meta redirects
continuous-integration/drone/push Build is passing Details
2022-02-01 23:32:49 +01:00
pictuga 4d64afe9cb crawler: fix regression from d6b90448f3
continuous-integration/drone/push Build is failing Details
2022-02-01 23:18:16 +01:00
pictuga d3b623482d pytest: crawler 2022-02-01 23:16:43 +01:00
pictuga 32645548c2 pytest: first batch with test_feeds
continuous-integration/drone/push Build is failing Details
And multiple related fixes
2022-01-31 08:32:34 +01:00
pictuga d6b90448f3 crawler: improve handling of non-ascii urls 2022-01-30 23:27:49 +01:00
pictuga da81edc651 log to stderr
continuous-integration/drone/push Build is failing Details
2022-01-26 07:57:57 +01:00
pictuga 4f2895f931 cli: update `--help`
continuous-integration/drone/push Build is failing Details
2022-01-25 22:36:57 +01:00
pictuga b2b04691d6 Ability to pass custom data_files location 2022-01-25 22:36:34 +01:00
pictuga bfaf7b0fac feeds: clean up default `item_link`
continuous-integration/drone/push Build is failing Details
To be supported by feeds' `_rule_parse`
2022-01-24 16:16:14 +00:00
pictuga 32d9bc9d9d feeds: proceed with conversion when rules do not match
continuous-integration/drone/push Build is failing Details
2022-01-24 09:34:57 +00:00
pictuga b138f11771 util: support more `data_files` location
continuous-integration/drone/push Build is passing Details
2022-01-23 12:40:18 +01:00
pictuga a01258700d More ordering options
continuous-integration/drone/push Build was killed Details
2022-01-23 12:27:07 +01:00
pictuga 4d6d3c9239 wsgi: limit supported mimetypes & return actual mimetype
continuous-integration/drone/push Build is passing Details
2022-01-23 11:44:07 +01:00
pictuga e81f6b173f readabilite: remove code duplicate 2022-01-23 11:41:32 +01:00
pictuga fe5dbf1ce0 wsgi: reuse mimetype table from crawler 2022-01-22 13:22:39 +01:00
pictuga fdf9acd32b helper: fix reload code
continuous-integration/drone/push Build is passing Details
2022-01-19 13:44:15 +01:00
pictuga d05706e056 crawler: fix typo
continuous-integration/drone/push Build was killed Details
2022-01-19 13:41:12 +01:00
pictuga e88a823ada feeds: better handle rulesets without a 'mode' specified
continuous-integration/drone/push Build is failing Details
2022-01-19 13:08:33 +01:00
pictuga 750850c162 crawler: avoid too many .append() 2022-01-19 13:04:33 +01:00
pictuga c8669002e4 feeds: exotic xpath in html as well
continuous-integration/drone/push Build is passing Details
2022-01-17 14:22:48 +00:00
pictuga c524e54d2d feeds: support some exotic xpath rules returning a single string
continuous-integration/drone/push Build is passing Details
2022-01-17 13:59:58 +00:00
pictuga ef14567d87 Handle `morss-helper` with `setup.py`
continuous-integration/drone/push Build is passing Details
2022-01-08 16:10:51 +01:00
pictuga fb643f5ef1 readabilite: remove unneeded reference to `features` (overriden by `builder`)
continuous-integration/drone/push Build is passing Details
2022-01-03 18:01:12 +00:00
pictuga dbdca910d8 readabilite: fix new parser code & drop PIs
continuous-integration/drone/push Build was killed Details
2022-01-03 17:51:49 +00:00
pictuga 9eb19fac04 readabilite: use custom html parser within bs4's lxml parser
continuous-integration/drone/push Build is passing Details
Solves the following obscure error:
ValueError: Invalid PI name 'b'xml''
2022-01-03 16:26:17 +00:00
pictuga d424e394d1 readabilite: use lxml bs4 parser for speed
continuous-integration/drone/push Build is passing Details
2022-01-01 14:52:48 +01:00
pictuga 3f92787b38 readabilite: limit html comments related issues
continuous-integration/drone/push Build is passing Details
2022-01-01 13:58:42 +01:00
pictuga afc31eb6e9 readabilite: avoid double parsing of html
continuous-integration/drone/push Build is passing Details
2022-01-01 12:51:30 +01:00
pictuga 87d2fe772d wsgi: fix py2 compatibility 2022-01-01 12:35:41 +01:00
pictuga 917aa0fbc5 crawler: do not re-save cached response
continuous-integration/drone/push Build is passing Details
Otherwise cache never gets invalidated!
2021-12-31 19:28:11 +01:00
pictuga 3e2b81286f xsl: add link to favicon
To limit error output when failing to fetch favicon.ico
2021-12-31 19:25:53 +01:00
pictuga 15430a2b83 helper: restore run if no param passed
continuous-integration/drone/push Build is passing Details
2021-12-29 23:35:16 +01:00
pictuga ecdb74812d Make helper & main.py executable
continuous-integration/drone/push Build is passing Details
2021-12-29 15:47:05 +01:00
pictuga 2c7844942c drone: re order deploy commands
continuous-integration/drone/push Build was killed Details
2021-12-29 15:41:29 +01:00
pictuga e12cb4567a helper: more debug options 2021-12-29 15:41:03 +01:00
pictuga b74365b121 Make helper more posix compliant 2021-12-29 15:40:43 +01:00
pictuga 2020543469 Make morss-helper executable 2021-12-29 15:37:12 +01:00
pictuga 676be4a4fe helper: work around for systems only having py3 binary
continuous-integration/drone/push Build is running Details
2021-12-29 14:07:12 +01:00
pictuga 8870400a6e Clean up morss-helper
continuous-integration/drone/push Build is failing Details
2021-12-28 16:30:20 +01:00
pictuga 8e9cc541b0 Turns out exec array is not supported in HEALTHCHECK
continuous-integration/drone/push Build is failing Details
2021-12-28 15:23:40 +01:00
pictuga 2a7a1b83ec Use alpine:edge to have up-to-date py packages
continuous-integration/drone/push Build is passing Details
2021-12-28 13:41:42 +01:00
pictuga 106f59afa1 docker: shift HEALTHCHECK to helper
continuous-integration/drone/push Build is failing Details
2021-12-27 16:08:55 +01:00
pictuga ee514e2da3 helper: remove unneeded sudo
continuous-integration/drone/push Build is passing Details
2021-12-25 22:00:22 +00:00
pictuga e7578e859a Clean up install/exec
continuous-integration/drone/push Build is passing Details
2021-12-25 18:21:55 +01:00
pictuga 3bcb8db974 Improve cloud-init (append & env var) 2021-12-25 11:02:27 +01:00
pictuga 7751792942 Shift htaccess to README 2021-12-24 18:03:55 +01:00
pictuga 6e2e5ffa00 README: cloud-init indication for env var
continuous-integration/drone/push Build is passing Details
2021-12-24 11:44:56 +01:00