Commit Graph

543 Commits (d07aa566ed7fabd84965dd657674310f735e8c23)

Author SHA1 Message Date
pictuga 51f1d330a4 Fn to access data_files & pkg files
continuous-integration/drone Build is running Details
continuous-integration/drone/push Build is passing Details
2021-12-05 12:09:01 +01:00
pictuga eb47aac6f1 morss: respect timeout settings in all cases
continuous-integration/drone/push Build is failing Details
Special treatment of feed fetch not justified and not documented
2021-11-25 22:13:38 +01:00
pictuga eca546b890 Change HTTP error code to 404
continuous-integration/drone/push Build is failing Details
To tell them apart from 'true' 500 errors
2021-11-25 21:34:46 +01:00
pictuga d8cc07223e readabilite: fix bug when nothing above threshold
continuous-integration/drone/push Build is failing Details
2021-11-23 20:53:00 +01:00
pictuga 765e0ba728 Pass py error msg in http headers
continuous-integration/drone/push Build is passing Details
2021-11-22 23:22:13 +01:00
pictuga 6ec3fb47d1 readabilite: .strip() first to save time
continuous-integration/drone/push Build is passing Details
2021-11-15 21:54:07 +01:00
pictuga 1083f3ffbc crawler: make sure to use HTTPMessage
continuous-integration/drone/push Build is passing Details
2021-11-11 10:21:48 +01:00
pictuga 7eeb1d696c crawler: clean up code
continuous-integration/drone/push Build is passing Details
2021-11-10 23:25:03 +01:00
pictuga e42df98f83 crawler: fix regression brought with 44a6b2591
continuous-integration/drone/push Build is passing Details
2021-11-10 23:08:31 +01:00
pictuga cb21871c35 crawler: clean up caching code
continuous-integration/drone/push Build is passing Details
2021-11-08 22:02:23 +01:00
pictuga c71cf5d5ce caching: fix diskcache implementation 2021-11-08 21:57:43 +01:00
pictuga 44a6b2591d crawler: cleaner http header object import 2021-11-07 19:44:36 +01:00
pictuga a890536601 morss: comment code a bit 2021-11-07 18:26:07 +01:00
pictuga 8de309f2d4 caching: add diskcache backend 2021-11-07 18:15:20 +01:00
pictuga cbf7b3f77b caching: simplify sqlite code 2021-11-07 18:14:18 +01:00
pictuga d023ec8d73 Change default port to 8000 2021-10-19 22:19:59 +02:00
pictuga 5473b77416 Post-clean up isort
continuous-integration/drone/push Build is passing Details
2021-09-21 08:11:04 +02:00
pictuga 0365232a73 readabilite: custom xpath for article detection
continuous-integration/drone/push Build is failing Details
2021-09-21 08:04:45 +02:00
pictuga a523518ae8 cache: avoid name collision 2021-09-21 08:04:45 +02:00
pictuga 52c48b899f readability: better var names 2021-09-21 08:04:45 +02:00
pictuga 9649cabb1b morss: do not crash on empty pages 2021-09-21 08:04:45 +02:00
pictuga 10535a17c5 cache: fix isort 2021-09-21 08:04:45 +02:00
pictuga 7d86972e58 Add Redis cache backend 2021-09-21 08:04:45 +02:00
pictuga 5da7121a77 Fix Options class behaviour 2021-09-21 08:04:45 +02:00
pictuga bb82902ad1 Move cache code to its own file 2021-09-21 08:04:45 +02:00
pictuga 04afa28fe7 crawler: cache pickle'd array 2021-09-21 08:04:45 +02:00
pictuga 75bb69f0fd Make mysql optdep 2021-09-21 08:04:45 +02:00
pictuga 97d9dda547 crawler: support 308 redirects 2021-09-21 08:04:45 +02:00
pictuga 6c1f8da692 ci: added pylint (triggered upon error w/ score < 8 only) 2021-09-21 08:04:45 +02:00
pictuga 4fd730b983 Further isort implementation 2021-09-21 08:04:45 +02:00
pictuga 0b3e6d7749 Apply isort 2021-09-21 08:04:23 +02:00
pictuga 06e0ada95b Allow POST requests 2021-09-08 20:43:21 +02:00
pictuga 37f5a92b05 wsgi: fix apache / workload 2021-09-06 22:01:48 +02:00
pictuga 24c26d3850 wsgi: handle url decoding 2021-08-31 21:51:21 +02:00
pictuga 8f24214915 crawler: better name for custom fns 2021-08-29 00:22:40 +02:00
pictuga d5942fe5a7 feeds: fix issues when mode not explicited in ruleset 2021-08-29 00:20:29 +02:00
pictuga 6f50443995 morss: Options return None instead of False if no match
Better for default fn values
2021-08-29 00:19:09 +02:00
pictuga 5582fbef31 crawler: comment 2021-08-29 00:18:50 +02:00
pictuga da5442a1dc feedify: support any type (json, xml, html) 2021-08-29 00:17:28 +02:00
pictuga f9d7794bcc feedifi.ini: json time zone handling 2021-04-23 20:43:00 +02:00
pictuga e37c8346d0 feeds: add fallback for time parser 2021-04-22 21:57:16 +02:00
pictuga 3a1d564992 feeds: fix time zone handling 2021-04-22 21:51:00 +02:00
pictuga 6880a443e0 crawler: improve CacheHandler code 2021-03-25 23:54:08 +01:00
pictuga 7342ab26d2 crawler: comment on how urllib works 2021-03-25 23:49:58 +01:00
pictuga 981da9e66a crawler: SQLITE_PATH point to .db file instead of folder 2021-03-25 23:48:21 +01:00
pictuga 01a7667032 Fix error due to remaining log force code 2021-01-14 00:51:47 +01:00
pictuga 3e886caaab crawler: drop encoding setting 2020-10-30 22:41:16 +01:00
pictuga ad927e03a7 crawler: use regex instead of lxml
Less reliable but should be faster
2020-10-30 22:21:19 +01:00
pictuga 0efb096fa7 crawler: shift gzip & encoding-fix to intermediary handler 2020-10-30 22:16:51 +01:00
pictuga 9ab2e488ef crawler: add intermediary handlers 2020-10-30 22:15:35 +01:00