 pictuga
|
a890536601
|
morss: comment code a bit
|
2021-11-07 18:26:07 +01:00 |
|
pictuga
|
8de309f2d4
|
caching: add diskcache backend
|
2021-11-07 18:15:20 +01:00 |
|
pictuga
|
cbf7b3f77b
|
caching: simplify sqlite code
|
2021-11-07 18:14:18 +01:00 |
|
pictuga
|
d023ec8d73
|
Change default port to 8000
|
2021-10-19 22:19:59 +02:00 |
|
pictuga
|
5473b77416
|
Post-clean up isort
continuous-integration/drone/push Build is passing
Details
|
2021-09-21 08:11:04 +02:00 |
|
pictuga
|
0365232a73
|
readabilite: custom xpath for article detection
continuous-integration/drone/push Build is failing
Details
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
a523518ae8
|
cache: avoid name collision
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
52c48b899f
|
readability: better var names
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
9649cabb1b
|
morss: do not crash on empty pages
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
10535a17c5
|
cache: fix isort
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
7d86972e58
|
Add Redis cache backend
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
5da7121a77
|
Fix Options class behaviour
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
bb82902ad1
|
Move cache code to its own file
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
04afa28fe7
|
crawler: cache pickle'd array
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
75bb69f0fd
|
Make mysql optdep
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
97d9dda547
|
crawler: support 308 redirects
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
6c1f8da692
|
ci: added pylint (triggered upon error w/ score < 8 only)
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
4fd730b983
|
Further isort implementation
|
2021-09-21 08:04:45 +02:00 |
|
pictuga
|
0b3e6d7749
|
Apply isort
|
2021-09-21 08:04:23 +02:00 |
|
pictuga
|
06e0ada95b
|
Allow POST requests
|
2021-09-08 20:43:21 +02:00 |
|
pictuga
|
37f5a92b05
|
wsgi: fix apache / workload
|
2021-09-06 22:01:48 +02:00 |
|
pictuga
|
24c26d3850
|
wsgi: handle url decoding
|
2021-08-31 21:51:21 +02:00 |
|
pictuga
|
8f24214915
|
crawler: better name for custom fns
|
2021-08-29 00:22:40 +02:00 |
|
pictuga
|
d5942fe5a7
|
feeds: fix issues when mode not explicited in ruleset
|
2021-08-29 00:20:29 +02:00 |
|
pictuga
|
6f50443995
|
morss: Options return None instead of False if no match
Better for default fn values
|
2021-08-29 00:19:09 +02:00 |
|
pictuga
|
5582fbef31
|
crawler: comment
|
2021-08-29 00:18:50 +02:00 |
|
pictuga
|
da5442a1dc
|
feedify: support any type (json, xml, html)
|
2021-08-29 00:17:28 +02:00 |
|
pictuga
|
f9d7794bcc
|
feedifi.ini: json time zone handling
|
2021-04-23 20:43:00 +02:00 |
|
pictuga
|
e37c8346d0
|
feeds: add fallback for time parser
|
2021-04-22 21:57:16 +02:00 |
|
pictuga
|
3a1d564992
|
feeds: fix time zone handling
|
2021-04-22 21:51:00 +02:00 |
|
pictuga
|
6880a443e0
|
crawler: improve CacheHandler code
|
2021-03-25 23:54:08 +01:00 |
|
pictuga
|
7342ab26d2
|
crawler: comment on how urllib works
|
2021-03-25 23:49:58 +01:00 |
|
pictuga
|
981da9e66a
|
crawler: SQLITE_PATH point to .db file instead of folder
|
2021-03-25 23:48:21 +01:00 |
|
pictuga
|
01a7667032
|
Fix error due to remaining log force code
|
2021-01-14 00:51:47 +01:00 |
|
pictuga
|
3e886caaab
|
crawler: drop encoding setting
|
2020-10-30 22:41:16 +01:00 |
|
pictuga
|
ad927e03a7
|
crawler: use regex instead of lxml
Less reliable but should be faster
|
2020-10-30 22:21:19 +01:00 |
|
pictuga
|
0efb096fa7
|
crawler: shift gzip & encoding-fix to intermediary handler
|
2020-10-30 22:16:51 +01:00 |
|
pictuga
|
9ab2e488ef
|
crawler: add intermediary handlers
|
2020-10-30 22:15:35 +01:00 |
|
pictuga
|
b525ab0d26
|
crawler: fix typo
|
2020-10-30 22:12:43 +01:00 |
|
pictuga
|
bd0bca69fc
|
crawler: ignore ssl via env var
|
2020-10-03 19:57:08 +02:00 |
|
pictuga
|
8abd951d40
|
More sensible default values for cache autotrim (1k entries, 1min)
|
2020-10-03 19:55:57 +02:00 |
|
pictuga
|
2514fabd38
|
Replace memory-leak-prone Uniq with @uniq_wrapper
|
2020-10-03 19:43:55 +02:00 |
|
pictuga
|
8cb7002fe6
|
feeds: make it possible to append empty items
And return the newly appended items, to make it easy to edit them
|
2020-10-03 16:56:07 +02:00 |
|
pictuga
|
6966e03bef
|
Clean up itemClass code
To avoid globals()
|
2020-10-03 16:25:29 +02:00 |
|
pictuga
|
9ce6acba20
|
Fix gunicorn related typo
|
2020-10-01 00:07:41 +02:00 |
|
pictuga
|
056a1b143f
|
crawler: autotrim: make ctrl+c working
|
2020-10-01 00:04:36 +02:00 |
|
pictuga
|
eed949736a
|
crawler: add ability to limit cache size
|
2020-09-30 23:59:55 +02:00 |
|
pictuga
|
2fc7cd391c
|
Shift __main__'s wsgi code where it belongs
|
2020-09-30 23:24:51 +02:00 |
|
pictuga
|
d9f46b23a6
|
crawler: default value for MYSQL_HOST (localhost)
|
2020-09-30 13:17:02 +02:00 |
|
pictuga
|
bbada0436a
|
Quick guide to ignore SSL certs
|
2020-09-27 16:48:22 +02:00 |
|
pictuga
|
039a672f4e
|
wsgi: clean up url reconstruction
|
2020-09-27 16:28:26 +02:00 |
|
pictuga
|
9ecf856f10
|
Add :resolve to remove (some?) tracking links
|
2020-09-15 22:57:52 +02:00 |
|
pictuga
|
56e0c2391d
|
Missing import for served files
|
2020-08-28 20:53:03 +02:00 |
|
pictuga
|
679f406a12
|
Default mimetype for served files
|
2020-08-28 20:52:43 +02:00 |
|
pictuga
|
f6d641eeef
|
Serve any file in www/
Also fixes #41
|
2020-08-28 20:45:39 +02:00 |
|
pictuga
|
2456dd9bbc
|
Fix broken pieces
Including #43
|
2020-08-28 19:38:48 +02:00 |
|
pictuga
|
0f33db248a
|
Add license info in each file
|
2020-08-26 20:08:22 +02:00 |
|
pictuga
|
75935114e4
|
Remove leftover code
|
2020-08-23 19:07:12 +02:00 |
|
pictuga
|
aa9143302b
|
Remove now-unused isInt code
|
2020-08-23 18:51:09 +02:00 |
|
pictuga
|
0d62a7625b
|
Define http port via env vars as well
|
2020-08-23 18:50:18 +02:00 |
|
pictuga
|
bd0efb1529
|
crawler: missing os import
|
2020-08-23 18:45:44 +02:00 |
|
pictuga
|
47a17614ef
|
Rename morss/cgi.py into morss/wsgi.py
To avoid name collision with the built-in cgi lib
|
2020-08-23 18:44:49 +02:00 |
|
pictuga
|
4dfebe78f7
|
Pick caching backend via env vars
|
2020-08-23 18:43:18 +02:00 |
|
pictuga
|
dcd3e4a675
|
cgi.py: add missing impots
|
2020-08-23 18:31:05 +02:00 |
|
pictuga
|
e968b2ea7f
|
Remove leftover :debug code
|
2020-08-23 16:59:34 +02:00 |
|
pictuga
|
0ac590c798
|
Set MAX_/LIM_* settings via env var
|
2020-08-23 16:09:58 +02:00 |
|
pictuga
|
f65fb45030
|
:debug completely deprecated in favour of DEBUG=
|
2020-08-23 14:33:32 +02:00 |
|
pictuga
|
6dd40e5cc4
|
cli.py: fix Options code
|
2020-08-23 14:25:09 +02:00 |
|
pictuga
|
0acfce5a22
|
cli.py: remove log
|
2020-08-23 14:24:57 +02:00 |
|
pictuga
|
97ccc15db0
|
cgi.py: rename parseOptions to parse_options
|
2020-08-23 14:24:23 +02:00 |
|
pictuga
|
7a560181f7
|
Use env var for DEBUG
|
2020-08-23 14:23:45 +02:00 |
|
pictuga
|
baccd3b22b
|
Move parseOptions to cgi.py
As it is no longer used in cli.py
|
2020-08-22 00:37:34 +02:00 |
|
pictuga
|
f79938ab11
|
Add :silent to readme & argparse
|
2020-08-22 00:02:08 +02:00 |
|
pictuga
|
5b8bd47829
|
cli.py: remove draft code
|
2020-08-21 23:59:12 +02:00 |
|
pictuga
|
b5b355aa6e
|
readabilite: increase penalty for high link density
|
2020-08-21 23:55:04 +02:00 |
|
pictuga
|
bd182bcb85
|
Move cli code to argParse
Related code changes (incl. :format=xyz)
|
2020-08-21 23:52:56 +02:00 |
|
pictuga
|
c7c2c5d749
|
Removed unused filterOptions code
|
2020-08-21 23:23:33 +02:00 |
|
pictuga
|
c6b52e625f
|
split morss.py into __main__/cgi/cli.py
Should hopefully allow cleaner code in the future
|
2020-08-21 22:17:55 +02:00 |
|
pictuga
|
c6d3a0eb53
|
readabilite: clean up code
|
2020-07-15 00:49:34 +02:00 |
|
pictuga
|
6021b912ff
|
morss: fix item removal
Usual issue when editing a list while looping over it
|
2020-07-06 19:25:48 +02:00 |
|
pictuga
|
f18a128ee6
|
Change :first for :newest
i.e. toggle default for the more-obvious option
|
2020-07-06 19:25:17 +02:00 |
|
pictuga
|
64af86c11e
|
crawler: catch html parsing errors
|
2020-07-06 12:25:38 +02:00 |
|
pictuga
|
15951d228c
|
Add :first to NOT sort items by date
|
2020-07-06 11:39:08 +02:00 |
|
pictuga
|
c1b1f5f58a
|
morss: restrict iframe use from :get to avoid abuse
|
2020-06-09 12:33:37 +02:00 |
|
pictuga
|
985185f47f
|
morss: more flexible feed creator auto-detection
|
2020-06-08 13:03:24 +02:00 |
|
pictuga
|
3190d1ec5a
|
feeds: remove useless if(len) before loop
|
2020-06-02 13:57:45 +02:00 |
|
pictuga
|
ce4cf01aa6
|
crawler: clean up encoding detection code
|
2020-05-27 21:35:24 +02:00 |
|
pictuga
|
dcfdb75a15
|
crawler: fix chinese encoding support
|
2020-05-27 21:34:43 +02:00 |
|
pictuga
|
4ccc0dafcd
|
Basic help for sub-lib interactive use
|
2020-05-26 19:34:20 +02:00 |
|
pictuga
|
2fe3e0b8ee
|
feeds: clean up other stylesheets before putting ours
|
2020-05-26 19:26:36 +02:00 |
|
pictuga
|
68c46a1823
|
morss: remove deprecated twitter/fb link handling
|
2020-05-13 12:31:09 +02:00 |
|
pictuga
|
91be2d229e
|
morss: ability to use first link from desc instead of default link
|
2020-05-13 12:29:53 +02:00 |
|
pictuga
|
038f267ea2
|
Rename :theforce into :force
|
2020-05-13 11:49:15 +02:00 |
|
pictuga
|
22005065e8
|
Use etree.tostring 'method' arg
Gives appropriately formatted html code.
Some pages might otherwise be rendered as blank.
|
2020-05-13 11:44:34 +02:00 |
|
pictuga
|
7d0d416610
|
morss: cache articles for 24hrs
Also make it possible to refetch articles, regardless of cache
|
2020-05-12 21:10:31 +02:00 |
|
pictuga
|
5dac4c69a1
|
crawler: more code comments
|
2020-05-12 20:44:25 +02:00 |
|
pictuga
|
36e2a1c3fd
|
crawler: increase size limit from 100KiB to 500
I'm looking at you, worldbankgroup.csod.com/ats/careersite/search.aspx
|
2020-05-12 19:34:16 +02:00 |
|
pictuga
|
83dd2925d3
|
readabilite: better parsing
Keeping blank_text keeps the tree more as-it, making the final output closer to expectations
|
2020-05-12 14:15:53 +02:00 |
|
pictuga
|
e09d0abf54
|
morss: remove deprecated peace of code
|
2020-05-07 16:05:30 +02:00 |
|
pictuga
|
ff26a560cb
|
Shift safari work around to morss.py
|
2020-05-07 16:04:54 +02:00 |