 pictuga
|
f65fb45030
|
:debug completely deprecated in favour of DEBUG=
|
2020-08-23 14:33:32 +02:00 |
|
pictuga
|
6dd40e5cc4
|
cli.py: fix Options code
|
2020-08-23 14:25:09 +02:00 |
|
pictuga
|
0acfce5a22
|
cli.py: remove log
|
2020-08-23 14:24:57 +02:00 |
|
pictuga
|
97ccc15db0
|
cgi.py: rename parseOptions to parse_options
|
2020-08-23 14:24:23 +02:00 |
|
pictuga
|
7a560181f7
|
Use env var for DEBUG
|
2020-08-23 14:23:45 +02:00 |
|
pictuga
|
baccd3b22b
|
Move parseOptions to cgi.py
As it is no longer used in cli.py
|
2020-08-22 00:37:34 +02:00 |
|
pictuga
|
f79938ab11
|
Add :silent to readme & argparse
|
2020-08-22 00:02:08 +02:00 |
|
pictuga
|
5b8bd47829
|
cli.py: remove draft code
|
2020-08-21 23:59:12 +02:00 |
|
pictuga
|
b5b355aa6e
|
readabilite: increase penalty for high link density
|
2020-08-21 23:55:04 +02:00 |
|
pictuga
|
bd182bcb85
|
Move cli code to argParse
Related code changes (incl. :format=xyz)
|
2020-08-21 23:52:56 +02:00 |
|
pictuga
|
c7c2c5d749
|
Removed unused filterOptions code
|
2020-08-21 23:23:33 +02:00 |
|
pictuga
|
c6b52e625f
|
split morss.py into __main__/cgi/cli.py
Should hopefully allow cleaner code in the future
|
2020-08-21 22:17:55 +02:00 |
|
pictuga
|
c6d3a0eb53
|
readabilite: clean up code
|
2020-07-15 00:49:34 +02:00 |
|
pictuga
|
6021b912ff
|
morss: fix item removal
Usual issue when editing a list while looping over it
|
2020-07-06 19:25:48 +02:00 |
|
pictuga
|
f18a128ee6
|
Change :first for :newest
i.e. toggle default for the more-obvious option
|
2020-07-06 19:25:17 +02:00 |
|
pictuga
|
64af86c11e
|
crawler: catch html parsing errors
|
2020-07-06 12:25:38 +02:00 |
|
pictuga
|
15951d228c
|
Add :first to NOT sort items by date
|
2020-07-06 11:39:08 +02:00 |
|
pictuga
|
c1b1f5f58a
|
morss: restrict iframe use from :get to avoid abuse
|
2020-06-09 12:33:37 +02:00 |
|
pictuga
|
985185f47f
|
morss: more flexible feed creator auto-detection
|
2020-06-08 13:03:24 +02:00 |
|
pictuga
|
3190d1ec5a
|
feeds: remove useless if(len) before loop
|
2020-06-02 13:57:45 +02:00 |
|
pictuga
|
ce4cf01aa6
|
crawler: clean up encoding detection code
|
2020-05-27 21:35:24 +02:00 |
|
pictuga
|
dcfdb75a15
|
crawler: fix chinese encoding support
|
2020-05-27 21:34:43 +02:00 |
|
pictuga
|
4ccc0dafcd
|
Basic help for sub-lib interactive use
|
2020-05-26 19:34:20 +02:00 |
|
pictuga
|
2fe3e0b8ee
|
feeds: clean up other stylesheets before putting ours
|
2020-05-26 19:26:36 +02:00 |
|
pictuga
|
68c46a1823
|
morss: remove deprecated twitter/fb link handling
|
2020-05-13 12:31:09 +02:00 |
|
pictuga
|
91be2d229e
|
morss: ability to use first link from desc instead of default link
|
2020-05-13 12:29:53 +02:00 |
|
pictuga
|
038f267ea2
|
Rename :theforce into :force
|
2020-05-13 11:49:15 +02:00 |
|
pictuga
|
22005065e8
|
Use etree.tostring 'method' arg
Gives appropriately formatted html code.
Some pages might otherwise be rendered as blank.
|
2020-05-13 11:44:34 +02:00 |
|
pictuga
|
7d0d416610
|
morss: cache articles for 24hrs
Also make it possible to refetch articles, regardless of cache
|
2020-05-12 21:10:31 +02:00 |
|
pictuga
|
5dac4c69a1
|
crawler: more code comments
|
2020-05-12 20:44:25 +02:00 |
|
pictuga
|
36e2a1c3fd
|
crawler: increase size limit from 100KiB to 500
I'm looking at you, worldbankgroup.csod.com/ats/careersite/search.aspx
|
2020-05-12 19:34:16 +02:00 |
|
pictuga
|
83dd2925d3
|
readabilite: better parsing
Keeping blank_text keeps the tree more as-it, making the final output closer to expectations
|
2020-05-12 14:15:53 +02:00 |
|
pictuga
|
e09d0abf54
|
morss: remove deprecated peace of code
|
2020-05-07 16:05:30 +02:00 |
|
pictuga
|
ff26a560cb
|
Shift safari work around to morss.py
|
2020-05-07 16:04:54 +02:00 |
|
pictuga
|
f685139137
|
crawler: use UPSERT statements
Avoid potential race conditions
|
2020-05-03 21:27:45 +02:00 |
|
pictuga
|
73b477665e
|
morss: separate :clip with <hr> instead of stars
|
2020-05-02 19:19:54 +02:00 |
|
pictuga
|
b425992783
|
morss: don't follow alt=rss with custom feeds
To have the same page as with :get=page and to avoid shitty feeds
|
2020-05-02 19:18:58 +02:00 |
|
pictuga
|
271ac8f80f
|
crawler: comment code a bit
|
2020-05-02 19:18:01 +02:00 |
|
pictuga
|
64e41b807d
|
crawler: handle http:/ (single slash)
Fixing one more corner case! malayalam.oneindia.com
|
2020-05-02 19:17:15 +02:00 |
|
pictuga
|
27a42c47aa
|
morss: use final request url
Code is not very elegant...
|
2020-04-28 22:30:21 +02:00 |
|
pictuga
|
c27c38f7c7
|
crawler: return dict instead of tuple
|
2020-04-28 22:29:07 +02:00 |
|
pictuga
|
a1dc96cb50
|
feeds: remove mimetype from function call as no longer used
|
2020-04-28 22:07:25 +02:00 |
|
pictuga
|
749acc87fc
|
Centralize url clean up in crawler.py
|
2020-04-28 22:03:49 +02:00 |
|
pictuga
|
cb69e3167f
|
crawler: accept non-ascii urls
Covering one more corner case!
|
2020-04-28 14:47:23 +02:00 |
|
pictuga
|
c3f06da947
|
morss: process(): specify encoding for clarity
|
2020-04-28 14:45:00 +02:00 |
|
pictuga
|
44a3e0edc4
|
readabilite: specify in- and out-going encoding
|
2020-04-28 14:44:35 +02:00 |
|
pictuga
|
818cdaaa9b
|
Make it possible to call sub-libs in non interactive mode
Run `python -m morss.feeds http://lemonde.fr` and so on
|
2020-04-27 18:00:14 +02:00 |
|
pictuga
|
2806c64326
|
Make it possible to directly run sub-libs (feeds, crawler, readabilite)
Run `python -im morss.feeds http://website.sample/rss.xml` and so on
|
2020-04-27 17:19:31 +02:00 |
|
pictuga
|
f6bc23927f
|
readabilite: drop dangerous tags (script, style)
|
2020-04-25 12:25:02 +02:00 |
|
pictuga
|
c86572374e
|
readabilite: minimum score requirement
|
2020-04-25 12:24:36 +02:00 |
|
pictuga
|
59ef5af9e2
|
feeds: fix bug when deleting attr in html
|
2020-04-24 22:12:05 +02:00 |
|
pictuga
|
6a0531ca03
|
crawler: randomize user agent
|
2020-04-24 11:28:39 +02:00 |
|
pictuga
|
8187876a06
|
crawler: stop at first alternative link
Should save a few ms and the first one is usually (?) the most relevant/generic
|
2020-04-23 11:23:45 +02:00 |
|
pictuga
|
325a373e3e
|
feeds: add SyntaxError catch
|
2020-04-20 16:15:15 +02:00 |
|
pictuga
|
2719bd6776
|
crawler: fix chinese encoding
|
2020-04-20 16:14:55 +02:00 |
|
pictuga
|
ec8edb02f1
|
Various small bug fixes
|
2020-04-19 12:54:02 +02:00 |
|
pictuga
|
d01b943597
|
Remove leftover threading var
|
2020-04-19 12:51:11 +02:00 |
|
pictuga
|
b361aa2867
|
Add timeout to :get
|
2020-04-19 12:50:26 +02:00 |
|
pictuga
|
4ce3c7cb32
|
Small code clean ups
|
2020-04-19 12:50:05 +02:00 |
|
pictuga
|
7e45b2611d
|
Disable multi-threading
Impact was mostly negative due to locks
|
2020-04-19 12:29:52 +02:00 |
|
pictuga
|
036e5190f1
|
crawler: remove unused code
|
2020-04-18 21:40:02 +02:00 |
|
pictuga
|
e99c5b3b71
|
morss: more sensible default MAX/LIM values
|
2020-04-18 17:21:45 +02:00 |
|
pictuga
|
7375adce33
|
sheet.xsl: fix & improve
|
2020-04-15 23:34:28 +02:00 |
|
pictuga
|
fe82b19c91
|
Merge .xsl & html template
Turns out they somehow serve a similar purpose
|
2020-04-15 22:30:45 +02:00 |
|
pictuga
|
0b31e97492
|
morss: remove debug code in http file handler
|
2020-04-14 23:20:03 +02:00 |
|
pictuga
|
b0ad7c259d
|
Add README & LICENSE to data_files
|
2020-04-14 19:34:12 +02:00 |
|
pictuga
|
59139272fd
|
Auto-detect the location of www/
Either ../www or /usr/share/morss
Adapted README accordingly
|
2020-04-14 18:07:19 +02:00 |
|
pictuga
|
e6b7c0eb33
|
Fix app definition for uwsgi
|
2020-04-13 15:30:09 +02:00 |
|
pictuga
|
67c096ad5b
|
feeds: add fake path to default html parser
Without it, some websites were accidentally matching it (false positives)
|
2020-04-12 13:00:56 +02:00 |
|
pictuga
|
f018437544
|
crawler: make mysql backend thread safe
|
2020-04-12 12:53:05 +02:00 |
|
pictuga
|
8e5e8d24a4
|
Timezone fixes
|
2020-04-10 20:33:59 +02:00 |
|
pictuga
|
ee78a7875a
|
morss: focus on the most recent feed items
|
2020-04-10 16:08:13 +02:00 |
|
pictuga
|
9e7b9d95ee
|
feeds: properly use html template
|
2020-04-09 20:00:51 +02:00 |
|
pictuga
|
987a719c4e
|
feeds: try all parsers regardless of contenttype
Turns out some websites send the wrong contenttype (json for html, html for xml, etc.)
|
2020-04-09 19:17:51 +02:00 |
|
pictuga
|
47b33f4baa
|
morss: specify server output encoding
|
2020-04-09 19:10:45 +02:00 |
|
pictuga
|
3c7f512583
|
feeds: handle several errors
|
2020-04-09 19:09:10 +02:00 |
|
pictuga
|
a32f5a8536
|
readabilite: add debug option (also used by :get)
|
2020-04-09 19:08:13 +02:00 |
|
pictuga
|
63a06524b7
|
morss: various encoding fixes
|
2020-04-09 19:06:51 +02:00 |
|
pictuga
|
b0f80c6d3c
|
morss: fix csv output encoding
|
2020-04-09 19:05:50 +02:00 |
|
pictuga
|
78cea10ead
|
morss: replace :getpage with :get
Also provides readabilite debugging
|
2020-04-09 18:43:20 +02:00 |
|
pictuga
|
e5a82ff1f4
|
crawler: drop auto-referer
Was solving some issues. But creating even more issues.
|
2020-04-07 10:39:21 +02:00 |
|
pictuga
|
f3d1f92b39
|
Detect encoding everytime
|
2020-04-07 10:38:36 +02:00 |
|
pictuga
|
7691df5257
|
Use wrapper for http calls
|
2020-04-07 10:30:17 +02:00 |
|
pictuga
|
f1d0431e68
|
morss: drop :html, replaced with :reader
README updated accordingly
|
2020-04-07 09:23:29 +02:00 |
|
pictuga
|
a09831415f
|
feeds: fix bug when mimetype matches nothing
|
2020-04-06 18:53:07 +02:00 |
|
pictuga
|
bfad6b7a4a
|
readabilite: clean before counting
To remove links which are not kept anyway
|
2020-04-06 16:55:39 +02:00 |
|
pictuga
|
6b8c3e51e7
|
readabilite: fix threshold feature
Awkward typo...
|
2020-04-06 16:52:06 +02:00 |
|
pictuga
|
dc9e425247
|
readabilite: don't clean-out the top 10% nodes
Loosen up the code once again to limit over-kill
|
2020-04-06 14:26:28 +02:00 |
|
pictuga
|
2f48e18bb1
|
readabilite: put scores directly in html node
Probably slower but makes code somewhat cleaner...
|
2020-04-06 14:21:41 +02:00 |
|
pictuga
|
a82ec96eb7
|
Delete feedify.py leftover code
iTunes integration untested, unreliable and not working...
|
2020-04-05 22:16:52 +02:00 |
|
pictuga
|
aad2398e69
|
feeds: turns out lxml.etree doesn't have drop_tag
|
2020-04-05 21:50:38 +02:00 |
|
pictuga
|
eeac630855
|
crawler: add more "realistic" headers
|
2020-04-05 21:11:57 +02:00 |
|
pictuga
|
e136b0feb2
|
readabilite: loosen the slayer
Previous impl. lead to too many empty results
|
2020-04-05 20:47:30 +02:00 |
|
pictuga
|
6cf32af6c0
|
readabilite: also use BS
|
2020-04-05 20:46:42 +02:00 |
|
pictuga
|
568e7d7dd2
|
feeds: make BS's output bytes for lxml's sake
|
2020-04-05 20:46:04 +02:00 |
|
pictuga
|
3617f86e9d
|
morss: make cgi_encore more robust
|
2020-04-05 16:43:11 +02:00 |
|
pictuga
|
d90756b337
|
morss: drop 'keep' option
Because the Firefox behaviour it is working around is no longer in use
|
2020-04-05 16:37:27 +02:00 |
|
pictuga
|
40c69f17d2
|
feeds: parse html with BS
More robust & to make it consistent with :getpage
|
2020-04-05 16:12:41 +02:00 |
|
pictuga
|
99461ea185
|
crawler: fix var name issues (private_cache)
|
2020-04-05 16:11:36 +02:00 |
|
pictuga
|
bf86c1e962
|
crawler: make AutoUA match http(s) type
|
2020-04-05 16:07:51 +02:00 |