I thought it would be fun to start a FreshRSS power users thread. Share your tips, tricks, setups, hard-to-find feeds, etc. Search filters are also welcome. Be sure to use code blocks.
- General AI search filter
/A\.?I\.?\b/ OR /Palantir/ OR /Anthropic/ OR /OpenClaw/i [example]
- Betteridge’s law of headlines search filter (add to your News categories to mark binary question headlined articles as read)
intitle:/^(?!.*(wh|how)).*\?\s*$/i
I’ll make a list of useful posts here to help navigating the thread.
Table of contents
Noteworthy (external) links
3 Likes
What to do when a feed won’t work?
In some cases, you may not be able to add a feed due to misconfigurations on the target webserver, SSL certificate failures (you can relax certificate checking in FreshRSS feeds), or overly aggressive anti-bot measures.
Things to try:
- (under Advanced when adding a new feed or modifying and existing one in FreshRSS)
- Change the user agent when adding a feed
- Some interesting user agents to try:
- The idea is to try and send a user agent of a “preferred user”
Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com)
- You could also try mobile agents too
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X; Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.98 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://google.com)
- Purportedly this is the kind of user agent string sent by Bloomberg Terminal but I wouldn’t know
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.98 Safari/537.36 Bloomberg/Professional
- Use FreshRSS to send HTTP Headers with your request. Send a bogus referer string like
Referer: https://google.com or the website’s homepage itself to try and trick the server into believing it’s not automated traffic.
- Use an rss feed proxy (https://feedburner.google.com/) - for whatever reason, I have had success getting feed content using this service when everything else failed. Yes, Alphabet Inc. is probably using your feeds to somehow make the world a worse place. However-and I suspect it may have to do with the fact that a lot of sites used to distribute their feeds through FeedBurner-it just seems to work™. Caveat emptor, Google could shut this service down at any time.
- Use a software solution like one of the following:
What if a website doesn’t publish an RSS feed?
DIY! Try to find a page (archives, latest, featured, etc.) that just lists articles. Then:
- Use FreshRSS feed type “HTML + XPath” to try and scrape the page. The article linked in the FreshRSS docs provides a nice 'n easy tutorial to follow.
- Note that you can use CSS selector content retrieval in FreshRSS to get the full article content, which is good in cases where no short description is published on an archive page, for example.
- Use a software solution like one of the following (ordered roughly from heaviest to lightest resource use):
- Use a web service like one of the following (ordered from least to most cool):
- PolitePaul - hint: you can use CSS selector content fetching in FreshRSS to replace the entries entirely, thereby removing “Feed provided by” or ad injections for freemium servies
- AnyRSS
- RSS-Bridge public instance warning: often rate-limited
- RSS-Proxy can be self hosted
- OpenRSS
very cool, check it out if you like newsreaders
Quality RSS feeds
Quality being in the eye of the beholder, I guess…
- https://theconversation.com there are many different editions, multilingual etc (AU, BR, CA, ES, EU, FR, ID, NZ, UK). “Academic rigour with a journalistic flair.” But for the benefit of what I assume is a primarily US audience:
https://theconversation.com/us/articles.atom
https://theconversation.com/us/business/articles.atom
https://theconversation.com/us/technology/articles.atom
# ... etc (append /articles.atom to your preferred sections) ...
- https://daily.jstor.org “JSTOR Daily provides context for current events using scholarship found in JSTOR, a digital library of academic journals, books, and other material. We publish articles grounded in peer-reviewed research and provide free access to that research for all of our readers.”
https://daily.jstor.org/feed/
https://daily.jstor.org/type/long-reads/feed/
https://daily.jstor.org/category/education-and-society/religion/feed/
# ... etc (append /feed to your preferred sections) ...
https://www.historyisnowmagazine.com/blog?format=rss
https://www.historyisnowmagazine.com/blog?tag=World+War+1&format=rss
https://www.historyisnowmagazine.com/blog?category=20th+century&format=rss
# ... etc (append &format=rss to your preferred sections) ...
1 Like
Can use it to add youtube feeds & track channels of interest without need of being logged in. (also works with nebula, at least some peertube instances)
Just add the channel url, e.g.:
https://www.youtube.com/channel/UCOWcZ6Wicl-1N34H0zZe38w
and a filter, to hide the chaff:
inurl:shorts
1 Like
Writing XPath selectors by hand
The process to scrape a website by XPath + HTML is not too difficult. You should familiarize yourself with the developer tools of your favorite browser. Most use similar terminology. A basic understanding of HTML will also be helpful. Instead of view-source: you will use the “Inspect element” feature (right click) in Chrome/Firefox etc. This splits the page in two, highlighting the element displayed on the page and its associated code in the frame below. The trick is to right click on a headline in a list of articles, and in the HTML frame, try to select a few parent elements to see if you can capture a title, author name and publication date. At least you will want the title, author is nice for filtering and date is largely unnecessary in the context of scraping for news.
Some useful XPath 1.0 patterns to make life much easier for advanced users…
//article[contains(@class, 'article-card-')] matches <article class="article-card-featured">, <article class="article-card-byline">, etc. Roughly equivalent to article-card-*.concat('https://archive.ph/{TARGET_SITE}', ./a/@href) Typically you would use concat to fix relative links (insert the https://{TARGET_SITE} instead) but this would be one way of bypassing a certain very annoying inducement to subscribe to read more….// for finding any descendant of the current context node. E.g. once you find the article container, .//img/@src will get you the thumbnail associated with each article. This is useful when author, date or images are encased in multiple divs. Keeps your XPath short and sweet.
1 Like
Convert newsletters to RSS
With a bit of trickery you can convert most newsletters (Morningstar, ISW, etc.) to an RSS feed using https://emailtofeed.com There is some interesting investing info only available in newsletter format, and some websites like the ISW publish a newsletter rather than RSS Feed. In some cases you may need to use a throwaway GMail account and configure forwarding addresses to bypass various newsletter providers’ checks. I would not put any personal information into Newsletters you send here since they are essentially exposed to the internet without password protection (and anyone can fire a malicious email at those addresses).
1 Like
Thanks for this useful thread. Here’s some of my RSS links that others might like to use.
Space
Train News
Weather
Tech
Most news sites also seem to support RSS, but only a title and relevant image without the rest of the story. Haven’t figured that one out yet.
1 Like
