How to crawl RSS feeds
Posted on: October 13, 2023
RSS stands for Rich site summary and this used to publish frequently updated content. RSS feed is an essential file in xml format. The content in rss feed mainly title, headlines, links and summaries.
We are going to use BeautifulSoup library along with the requests to crawl and parse rss feeds
Install libraries:
pip install requests beautifulsoup4
https://feeds.feedburner.com/gadgets360-latest.xml using this rss feed to crawl. Replace with required url of rss feed you want to crawl.
What we are crawling from RSS feed:
title, url, summary and updated at
Fetch and Parse RSS Feeds:
import pdb
import requests
from bs4 import BeautifulSoup
url = "https://feeds.feedburner.com/gadgets360-latest.xml"
rqs = requests.get(url)
#beautiful soup documentation https://www.crummy.com/software/BeautifulSoup/bs4/doc/
soup = BeautifulSoup(rqs.content, 'xml').select("item")
data = []
for article in soup:
title = article.find("title").get_text()
link = article.find("link").get_text()
pubDate = article.find("pubDate").get_text()
description = article.find("description").get_text()
content = article.find("content:encoded").get_text().strip()
data.append(
{"title": title, "link": link, "pubDate": pubDate,
"description": description, "content": content}
)
print(data)
Process data:
Now we are displaying data in console itself with array of dict. This may include extracting specific information from the feed, saving it to a database, or performing any other relevant actions.
Error handling:
Make sure to implement error handling in case the RSS feed is unavailable or the XML is malformed.
Scheduling:
If you want to continuously crawl and update RSS feeds, you can use scheduling tools like cron jobs or Windows Task Scheduler to run your script at specific intervals.
Extracting large data:
To extract large data first need to gather the feed urls from source. Now we taken only single feeds related to gadgets but there are many feeds related news, top stories, sports ... etc. Extracting millions of news articles can be a daunting task, but craw feeds can make it easier.
Demo data:
[
{'title': 'Vivo Y200 5G Launch Timeline, Specifications Teased; Could Debut by October End', 'link': 'https://www.gadgets360.com/mobiles/news/vivo-y200-5g-launch-timeline-specifications-teased-could-debut-october-end-4479863', 'pubDate': 'Sat, 14 Oct 2023 08:23:07 +0530', 'description': 'Vivo Y200 5G is expected to be priced under Rs. 24,000 in India', 'content': '<p><a href="https://www.gadgets360.com/vivo-y100-price-in-india-115909">Vivo Y100</a> was launched in India in February this year, sporting a 6.38-inch AMOLED display with a 90Hz refresh rate. The Chinese smartphone company is expected to launch its successor — the Vivo Y200 5G — soon. The upcoming smartphone was reportedly spotted on Google Play Console a few days back. Now, a new report has unveiled the specifications, launch timeline, and price range of the smartphone. Vivo also teased the smartphone recently, hinting at its launch in the coming days. </p><p>A new <a href="https://www.mysmartprice.com/gear/vivo-y200-5g-specifications-launch-timeline-price-range-revealed-exclusive/">report</a> by MySmartPrice has cited industry sources to reveal the Vivo Y200 5G launch timeline, which could debut in India by the end of this month. Moreover, it is expected to be priced under Rs. 24,000 in India. For specifications, the smartphone is said to feature an AMOLED display. The report also suggested that the smartphone could get a 64-megapixel primary camera with OIS support.</p><p>A previous report by The Tech Outlook also <a href="http://www.thetechoutlook.com/news/new-release/gadgets-release/mobile/exclusive-source-vivo-y200-key-specifications-leaked-via-training-materials-launching-next-month/">revealed</a> some specifications about the Vivo Y200 5G, hinting that the device could be powered by a Snapdragon 4 Gen 1 SoC paired with 8GB RAM. For dimply, it is said to feature a 6.67-inch screen and could run on Android 13-based Funtouch OS 13 out-of-the-box. It is also expected to pack a 4,800mAh battery with 44W fast charging support. </p><p>The smartphone was <a href="https://www.mysmartprice.com/gear/vivo-y200-5g-google-play-console-listing-render-specifications-revealed/">spotted</a> on Google Play Console, as reported by MySmartPrice. The details on the site corroborated the specifications mentioned above. Vivo Y200 5G was shown to sport a triple-camera setup, while on the front, the device could feature a punch-hole cutout at the centre of the screen. </p><p>The company has already confirmed the launch of Vivo Y200 5G soon. In an X <a href="https://twitter.com/Vivo_India/status/1711343304384856187?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1711343304384856187%7Ctwgr%5Ed9a8803d25a0cc20b7eb558192391c1f8a87b001%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.mysmartprice.com%2Fgear%2Fvivo-y200-india-launch-officially-teased%2F">post</a>, the smartphone was teased to be \'coming soon\'. However, the company did not mention the launch date for the smartphone. </p><p><iframe allowtransparency="true" frameborder="0" height="500" scrolling="no" src="https://ndtv.com/video/embed-player/category/embed/site/classic/id/724967/autostart/0/pWidth/100/pHeight/100" width="100%"></iframe></p><hr /><div><em>Is the iQoo Neo 7 Pro the best smartphone you can buy under Rs. 40,000 in India? We discuss the company\'s recently launched handset and what it has to offer on the latest episode of <a href="https://www.gadgets360.com/tags/orbital">Orbital</a>, the Gadgets 360 podcast. Orbital is available on <a href="https://open.spotify.com/show/1TQIHsTESxrsHleJVhUKcr" rel="nofollow" target="_blank">Spotify</a>, <a href="https://gaana.com/podcast/orbital-season-1" rel="nofollow" target="_blank">Gaana</a>, <a href="https://www.jiosaavn.com/shows/orbital/1/r9VmNAJ211U" rel="nofollow" target="_blank">JioSaavn</a>, <a href="https://podcasts.google.com/feed/aHR0cHM6Ly9jZG4uZ2FkZ2V0czM2MC5jb20vY29udGVudC9kYXRhL3htbC9vcmJpdGFsLnhtbA" rel="nofollow" target="_blank">Google Podcasts</a>, <a href="https://itunes.apple.com/in/podcast/orbital-gadgets-360-podcast/id1121036073" rel="nofollow" target="_blank">Apple Podcasts</a>, <a href="https://music.amazon.in/podcasts/e3a8ca64-3005-4427-a89e-b3e8ec828c3b/Orbital" rel="nofollow" target="_blank">Amazon Music</a> and wherever you get your podcasts.</em><br /><iframe allow="autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture" allowfullscreen="" frameborder="0" height="152" loading="lazy" src="https://open.spotify.com/embed/episode/1WiXEJYLmG1tnG7dN1HvAK?utm_source=generator" style="border-radius:12px" width="100%"></iframe></div>'},
{'title': 'Google Pixel 8 Pro Tops DxOMark Display Test Ranking', 'link': 'https://www.gadgets360.com/mobiles/news/google-pixel-8-pro-tops-dxomark-display-test-ranking-4479130', 'pubDate': 'Sat, 14 Oct 2023 08:21:06 +0530', 'description': 'Pixel 8 Pro was launched on October 4, while it went on sale in India on October 12', 'content': '<p><a href="https://www.gadgets360.com/google-pixel-8-price-in-india-120540">Google Pixel 8</a> was launched earlier this month along with <a href="https://www.gadgets360.com/google-pixel-8-pro-price-in-india-120544">Pixel 8 Pro</a>. The smartphones come with dual-SIM support and run on Android 14 out-of-the-box. The Pro variant of the Pixel 8 series houses a 6.7-inch Quad-HD (1,344x2,992 pixels) resolution and a 120Hz refresh rate. The smartphone has also recently performed well on DxOMark\'s <a href="https://www.dxomark.com/smartphones/Google/Pixel-8-Pro#display">display rankings</a>, securing the top position. From its smooth touch to its viewing experience, the Pixel 8 Pro has scored high in all departments on the ranking website. </p><p>A now-removed report from PhoneArena (via <a href="https://www.gizmochina.com/2023/10/13/google-pixel-8-pro-dxomark-display-tests/">Gizmochina</a>) cited Google Pixel 8 Pro\'s top position in the DxOMark\'s display ranking. As per the official website of the rankings, the smartphone offers good screen readability in all environments, with pleasant screen colour and solid HDR video experience. The Pixel 8 Pro has also been appreciated for its smooth web browsing and quality sound from built-in speakers. The website also mentioned one con of the latest smartphone, which includes limited audio zoom performance.</p><p><strong>Also read:</strong> <a href="https://www.gadgets360.com/mobiles/reviews/google-pixel-8-and-pixel-8-pro-first-impressions-price-in-india-flipkart-4471814">Google Pixel 8 and Pixel 8 Pro First Impressions: Going Big on AI</a></p><p>The smartphone has secured a top display score of 164, while the audio score is 142. The Pixel 8 Pro was <a href="https://www.gadgets360.com/mobiles/news/google-pixel-8-pixel-8-pro-price-in-india-specifications-features-4449112">launched</a> on October 4, and it went on sale in India on October 12. It is priced at Rs. 1,06,999 for the lone 12GB RAM + 128GB storage model, available in Bay, Obsidian, and Porcelain colour options.</p><p>It runs on Android 14 out-of-the-box and features a 6.7-inch Quad-HD (1344x2992 pixels) resolution. It is powered by Google\'s nona-core Tensor G3 SoC and the Titan M2 security chip. The Pixel 8 Pro packs a 5,050mAh battery with support for 30W wired charging.</p><p>For optics, the Pixel 8 Pro houses a 50-megapixel primary camera with a Samsung GN2 sensor with an f/1.68 aperture, along with a 64-megapixel ultra-wide-angle camera. There is a third 48-megapixel telephoto camera with a Samsung GM5 sensor. For selfies, the smartphone gets an 11-megapixel camera. </p><p><iframe allowtransparency="true" frameborder="0" height="500" scrolling="no" src="https://ndtv.com/video/embed-player/category/embed/site/classic/id/729995/autostart/0/pWidth/100/pHeight/100" width="100%"></iframe></p><hr /><div><em>From the launch of the Infinix GT 10 Pro to Amazon\'s latest mega-sale, we discuss the most noteworthy technology news events of the week on the latest episode of <a href="https://www.gadgets360.com/tags/orbital">Orbital</a>, the Gadgets 360 podcast. Orbital is available on <a href="https://open.spotify.com/show/1TQIHsTESxrsHleJVhUKcr" rel="nofollow" target="_blank">Spotify</a>, <a href="https://gaana.com/podcast/orbital-season-1" rel="nofollow" target="_blank">Gaana</a>, <a href="https://www.jiosaavn.com/shows/orbital/1/r9VmNAJ211U" rel="nofollow" target="_blank">JioSaavn</a>, <a href="https://podcasts.google.com/feed/aHR0cHM6Ly9jZG4uZ2FkZ2V0czM2MC5jb20vY29udGVudC9kYXRhL3htbC9vcmJpdGFsLnhtbA" rel="nofollow" target="_blank">Google Podcasts</a>, <a href="https://itunes.apple.com/in/podcast/orbital-gadgets-360-podcast/id1121036073" rel="nofollow" target="_blank">Apple Podcasts</a>, <a href="https://music.amazon.in/podcasts/e3a8ca64-3005-4427-a89e-b3e8ec828c3b/Orbital" rel="nofollow" target="_blank">Amazon Music</a> and wherever you get your podcasts.</em><br /><iframe allow="autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture" allowfullscreen="" frameborder="0" height="152" loading="lazy" src="https://open.spotify.com/embed/episode/6z5xMJzCyQw7yNIu1T5Vek?utm_source=generator" style="border-radius:12px" width="100%"></iframe></div>'}
]
Remember to respect the terms of use and licensing agreements for the RSS feeds you crawl, as some websites may have restrictions on how their content can be used.
If you need to crawl multiple feeds or to extract data in bulk or require more advanced features, you might want to explore other Python libraries and frameworks like Scrapy for more robust web scraping and crawling capabilities.
Find a right dataset that you are looking for from crawl feeds store.
Submit data request if not able to find right dataset.
Custom request