On the fly OPML in Feeds

I’ve recently enjoyed following Terry on his quest to make RSS more accessible through thoughtful products. One of the latest is Sourcefeed, an RSS only publishing platform. You make an RSS feed, and instead of seeing a list of posts on that person’s page, you just get a link to the RSS feed. This way you can write content directly to someone else’s reader as long as they’re following you.

One of the recent updates was an OPML file for all of Sourcefeed’s users. For those who don’t know, OPML is a file format to help transport a list of different RSS feeds you can follow. I really wanted to view some of the posts from these feeds without subscribing to all of them just yet. I already have a feature in Feeds which does something similar where I can add ?url=https://someperson.com/rss.xml to the hosted instance of my feeds app and preview that feed of posts without subscribing directly.

I decided to add the same thing with OPML, although this was a bit more complicated. I needed to fetch the list of feeds, fetch a few posts from each feed, then pipe them into an order list of posts so I could preview them. If the list gets pretty long, then I could be waiting a hot minute before it would be ready due to how many requests my server is making to grab each feed. Thankfully, there’s a pretty sick solution.

Feeds was recently migrated to Go instead of Rust (more on that in a future post), and one cool feature that’s super accessible is Go routines. These enable concurrency for multiple tasks, and the ability to craft how they are completed and managed. In my use case of needing to fetch all the feeds in the OPML file, Go can fetch them all at once and then organize the results. The code looks something like the following.

func previewURLs(ctx context.Context, urls []string, perFeed int, log *slog.Logger) []FeedPreviewItem {
	var wg sync.WaitGroup
	var mu sync.Mutex
	items := []FeedPreviewItem{}
	for _, raw := range urls {
		feedURL := strings.TrimSpace(raw)
		if feedURL == "" {
			continue
		}
		wg.Add(1)
		go func() {
			defer wg.Done()
			res, err := fetchFeed(ctx, feedURL, "", "")
			if err != nil {
				log.Warn("preview fetch failed", "url", feedURL, "err", err)
				return
			}
			feedTitle := res.Title
			local := make([]FeedPreviewItem, 0, len(res.Entries))
			for _, entry := range res.Entries {
				if perFeed > 0 && len(local) >= perFeed {
					break
				}
				author := feedTitle
				if entry.Author != "" && feedTitle != "" {
					author = feedTitle + " - " + entry.Author
				} else if entry.Author != "" {
					author = entry.Author
				}
				local = append(local, FeedPreviewItem{Title: entry.Title, Link: entry.Link, Author: author, Published: entry.PublishedAt})
			}
			mu.Lock()
			items = append(items, local...)
			mu.Unlock()
		}()
	}
	wg.Wait()
	slices.SortFunc(items, func(a, b FeedPreviewItem) int {
		switch {
		case a.Published > b.Published:
			return -1
		case a.Published < b.Published:
			return 1
		default:
			return 0
		}
	})
	return items
}

The result is being able to preview an OPML file with 45 feeds in less than a second; not bad!

Really happy with how this turned out and how these little programs continue to grow. If you’re interested in the source code you can find it here!