Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@ with malformed feeds. `lib/simple-rss/xml_element.rb` holds shared XML tokenizat
and scoped element metadata. The optional normalized view lives in
`entry_normalizer.rb` and the format-independent `normalized_entry.rb` value object
under `lib/simple-rss/`. Preserve the existing raw parser and serialization
contracts when extending normalization.
contracts when extending normalization. `json_feed.rb` validates JSON Feed
structure and preserves the original document; `json_entry_normalizer.rb` maps
JSON fields into the same immutable entry type. Keep format-specific extraction
separate and use the existing JSON standard library dependency.

**Tag Syntax** (extend via `SimpleRSS.item_tags <<`):
- `tag` - simple element extraction
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ repository begins with a 1.1 import, so earlier releases are not reconstructed.

## Unreleased

- Parse JSON Feed 1.0 and 1.1 through `parse` and `fetch`, using the same
normalized entry interface as RSS/Atom. Support titleless and empty feeds,
inherited authors, opaque/numeric IDs, separate content and dates, tags, and
multiple attachments. Preserve original metadata and extensions in `raw_json`
and normalized entry `raw`; report recoverable date/number issues. Keep XML
serialization unchanged and reject JSON-to-XML conversion explicitly.
Fetch detects JSON regardless of Content-Type and retains conditional GET.
([#60](https://github.com/cardmagic/simple-rss/issues/60))

- Add `normalized_entries`, an optional immutable RSS/Atom view with consistent
article URLs, separate publication/update dates, typed content and summaries,
category terms, authors, and associated attachment metadata. Preserve raw
Expand Down
131 changes: 123 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
[![CI](https://github.com/cardmagic/simple-rss/actions/workflows/ruby.yml/badge.svg)](https://github.com/cardmagic/simple-rss/actions/workflows/ruby.yml)
[![License: LGPL](https://img.shields.io/badge/License-LGPL-blue.svg)](https://opensource.org/licenses/LGPL-3.0)

A simple, flexible, extensible, and liberal RSS and Atom reader for Ruby. Designed to be backwards compatible with Ruby's standard RSS parser while handling malformed feeds gracefully.
A simple, flexible, extensible, and liberal RSS, Atom, and JSON Feed reader for Ruby. Designed to be backwards compatible with Ruby's standard RSS parser while handling malformed feeds gracefully.

## Features

- Parses both RSS and Atom feeds
- Parses RSS, Atom, and JSON Feed 1.0/1.1
- Tolerant of malformed XML (regex-based parsing)
- Built-in URL fetching with conditional GET support (ETags, Last-Modified)
- JSON and XML serialization
Expand All @@ -27,7 +27,7 @@ The 2.x releases add:

- **JSON Serialization** - Export feeds with `to_json`, `to_hash`, and Rails-compatible `as_json`. Time objects serialize to ISO 8601.

- **XML Serialization** - Convert any parsed feed to clean RSS 2.0 or Atom XML with `to_xml(format: :rss2)` or `to_xml(format: :atom)`.
- **XML Serialization** - Convert parsed XML feeds to clean RSS 2.0 or Atom XML with `to_xml(format: :rss2)` or `to_xml(format: :atom)`.

- **Array Tags** - Collect all occurrences of a tag (like multiple categories) with the `array_tags:` option.

Expand Down Expand Up @@ -178,7 +178,7 @@ order, regardless of their dates.

### Normalized Entries

Use `normalized_entries` when an importer or digest should handle RSS and Atom
Use `normalized_entries` when an importer or digest should handle RSS, Atom, and JSON Feed
through the same fields:

```ruby
Expand Down Expand Up @@ -283,18 +283,133 @@ options raise `ArgumentError`.
For migration, replace format-specific expressions such as
`item[:link_alternate] || item[:link]` with `entry.url`, while retaining
`entry.raw` for existing custom fields. The runnable [digest example](examples/digest.rb)
reads either format without testing which one it received:
reads all three formats without testing which one it received:

```bash
ruby -Ilib examples/digest.rb test/data/normalized_rss.xml
ruby -Ilib examples/digest.rb test/data/normalized_atom.xml
ruby -Ilib examples/digest.rb test/data/json_feed_1_1.json
```

The mapping follows the [Atom specification](https://www.rfc-editor.org/rfc/rfc4287.html),
[RSS specification](https://www.rssboard.org/rss-specification), and
[XML Base rules](https://www.w3.org/TR/xmlbase/). It is a tolerant extraction view,
not a standards validator. JSON Feed parsing is tracked separately in
[#60](https://github.com/cardmagic/simple-rss/issues/60).
not a standards validator. JSON Feed has the separate rules below.

### JSON Feed Parsing

JSON Feed 1.0 and 1.1 use the same `parse`, IO, and `fetch` entry points:

```ruby
require "simple-rss"
require "json"

source = JSON.generate(
version: "https://jsonfeed.org/version/1.1",
title: "Example",
authors: [{ name: "Example Editor" }],
items: [{
id: "post:42",
url: "https://example.com/posts/42",
content_text: "Hello from JSON Feed",
date_published: "2026-09-12T10:00:00Z",
tags: ["ruby", "feeds"]
}]
)
feed = SimpleRSS.parse(source)
entry = feed.normalized_entries.first

feed.feed_type # => :json_feed
entry.identifier # => "post:42"
entry.content_text # => "Hello from JSON Feed"
entry.authors.first[:name] # => "Example Editor" (inherited)
entry.categories # => ["ruby", "feeds"]
entry.published_at # => a Time
entry.raw["id"] # => "post:42"

feed = SimpleRSS.fetch("https://example.com/feed.json", timeout: 10)
feed.next_url # Pagination metadata only; never fetched automatically
feed.raw_json # Frozen original JSON document, including extensions
```

`fetch` detects the response body independently of Content-Type and sends an
Accept header covering all three formats. Custom headers can override Accept.
ETag, Last-Modified, redirects, and `nil` for HTTP 304 work as for XML feeds.

| Normalized field | JSON Feed mapping |
| --- | --- |
| `identifier` | `id`, preserved as an opaque string; numeric IDs become strings |
| `url`, `external_url` | Separate permalink and linkblog destination; no ID fallback |
| `published_at`, `updated_at` | `date_published`, `date_modified`, parsed separately as RFC 3339 |
| `content_html`, `content_text` | Corresponding fields, without decoding HTML entities or deriving one from the other |
| `summary`, `summary_type` | Plain text `summary`, with type `:text` |
| `image`, `banner_image` | Corresponding image URLs |
| `categories`, `category_details` | Nonblank `tags`, trimmed and deduplicated for categories; duplicate details retained |
| `authors` | Item authors, otherwise feed authors; includes `name`, `url`, `avatar`, and raw metadata |
| `language` | 1.1 item language, otherwise feed language |
| `attachments` | All attachments with URL, MIME type as `media_type`, title, size, duration, and raw metadata |

In 1.0, authors come from singular `author`. In 1.1, `authors` takes precedence
over deprecated `author` within the same object; an item's authors take
precedence over the feed's. An explicit empty `authors` array prevents
inheritance. The later `authors` and `language` fields are retained as raw data
but not normalized in a 1.0 document. Matching attachment titles preserve the
publisher's grouping of alternate formats. `external_url`, `image`,
`banner_image`, and `language` are additive normalized fields; XML entries
currently return `nil` for these fields.

Relative JSON URLs resolve against the supplied/fetched `source_url`, or the
JSON `feed_url` when no source URL is supplied. Per-call `source_url` overrides
still work. Missing bases and invalid URLs remain inspectable through `issues`
and raw data, using the same issue codes as XML. `content_base_url` is the item
URL when available, otherwise the source URL or feed URL. Content links are not
rewritten; no articles, attachments, hubs, or pagination URLs are fetched.

Parsing requires a supported version, string feed title, an items array, and
objects with nonblank string/numeric IDs and at least one string content field.
Titleless items and empty feeds are supported. Malformed JSON, missing required
fields, malformed author/tag/attachment structures, and wrong known field types
raise `SimpleRSSError` with a field path. Invalid required item data rejects the
whole feed; items are never assigned invented IDs or returned partially parsed.
Invalid optional dates and attachment numbers are preserved in raw data and
reported in `issues`, with `nil` normalized values. Dates are never substituted
with the current time. `effective_at`, `latest`, and `items_since` safely use a
valid modification date when publication is invalid or absent.

UTF-8 strings and readable IO accept ordinary leading JSON whitespace and one
UTF-8 BOM at the very start, before whitespace. Embedded or repeated BOMs are
rejected. `source` preserves the original input. This is a parser, not a complete
standards validator: it does not validate URL reachability, language tags, ID
uniqueness across updates, or publisher extension schemas.
`SimpleRSS.valid?(source)` reports parseability. A parsed JSON feed's instance
`valid?` is true, including an empty feed. XML retains its historical distinction:
the class method accepts parseable empty feeds, but instance `valid?` requires
items and a title or link.

For JSON feeds, `raw_json` is an immutable snapshot of the entire decoded
document with string keys. Each normalized entry's `raw` is its original JSON
item, also with string keys; `raw_xml` is `nil`. Original numeric IDs, date
strings, nested data, and unknown fields remain intact. Normalized entries are
immutable snapshots based on the original JSON, so edits to `items` do not
rewrite their normalized fields. Reordering or deduplication retains the source
association; inserting an unrelated item raises an error during normalization.

`items` remains an array of hashes with symbol keys and dot access. It retains
JSON fields and adds compatibility aliases: string `id`/`guid`, `link`,
`description`, `content`, category arrays, publication/update dates, and the first
attachment's enclosure metadata. Nested original JSON values are frozen.
XML-only tag configuration and `array_tags` do not affect JSON; passing XML
`mappings` to JSON normalization raises `ArgumentError`.

`as_json`, `to_hash`, and `to_json` export that Ruby object view: original feed
metadata plus compatibility fields and the current items, with times converted
to ISO 8601. These operations preserve extensions but are **not JSON Feed
exporters**. Use `raw_json` to inspect the original document. XML serialization
behavior is unchanged; calling `to_xml` on a JSON feed raises `SimpleRSSError`
until a separate conversion contract exists.

These mappings follow the [JSON Feed 1.0 specification](https://www.jsonfeed.org/version/1/)
and [JSON Feed 1.1 specification](https://www.jsonfeed.org/version/1.1/).

### JSON Serialization

Expand All @@ -315,7 +430,7 @@ feed.as_json

### XML Serialization

Convert parsed feeds to standard RSS 2.0 or Atom format:
Convert parsed XML feeds to standard RSS 2.0 or Atom format:

```ruby
feed = SimpleRSS.parse(xml)
Expand Down
1 change: 1 addition & 0 deletions Steepfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ target :lib do
check "lib"

library "time"
library "date"
library "cgi"
library "json"
library "uri"
Expand Down
46 changes: 45 additions & 1 deletion lib/simple-rss.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ class SimpleRSS # rubocop:disable Metrics/ClassLength
# @rbs @items: Array[Hash[Symbol, untyped]]
# @rbs @source: String
# @rbs @options: Hash[Symbol, untyped]
# @rbs @json_feed: JsonFeed?
# @rbs @etag: String?
# @rbs @last_modified: String?
# @rbs @entry_contexts: Hash[Hash[Symbol, untyped], Hash[Symbol, untyped]]
Expand All @@ -24,6 +25,9 @@ class SimpleRSS # rubocop:disable Metrics/ClassLength
attr_reader :etag #: String?
attr_reader :last_modified #: String?
attr_reader :source_url #: String?
attr_reader :raw_json #: Hash[String, untyped]?
attr_reader :home_page_url, :feed_url, :favicon, :next_url, :user_comment #: String?
attr_reader :authors, :hubs, :expired #: untyped
alias entries items #: Array[Hash[Symbol, untyped]]

@@feed_tags = %i[
Expand Down Expand Up @@ -66,6 +70,8 @@ def initialize(source, options = {})
@options = {} #: Hash[Symbol, untyped]
@options.update(options)
@source_url = options[:source_url]
@json_feed = nil
@raw_json = nil
@entry_contexts = {} #: Hash[Hash[Symbol, untyped], Hash[Symbol, untyped]]
@entry_contexts.compare_by_identity

Expand All @@ -80,6 +86,13 @@ def channel

# @rbs (?source_url: String?, ?mappings: Hash[Symbol, untyped]) -> Array[NormalizedEntry]
def normalized_entries(source_url: nil, mappings: {})
json_feed = @json_feed
if json_feed
raise ArgumentError, "XML mappings are not supported for JSON Feed" unless mappings.empty?

return items.map { |item| json_feed.normalized_entry(item, source_url: source_url || @source_url) }
end
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Outdated

EntryNormalizer.validate_mappings(mappings)
feed_authors = normalized_feed_authors
items.map do |item|
Expand Down Expand Up @@ -117,6 +130,8 @@ def latest(count = 10)

# @rbs () -> Symbol
def feed_type
return :json_feed if @json_feed

atom_namespaced_feed = source.match?(/<(atom:)?feed\b[^>]*xmlns(:\w+)?=['"][^'"]*atom/i)
return :atom if atom_namespaced_feed
return :rss2 if source.match?(/<rss[^>]*version=['"]2/i)
Expand All @@ -128,6 +143,8 @@ def feed_type

# @rbs () -> bool
def valid?
return true if @json_feed

return false if items.empty?

title_value = instance_variable_get(:@title)
Expand Down Expand Up @@ -212,7 +229,8 @@ def images

# @rbs (?Hash[Symbol, untyped]) -> Hash[Symbol, untyped]
def as_json(_options = {})
hash = {} #: Hash[Symbol, untyped]
raw_json = @raw_json
hash = raw_json ? raw_json.transform_keys(&:to_sym) : {} #: Hash[Symbol, untyped]

@@feed_tags.each do |tag|
tag_cleaned = clean_tag(tag)
Expand All @@ -237,6 +255,8 @@ def to_json(*)

# @rbs (?format: Symbol) -> String
def to_xml(format: :rss2)
raise SimpleRSSError, "JSON Feed to XML conversion is not supported" if @json_feed

case format
when :rss2 then to_rss2_xml
when :atom then to_atom_xml
Expand Down Expand Up @@ -339,6 +359,7 @@ def build_http(uri, options)
# @rbs (untyped, Hash[Symbol, untyped]) -> untyped
def build_request(uri, options)
request = Net::HTTP::Get.new(uri)
request["Accept"] = "application/feed+json, application/rss+xml, application/atom+xml, application/json, application/xml, text/xml, */*"
request["User-Agent"] = "SimpleRSS/#{VERSION}"

# Conditional GET headers
Expand Down Expand Up @@ -386,6 +407,27 @@ def normalized_feed_authors

# @rbs () -> void
def parse
prefix = @source.b.sub(/\A\xEF\xBB\xBF/n, "").lstrip
return parse_xml unless prefix.match?(/\A(?:[\{\["0-9-]|true\b|false\b|null\b)/n)

json_feed = JsonFeed.new(@source)
@json_feed = json_feed
@raw_json = json_feed.document
JsonFeed::FEED_FIELDS.each do |field|
instance_variable_set("@#{field}", json_feed.document[field])
self.class.attr_reader(field)
end
@link = json_feed.document["home_page_url"]
self.class.attr_reader(:link)
@items = json_feed.items
@items.each do |item|
item.define_singleton_method(:method_missing) { |name, *_args| self[name] }
add_item_media_helpers(item)
end
end

# @rbs () -> void
def parse_xml
raise SimpleRSSError, "Poorly formatted feed" unless @source =~ %r{<(channel|feed).*?>.*?</(channel|feed)>}mi

# Feed's title and link
Expand Down Expand Up @@ -916,6 +958,8 @@ def unescape(content)

require_relative "simple-rss/xml_element"
require_relative "simple-rss/normalized_entry"
require_relative "simple-rss/json_feed"
require_relative "simple-rss/json_entry_normalizer"
require_relative "simple-rss/entry_normalizer"

class SimpleRSSError < StandardError # rubocop:disable Style/OneClassPerFile
Expand Down
Loading
Loading