Skip to content

Commit

Permalink
Updated feed endpoint implementation (#307)
Browse files Browse the repository at this point in the history 
refs https://linear.app/ghost/issue/AP-704

Updated the feed endpoint implementation to handle some edge cases discovered
when updating the client to utilise the endpoint and moves towards a separate
endpoint for feed and inbox, so that the implementation is not in the client.

Co-authored-by: Fabien O'Carroll <fabien@allou.is>
  • Loading branch information
@mike182uk @allouis
mike182uk and allouis authored Feb 11, 2025
1 parent 338fa2f commit ffeda18
Show file tree
Hide file tree
Showing 11 changed files with 635 additions and 83 deletions.
130 changes: 130 additions & 0 deletions features/feed.feature
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
Feature: Feed
In order to see posts from accounts I follow
As a user
I want to query my feed

Background:
Given an Actor "Person(Alice)"
And we follow "Alice"
And the request is accepted
And a "Accept(Follow(Alice))" Activity "Accept" by "Alice"
And "Alice" sends "Accept" to the Inbox
And "Accept" is in our Inbox

Scenario: Querying the inbox
Given a "Create(Article)" Activity "Article1" by "Alice"
And "Alice" sends "Article1" to the Inbox
And "Article1" is in our Inbox
And a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/inbox"
Then the request is accepted
And the feed contains "Article1"
And the feed does not contain "Note1"

Scenario: Querying the feed filtered by type: Note
Given a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
And a "Create(Article)" Activity "Article1" by "Alice"
And "Alice" sends "Article1" to the Inbox
And "Article1" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"
And the feed does not contain "Article1"

Scenario: Feed only includes posts
Given a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
And a "Create(Article)" Activity "Article1" by "Alice"
And "Alice" sends "Article1" to the Inbox
And "Article1" is in our Inbox
And a "Like(Note1)" Activity "Like1" by "Alice"
And "Alice" sends "Like1" to the Inbox
And "Like1" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"
And the feed does not contain "Like1"
When an authenticated request is made to "/.ghost/activitypub/inbox"
Then the request is accepted
And the feed contains "Article1"
And the feed does not contain "Like1"

Scenario: Feed is paginated
Given a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
And a "Create(Note)" Activity "Note2" by "Alice"
And "Alice" sends "Note2" to the Inbox
And "Note2" is in our Inbox
And a "Create(Note)" Activity "Note3" by "Alice"
And "Alice" sends "Note3" to the Inbox
And "Note3" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/feed?limit=2"
Then the request is accepted
And the feed contains "Note3"
And the feed contains "Note2"
And the feed does not contain "Note1"
And the feed has a next cursor
When an authenticated request is made to "/.ghost/activitypub/feed?limit=3"
Then the request is accepted
And the feed contains "Note1"

Scenario: Requests with limit over 100 are rejected
When an authenticated request is made to "/.ghost/activitypub/feed?limit=200"
Then the request is rejected with a 400

Scenario: Feed includes our own posts
When we create a note "Note1" with the content
"""
Hello World
"""
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"

Scenario: Feed includes posts we reposted
Given a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
And we repost the object "Note1"
And the request is accepted
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"

Scenario: Feed includes posts from followed accounts
Given a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"

Scenario: Feed includes reposts from followed accounts
Given an Actor "Person(Bob)"
And a "Note" Object "Note1" by "Bob"
And a "Announce(Note1)" Activity "Repost1" by "Alice"
And "Alice" sends "Repost1" to the Inbox
And "Repost1" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"

Scenario: Feed excludes replies
Given a "Create(Note)" Activity "Note1" by "Alice"
And "Alice" sends "Note1" to the Inbox
And "Note1" is in our Inbox
And a "Note" Object "Reply1" by "Alice"
And "Reply1" is a reply to "Note1"
And a "Create(Reply1)" Activity "ReplyCreate" by "Alice"
And "Alice" sends "ReplyCreate" to the Inbox
And "ReplyCreate" is in our Inbox
When an authenticated request is made to "/.ghost/activitypub/feed"
Then the request is accepted
And the feed contains "Note1"
And the feed does not contain "ReplyCreate"
48 changes: 48 additions & 0 deletions features/step_definitions/stepdefs.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1662,3 +1662,51 @@ Given('{string} has Object {string}', function (activityName, objectName) {

this.activities[activityName] = { ...activity, object };
});

When('we request the feed with the next cursor', async function () {
const responseJson = await this.response.clone().json();
const nextCursor = responseJson.next;

this.response = await fetchActivityPub(
`http://fake-ghost-activitypub/.ghost/activitypub/feed/index?next=${encodeURIComponent(nextCursor)}`,
{
headers: {
Accept: 'application/json',
},
},
);
});

Then('the feed contains {string}', async function (activityOrObjectName) {
const responseJson = await this.response.clone().json();
const activity = this.activities[activityOrObjectName];
const object = this.objects[activityOrObjectName];
let found;

if (activity) {
found = responseJson.posts.find(
(post) => post.id === activity.object.id,
);
} else if (object) {
found = responseJson.posts.find((post) => post.id === object.id);
}

assert(found, `Expected to find ${activityOrObjectName} in feed`);
});

Then('the feed does not contain {string}', async function (activityName) {
const responseJson = await this.response.clone().json();
const activity = this.activities[activityName];

const found = responseJson.posts.find(
(post) => post.id === activity.object.id,
);

assert(!found, `Expected not to find ${activityName} in feed`);
});

Then('the feed has a next cursor', async function () {
const responseJson = await this.response.clone().json();

assert(responseJson.next, 'Expected feed to have a next cursor');
});
14 changes: 12 additions & 2 deletions src/app.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import {
withContext,
} from '@logtape/logtape';
import * as Sentry from '@sentry/node';
import { PostType } from 'feed/types';
import { Hono, type Context as HonoContext, type Next } from 'hono';
import { cors } from 'hono/cors';
import jwt from 'jsonwebtoken';
Expand Down Expand Up @@ -846,7 +847,7 @@ app.get(
spanWrapper(handleGetProfilePosts),
);
app.get(
'/.ghost/activitypub/thread/:activity_id',
'/.ghost/activitypub/thread/:object_id',
spanWrapper(handleGetActivityThread),
);
app.get(
Expand All @@ -862,7 +863,16 @@ app.get(
app.get(
'/.ghost/activitypub/feed',
requireRole(GhostRole.Owner),
spanWrapper(createGetFeedHandler(feedService, accountService)),
spanWrapper(
createGetFeedHandler(feedService, accountService, PostType.Note),
),
);
app.get(
'/.ghost/activitypub/inbox',
requireRole(GhostRole.Owner),
spanWrapper(
createGetFeedHandler(feedService, accountService, PostType.Article),
),
);
/** Federation wire up */

Expand Down
2 changes: 2 additions & 0 deletions src/constants.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ export const ACTOR_DEFAULT_NAME = 'Local Ghost site';
export const ACTOR_DEFAULT_ICON = 'https://ghost.org/favicon.ico';
export const ACTOR_DEFAULT_SUMMARY = 'This is a summary';

export const ACTIVITY_TYPE_ANNOUNCE = 'Announce';
export const ACTIVITY_TYPE_CREATE = 'Create';
export const ACTIVITY_OBJECT_TYPE_ARTICLE = 'Article';
export const ACTIVITY_OBJECT_TYPE_NOTE = 'Note';

Expand Down
55 changes: 55 additions & 0 deletions src/db.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,51 @@ export async function getActivityMeta(
return map;
}

// This is a variant of getActivityMeta that does not use a join on itself in
// order to also fetch replies. This is fixes a long standing issue where replies
// are not correctly being fetched. This is used by the new feed endpoint which
// does not need to fetch replies. Why not just update getActivityMeta? That
// method is used by the notifications section of the client which needs replies
// and somehow works around its quirks of them not being correctly fetched :s
// Seeming though this is method is only going to be temporary until we have the
// posts table, I thought it would be easier to do this instead of updating
// getActivityMeta and potentially breaking notifications
export async function getActivityMetaWithoutJoin(
uris: string[],
): Promise<Map<string, ActivityMeta>> {
const results = await client
.select(
'key',
'id',
client.raw('JSON_EXTRACT(value, "$.actor.id") as actor_id'),
client.raw('JSON_EXTRACT(value, "$.type") as activity_type'),
client.raw('JSON_EXTRACT(value, "$.object.type") as object_type'),
client.raw(
'COALESCE(JSON_EXTRACT(value, "$.object.inReplyTo.id"), JSON_EXTRACT(value, "$.object.inReplyTo")) as reply_object_url',
),
)
.from('key_value')
.whereIn(
'key',
uris.map((uri) => `["${uri}"]`),
);

const map = new Map<string, ActivityMeta>();

for (const result of results) {
map.set(result.key.substring(2, result.key.length - 2), {
id: result.id,
actor_id: result.actor_id,
activity_type: result.activity_type,
object_type: result.object_type,
reply_object_url: result.reply_object_url,
reply_object_name: '',
});
}

return map;
}

export async function getActivityChildren(activity: ActivityJsonLd) {
const objectId = activity.object.id;

Expand Down Expand Up @@ -203,3 +248,13 @@ export async function getActivityParents(activity: ActivityJsonLd) {

return parents;
}

export async function getActivityForObject(objectId: string) {
const result = await client
.select('value')
.from('key_value')
.where(client.raw(`JSON_EXTRACT(value, "$.object.id") = "${objectId}"`))
.andWhere(client.raw(`JSON_EXTRACT(value, "$.type") = "Create"`));

return result[0].value;
}
69 changes: 42 additions & 27 deletions src/feed/feed.service.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ import type { FedifyRequestContext } from '../app';
import {
ACTIVITY_OBJECT_TYPE_ARTICLE,
ACTIVITY_OBJECT_TYPE_NOTE,
ACTIVITY_TYPE_ANNOUNCE,
ACTIVITY_TYPE_CREATE,
} from '../constants';
import { getActivityMeta } from '../db';
import { getActivityMetaWithoutJoin } from '../db';
import { type Activity, buildActivity } from '../helpers/activitypub/activity';
import { spanWrapper } from '../instrumentation';
import { PostType } from './types';
Expand Down Expand Up @@ -59,32 +61,45 @@ export class FeedService {

let activityRefs = [...inboxRefs, ...outboxRefs];

// Filter the activityRefs by the provided post type
const activityMeta = await getActivityMeta(activityRefs);
activityRefs = activityRefs
// If we can't find the meta data in the database for an activity, we
// skip it as this is unexpected
.filter((ref) => activityMeta.has(ref))
// Filter the activityRefs by the provided post type if provided. If
// no post type is provided, we include all articles and notes
.filter((ref) => {
const meta = activityMeta.get(ref);

if (options.postType === null) {
return [
ACTIVITY_OBJECT_TYPE_ARTICLE,
ACTIVITY_OBJECT_TYPE_NOTE,
].includes(meta!.object_type);
}

if (options.postType === PostType.Article) {
return meta!.object_type === ACTIVITY_OBJECT_TYPE_ARTICLE;
}

if (options.postType === PostType.Note) {
return meta!.object_type === ACTIVITY_OBJECT_TYPE_NOTE;
}
});
const activityMeta = await getActivityMetaWithoutJoin(activityRefs);
activityRefs = activityRefs.filter((ref) => {
const meta = activityMeta.get(ref);

// If we can't find the meta data in the database for an activity,
// we skip it as this is unexpected
if (!meta) {
return false;
}

// The feed should only contain Create and Announce activities
if (
meta.activity_type !== ACTIVITY_TYPE_CREATE &&
meta.activity_type !== ACTIVITY_TYPE_ANNOUNCE
) {
return false;
}

// The feed should not contain replies
if (meta.reply_object_url !== null) {
return false;
}

// Filter by the provided post type
if (options.postType === null) {
return [
ACTIVITY_OBJECT_TYPE_ARTICLE,
ACTIVITY_OBJECT_TYPE_NOTE,
].includes(meta!.object_type);
}

if (options.postType === PostType.Article) {
return meta!.object_type === ACTIVITY_OBJECT_TYPE_ARTICLE;
}

if (options.postType === PostType.Note) {
return meta!.object_type === ACTIVITY_OBJECT_TYPE_NOTE;
}
});

// Sort the activity refs by the latest first (yes using the ID which
// is totally gross but we have no other option at the moment)
Expand Down
Loading

0 comments on commit ffeda18

Please sign in to comment.