Refactor appview media embeds (bluesky-social#1343)

* refactor appview services * tidy types * Allow moderators to take and reverse actor takedowns (bluesky-social#1330) allow moderators to take and reverse actor takedowns * ✨ Allow searching reports by moderator did (bluesky-social#1283) * ✨ Allow searching reports by moderator did * ✅ Remove .only flags on tests * ✅ Update snapshot * ✅ Add checks for did match in actions * v0.4.1 * Make sequencer leader behavior optional on pds (bluesky-social#1250) * make sequencer leader behavior optional on pds * tidy * use 127.0.0.1 in with-test-db.sh for colima (bluesky-social#1297) So, since Docker Desktop has licensing issues, some folks use colima for running containers on their macOS machines (The licensing exempted CLI-only version of Docker only exists on Linux). Unfortunately, colima binds host ports only on the IPv4 localhost address (`127.0.0.1`) while the atproto postgres clients will attempt to connect to the IPv6 localhost address (`::1`) that macOS sets in /etc/hosts. See abiosoft/colima#583 and lima-vm/lima#1330 for the tickets against colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses and so doesn't have this issue.) To workaround this silly issue, we can use `localhost` within the docker containers and docker-compose, but need to set the `DB_POSTGRES_URL` env var to use the IPv4 localhost explicitly. (Asking folks to edit /etc/hosts causes other tools to break and will be overridden on each OS upgrade.) * Subscription util tests (bluesky-social#1295) * consecutive list tests * flesh out subscription util tests --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> * Content reporting on record fields (bluesky-social#1351) * content reporting on record fields * fix test * tests * tidy * Check rkey contents just for non-tids (bluesky-social#1353) check rkey content for non-tids * ✨ Added new procedure for sending admin email (bluesky-social#1312) * 🚧 Added new lexicon for sending admin email * ✨ Add moderation mailer * ✨ Switch to text email content from html * 🧹 Cleanup some early implementation code and reflect PR reivew * ✨ Use smtp host instead of gmail service config * ✨ Move to using single smtp url * v0.4.2 * Patch up a couple sqlite tests (bluesky-social#1355) patch up a couple sqlite tests * Increase appview keepalive (bluesky-social#1360) increase keepalive to 90s * Appview - serve feed skeletons (bluesky-social#1265) * proxy timeline skeleton construction to appview * add getFeedSkeleton to appview * mount route * smart proxy feed skeletons * tests * proper error code * only proxy specific feeds * build branch * update proxyable feed logic, should use feed publisher rather than generator * fix feed proxy tests, configure feed publisher (in addition to generator) * hotfix: prevent user-supplied rkey on posts with createRecord (bluesky-social#1313) * prevent user-supplied rkey on posts with createRecord * allow empty-string rkey parameter Co-authored-by: devin ivy <devinivy@gmail.com> --------- Co-authored-by: devin ivy <devinivy@gmail.com> * add slurs to reserved words (bluesky-social#1318) * add slurs to reserved words (bluesky-social#1314) * Update reserved.ts Add slurs to reserved words * Update reserved.ts fix typo * Update reserved.ts to clean up the slur list * linting * pluralise --------- Co-authored-by: jess <jesslarkdev+gh@gmail.com> * identifier: tweaks and additions to slur list (bluesky-social#1319) * Refactor appview repo subscription for memleak (bluesky-social#1308) * refactor to remove closure in loop * move consecutive item out of p-queue * Handle validation improvements (bluesky-social#1336) * Handle matches and false positives for unacceptable words in handles * move handle validation logic to pds * missed merge * add cfg flag * encode lists * fix build issues * move words to cfg * tidy --------- Co-authored-by: Jaz Volpert <ericvolp12@gmail.com> * Allow moderators to take and reverse actor takedowns (bluesky-social#1330) allow moderators to take and reverse actor takedowns * ✨ Allow searching reports by moderator did (bluesky-social#1283) * ✨ Allow searching reports by moderator did * ✅ Remove .only flags on tests * ✅ Update snapshot * ✅ Add checks for did match in actions * v0.4.1 * Make sequencer leader behavior optional on pds (bluesky-social#1250) * make sequencer leader behavior optional on pds * tidy * use 127.0.0.1 in with-test-db.sh for colima (bluesky-social#1297) So, since Docker Desktop has licensing issues, some folks use colima for running containers on their macOS machines (The licensing exempted CLI-only version of Docker only exists on Linux). Unfortunately, colima binds host ports only on the IPv4 localhost address (`127.0.0.1`) while the atproto postgres clients will attempt to connect to the IPv6 localhost address (`::1`) that macOS sets in /etc/hosts. See abiosoft/colima#583 and lima-vm/lima#1330 for the tickets against colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses and so doesn't have this issue.) To workaround this silly issue, we can use `localhost` within the docker containers and docker-compose, but need to set the `DB_POSTGRES_URL` env var to use the IPv4 localhost explicitly. (Asking folks to edit /etc/hosts causes other tools to break and will be overridden on each OS upgrade.) * Subscription util tests (bluesky-social#1295) * consecutive list tests * flesh out subscription util tests --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> * Content reporting on record fields (bluesky-social#1351) * content reporting on record fields * fix test * tests * tidy * Check rkey contents just for non-tids (bluesky-social#1353) check rkey content for non-tids * ✨ Added new procedure for sending admin email (bluesky-social#1312) * 🚧 Added new lexicon for sending admin email * ✨ Add moderation mailer * ✨ Switch to text email content from html * 🧹 Cleanup some early implementation code and reflect PR reivew * ✨ Use smtp host instead of gmail service config * ✨ Move to using single smtp url * v0.4.2 * Patch up a couple sqlite tests (bluesky-social#1355) patch up a couple sqlite tests * enable feeds & build branch * disable branch building & enable without proxy header --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> Co-authored-by: David Buchanan <d@vidbuchanan.co.uk> Co-authored-by: jess <jesslarkdev+gh@gmail.com> Co-authored-by: bnewbold <bnewbold@robocracy.org> Co-authored-by: Jaz Volpert <ericvolp12@gmail.com> Co-authored-by: Foysal Ahamed <cfaion341@gmail.com> Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com> * add optional ilike query to getPopularFeedGenerators * fix lint * fix lint * handle sqlite too in pds only * fix lint * v0.4.3 * Include takendown posts for admins (feature branch) (bluesky-social#1361) * 🚧 WIP including takendown posts on author feed * ✨ Add takedown id on posts when including taken down posts * 🧹 Cleanup the auth verifier and other bsky package code * ✅ Add test for admin getAuthorFeed * 🧹 Cleanup lexicon and exclude takedownId * more explicit plumbing for post hydration w/o requester or with takedown info * pass along flag for soft-deleted actors * cleanup getAuthorFeed w/ auth * reorg getAuthorFeed logic around role/access-based auth --------- Co-authored-by: Foysal Ahamed <cfaion341@gmail.com> * ✨ Disable signing up with invite code from takendown account (bluesky-social#1350) * fix viewer data --------- Co-authored-by: Devin Ivy <devinivy@gmail.com> Co-authored-by: Foysal Ahamed <cfaion341@gmail.com> Co-authored-by: Jeff Hodges <jeff@somethingsimilar.com> Co-authored-by: David Buchanan <d@vidbuchanan.co.uk> Co-authored-by: jess <jesslarkdev+gh@gmail.com> Co-authored-by: bnewbold <bnewbold@robocracy.org> Co-authored-by: Jaz Volpert <ericvolp12@gmail.com> Co-authored-by: Eric Bailey <git@esb.lol>
mloar · Jul 24, 2023 · adf69c7 · adf69c7
1 parent 6c03374
commit adf69c7
Show file tree

Hide file tree

Showing 15 changed files with 537 additions and 383 deletions.
diff --git a/packages/bsky/src/api/app/bsky/feed/getActorFeeds.ts b/packages/bsky/src/api/app/bsky/feed/getActorFeeds.ts
@@ -39,9 +39,13 @@ export default function (server: Server, ctx: AppContext) {
       ])
       const profiles = { [creatorProfile.did]: creatorProfile }
 
-      const feeds = feedsRes.map((row) =>
-        feedService.views.formatFeedGeneratorView(row, profiles),
-      )
+      const feeds = feedsRes.map((row) => {
+        const feed = {
+          ...row,
+          viewer: viewer ? { like: row.viewerLike } : undefined,
+        }
+        return feedService.views.formatFeedGeneratorView(feed, profiles)
+      })
 
       return {
         encoding: 'application/json',

diff --git a/packages/bsky/src/api/app/bsky/feed/getFeed.ts b/packages/bsky/src/api/app/bsky/feed/getFeed.ts
@@ -16,7 +16,7 @@ import { QueryParams as GetFeedParams } from '../../../../lexicon/types/app/bsky
 import { OutputSchema as SkeletonOutput } from '../../../../lexicon/types/app/bsky/feed/getFeedSkeleton'
 import { Server } from '../../../../lexicon'
 import AppContext from '../../../../context'
-import { FeedRow } from '../../../../services/types'
+import { FeedRow } from '../../../../services/feed/types'
 import { AlgoResponse } from '../../../../feed-gen/types'
 
 export default function (server: Server, ctx: AppContext) {

diff --git a/packages/bsky/src/api/app/bsky/feed/getFeedGenerator.ts b/packages/bsky/src/api/app/bsky/feed/getFeedGenerator.ts
@@ -16,7 +16,7 @@ export default function (server: Server, ctx: AppContext) {
 
       const feedService = ctx.services.feed(ctx.db)
 
-      const got = await feedService.getFeedGeneratorViews([feed], viewer)
+      const got = await feedService.getFeedGeneratorInfos([feed], viewer)
       const feedInfo = got[feed]
       if (!feedInfo) {
         throw new InvalidRequestError('could not find feed')
@@ -45,7 +45,7 @@ export default function (server: Server, ctx: AppContext) {
         )
       }
 
-      const profiles = await feedService.getActorViews(
+      const profiles = await feedService.getActorInfos(
         [feedInfo.creator],
         viewer,
       )

diff --git a/packages/bsky/src/api/app/bsky/feed/getFeedGenerators.ts b/packages/bsky/src/api/app/bsky/feed/getFeedGenerators.ts
@@ -10,11 +10,11 @@ export default function (server: Server, ctx: AppContext) {
 
       const feedService = ctx.services.feed(ctx.db)
 
-      const genViews = await feedService.getFeedGeneratorViews(feeds, requester)
-      const genList = Object.values(genViews)
+      const genInfos = await feedService.getFeedGeneratorInfos(feeds, requester)
+      const genList = Object.values(genInfos)
 
       const creators = genList.map((gen) => gen.creator)
-      const profiles = await feedService.getActorViews(creators, requester)
+      const profiles = await feedService.getActorInfos(creators, requester)
 
       const feedViews = genList.map((gen) =>
         feedService.views.formatFeedGeneratorView(gen, profiles),

diff --git a/packages/bsky/src/api/app/bsky/feed/getPostThread.ts b/packages/bsky/src/api/app/bsky/feed/getPostThread.ts
@@ -2,12 +2,11 @@ import { InvalidRequestError } from '@atproto/xrpc-server'
 import { Server } from '../../../../lexicon'
 import AppContext from '../../../../context'
 import {
-  ActorViewMap,
-  FeedEmbeds,
   FeedRow,
-  PostInfoMap,
-} from '../../../../services/types'
-import { FeedService } from '../../../../services/feed'
+  ActorInfoMap,
+  PostEmbedViews,
+} from '../../../../services/feed/types'
+import { FeedService, PostInfoMap } from '../../../../services/feed'
 import { Labels } from '../../../../services/label'
 import {
   BlockedPost,
@@ -41,14 +40,14 @@ export default function (server: Server, ctx: AppContext) {
         throw new InvalidRequestError(`Post not found: ${uri}`, 'NotFound')
       }
       const relevant = getRelevantIds(threadData)
-      const [actors, posts, embeds, labels] = await Promise.all([
-        feedService.getActorViews(Array.from(relevant.dids), requester, {
+      const [actors, posts, labels] = await Promise.all([
+        feedService.getActorInfos(Array.from(relevant.dids), requester, {
           skipLabels: true,
         }),
-        feedService.getPostViews(Array.from(relevant.uris), requester),
-        feedService.embedsForPosts(Array.from(relevant.uris), requester),
+        feedService.getPostInfos(Array.from(relevant.uris), requester),
         labelService.getLabelsForSubjects([...relevant.uris, ...relevant.dids]),
       ])
+      const embeds = await feedService.embedsForPosts(posts, requester)
 
       const thread = composeThread(
         threadData,
@@ -76,8 +75,8 @@ const composeThread = (
   threadData: PostThread,
   feedService: FeedService,
   posts: PostInfoMap,
-  actors: ActorViewMap,
-  embeds: FeedEmbeds,
+  actors: ActorInfoMap,
+  embeds: PostEmbedViews,
   labels: Labels,
 ) => {
   const post = feedService.views.formatPostView(

diff --git a/packages/bsky/src/api/app/bsky/feed/getPosts.ts b/packages/bsky/src/api/app/bsky/feed/getPosts.ts
@@ -1,7 +1,6 @@
 import * as common from '@atproto/common'
 import { Server } from '../../../../lexicon'
 import AppContext from '../../../../context'
-import { AtUri } from '@atproto/uri'
 import { PostView } from '@atproto/api/src/client/types/app/bsky/feed/defs'
 
 export default function (server: Server, ctx: AppContext) {
@@ -10,32 +9,15 @@ export default function (server: Server, ctx: AppContext) {
     handler: async ({ params, auth }) => {
       const requester = auth.credentials.did
 
-      const feedService = ctx.services.feed(ctx.db)
-      const labelService = ctx.services.label(ctx.db)
-
       const uris = common.dedupeStrs(params.uris)
-      const dids = common.dedupeStrs(
-        params.uris.map((uri) => new AtUri(uri).hostname),
-      )
 
-      const [actors, postViews, embeds, labels] = await Promise.all([
-        feedService.getActorViews(Array.from(dids), requester, {
-          skipLabels: true,
-        }),
-        feedService.getPostViews(Array.from(uris), requester),
-        feedService.embedsForPosts(Array.from(uris), requester),
-        labelService.getLabelsForSubjects([...uris, ...dids]),
-      ])
+      const postViews = await ctx.services
+        .feed(ctx.db)
+        .getPostViews(uris, requester)
 
       const posts: PostView[] = []
       for (const uri of uris) {
-        const post = feedService.views.formatPostView(
-          uri,
-          actors,
-          postViews,
-          embeds,
-          labels,
-        )
+        const post = postViews[uri]
         if (post) {
           posts.push(post)
         }

diff --git a/packages/bsky/src/api/app/bsky/unspecced/getPopularFeedGenerators.ts b/packages/bsky/src/api/app/bsky/unspecced/getPopularFeedGenerators.ts
@@ -43,13 +43,13 @@ export default function (server: Server, ctx: AppContext) {
 
       const res = await builder.execute()
 
-      const genInfos = await feedService.getFeedGeneratorViews(
+      const genInfos = await feedService.getFeedGeneratorInfos(
         res.map((feed) => feed.uri),
         requester,
       )
 
       const creators = Object.values(genInfos).map((gen) => gen.creator)
-      const profiles = await feedService.getActorViews(creators, requester)
+      const profiles = await feedService.getActorInfos(creators, requester)
 
       const genViews: GeneratorView[] = []
       for (const row of res) {

diff --git a/packages/bsky/src/api/app/bsky/util/feed.ts b/packages/bsky/src/api/app/bsky/util/feed.ts
@@ -1,5 +1,5 @@
 import { TimeCidKeyset } from '../../../../db/pagination'
-import { FeedRow } from '../../../../services/types'
+import { FeedRow } from '../../../../services/feed/types'
 
 export enum FeedAlgorithm {
   ReverseChronological = 'reverse-chronological',

diff --git a/packages/bsky/src/feed-gen/types.ts b/packages/bsky/src/feed-gen/types.ts
@@ -1,6 +1,6 @@
 import AppContext from '../context'
 import { QueryParams as SkeletonParams } from '../lexicon/types/app/bsky/feed/getFeedSkeleton'
-import { FeedRow } from '../services/types'
+import { FeedRow } from '../services/feed/types'
 
 export type AlgoResponse = {
   feedItems: FeedRow[]

diff --git a/packages/bsky/src/feed-gen/whats-hot.ts b/packages/bsky/src/feed-gen/whats-hot.ts
@@ -6,7 +6,7 @@ import { GenericKeyset, paginate } from '../db/pagination'
 import AppContext from '../context'
 import { valuesList } from '../db/util'
 import { sql } from 'kysely'
-import { FeedItemType } from '../services/types'
+import { FeedItemType } from '../services/feed/types'
 
 const NO_WHATS_HOT_LABELS: NotEmptyArray<string> = [
   '!no-promote',