Pagination for the regular fetch() on feeds. #33

stevenlafl · 2023-07-10T05:03:33Z

Right now here is the WIP:
https://github.com/stevenlafl/threads-web-client/blob/master/src/pages/api/feed.ts

and associated Feed component: https://github.com/stevenlafl/threads-web-client/blob/master/src/app/Feed.tsx

I need to get it to display more. It looks like I'm limited to 8, and refreshing may or may not get me a similar result set. Is there pagination available?

elijah-wright · 2023-07-10T05:17:35Z

yeah I saw pagination in there but wanted to get the base stuff in before I worried about that stuff. there should be a token in there that you can use but idk how you use it. like I said, working on getting my setup back up

fwiw i'm also working on a web client so the stuff that you're bringing up is stuff i'm experiencing

elijah-wright · 2023-07-10T05:38:02Z

@stevenlafl the endpoint for the timeline is gzipped for some reason but look at line 7 of this file:
https://github.com/threadsjs/threads.js/blob/main/src/managers/FeedManager.js
you need to pass the max_id you get from the response of this endpoint to the body of the next one.

stevenlafl · 2023-07-10T06:15:36Z

Like this?

FeedManager.js

const RESTManager = require('./RESTManager');

class FeedManager extends RESTManager {
	async fetch(max_id) {
		return await this.request('/api/v1/feed/text_post_app_timeline/', {
			method: 'POST',
			body: 'pagination_source=text_post_feed_threads' + (max_id ? '&max_id=' + max_id : ''),
		})
	}

	...
}

module.exports = FeedManager;

index.d.ts

...
declare module '@threadsjs/threads.js/src/managers/FeedManager.js' {
	import RESTManager from "@threadsjs/threads.js/src/managers/RESTManager.js";
	export default class FeedManager extends RESTManager {
		fetch(max_id?: number): Promise<any>;
		fetchThreads(user: string): Promise<any>;
		fetchReplies(user: string): Promise<any>;
		recommended(): Promise<any>;
	}
}
...

I went ahead and used the RESTManager in my implementation for now for infinite scroll. I am not sure if it works for certain, but it does get me a new result set without erroring.

thisisdice · 2023-07-10T06:26:31Z

I might be wrong but I don’t think that’s paginating. I think your fetching a new feed on each call which might eventually flag as it doesn’t follow normal usage. The app paginates as you scroll taking a gzipbody with values such as the below.

{"media id*: "3143463776240999995 180569377", "version": 24 , "media pet": 1.0, "time info: {"10': 8920, "25*: 8920, "50": 8920, "75": 8716}, 'was_share tapped": false), {"media_id": "3143527582710965035_403375573"
"version": 24, "media pct": 0.9230769, "time info': {10: 865, "25*: 865, "50: 865, "75": 254}, "was_share tapped" :false},
{"media_id": "3143533596829412672 40730850"
*version": 24, "media pct":0.9594017, "time info": (*10": 347, "25":347, "50": 347, "75" :3473, "was_share_tapped" false),
{"media id': "3143519892437762445 403375573"
"version"
24, "media pet" :1.0, "time info" (*10" 1689, "25 1689, "50' 1079, "75":6197, "was_share_tapped" false),
{"media_id": "3143528201681201769 403375573"
"'version":24, "media_pct":1.0, "time_ info" ('10" :8920, 25":8514, "50": 8514, "75": 8514}, "was_share_tapped": false),
('media id":"3143523828357551367 180569377"
"*version": 24, "media_pct' :1.0, "time info": (™10": 1476, "25":1476, "50" :1476, "75" :1476), "was_share_tapped" false),
{"media id': "3143528196759601588 27288745"
"version": 24, "media pct": 1.0, "time info': (™10": 1274, "25":1274, "50": 1274, "75" :1274}, "was_share_tapped": false),
"media id": "3143503512472939477 4053226" , "version": 24, "media pct: 1.0,
"time info": (*10" 206, "25": 206, "50" 206, *75":2067, "was_share_tapped": false),
{"media id': "3143486442730059357 51091500"
"version": 24, "media pct: 1.0, "time_info': {10":205, "25" :205, "50': 205, "75": 205}, "was_share_tapped": false}.
{"media id": "3143468471916764278 395761959"
"media pet":1.0, "time_ info": ("10":205, "25":205, "50": 205, "75": 205}, "was_share_tapped": false).
'media id" "3143530367872313738 4094723", "Version'" 24, 'media pet" 0.93615873, 'time info" ("10" 205, "25 - 205, 50": 205, "75" 143),
"was_share_tapped" :false),
("media id': "3143459173832072096_3225079059
"version" :24, "media pet': 1.0, "time_ info': (*10": 8716, "25" :8716, "50":8716, "75" :8716), "was share_tapped":false)
]

elijah-wright · 2023-07-10T06:54:31Z

I'll take a look at this in the morning

elijah-wright · 2023-07-10T14:23:51Z

what thisisdice shared is the analytics that Meta is getting from your scrolling. so if you looked at a post for longer, it notes that down. idk what that stuff means but that's just it. there's a lot of analytics to threads lol, in that timeline endpoint Meta even gets your battery level

when you do a POST request to the timeline endpoint, what changes in the body when you pull to refresh and when you paginate is the reason (which changes to "pagination") and the max_id param, which you get from next_max_id from the previous response.

I'll see what I can do today about this

stevenlafl · 2023-07-10T14:29:15Z

One more thing to be aware of is that in implementing viewing replies with posts.fetch() we also need pagination. I see:

thisisdice · 2023-07-10T17:42:18Z

@elijah-wright @sooluh @stevenlafl I’m able to pull endpoints, requests and responses but I’m not the best at deciphering them into reusable components. Any suggestions for a collaborative effort to get as many of them in welcome! Should I start dumping them here for others to have a go?

micheltucker · 2023-07-10T19:12:30Z

@stevenlafl did you figure out how to get the proper feed? I am not even sure the meta engineers know. Every time I open the app I see something new 🤣

elijah-wright · 2023-07-10T19:16:43Z

@thisisdice sure

stevenlafl · 2023-07-10T21:00:56Z

@stevenlafl did you figure out how to get the proper feed? I am not even sure the meta engineers know. Every time I open the app I see something new 🤣

Hah, I don't see duplicates as I scroll down. Doesn't mean what I did was correct, I just have to trust @elijah-wright. But I have it to where I can refresh and see new stuff if I want. Otherwise just keep scrolling. Even if there are duplicates, I've got it to where it removes them before trying to display.

I managed to get it to display things like this (quote posts only showing links with no text in body of post), or even quotes of replies.

When I reply to a post, I get back the full renderable post, so that feeds right back into my component structure. A few quirks but it works well. @elijah-wright is a wizard for making any of this even possible.

@birobirobiro has been assisting me with some of the styling too.

elijah-wright · 2023-07-11T07:18:58Z

thank you lol!

CryptaBean · 2023-07-11T11:14:47Z

https://i.instagram.com/api/v1/feed/text_post_app_timeline/

[decoded gzip] URLEncoded form
has_camera_permission: 0
feed_view_info:        [{"media_id":"3143468282492089976_702310471","version":24,"media_pct":1.0,"time_info":{"10":37820642,"25":37820642,"50":37820440,"75":37820239},"was_share_tapped":false},{"media_id":"3143480192010578160_1626596167","version":24,"media_pct":0.37179488,"time_info":{"10":2481,"25":2073,"50":0,"75":0},"was_share_tapped":false}]
phone_id:              d9f9f6a3-****-4ac3-b4b1-0e6004702f82
reason:                cold_start_fetch **<---- this changes to pagination for subsequent** requests
battery_level:         75
timezone_offset:       3600
pagination_source:     text_post_feed_threads
device_id:            ****4c42-6663-****-884b-66418b4bb843
request_id:            ****8f5e-dea0-****-a3d6-68484e6faed1
is_pull_to_refresh:    0  **<---- this changes 1 for homepage refresh**
_uuid:                 ****4c42-6663-****-884b-66418b4bb843
is_charging:           0
is_dark_mode:          0
will_sound_on:         0
session_id:            ****8bd-4699-****-a42d-cf267108094b
bloks_versioning_id:   5f56efad68e1edec7801f630b5c122704ec5378adbee6609a448f105f34a9c73

https://i.instagram.com/api/v1/notifications/badge/

content-type: application/x-www-form-urlencoded; charset=UTF-8
phone_id:  d9f9f6a3-****-4ac3-b4b1-0e6004702f82
user_ids:  35048****
device_id: ****4c42-6663-****-884b-66418b4bb843
_uuid:     ****4c42-6663-****-884b-66418b4bb843

returns

json
{
    "badge_payload": {
        "35048****": {
            "badge_count_map": {
                "ac": 1
            },
            "total_count": 1
        }
    },
    "status": "ok"
}

stevenlafl · 2023-07-11T14:02:17Z

Confirmed that this is right, however the next_max_id ends with an equal sign which needs to be urlencoded (%3D)

So it becomes:

const RESTManager = require('./RESTManager');

class FeedManager extends RESTManager {
	async fetch(max_id) {
		return await this.request('/api/v1/feed/text_post_app_timeline/', {
			method: 'POST',
			body: 'pagination_source=text_post_feed_threads' + (max_id ? '&max_id=' + encodeURIComponent(max_id) : ''),
		})
	}

	...
}

module.exports = FeedManager;

#49

elijah-wright · 2023-07-11T17:31:17Z

should be fixed

elijah-wright closed this as completed Jul 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pagination for the regular fetch() on feeds. #33

Pagination for the regular fetch() on feeds. #33

stevenlafl commented Jul 10, 2023 •

edited

Loading

elijah-wright commented Jul 10, 2023 •

edited

Loading

elijah-wright commented Jul 10, 2023

stevenlafl commented Jul 10, 2023 •

edited

Loading

thisisdice commented Jul 10, 2023

elijah-wright commented Jul 10, 2023

elijah-wright commented Jul 10, 2023 •

edited

Loading

stevenlafl commented Jul 10, 2023

thisisdice commented Jul 10, 2023

micheltucker commented Jul 10, 2023

elijah-wright commented Jul 10, 2023

stevenlafl commented Jul 10, 2023 •

edited

Loading

elijah-wright commented Jul 11, 2023

CryptaBean commented Jul 11, 2023 •

edited

Loading

stevenlafl commented Jul 11, 2023 •

edited

Loading

elijah-wright commented Jul 11, 2023

Pagination for the regular fetch() on feeds. #33

Pagination for the regular fetch() on feeds. #33

Comments

stevenlafl commented Jul 10, 2023 • edited Loading

elijah-wright commented Jul 10, 2023 • edited Loading

elijah-wright commented Jul 10, 2023

stevenlafl commented Jul 10, 2023 • edited Loading

thisisdice commented Jul 10, 2023

elijah-wright commented Jul 10, 2023

elijah-wright commented Jul 10, 2023 • edited Loading

stevenlafl commented Jul 10, 2023

thisisdice commented Jul 10, 2023

micheltucker commented Jul 10, 2023

elijah-wright commented Jul 10, 2023

stevenlafl commented Jul 10, 2023 • edited Loading

elijah-wright commented Jul 11, 2023

CryptaBean commented Jul 11, 2023 • edited Loading

stevenlafl commented Jul 11, 2023 • edited Loading

elijah-wright commented Jul 11, 2023

stevenlafl commented Jul 10, 2023 •

edited

Loading

elijah-wright commented Jul 10, 2023 •

edited

Loading

stevenlafl commented Jul 10, 2023 •

edited

Loading

elijah-wright commented Jul 10, 2023 •

edited

Loading

stevenlafl commented Jul 10, 2023 •

edited

Loading

CryptaBean commented Jul 11, 2023 •

edited

Loading

stevenlafl commented Jul 11, 2023 •

edited

Loading