Alessio
24129c4852
REFACTOR: reduce technical debt, particularly that caused by singleton pattern in pkg/scraper
...
- ensure all scraper functions have a `api.XYZ` version and a package-level convenience function
- isolate `the_api` to top-level convenience functions, in preparation for removal
- move a bunch of scraper functions around to be nearby their related functions
- new ErrLoginRequired
- remove obsolete APIv1 stuff (Feed, TweetDetail)
- rename scraper function GetUserFeedGraphqlFor => GetUserFeed
- fix go.mod Go version incorrectly claiming it's compatible with Go 1.16 (should be Go 1.17)
2024-08-09 19:48:50 -07:00
Alessio
b4384f9cc4
Enable sending emoji reactions in DMs
2024-07-28 14:09:41 -07:00
Alessio
8aca12695b
Handle media download 404s gracefully
2024-07-28 12:50:00 -07:00
Alessio
d0c23b392b
Don't crash when receiving a Grok card
2024-07-28 08:07:33 -07:00
Alessio
98015ddf0c
REFACTOR: remove DMTrove type, merge it into TweetTrove
2024-07-28 08:01:58 -07:00
Alessio
ef15e8a306
Handle guest token / session initialization when not connected to internet
2024-07-14 13:20:44 -07:00
Alessio
0fd17f1af0
Fix getting tweets from deleted accounts
2024-07-14 12:52:07 -07:00
Alessio
d2045028a7
Improve closing message when a scrape ends due to rate-limiting
2024-06-11 12:32:58 -07:00
Alessio
a1faacaf6b
Don't throw away all previous results if you get rate-limited, just save them
2024-06-10 19:59:08 -07:00
Alessio
cac87ec973
Save tweets with unknown user as <UNKNOWN USER> instead of UserID 0, which causes foreign key errors
2024-06-10 15:47:13 -07:00
Alessio
14a7f88cdb
Add scraping and saving of bookmarks
2024-05-28 21:55:42 -07:00
Alessio
42bf8ec06a
Fix a bug sending empty POST bodies
2024-05-11 10:58:33 -07:00
Alessio
f927507089
Enable marking DMs as read
2024-05-10 22:09:48 -07:00
Alessio
5cbf96f379
Fix unusual bug when scraping tweets which only happens if the tweet has a link and the text is exactly 22 chars long
2024-05-05 11:59:58 -07:00
Alessio
5d727487ba
Fix lint error
2024-05-05 11:23:06 -07:00
Alessio
71230e3886
Fix 3 bugs in DMs
...
- escape HTML entities in messages so "<" doesn't scrape as "<"
- escape quotes in sent messages, so they no longer fail to send
- fix the web UI receiving the last 50 messages after every sent message instead of just the new ones
2024-05-05 10:58:39 -07:00
Alessio
341ddaa5c2
Add Participants view to group chats
2024-05-04 21:41:15 -07:00
Alessio
1b3c5d0ed3
Add timeout error handling for scraper requests to the request body download as well (rather than just headers)
2024-04-13 16:10:23 -07:00
158710df8d
Update User Feed graphql endpoint so it can get long tweets
2024-03-22 21:39:13 -07:00
750ad3031a
Fix lint error
2024-03-18 22:32:30 -07:00
898ad163ae
Add parsing of pinned tweets on scraped user feeds
2024-03-18 22:20:36 -07:00
ca7cf613f9
Remove unnecessary import
2024-03-18 21:16:38 -07:00
3967367eed
Fix more lint errors
2024-03-18 21:15:27 -07:00
69e0a35e57
Handle HTTP request timeouts
2024-03-16 19:55:05 -07:00
73c5803a47
Add downloading of DM embedded images, videos and links
2024-03-11 21:12:38 -07:00
0ad3cf8fb8
Fix lint errors
2024-03-11 14:08:07 -07:00
aa05708e20
Move media downloader from persistence to scraper package; add 429 Rate Limited error type
2024-03-11 12:57:58 -07:00
1ba4f91463
REFACTOR: replace 'log.Debug(fmt.Sprintf(...))' with 'log.Debugf(...)' and remove 'scraper.' prefix in utils_test.go
2024-03-10 19:14:27 -07:00
07233a7153
Fix parsing of other media items in DMs as 'urls'
2024-03-09 15:11:33 -08:00
f18a2e529c
Add test files for previous commit
2024-03-08 17:30:27 -08:00
0d12166966
Add parsing of DM images, videos and URLs
2024-03-08 17:22:52 -08:00
2d14621cc0
FIXUP: Add test file for previous commit
2024-03-03 14:58:41 -08:00
b076f4d2f2
Scraper: handle videos being geo-blocked, which was panicking the downloader
2024-03-03 14:56:36 -08:00
f39ad6b7cb
Fix lint error
2024-03-02 15:45:17 -08:00
73ffb90f63
Move API login flow to its own file; add support for secondary verification challenges
2024-03-02 15:43:02 -08:00
e0266667d6
Fix previous commit
2024-02-25 17:02:39 -08:00
5b6894eee2
Add List queries
2024-02-24 16:45:08 -08:00
a7ce15ebd2
Don't clobber interaction stats when fetching a deleted tweet
2024-02-18 14:52:28 -08:00
018df0d6a4
Rename 'for you' timeline to 'following only' to match Twitter's upstream changes
...
- Also add 'get_user_likes_all' cmd subcommand to get all users likes
2024-02-18 14:47:12 -08:00
84ff6c12b7
Change all timestamps to use milliseconds instead of seconds
2024-01-07 15:15:43 -06:00
dd68ee1fce
Add scraper funcs for previous commit
2023-12-26 19:54:08 -06:00
df8093bbd9
Add follows
2023-12-26 19:52:37 -06:00
92b166a4eb
REFACTOR: create 'GetPaginatedQuery' function to encapsulate queries with cursors
2023-12-26 15:54:41 -06:00
f3f2550d85
REFACTOR: create 'GetMore' function for paginated queries with cursors
2023-12-26 11:42:02 -06:00
e1bc6bba98
Enable sending DM messages
2023-12-24 14:21:57 -06:00
903483e305
Add function to send DMs
2023-11-26 21:25:04 -08:00
aaab11b1d0
Add DM message replying-to indicators/previews and render embedded tweets
2023-11-25 14:21:09 -08:00
dcab3de773
Add test file for previous commit
2023-11-19 23:41:05 -08:00
e5b00c334d
Add embedded tweet recognition to DMs
2023-11-19 23:33:04 -08:00
b73ee1f5d2
Add support for group DMs; also make UI highlight which chat is currently active
2023-11-19 20:37:22 -08:00
