21581b325a
Add support for parsing Tweet Detail in APIv2, including its unusual cursor format and conversation threads
2023-06-08 23:19:50 -03:00
693831704d
BUGFIX: spaces ended_at should be updated on saving now
2023-06-08 21:52:22 -03:00
d78f9c004d
REFACTOR: use a switch statement to add clarity for parsing non-tweet timeline entries
2023-06-08 20:43:07 -03:00
6192c37999
FIXUP: add test file for previous commit
2023-06-08 18:53:50 -03:00
0868d8d6d8
Implement expandable ("Show more...") tweets
2023-06-08 18:51:50 -03:00
eb566c0612
FIXUP: add test file for previous commit
2023-06-08 18:10:55 -03:00
e4e05008d5
Add parsing of home-conversation entry types (mini-threads that show up in a feed)
2023-06-08 18:08:23 -03:00
9a1ef814ef
Add debug logging of requested URLs
2023-06-08 16:04:24 -03:00
5c70d2cf49
Remove incorrect/redundant API type field
2023-06-08 14:22:04 -03:00
b36e7410c5
FIXUP: forgot this in the previous commit
2023-06-08 13:10:15 -03:00
a464aa8a98
REFACTOR: rename 'ParseTweetResponse' to 'ToTweetTrove' and put it in the api_types file
2023-06-08 13:08:46 -03:00
f7d383adf3
REFACTOR: move API types, requests and tests to their own 'api_types_spaces' files
2023-06-07 17:18:26 -03:00
b37977145a
Expand debug logging to make debugging scraping errors easier
...
- Print HTTP headers to the debug log on every request
- If parsing a APIV2 tweet panics, print the full JSON of that tweet entry to the debug log
2023-06-07 15:17:38 -03:00
2a74941707
Fix lint error
2023-06-05 18:10:02 -03:00
6df313558d
Fix lint errors
2023-06-03 09:17:12 -03:00
815a8180da
BUGFIX: don't clobber num_likes, num_retweets etc. if tweet is a stub
...
- Convert Tweet queries to sqlx, including new CommaSeparatedList type to convert Hashtags, Mentions and ReplyMentions
2023-06-03 08:30:49 -03:00
dea37c7556
Fix cursor bug in authenticated sessions
2023-06-03 07:51:39 -03:00
af93f44ed2
Two bug fixes
...
- Search is now authenticated only
- `fetch_tweet` and `search` subcommands now respect the `-n [number]` flag
2023-06-03 07:38:17 -03:00
1384aa73e9
BUGFIX: fix CSRF tokens getting out of sync on requests that load multiple pages of results
2023-03-09 22:49:55 -05:00
ee4b863d38
Remove integration test masquerading as a unit test
2023-02-17 13:22:09 -05:00
c09b77f497
Add ability to use saved sessions
2023-02-17 13:07:12 -05:00
Jaeger Aquila
28ca0ac4eb
added cmdline login functionality
2023-02-05 20:29:57 -05:00
390c831541
Add marshaling and unmarshaling of API objects so state can be persisted between runs
2023-01-01 21:57:08 -05:00
c3494cfe19
Add handle of the logged-in user to API
2023-01-01 21:45:58 -05:00
00a73836f5
Fix some lint errors
2023-01-01 20:14:26 -05:00
8a1b2fc801
Fix lint error
2022-12-30 21:23:45 -05:00
04842f8c72
REFACTOR: make login flow less verbose
2022-12-30 21:23:45 -05:00
1087809d63
Add fetching likes API request
2022-12-30 21:23:45 -05:00
c0366a0978
Add cookies and CSRF to authentication flow
2022-12-30 21:23:45 -05:00
Jaeger Aquila
2cf64db93e
some refactoring and tests
2022-12-30 21:23:45 -05:00
Jaeger Aquila
140024e903
initial login code
2022-12-30 21:23:45 -05:00
58f4026e88
Rename login script
2022-12-30 21:23:45 -05:00
62c499e95c
Tidy up login script, add request to fetch 'likes' (auth-guarded endpoint)
2022-12-30 21:23:45 -05:00
James Raleigh
55b25f4fbf
initial login test script in py
2022-12-30 21:23:45 -05:00
dac10fa13c
Tighten up no-worsening of spaces, add saving ShortUrl when scraping from feed
2022-12-23 00:00:46 -05:00
1a9a2cb745
Fix Spaces created-by-id not being scraped
2022-12-22 23:29:11 -05:00
Jaeger Aquila
ce11a70bf9
Finished refactoring
2022-12-10 13:30:16 -05:00
ec6ecb39cb
Remove UpdateQueryCursor helper function as it was inlined
2022-12-10 12:18:52 -05:00
fe57763568
Change entrypoint functions to use NewGuestSession constructor for API
2022-12-10 10:58:32 -05:00
3c9ed97c64
Fix lint error
2022-12-10 10:55:55 -05:00
6c5f8b8356
Update API call functions to use do_http
...
- move query-string modifications to individual API call functions:
"referrer=tweet", tweet "include_XYZ" params
- tidy up implementation of retries in `API.GetUser`
- fix out-param of `do_http` being a pointer to an interface when it should just be an interface
2022-12-10 10:49:25 -05:00
Jaeger Aquila
e44a83edae
fixed trailing whitespace
2022-12-09 23:03:22 -05:00
Jaeger Aquila
6816dcbe2f
generic http implem and fixing package errors
2022-12-09 23:03:22 -05:00
77e338b4f7
Add skeleton for API refactoring
2022-12-09 23:03:22 -05:00
f2af4b2403
Fix search
2022-12-09 23:02:12 -05:00
7bad18e9b1
Tidy up the formatting
2022-12-03 13:47:34 -05:00
James Raleigh
972fd50c5b
Fixed tests for DMCAed videos and other fixes
2022-12-02 20:38:57 -05:00
6830be3c87
Fix error where retweets were not parsing properly sometimes
2022-11-25 15:51:21 -05:00
412574e46f
Fix parsing of spaces in APIv2; improve no-worsening of Spaces
2022-11-25 13:22:20 -05:00
433dd5e0bb
Enable fetching of Space details
2022-11-24 19:15:21 -05:00
