|
|
648098352b
|
Fix scraper treating NotFoundError as a bug requiring retries
|
2022-02-27 23:35:44 -08:00 |
|
|
|
52370a4f9d
|
Recognized deleted users when scraping, create a user with a fake ID
|
2022-02-27 23:06:06 -08:00 |
|
|
|
772fa247f7
|
Add creation of unknown users with known handles (i.e., deleted accounts)
|
2022-02-26 22:09:27 -08:00 |
|
|
|
373b9ce16e
|
Add IsFollowed field on Users
|
2022-02-26 15:58:30 -08:00 |
|
|
|
19c4b14704
|
Use a more lint-approved case-insensitive comparator
|
2022-02-14 18:06:04 -08:00 |
|
|
|
82f5721ac2
|
Make user handle matching case-insensitive in tweet trove
|
2022-02-14 17:58:10 -08:00 |
|
|
|
8a70f55e1b
|
Update some more stuff to match API changes
- set X-Twitter-Client-Language header to "en"
- Add a new tombstone type (age-restricted)
|
2022-02-14 17:18:23 -08:00 |
|
|
|
9cd7f4171a
|
Add proper logging module, set logging level to 'info' by default (hide very verbose debug results)
|
2022-02-14 14:31:26 -08:00 |
|
|
|
e0f05cbeea
|
Refactor all scraper areas to use TweetTroves where applicable
|
2022-02-12 20:39:30 -08:00 |
|
|
|
4b2df1c628
|
Fix wrong 'Instruction' item being chosen following new API update :V
|
2022-02-12 15:51:55 -08:00 |
|
|
|
2773214855
|
Add test file for previous commit
|
2022-02-04 17:34:15 -08:00 |
|
|
|
09b8685f19
|
Add polls for APIv2
|
2022-02-04 17:05:27 -08:00 |
|
|
|
8d406e1b11
|
Minify that test file
|
2022-02-02 15:21:42 -08:00 |
|
|
|
359c166363
|
Add test file for previous commit
|
2022-02-02 14:53:10 -08:00 |
|
|
|
8892785aad
|
Add fetching of tombstoned users for APIv2
|
2022-02-02 14:49:46 -08:00 |
|
|
|
1e1e97b5ca
|
Add TweetTrove#FillMissingUserIDs helper method
|
2022-02-02 14:43:17 -08:00 |
|
|
|
3ca67418e0
|
Add test file for previous commit
|
2022-02-02 12:26:02 -08:00 |
|
|
|
df1a7d2388
|
Add recognizing bottom of feed
|
2022-02-02 12:10:24 -08:00 |
|
|
|
c00ff06b3f
|
URLs need to have a TweetID lol
|
2022-02-01 18:21:48 -08:00 |
|
|
|
32e1558800
|
Fix link parsing for retweets
|
2022-02-01 17:36:23 -08:00 |
|
|
|
9eb3e42539
|
Add link parsing for APIv2
|
2022-02-01 16:56:37 -08:00 |
|
|
|
1bc6aec3b5
|
Incorporate the new parsing structure into the actual program
|
2022-02-01 16:04:52 -08:00 |
|
|
|
b2199e9b59
|
Add Url#ShortText field
|
2022-02-01 15:48:43 -08:00 |
|
|
|
069ab83fba
|
Refactor scraper tests
- use Stretchr assert package for assertions
- use "dot imports" to remove "scraper.XYZ" from everything
|
2022-01-31 19:14:14 -08:00 |
|
|
|
babbdecf6c
|
Fix linting error
|
2022-01-30 18:03:09 -08:00 |
|
|
|
12be70f541
|
Add test files
|
2022-01-30 17:58:03 -08:00 |
|
|
|
c085a579d9
|
Add TombstoneUsers to TweetTrove
|
2022-01-30 17:55:18 -08:00 |
|
|
|
4708ffc3c9
|
Intermediate results for refactoring :V
|
2022-01-30 17:48:03 -08:00 |
|
|
|
f0b1acd595
|
Fix an error
|
2022-01-30 15:44:26 -08:00 |
|
|
|
3046dd56f4
|
Add TweetTrove#MergeWith method
|
2022-01-30 15:31:06 -08:00 |
|
|
|
f264aff265
|
Update test data since the previous private-user unprivated
|
2022-01-30 15:16:04 -08:00 |
|
|
|
68a8549f34
|
Rename api V2 test responses directory
|
2022-01-30 15:06:07 -08:00 |
|
|
|
4ccd6795d5
|
Fix import error
|
2022-01-30 14:51:14 -08:00 |
|
|
|
8c79a39c2e
|
API V2: parse User result
|
2022-01-30 14:40:17 -08:00 |
|
|
|
b261f095ad
|
Change main application to use new API for user feeds
|
2022-01-24 20:19:08 -08:00 |
|
|
|
fb421215df
|
Add parsing of new GraphQL twitter api for user feeds
|
2022-01-24 20:13:17 -08:00 |
|
|
|
dc24ce5ad4
|
Fix tweet URL parsing for urls with the stupid '?s=20' etc
|
2022-01-10 11:41:03 -05:00 |
|
|
|
d9f9c9de83
|
Integrate url expander into Tweet URL parser
|
2022-01-08 18:36:23 -05:00 |
|
|
|
043a7fd66f
|
Add link expander
|
2022-01-08 18:25:26 -05:00 |
|
|
|
e71e32228c
|
Get rid of tiny default profile images (default image is small enough, doesn't need a tiny version)
|
2022-01-07 17:33:26 -05:00 |
|
|
|
12f1cc6728
|
Fix some problems associated with banned users
|
2022-01-07 16:23:02 -05:00 |
|
|
|
c253486580
|
Fix lint error
|
2022-01-07 13:52:33 -05:00 |
|
|
|
d95f183e45
|
Add parsing of tombstoned quoted-tweets
|
2022-01-07 13:42:00 -05:00 |
|
|
|
6eac56183d
|
Allow fetching of banned users
|
2022-01-07 13:40:22 -05:00 |
|
|
|
1ea97e669a
|
Update tiny profile image logic for users with no profile image (including banned users)
|
2022-01-06 15:21:27 -05:00 |
|
|
|
1a9ba75355
|
Add checking for banned users
|
2022-01-06 14:39:31 -05:00 |
|
|
|
b1c7db6540
|
Add helper method to parse tweet URLs
|
2022-01-06 13:43:22 -05:00 |
|
|
|
bf2dbede94
|
Escape HTML entities in tweet text
|
2022-01-05 21:04:40 -05:00 |
|
|
|
f25f1ae024
|
Change profile image downloading to use tiny profile image if appropriate
|
2022-01-05 12:37:11 -05:00 |
|
|
|
80f6787aa0
|
Add getter for User tiny-profile-image URL
|
2022-01-04 13:15:47 -05:00 |
|
