19 Commits

Author SHA1 Message Date
159084006d Fix handling of reply-mentions and tweet text normalization 2021-09-27 18:12:28 -07:00
c865df8aca Add support for tweets with multiple URLs 2021-09-17 19:45:31 -07:00
e2ff9436fa Add player type of URL card 2021-09-17 17:35:55 -07:00
3f9c807efa Add Url type 2021-09-17 13:41:43 -07:00
a2554e9293 Enable 'fetch_tweet' full scrape 2021-08-22 17:55:21 -07:00
ea1264ddbd Fix misplaced comma 2021-08-22 17:38:13 -07:00
9eee64cca1 Catch random API 403 errors 2021-08-22 16:41:59 -07:00
5a0b8b5e36 Check for end-of-feed 2021-08-22 15:01:46 -07:00
58dc223f84 Use VideoID given by the Twitter API instead of a rowid 2021-08-04 23:41:58 -07:00
c803794a0f Change TweetID to be an int64 instead of a string 2021-08-04 14:49:20 -07:00
d81fae0013 Make UserID an alias for int64 rather than string 2021-08-04 02:00:58 -07:00
7a81b71cd3 Use ImageID from the API rather than rowid 2021-08-04 01:23:55 -07:00
494ca25dc4 Change method of scraping images
- break `Filename` field on Image into `RemoteURL` and `LocalFilename`
- new parsing function for an APIMedia item
2021-08-03 17:34:44 -07:00
647dd8aa6b Add scraping of videos from tweets 2021-07-25 14:51:17 -07:00
6c111ab976 BUGFIX: fix incorrect text normalization on tweet with multiple images 2021-07-23 19:14:16 -07:00
5290a0b549 Add linting 2021-06-14 13:08:38 -07:00
f85305b946 Refactor API response types, creating new APIUser type
- extracted this from both UserResponse struct and TweetResponse struct,
both of which return something almost identical
2021-06-13 12:43:34 -07:00
0738f77b55 Add helper methods to API types to help pre-process them 2021-05-23 20:58:31 -04:00
2c8fe25e78 Add tweet parsing 2021-05-22 18:20:18 -04:00