|
|
89179f2447
|
Tidy up the scraper tests a bit
|
2021-09-27 15:30:05 -07:00 |
|
|
|
0bb9ff6c6b
|
Add handling for URL cards with no link preview thumbnail image
|
2021-09-17 20:50:28 -07:00 |
|
|
|
9d10fd5942
|
Fix scraper test file to be minified
|
2021-09-17 20:12:51 -07:00 |
|
|
|
497a62050b
|
Add test file for previous commit
|
2021-09-17 20:02:33 -07:00 |
|
|
|
c865df8aca
|
Add support for tweets with multiple URLs
|
2021-09-17 19:45:31 -07:00 |
|
|
|
690dd99b8f
|
Clean up the michael malice feed test file
|
2021-09-17 18:08:16 -07:00 |
|
|
|
05c3f2289b
|
Add persistence for new Url type
|
2021-09-17 18:04:57 -07:00 |
|
|
|
79f098450e
|
Add 'HasCard' URL field
|
2021-09-17 17:39:04 -07:00 |
|
|
|
355ed7ccf5
|
Add test file for the previous commit
|
2021-09-17 17:36:35 -07:00 |
|
|
|
e2ff9436fa
|
Add player type of URL card
|
2021-09-17 17:35:55 -07:00 |
|
|
|
3f9c807efa
|
Add Url type
|
2021-09-17 13:41:43 -07:00 |
|
|
|
1f5ebf9f8d
|
Enable fetching tweets to the limit of scrollback
|
2021-08-22 18:22:06 -07:00 |
|
|
|
a2554e9293
|
Enable 'fetch_tweet' full scrape
|
2021-08-22 17:55:21 -07:00 |
|
|
|
ea1264ddbd
|
Fix misplaced comma
|
2021-08-22 17:38:13 -07:00 |
|
|
|
9eee64cca1
|
Catch random API 403 errors
|
2021-08-22 16:41:59 -07:00 |
|
|
|
5a0b8b5e36
|
Check for end-of-feed
|
2021-08-22 15:01:46 -07:00 |
|
|
|
4a3c12b04a
|
Update some docstrings, rename a few methods
|
2021-08-16 20:35:15 -07:00 |
|
|
|
2698bd4fd7
|
Fix linting error
|
2021-08-10 22:08:01 -07:00 |
|
|
|
932b42eafc
|
Should be using the full-sized profile images, not the 'normal' ones
|
2021-08-10 22:01:31 -07:00 |
|
|
|
655c968614
|
Add LocalPath fields for BannerImage and ProfileImage on a User
|
2021-08-10 19:51:56 -07:00 |
|
|
|
9734c09426
|
Add downloading for media in tweets
|
2021-08-07 16:51:38 -07:00 |
|
|
|
d1ba8b48f3
|
Add RemoteURL and LocalFilename to Images, remove Filename
|
2021-08-05 14:22:16 -07:00 |
|
|
|
9c9100d874
|
Add RemoteURL and LocalFilename fields to Video, remove Filename
|
2021-08-05 14:10:46 -07:00 |
|
|
|
8462365d70
|
Add tests for API video parsing
|
2021-08-04 23:49:45 -07:00 |
|
|
|
58dc223f84
|
Use VideoID given by the Twitter API instead of a rowid
|
2021-08-04 23:41:58 -07:00 |
|
|
|
c803794a0f
|
Change TweetID to be an int64 instead of a string
|
2021-08-04 14:49:20 -07:00 |
|
|
|
529c181397
|
Fix build errors
|
2021-08-04 02:14:06 -07:00 |
|
|
|
d81fae0013
|
Make UserID an alias for int64 rather than string
|
2021-08-04 02:00:58 -07:00 |
|
|
|
7a81b71cd3
|
Use ImageID from the API rather than rowid
|
2021-08-04 01:23:55 -07:00 |
|
|
|
494ca25dc4
|
Change method of scraping images
- break `Filename` field on Image into `RemoteURL` and `LocalFilename`
- new parsing function for an APIMedia item
|
2021-08-03 17:34:44 -07:00 |
|
|
|
2663387f29
|
Add filename helper utils to video and image types
|
2021-08-03 15:38:19 -07:00 |
|
|
|
083f5f72e0
|
Add media queries
|
2021-08-01 15:52:04 -07:00 |
|
|
|
81392996bb
|
Add video and image types
- in addition to `Filename` (string), includes IsDownloaded (bool)
|
2021-07-26 17:26:39 -07:00 |
|
|
|
0c87428c44
|
Make videos their own table; store as a slice on Tweet rather than text field
|
2021-07-25 15:42:43 -07:00 |
|
|
|
647dd8aa6b
|
Add scraping of videos from tweets
|
2021-07-25 14:51:17 -07:00 |
|
|
|
2086a5e983
|
FIXUP: rename Video field in Tweet
|
2021-07-24 21:52:39 -07:00 |
|
|
|
d386ba2cb4
|
Make String() methods more useful for terminal output
|
2021-07-24 10:55:44 -07:00 |
|
|
|
0221cbe865
|
Fix some unnecessary typecasting
|
2021-07-24 10:23:40 -07:00 |
|
|
|
96e2edfb8a
|
ID fields are now Hungarian-notated
- tweet.User is now a pointer to a User object; tweet.UserID holds the user id
|
2021-07-24 10:19:05 -07:00 |
|
|
|
6c111ab976
|
BUGFIX: fix incorrect text normalization on tweet with multiple images
|
2021-07-23 19:14:16 -07:00 |
|
|
|
b5c33c2ad0
|
Fix some types to be more clear
|
2021-07-22 14:16:40 -07:00 |
|
|
|
28b11548bf
|
Add actual scraping
|
2021-06-16 19:34:06 -07:00 |
|
|
|
f28d600024
|
Add user feed test data file
|
2021-06-16 13:15:53 -07:00 |
|
|
|
74ef2e4193
|
Create UserHandle type
|
2021-06-16 13:14:56 -07:00 |
|
|
|
90e5fdb8b1
|
Add parser for a TweetResponse
|
2021-06-15 15:18:09 -07:00 |
|
|
|
5290a0b549
|
Add linting
|
2021-06-14 13:08:38 -07:00 |
|
|
|
5b7d3dcdc4
|
Add Retweet type
|
2021-06-14 12:14:51 -07:00 |
|
|
|
02a2a0be33
|
Add parsing for user API type
|
2021-06-13 14:34:20 -07:00 |
|
|
|
73510288af
|
Add test file for user profile
|
2021-06-13 12:48:14 -07:00 |
|
|
|
f85305b946
|
Refactor API response types, creating new APIUser type
- extracted this from both UserResponse struct and TweetResponse struct,
both of which return something almost identical
|
2021-06-13 12:43:34 -07:00 |
|
