|
|
565b7eaf1e
|
Add Spaces type
|
2022-05-14 14:35:49 -07:00 |
|
|
|
d1d80a91cd
|
Add 'gofmt' linter
|
2022-03-13 17:13:40 -07:00 |
|
|
|
f41d072573
|
Improve error handling some more
|
2022-03-06 20:31:04 -08:00 |
|
|
|
18c9f79429
|
REFACTOR: LINT: add .golangci.yaml lint config file
- Add linting for line lengths
- Fix some linting errors
|
2022-03-06 15:06:06 -08:00 |
|
|
|
52370a4f9d
|
Recognized deleted users when scraping, create a user with a fake ID
|
2022-02-27 23:06:06 -08:00 |
|
|
|
8a70f55e1b
|
Update some more stuff to match API changes
- set X-Twitter-Client-Language header to "en"
- Add a new tombstone type (age-restricted)
|
2022-02-14 17:18:23 -08:00 |
|
|
|
9eb3e42539
|
Add link parsing for APIv2
|
2022-02-01 16:56:37 -08:00 |
|
|
|
d95f183e45
|
Add parsing of tombstoned quoted-tweets
|
2022-01-07 13:42:00 -05:00 |
|
|
|
1a9ba75355
|
Add checking for banned users
|
2022-01-06 14:39:31 -05:00 |
|
|
|
bf2dbede94
|
Escape HTML entities in tweet text
|
2022-01-05 21:04:40 -05:00 |
|
|
|
28e139a3e4
|
Add video durations and view counts
|
2021-12-24 16:26:34 -05:00 |
|
|
|
13299233c0
|
Add 'suspended' tombstone text; add 'violated twitter rules' and 'account no longer exists' tombstone types
|
2021-12-21 16:05:51 -05:00 |
|
|
|
ddb16c9e64
|
Remove integration test for tombstones because the guy un-privated for some reason
|
2021-12-12 16:14:11 -08:00 |
|
|
|
658f3c338f
|
Add parsing of twitter polls
|
2021-11-07 21:10:24 -08:00 |
|
|
|
6d7d940df4
|
Add search
|
2021-11-06 14:50:39 -07:00 |
|
|
|
c0ae23b4fd
|
Add 'unavailable' tombstone category
|
2021-11-01 15:47:52 -07:00 |
|
|
|
bb2ae81dfa
|
Fix the previous commit by actually adding the relevant code
|
2021-11-01 14:10:20 -07:00 |
|
|
|
e8a9c10c89
|
Add parsing of tombstones from api
|
2021-11-01 14:06:15 -07:00 |
|
|
|
4fa30e246d
|
Add dimension info to media content
|
2021-10-10 16:06:47 -07:00 |
|
|
|
dd22b25831
|
Fix lint error
|
2021-10-10 15:28:49 -07:00 |
|
|
|
3b56973355
|
Remove reply-mentions for threads
|
2021-10-10 15:22:48 -07:00 |
|
|
|
159084006d
|
Fix handling of reply-mentions and tweet text normalization
|
2021-09-27 18:12:28 -07:00 |
|
|
|
c865df8aca
|
Add support for tweets with multiple URLs
|
2021-09-17 19:45:31 -07:00 |
|
|
|
e2ff9436fa
|
Add player type of URL card
|
2021-09-17 17:35:55 -07:00 |
|
|
|
3f9c807efa
|
Add Url type
|
2021-09-17 13:41:43 -07:00 |
|
|
|
a2554e9293
|
Enable 'fetch_tweet' full scrape
|
2021-08-22 17:55:21 -07:00 |
|
|
|
ea1264ddbd
|
Fix misplaced comma
|
2021-08-22 17:38:13 -07:00 |
|
|
|
9eee64cca1
|
Catch random API 403 errors
|
2021-08-22 16:41:59 -07:00 |
|
|
|
5a0b8b5e36
|
Check for end-of-feed
|
2021-08-22 15:01:46 -07:00 |
|
|
|
58dc223f84
|
Use VideoID given by the Twitter API instead of a rowid
|
2021-08-04 23:41:58 -07:00 |
|
|
|
c803794a0f
|
Change TweetID to be an int64 instead of a string
|
2021-08-04 14:49:20 -07:00 |
|
|
|
d81fae0013
|
Make UserID an alias for int64 rather than string
|
2021-08-04 02:00:58 -07:00 |
|
|
|
7a81b71cd3
|
Use ImageID from the API rather than rowid
|
2021-08-04 01:23:55 -07:00 |
|
|
|
494ca25dc4
|
Change method of scraping images
- break `Filename` field on Image into `RemoteURL` and `LocalFilename`
- new parsing function for an APIMedia item
|
2021-08-03 17:34:44 -07:00 |
|
|
|
647dd8aa6b
|
Add scraping of videos from tweets
|
2021-07-25 14:51:17 -07:00 |
|
|
|
6c111ab976
|
BUGFIX: fix incorrect text normalization on tweet with multiple images
|
2021-07-23 19:14:16 -07:00 |
|
|
|
5290a0b549
|
Add linting
|
2021-06-14 13:08:38 -07:00 |
|
|
|
f85305b946
|
Refactor API response types, creating new APIUser type
- extracted this from both UserResponse struct and TweetResponse struct,
both of which return something almost identical
|
2021-06-13 12:43:34 -07:00 |
|
|
|
0738f77b55
|
Add helper methods to API types to help pre-process them
|
2021-05-23 20:58:31 -04:00 |
|
|
|
2c8fe25e78
|
Add tweet parsing
|
2021-05-22 18:20:18 -04:00 |
|
