b8b3b56c8e
Add woodpecker config, update tests due to an account getting banned
2022-10-14 23:32:14 -04:00
e932dd33d5
Add parsing Spaces for API v2
2022-05-14 17:36:47 -07:00
1b675e8200
Add db operations for Spaces
2022-05-14 16:04:09 -07:00
54857f40cd
Add parsing of tweets with Space links in them
2022-05-14 15:02:58 -07:00
565b7eaf1e
Add Spaces type
2022-05-14 14:35:49 -07:00
d98f02954e
Add support for 'mobile' twitter links
2022-05-14 13:51:51 -07:00
d1d80a91cd
Add 'gofmt' linter
2022-03-13 17:13:40 -07:00
223734d001
Remove all references to deprecated 'ioutil' package
2022-03-13 16:33:17 -07:00
f08da27cc2
Enable 'wrapcheck' linter
2022-03-13 16:13:16 -07:00
482f3ae127
Enable the nolintlint linter
2022-03-13 13:05:37 -07:00
f41d072573
Improve error handling some more
2022-03-06 20:31:04 -08:00
15d7cd77a1
Do slightly more proper error handling
2022-03-06 19:27:30 -08:00
2f784c779c
REFACTOR: Add sqlx dependency
2022-03-06 19:17:43 -08:00
40e585e1a6
Fix blunder
2022-03-06 18:17:00 -08:00
26ab9ec937
Create new Timestamp helper type
2022-03-06 18:09:43 -08:00
7edc8ad5d3
Some whitespace changes :V
2022-03-06 17:07:05 -08:00
1582dfe44e
Fix some lint problems
2022-03-06 15:36:07 -08:00
18c9f79429
REFACTOR: LINT: add .golangci.yaml lint config file
...
- Add linting for line lengths
- Fix some linting errors
2022-03-06 15:06:06 -08:00
872fc4addf
Add test file for previous commit
2022-03-02 14:37:37 -08:00
f22f15f3d9
Prevent quoted-tweet URLs from being added as URLs (they should just be quote-tweets)
2022-03-02 14:34:42 -08:00
aa961b9ff4
Fix handling of empty entries in the feed
...
- e.g., retweets where the author then deleted the original tweet
- I think this is a bug in the Twitter API (it doesn't make sense to return this data) but we have to handle it anyway
2022-03-01 11:51:34 -08:00
508670248e
Add list_followed subcommand
2022-02-28 16:06:58 -08:00
3c2aee5016
Add test file for previous commit
2022-02-28 13:50:19 -08:00
0909efb18b
Add support for TweetWithVisibilityResults API type, which has a different JSON structure for some reason
2022-02-28 13:44:42 -08:00
648098352b
Fix scraper treating NotFoundError as a bug requiring retries
2022-02-27 23:35:44 -08:00
52370a4f9d
Recognized deleted users when scraping, create a user with a fake ID
2022-02-27 23:06:06 -08:00
772fa247f7
Add creation of unknown users with known handles (i.e., deleted accounts)
2022-02-26 22:09:27 -08:00
373b9ce16e
Add IsFollowed field on Users
2022-02-26 15:58:30 -08:00
19c4b14704
Use a more lint-approved case-insensitive comparator
2022-02-14 18:06:04 -08:00
82f5721ac2
Make user handle matching case-insensitive in tweet trove
2022-02-14 17:58:10 -08:00
8a70f55e1b
Update some more stuff to match API changes
...
- set X-Twitter-Client-Language header to "en"
- Add a new tombstone type (age-restricted)
2022-02-14 17:18:23 -08:00
9cd7f4171a
Add proper logging module, set logging level to 'info' by default (hide very verbose debug results)
2022-02-14 14:31:26 -08:00
e0f05cbeea
Refactor all scraper areas to use TweetTroves where applicable
2022-02-12 20:39:30 -08:00
4b2df1c628
Fix wrong 'Instruction' item being chosen following new API update :V
2022-02-12 15:51:55 -08:00
2773214855
Add test file for previous commit
2022-02-04 17:34:15 -08:00
09b8685f19
Add polls for APIv2
2022-02-04 17:05:27 -08:00
8d406e1b11
Minify that test file
2022-02-02 15:21:42 -08:00
359c166363
Add test file for previous commit
2022-02-02 14:53:10 -08:00
8892785aad
Add fetching of tombstoned users for APIv2
2022-02-02 14:49:46 -08:00
1e1e97b5ca
Add TweetTrove#FillMissingUserIDs helper method
2022-02-02 14:43:17 -08:00
3ca67418e0
Add test file for previous commit
2022-02-02 12:26:02 -08:00
df1a7d2388
Add recognizing bottom of feed
2022-02-02 12:10:24 -08:00
c00ff06b3f
URLs need to have a TweetID lol
2022-02-01 18:21:48 -08:00
32e1558800
Fix link parsing for retweets
2022-02-01 17:36:23 -08:00
9eb3e42539
Add link parsing for APIv2
2022-02-01 16:56:37 -08:00
1bc6aec3b5
Incorporate the new parsing structure into the actual program
2022-02-01 16:04:52 -08:00
b2199e9b59
Add Url#ShortText field
2022-02-01 15:48:43 -08:00
069ab83fba
Refactor scraper tests
...
- use Stretchr assert package for assertions
- use "dot imports" to remove "scraper.XYZ" from everything
2022-01-31 19:14:14 -08:00
babbdecf6c
Fix linting error
2022-01-30 18:03:09 -08:00
12be70f541
Add test files
2022-01-30 17:58:03 -08:00
