TODO user-images-extract-to-table: User profile images and banner images => extract to table. Add `is_downloaded` column. Add test for double-downloading a user, shouldn't change the write-times on these files TODO update-all: For each user in userslist.txt: - update user - download new images if necessary - fetch tweets since last update TODO implement-search TODO fetch-older-tweets: get tweets older than 3200 ago TODO https://twitter.com/michaelmalice/status/1461031030278742020 => "account no longer exists" tombstone TODO cleanup-main: The `cmd/twitter/main.go` file is becoming a disaster, it needs a cleanup badly. lots of copy-pastes that should be functions, code that should be moved to the scraper / persistence modules, etc TODO broadcast-cards panic: Unknown card type: 3691233323:periscope_broadcast panic: Unknown card type: 745291183405076480:broadcast TODO consecutive-tombstones Conversation threads with two or more consecutive tombstones don't get scraped correctly Video size reduction: - add "bitrate" and "available bitrates" fields - option to upgrade or downgrade video quality TODO tombstone updating: - "Tweet is unavailable" tombstones are not always accurate. In user feed, they show as "not available" but the real reason is "deleted" etc; should update the tombstone type when fetching the conversation view - tweet-to-stub: If a tweet is deleted after getting a non-stub version, should still show a tombstone (or a modified one at least) - stub-to-tweet: If a user un-privates after getting a stub-version (or tweet becomes available for other reasons), stub versions should write the real content and "stub" status should be removed TODO: Problem tweets - https://twitter.com/HazardHarringto/status/1481808740445741062: replies not loading - https://twitter.com/TyCardon/status/1480640777281839106: "Media has been disabled in response to report by copyright owner" - https://twitter.com/CovfefeAnon/status/1427693931047837702: parent thread not loading (possible GUI problem) - "account no longer exists" tombstone not being collected TODO: videos-view-count - videos don't parse properly in APIv2 TODO: APIv2-pinned-tweets - feed with pinned tweet