Changelog ========= v0.0.0 ------ - Initial release. v0.0.1 ------ - Added polls v0.0.2 ------ - Add `is_conversation_scraped` field to Tweets - Add some new tombstone types: "tweet violated twitter rules", "account no longer exists" - No longer re-download media content (tweet content, user profile / banner images) if not needed - Download video thumbnail images as part of video downloading v0.0.3 ------ - Add video `duration` and `view_count` fields - Add `--version` command line option - Improve user image scraping for rando users (download less eagerly) to speed up browsing and save disk space: - Download tiny versions of profile images (48x48 pixels) - Don't download banner images at all v0.0.4 ------ - Escape HTML entities in tweet text - Add support for banned users - Improve tombstone recognition (quote-tweets) - Fix spacing issue around tiny profile images v0.0.5 ------ - BUGFIX: fix parsing of tweet URLs with query params ('?s=20', etc) - BUGFIX: fix bug in scraping users with "600x200" banner images that caused them not to be saved correctly v0.1.0 ------ - Add support for new GraphQL twitter API (dubbed "APIv2" in code). User feed scraping now uses this API - Embedded links (URLs) now have their short versions saved in the DB. v0.1.1 ------ - BUGFIX: fix selection of primary "TimelineAddEntries" field in APIv2 following breaking API change - Add `-h, --help` command line flags to display help message and gracefully exit v0.1.2 ------ - BUGFIX: fix error when trying to match mixed-case user handles in tombstone tweets - BUGFIX: set "Client Language" header to "English" in all cases - Add an option to show debug logging; hide it by default v0.1.3 ------ - Add following users - Add `list_followed` subcommand - Add support for tweets that quote-tweet something from a user who deleted their account v0.1.4 ------ - BUGFIXES: Fixed some annoying bugs caused by Twitter's poor API - ignore empty entries in a user feed instead of crashing - quote tweets no longer include their quoted tweet as an embedded link v0.1.5 ------ - BUGFIX: users changing their profile images should be not broken now - Major refactoring (testing, massive linting improvements, proper Go dependencies like sqlx, etc) - Improve tombstone handling - Update tweet content when a tombstone becomes available - Update tombstone type of a tweet if it changes - If a tweet gets hidden / deleted, it can become a tombstone without losing its content - Enable support for users that change their handles v0.1.6 ------ - Add support for `mobile.twitter.com` tweet URLs - Add basic support for Spaces (i.e., can now basically detect whether a tweet links to a Space or not) v0.1.7 ------ - Add support for multiple videos as well as videos and images in the same tweet - Restructure media file storage into sub-directories to improve performance when retrieving them v0.1.8 ------ - Add proper support for fetching Spaces (with full detail) - Add checking for DMCA takedowns of video content - Add index for fetching tweet threads (index `in_reply_to_id`) - Add indexes to tweet content types to improve fetching performance (index `tweet_id`) - BUGFIX: can now fetch retweets with "visibility results" - BUGFIX: search changed how cursors work at end-of-feed v0.2.0 ------ - Add index on tweets user_id column to accelerate loading of user feeds - Implement login and authenticated requests! v0.2.1 ------ - BUGFIX: fix cursors being broken in authenticated sessions - BUGFIX: fix stats for likes, RTs, etc. getting erased if a tweet gets deleted or hidden - BUGFIX: `fetch_tweet` and `search` subcommands no longer ignore the `-n [number]` flag - ".session" extension for the `--session` flag is now optional, allowing tab-autocompletion v0.2.2 ------ - Add support for Expandable (i.e., "Show more") tweets - Fix an issue with half the tweets on user feeds not getting fetched while logged in v0.3.0 ------ - Now use APIv2 for Tweet Detail pages as well as user feeds. - BUGFIX: fix a lot of bugs that were unfixable using the old API - BUGFIX: fix spaces not updating ended_at time v0.3.1 ------ - HOTFIX: it looks like Twitter changed the label for a timeline module from "homeConversation" to "profile-conversation"! v0.3.2 ------ - BUGFIX: fix duplicate space participants bug (participants in a space should be unique) - Add fetching of a user's Likes tab - Implement "liking" and "unliking" v0.4.0 ------ - Add Web front-end served by `twitter webserver` - Some small improvements to scraping of Spaces v0.4.1 ------ - BUGFIX: fix Spaces showing times 50,000 years in the future (timestamps in millis rather than seconds) - Few UI improvements - fix link preview images overflowing the feed column - removed gray thread-connector line showing up in QTs - Make the search bar work for searches - Add scraper request for HomeTimeline v0.4.2 ------ - BUGFIX: fix long tweets displaying with paragraphs in wrong order - BUGFIX: fix scrolling issues when using "back" button / history - Search bar improvements: - can now paste twitter.com / mobile.twitter.com links into the search bar - go directly to user feed by entering "@elonmusk" (for example) - UI improvements: - Make entities (#hashtags, @mentions) clickable in tweets and user bios - Make tombstones show - Other misc improvements - Add manual re-scrape button to user feeds - Add automatic timeline scraping in background when logged in v0.5.0 ------ - Add support for user "likes" - Logged-in user's "liked" tweets show filled-in heart in the UI - Can "like" (and "unlike") a tweet by clicking on its heart icon - UI improvements: - Show entire threads in UI - Add tabs to User Feed: "Tweets", "Tweets with replies", "Media", "Likes" - Search now supports sort orders: most recent, least recent, most likes, most retweets - Add support for a number of upstream stuff: "x.com" links, "Tweets" renamed to "Posts", new API formats, edited tweets - Bugfixes v0.5.1 ------ - Add a preliminary Lists page which shows all offline-follows - Add search page support for searching users - Automatic background scraping of logged-in user's Likes v0.6.0 ------ - Add DMs - Bugfixes: - Fix crashing because of the composer button when scraping logged-in user's threads - Fix crashing when loading the user feed of a user who isn't scraped yet - UI improvements: - Make page loads much faster by moving image/video downloading to non-blocking background tasks - Get rid of dropdown button on tweets (buttons are just inline now) v0.6.1 ------ - UI improvements: - add the `--auto-open` flag for `twitter webserver` which opens the homepage in a browser on launch - search bar preserves its text between searches - spinner now shows when loading / fetching tweets - on errors, an error toast notification now shows - Add followers/followees pages and queries v0.6.2 ------ - make all timestamps across the app use milliseconds (previously used both seconds or milliseconds in different places) - add `get_user_likes_all` subcommand - UI improvements: - make end-of-feed indicator look better - add favicon - Bugfixes: - fix clobbering of interaction stats if a tweet is deleted (previously they'd all get set to 0) - fix double HTMX spinner on tweets with quoted tweet - fix focused tweets being clickable v0.6.3 ------ - Add Lists - Add `--default-profile` flag which will use and create a profile in a default location (based on OS) - UI improvements: - make scrolling in DMs work way better - hovering over a tweet now highlights it with a light background color - Create a `Twitter.desktop` file in dpkg build v0.6.4 ------ - Can now re-scrape search results from the web UI - UI improvements: - feed "Show More" button looks nicer - online-login form now has a HTMX spinner - Login flow improved: - now auto-scrapes timeline and list of people you follow after logging in - command-line supports 2FA challenges - web UI notifies login failure if challenged for 2FA instead of failing silently - Home page now has a "User Feed" timeline (shows people logged-in user follows) in addition to the "Offline Timeline" v0.6.5 ------ - UI improvements: - Add image full-size on-click viewer - Add images, videos and links to DMs v0.6.6 ------ - BUGFIX: fix server crashing if a media download HTTP request times out - BUGFIX: scraping user feeds now gets long tweets properly - UI improvements: - Fix image sizing issues in link preview images and DM images - Add support for pinned tweets (both fetching and displaying) - New search bar filters: "tombstone:" and "followed_by:" - Add boilerplate for Progressive Web App v0.6.7 ------ - UI improvements: - various bugfixes - make tweets take up less space - Add "list:" search bar filter - BUGFIX: Fix pinned tweets with a quoted-tweet not showing the quoted tweet v0.6.8 ------ - UI improvements: - fix some buggy styling around the pin icon on pinned tweets and error toast messages - BUGFIX: fix scraper crashing when a media download times out (e.g., long video) v0.6.9 ------ - UI improvements: - GIFs now show differently than videos; can be paused and restarted on click - Add unread indicators to DMs, and show unread messages count in nav sidebar - DM messages now have a message preview if the last message was just an attachment (image, tweet, or video) v0.6.10 ------- - UI improvements: - add "Copy Link" button to tweets - add "Go Online" button to nav sidebar - add header section for DMs with open-in-external button - Clicking on the "replying-to" button on a DM scrolls that message into view - Add DM pagination / show-more button v0.6.11 ------- - Add 2 buttons in DM chat header: mark as "read", manually re-scrape - DM list now updates itself live while on the Messages page v0.6.12 ------- - Add support for scraping bookmarks (not making them, yet) - Publish statically linked builds only - Various minor bugfixes v0.6.13 ------- - BUGFIX: no longer throws away all scrape results if a pagination request gets rate-limited - BUGFIX: can now start the webserver without an internet connection - UI improvements: - image carousel now closes if you click outside of it v0.6.14 ------- - Add sending replies and emoji reaccs for DMs - Bookmarks are now scrapable from the web UI - BUGFIX: don't crash when an image or video download 404s v0.6.15 ------- - BUGFIX: `webserver` subcommand now respects the `--session` flag - More types of scraping errors now show toasts in the UI: rate limiting, expired session - New `--delay` flag added to introduce a delay in paginated scrapes v0.6.16 ------- - Add notifications v0.6.17 ------- - BUGFIX: fix a series of bugs relating to inactive users and duplicate user handles v0.6.18 ------- - BUGFIX: fix scraper crashing on "composer" entry in threads - BUGFIX: fix "User followed you" notifications sometimes parsing as "New recommended post" - BUGFIX: emojis with skin tones now work in DMs - UI improvements: - Notifications page is now paginated in web UI v0.6.19 ------- - BUGFIX: fix the scraper `GetUser` function which was broken by an upstream change - UI improvements: - Profile and banner images in user feed header can now be clicked to open full-size versions - Reply previews in DMs now include the username who's being replied to - Notifications can now be marked as "read" via command line and web UI v0.6.20 ------- - BUGFIX: fix sending a reacc causing an error if it's not on the latest message in the chat - BUGFIX: various fixes around error toasts - UI improvements: - reaccs: add a little circle around the logged-in user's reaccs, to indicate them better - "refresh chat room" button now puts a spinner on the chat while it's refreshing v0.6.21 ------- - UI improvements: - on loading a "misshapen" chat image, it will try to reload it a few times - Windows: - Start menu shortcut should work now - "Install for current user only" should now create desktop shortcut properly v0.6.22 ------- - Add background scraping jobs for Bookmarks and logged-in user's User Feed - Manually rescraping a User Feed now re-downloads user's profile and banner images as well v0.7.0 ------ - PKG: Type definitions are now in `pkg/persistence` instead of `pkg/scraper` - BUGFIX: fix notifications page not having user's "likes" filled out on tweets - Compound notifications now tell you how many tweets were "liked", how many users followed you, etc v0.7.1 ------ - BUGFIX: fix banned users getting their user info (followers count, profile image, display name) getting erased - PKG: webserver is now a public package