Rate limits to safely scrape and automate
Have you ever noticed that after a few months (or days) of extensive scrolling, posting, and liking, your social network account disconnects you and makes you login again? While Silicon Valley probably loves the time you spend on social media, there are still limits to the amount of activity that they believe to be humanly possible. Social networks disconnect your account to make you re-identify yourself and confirm the security of your account.
How does this affect your Phantom’s automations? As a Phantom is a simple bot that acts on your behalf, the network on which you’re automating will believe you’re the one manually performing each action. Phantoms gain access to your account through your browser’s session cookie, which expires every time you’re logged out of the account. (For more on that, check out How to get your cookies with Phantombuster). This means you have to reconnect the session cookie to your Phantom to allow the Phantom to continue its work.
Disconnected again? Your cookies have expired
Session cookies naturally expire every couple of months, so occasionally reconnecting your cookie to your Phantom is inevitable. If you find yours expiring too often and you're not logging out yourself, it could be that the network is logging you out as your activity rate is too high.
To prevent this, we recommend reducing your activity, keeping in mind the rate limits for different websites.
You should always make sure to spread your actions over several launches throughout the day, within business hours where possible. Do this by setting your Phantom to automatic, repeated launches.
It's also important to note that if you're just starting to automate your account, you should start with a similar amount of actions as you would normally do manually and increase gradually.
Basic rate limits to keep in mind
Regardless of the network you’re using, remember that these limits apply to the total number of actions done on your account in a single day. This includes actions you might be doing manually, as well as the sum of all of your Phantoms’ actions.
LinkedIn depends on whether or not you’re paying for your LinkedIn account. If you have a regular, free account, we recommend processing a maximum of around 80 profiles per day. With paid, premium, or Sales Navigator accounts, it’s possible to do ~150.
Instagram limits depend more on the actions you’re performing. For example, if you’re looking at extracting a large number of followers with the Instagram Follower Collector, accounts a profile is following with the Instagram Following Collector, or posts associated with hashtags from the Instagram Hashtag Collector -- all of these Phantoms can take around 5,000 - 9,000 in one launch, with up to 20 launches per day. Direct actions, however, such as liking, commenting, or auto-following accounts, are much more limited. For the entire breakdown, see Every social network limitation any digital-marketer should know.
Facebook is a tricky one, as it absolutely requires proxies to ensure your Phantom is able to connect to your account in the first place. Luckily, we have a couple of articles explaining that already: How and why to use a proxy. Be sure to check out Troubleshooting with proxies to avoid common user errors for proxy configurations.
Once you’ve set up your proxy, you’ll still need to follow the recommended rate limits for automation. Similar to Instagram, extracting profiles from a large list is fairly easy -- the Facebook Group Extractor can take up to 10,000 members from a group. Liking, messaging, and profile scraping is much more restricted. We recommend scraping 5 per hour with the Facebook Profile Scraper, liking 5 per hour with the Facebook Auto Liker, and messaging 3 per hour with the Facebook Message Sender.
Twitter’s limits vary yet again, but we still have our best practice suggestions! Stick to 10 accounts per launch, 5-8 launches per day with the Twitter Auto Follow, Twitter Auto Retweeter, and Twitter Message Sender. Like up to 1,000 tweets per day with the Twitter Auto Liker, and scrape 5,000 followers every 2 hours with the Twitter Follower Collector.
Did we miss a limit? For any limits unaddressed here, be sure to check out our Blog for the complete list: Every social network limitation any digital-marketer should know.