How To Scrape Linkedin
How to scrape and collect data from LinkedIn?
Scrape and collect data from LinkedIn sounds interesting. With over 22 million members, it’s one of the largest professional social networks in the known to professionals, LinkedIn allows them to develop their professional network, but also to prospect or look for a job.
The social network is full of interesting data for for finding professional profiles or for recruiting. ScrapingBot offers a LinkedIn scraper to scrape and collect public data from LinkedIn profile pages and LinkedIn company the data you want in JSON, without any blocking.
Example of the data you can collect:
LinkedIn profile: URL, name, position, current company name & link, avatar, about, city, following, education details, posts, experience, education, certifications, courses, languages, groups.
LinkedIn company profile: URL, name, sphere, followers, employees number, about, locations, employees updates, websites, industries, company size, headquarters, type, founded
How to start scraping LinkedIn?
ScrapingBot is here to help you scrape LinkedIn profiles, company profiles and job results as easily and efficiently as possible.
Start scraping LinkedIn in 3 steps:
1. Create an account
Want to start scraping LinkedIn right now? No problem, ScrapingBot offers FREE ACCESS with 100 credits per month to get started. No payment information required, and no any you already have an account, just log in.
From the homepage, click on “FREE PLAN” or “Register” to create a free account.
2. Configure your scraping
Once your account has been created, go to Documentation, to the “Data Scraper API” section to be able to start scraping what you want.
Setting up the endpoint is a two-step process with two API ‘s a little longer than usual because social networks have many protections to avoid robots and get scraped. These two calls make it possible to recover the data without being blocked.
Step 1: Get your Response ID
First, you must make a first POST request to get the ResponseID which will be used do this, you must enter the following parameters in the body, then run a first API call:
scraper(“linkedinProfile” to scrape LinkedIn profiles, “linkedinCompanyProfile” to scrape LinkedIn company profiles)urlThe URL of the LinkedIn profile you want to scrape.
Here is the Request Endpoint:
Step 2: Configure your second request
Now that you have your responseId, you will be able to configure a GET request to ask for the response.
Here is the Response Endpoint:
Complete the Response Endpoint with these parameters:
responseIdInsert here the Response ID that you had with the POST API callscraperYou must fill in the same as that used previously for the POST API call (“linkedinProfile” or “linkedinCompanyProfile”)
When it’s ready, execute the API call. The result you will get will tell you if the scraping is ready to use or it’s not, you will get this pending message:
{status: “pending”, message: “Scraping is not finished for this request, try again in a few”}
In this case, you have to try again until it’s ready.
3. Let’s scraping!
Your web scraping setup is now ready to use! You can start collecting data from Linkedin Profiles and Company Profiles.
NB: If you need more help, a sample code to execute the call to this API is available in the documentation right here (“Data Scraper API” section).
How To Scrape LinkedIn And Use The Data Effectively – Zopto
Internet users generate 2. 5 quintillion bytes of data every single day. By 2025, there will be 150 trillion gigabytes of data to analyze.
And while it’s tough to picture how much data that actually is, it’s easy to see there is a lot to be learned by gathering information on the internet. Not to mention endless information on potential clients just waiting to be reached out to.
But who has the time to comb through all that data for lead generation? Manually looking for leads, collecting data, and turning it into a viable list of potential customers is time-consuming and a waste of valuable hours.
That’s where data scraping comes in.
In this post, we’ll show you what data scraping is, how to scrape data from LinkedIn (safely and legally), and how it will drastically boost your lead generation efforts.
Already got a list of potential leads? Let’s turn them into a steady stream of customers every month. Check out Zopto’s free demo to see how it works.
What Exactly is Data Scraping?
Instead, web scraping will literally scrap the web for you, collect data you’re looking for, and then structure that data into an easy-to-understand format.
Web scraping services automate this process so it can be done on autopilot. Meaning you can automatically find the best leads (including all their publicly available information) and then use this to make your lead generation much faster.
Why Should You Try Scraping Data from LinkedIn?
Scraping the entire internet for data is going to churn up generic information, even with the help of data scraping services.
For B2B businesses, focusing on LinkedIn for data scraping will yield much better results. This platform is full of high-level business executives and employees that make up targeted audiences in almost every B2B market.
With almost 740 million users, 90 million senior-level influencers, and around 30 million companies, it’s a gold mine of information to help you grow your business faster.
Scraping data from LinkedIn gives you the opportunity to leverage all this data to create a tailored list of potential leads with all the information you need to create effective outreach campaigns.
The real question should be, why wouldn’t you try scraping data from LinkedIn?
What LinkedIn Has to Say About Scraping
This is where the waters can get a little muddy. Officially, LinkedIn isn’t a fan of data scraping and discourages scrapers from the platform to protect user data and stop the space from being flooded with bad actors.
However, it’s completely legal to scrape public data from the platform. In fact, back in 2019, LinkedIn lost a Supreme court case against a San Francisco startup called hiQ Labs over the matter.
After blocking hiQ Labs from scraping data for research purposes, the startup filed an injunction against LinkedIn which was upheld in the 9th US Circuit Court of Appeals.
Basically, they ruled that people who make data publicly available on a social site don’t have a reasonable expectation of privacy, and so data scraping wasn’t violating any privacy rights.
This set a precedent that scraping publicly available user data doesn’t go against the terms of service and so won’t get you into trouble with the platform.
LinkedIn Data Scraping Tools and Methods
When it comes to how to scrape LinkedIn, there are tools designed to make it much easier. Here are the top five currently being used:
1.
is easily one of the best scraping tools available. It extracts data and arranges it in CSV files, excel sheets and APIs, depending on what works for you.
With customizable extractors and the ability to build your own, it gives you great control over the type of information you target. It also generates reports showing you trends and changes on the websites you’re scraping, which is a great bonus.
The price for this tool ranges from $199 to $9999 depending on the size of your organization, but there is a free trial to get started with.
2.
This one is a chrome extension and a popular lead gen tool for social media platforms in particular.
It works with LinkedIn Sales Navigator to do targeted searches for specific data and then exports everything to a CSV you can seamlessly integrate into your outreach software.
It’s free to use for up to 800 scans and then just $9 a month for up to 1200, which is great for startups and small businesses looking to test the waters of data scraping.
3. Octoparse
This scraping tool claims to mimic human behavior when scraping and uses multiple IPs to avoid getting blocked by any social sites. Data extracted is stored in the cloud so it’s easy enough to transfer to your outreach software, but doesn’t provide a CSV which is usually simpler to use.
Plans range from $75 to $249 a month depending on the amount of data you want to export but there is a free version to try up to 10, 000 exports.
4. FindThatLead
FindThatLead is a LinkedIn profile scraper that focuses on extracting emails of potential leads so you have somewhere to send your outreach campaigns.
The software has a full dashboard but also comes with a Chrome extension which does a bulk profile crawl to extract emails directly from LinkedIn whenever you’re surfing and exports them directly to your dashboard ready to use.
This one doesn’t scrape any additional data though which does come in useful when you want to personalize your outreach campaigns.
Prices range from $29 to $399 a month, but the free plan will generate ten emails a day which isn’t bad as a starting point.
5.
Web Scraper is the final tool on the list which is both a Chrome extension and a cloud-based data scraper.
It specializes in web page data extraction so will scrape for text, links, files, and more. It also allows you to create sitemaps to navigate different sites (including LinkedIn).
This tool works on credits, and pricing ranges from $50 to $250 for up to 2 million credits.
Scraping Data from LinkedIn Groups
Whatever LinkedIn scraper you decide to use, focusing on data from LinkedIn groups is a great strategy.
You’ll find multiple LinkedIn groups in any B2B vertical and they are packed with potential customers. Building and managing a list of leads from this goldmine of data is an easy way to get a warm list of clients who are perfect for your product or service.
Not to mention you’ll get far more detailed information from groups. Rather than just a name and email, you’ll be able to scrap job titles, three top skills, company name, previous companies, and the different groups they’re a member of.
All of this additional information will help you personalize your outreach campaigns and make real connections with potential clients straight away.
Scraping Data from LinkedIn Profiles
As well as groups, individual LinkedIn user profiles offer great information for your lead generation.
Whenever you visit a public LinkedIn profile, you can use any of the Chrome extension tools to automatically scrape names, email addresses, job titles, seniority, skills, industry – pretty much any information about the person on their personal or company profile.
Once you have that information, lead gen becomes much quicker and more successful.
How Scraping Data from LinkedIn can Boost Your Lead Gen
Scraping data is just the first step in your lead generation efforts. Once you’ve found a tool that works for your company and begin generating a list of potential leads, you can use this information to create much more effective outreach campaigns.
From data scraping you’ll not only have contact information such as name, email address, or phone number, you’ll also have more in-depth information such as job title, skills, awards, etc.
Using Zopto’s automation tools outreach campaign software, you can personalize your outreach messages using this demographic information.
Rather than sending out the same generic emails to everyone and getting hardly any replies, you can automate personalized drip campaigns that will bring in a consistent stream of warm leads every month.
The best part is once you’ve integrated your list of potential customers you got from your LinkedIn scraper, the outreach process is incredibly easy to set up and totally automated.
From your Zopto dashboard, you can easily see who you’ve reached out to, who has replied, and who you need to follow up with.
It takes the time and effort out of outreach so you can focus on what matters – closing deals and providing the best service.
Make Your Scraped Data Work for You
Scraping data is the easy part – once you have it set up, it will help you collect a stream of data.
The key is making that data work for you and leveraging the information you extract. While a lot of B2B companies simply find contact information and cold email customers, the top businesses growing year on year know how to use this data more effectively.
With Zopto, you get the opportunity to make using your data much easier than ever before, giving you time to focus on more important stuff. Try out a free demo and start automating your campaigns today.
NickNick Biggs is a content marketer from Denver, CO. He helps B2B companies develop awesome content to connect with their audience. When he’s not working on content strategy, you might catch him out in the mountains, attempting a home reno project, or planning his next adventure.
How to Scrape Data from Linkedin Using Proxies
With over 500 million users, LinkedIn is the digital Rolodex of the modern age. If you don’t have an account you should probably get one. You can rub shoulders with major players in your industry, creep on old high school acquaintances, and strategize your next business ’s all for the normal user of LinkedIn, which I am, and which you should ever, for the scraper, LinkedIn has an entirely different meaning. Instead of connecting manually with people in an industry, scrapers see LinkedIn as a gold-filled mine of personal information. A mine with 500+ million (and growing) nuggets, all of which can be harvested in a variety of there are company profiles on LinkedIn, which is separate from individual users and adds an entire other element for a Scrape LinkedIn? The answer should be clear: to get all that information. User profiles have names, email addresses, industries, skill competencies, etc. Companies have a number of employees, job postings, current employees, and a host of other important nkedIn is a literal representation of people and companies in the workforce, and they keep their info up to date. This data is incredibly course, you can’t scrape all the data I listed above. But you can scrape some of LinkedIn Allow Scraping? Let’s all yell “NO! ” together so the point gets across. LinkedIn is very, very against scraping of any kind. It recognizes the worth of its customers in terms of analytics and privacy and will continue to fight tooth and nail to keep scrapers off the site. You can read it’s clear statement titled “Prohibition of Scraping Software” to get the that language is solid, this subject is best illustrated by the lawsuit LinkedIn took out against 100 anonymous data scrapers who did what you’re trying to do but did it poorly. The verdict of the case has not been decided at the time of writing, and it brings up many issues around scraping that are beyond the purview of this point I’m trying to make is that if you do plan to scrape LinkedIn, be very cautious. They really don’t want you to do it, so if you plan to you have to do it to Scrape LinkedInDoing it right consists of many factors. You need to think about:The applications required to do the scrapingThe parameters you need to set in the applicationsThe type of pages you will scrape on LinkedIn (public or private)The types of proxies to use, and how many proxies to useAn easy sample to scrape LinkedIn with pythonLinkedIn Crawling ApplicationsThere are specific applications meant for LinkedIn and LinkedIn alone — like Octoparse. Then there are multi-functional tools like Scrapebox (Good for public profiles) more: LinkedIn Scraper 101: How to Scrape LinkedIn Profiles with PythonChoosing an application is important, as many of them cost money. You’ll want to have a full understanding of the software itself, and then what you’re trying to get out of LinkedIn in order to make a solid return on your rameters within the Application Need to note! Once you’ve settled on an application you’ll need to adjust two key settings inside it. This is generally true for all scraping procedures, but specifically for LinkedIn as it is more sensitive than other websites. 1. ThreadsThreads in scraping software details the number of open connections you are using to scrape. The more threads the faster the scrape; the more threads the faster you will get flagged and very cautious use one thread per proxy. That’s what a true human does, so anything more than that will, at some point, become suspicious. However, plenty of scrapers use up to 10 threads per to LinkedIn’s extreme policy against scraping, I recommend staying to the single thread per proxy. Yes, it will slow results and cost more in the long run. In my view, those are costs built into scraping LinkedIn and avoiding a lawsuit. 2. TimeoutsThe second major factor in adjusting your application’s scrape settings is timeouts. Timeouts are the literal amount of time it takes for a server to respond to a proxy before the proxy starts a new your timeouts are set to 10 seconds, your proxy will send another request for information from the server after 10 seconds of it not scrapers set the timeout very low: 1 or 2 seconds. This produces a huge number of results because it creates new requests for information often, meaning you get results more ’t do this. Set your timeouts high, between 30-60 seconds. This gives the server a solid pause before that particular proxy sends another of it like a human: does a human reload a website’s home page every second if there is lag? Maybe, but they don’t do it a thousand times in a thousand seconds on setting your timeouts high you avoid a lot of the detection by LinkedIn and don’t overwhelm them with repeated to Avoid Proxies Get banned or blockedScraping Public Profiles on LinkedIn Through Search EnginesMoving away from the applications let’s get into LinkedIn itself. LinkedIn is primarily used as a private network. To see most of its information you have to create an account, log in, and start connecting with ever, it has plenty of public pages. These can be viewed without an account, and can, therefore, be scraped without logging in, for you can easily view the public LinkedIn profile without an are free to scrape public pages on LinkedIn like any normal scrape that starts with a search engine, You have to enter the correct search terms, like including “”, which will generate results in Google that point to specific LinkedIn scraper can then access the information available on these public pages and return it to you. You’ll be scraping both Google and LinkedIn in this context, so you’ll want to be careful not to set off the alarm bells for either of can get very specific with this, searching for an industry sector of company pages on LinkedIn through an engine, like Microsoft or Google or Apple. You would do this by scraping for “Apple LinkedIn” and then scraping the will only give you public pages though, and you may not want to be Rotating Backconnect Proxy to anonymous ScrapingIt’s not difficult to Scrape public files on LinkedIn on Google, Even just use Scrapebox that can help you handle it in many, if you just want to Scrape public profiles the best solution is to use backconnect rotating proxies for Scraping data on google and LinkedIn! Luminati – 72+ million residential IPs in proxy poolSmartproxy – 40+ million residential IPs in proxy poolShifter – 31+ million residential IPs in proxy poolRelated: How Backconnect Proxies Work? Scraping Private Profiles on LinkedInThe scraping of private accounts is the specific line in the sand that LinkedIn doesn’t want you to cross. It’s not happy that you scrape public pages, but they’re public, and there’s not much they can do about it from a legal ivate pages are another matter. When a person signs up with LinkedIn they are told their information will be kept private, not sold to other companies, and used for internal use only. When a scraper comes along to grab that information LinkedIn has a major problem on its hands. I don’t condone this activity if you’re using your scrape to sell an individual’s information. This basically means you’d be bypassing LinkedIn’s privacy clause, harvesting personal information from people, then selling it to companies for a profit. Not the coolest thing to are other reasons to scrape this information though. Maybe you’re on a job hunt and want to find programmers in a specific city or available jobs in a new state. You can scrape for research, too. Either of these seems fine to me, but the for-profit model doesn’ AccountsThe way to scrape private pages on LinkedIn is to create an account. Once you do this and actually log into LinkedIn you’ll be able to search as much as you want. Remember, this account isn’t for connecting with people, but as an access point to LinkedIn for a do this I recommend Octoparse. Their software allows you to log in to LinkedIn with an account and apply specific searches and scrapes with a drag and drop interface, all while showing you the LinkedIn page you’re on. It’s very nice visually if a little clunky to could figure out a way to do it with other applications but it won’t be as lated: How to Scrape Amazon reviews with OctoparseSearch and HarvestAfter creating the account, just figured out what you want to search. If you try and find Microsoft employees a ton of people will come up. You can have the scraper harvest any information that is available to you as a non-connection. Basically name, position, sometimes the email of the information is still private unless you connect with people, and if you do that you’re basically just running a normal LinkedIn Dedicated Proxy Per AccountBy doing the above you are using a direct automation tool within LinkedIn. The potential for getting caught here is huge, so make sure to follow the threads and timeouts rules, make sure you’re using one proxy IP address to create the account, and then scrape on that account. This is all about appearing like a human. Most humans don’t access LinkedIn from a different IP address every few hours. They access it from one IP address: their home you create the account with a proxy IP, use the same proxy IP to scrape on the account, and set all your parameters correctly you will greatly reduce the chances of getting blocked or and Number of ProxiesThe final element in all this is the types of proxies you use, and how many of them you use. This coincides pretty heavily with your budget because more proxies (and better ones) equals more cash. Keep that in mind for this whole you want to scrape private profiles of Linkedin accounts, you have to use dedicated proxies for each account! For you have to log in to views others private profiles, and Linkedin is so strict to IP，When you change the IP to login to account, you have to verify via email! You want elite private proxies for scraping LinkedIn. With a lawsuit underway, LinkedIn is not kidding around about punishing scrapers. This means you’ll want elite private proxies and only elite dedicated proxies offer the most anonymous and secure HEADER settings out of all the proxy types, and give you unfettered access and speeds. Shared proxies or free proxies (even lesser private proxies) are simply not secure or fast enough to do the more: The Difference Between Shared and Private ProxiesYou’ll also want to test your proxies to make sure they work with LinkedIn. Due to LinkedIn’s anti-scrape stance, it has a large list of blacklisted IPs. If your proxies are in this list they won’t work at all. Contact your provider to get these details, or test it out for yourself and then chat with of ProxiesDepending on the size of your scrape you’re going to need a number of them. The general rule of thumb is the more proxies the better, especially when scraping a difficult you stick to a single proxy per account and want to harvest a lot of data quickly, consider 50 accounts and 50 proxies as a place to get you want to do more proxies per account (which I don’t recommend), grab somewhere in the 100-200 range and rotate them often so they don’t get noticed, then blocked, banned, and fewer proxies you have the more often they’ll be detected. This is always an experiment, so make sure you test Many Proxies Do I Need for My Application? Wrapping UpScraping LinkedIn requires proxies and moxie. You have to really want to do it because it’s not going to be easy, and could result in blacklisted IPs or a lawsuit. As such, take precautionary measures. Understand why you’re scraping LinkedIn, and then reach those specific goals carefully.
