Php Web Scraper Library
8 Awesome PHP Web Scraping Libraries and Tools – DZone
Web scraping is something developers encounter on a daily basis.
There could be different needs as far as each scraping task is concerned. It could be a product or stock pricing.
In backend development, web scraping is quite popular. There are people who keep creating quality parsers and scrapers.
In this post, we will explore some of the libraries which can enable scraping websites and storing data in a manner that could be useful for your immediate needs.
In PHP, you can do scraping with some of these libraries:
Goutte
Simple HTML DOM
htmlSQL
cURL
Requests
HTTPful
Buzz
Guzzle
1. Goutte
Description:
The Goutte library is great for it can give you amazing support regarding how to scrape content using PHP.
Based on the Symfony framework, Goutte is a web scraping as well as web crawling library.
Goutte is useful because it provides APIs to crawl websites and scrape data from the HTML/XML responses.
Goutte is licensed under the MIT license.
Features:
It works well with big projects.
It is OOP based.
It carries a medium parsing speed.
Requirements:
Goutte depends on PHP 5. 5+ and Guzzle 6+.
Documentation:
Learn more:
2. Simple HTML DOM
Written in PHP5+, an HTML DOM parser is good because it enables you to access and use HTML quite easily and comfortably.
With it, you can find the tags on an HTML page with selectors pretty much like jQuery.
You can scrape content from HTML in a single line.
It is not as fast as some of the other libraries.
Simple HTML DOM is licensed under the MIT license.
It supports invalid HTML.
Require PHP 5+.
3. htmlSQL
Basically, it is a PHP library which is experimental. It is useful because it enables you to access HTML values with a SQL-like syntax.
What this means is that you don’t need to write complex functions or regular expressions in order to scrape specific values.
If you are someone who likes SQL, you would also love this experimental library.
How it will be useful is that you can leverage it for any kind of miscellaneous task and parsing a web page pretty quickly.
While it stopped receiving updates/support in 2006, htmlSQL remains a reliable library for parsing and scraping.
htmlSQL is licensed under the BSD license.
It provides relatively fast parsing, but it has a limited functionality.
Any flavor of PHP4+ should do.
Snoopy PHP class – Version 1. 2. 3 (optional – required for web transfers).
4. cURL
cURL is well-known as one of the most popular libraries (a built-in PHP component) for extracting data from web pages.
There is no requirement to include third-party files and classes as it is a standardized PHP-library.
When you want to use PHP’s cURL functions, all you need do is install the » libcurl package. PHP will need libcurl version 7. 10. 5 or later.
5. Requests
Description
Requests is an HTTP library written in PHP.
It is sort of based on the API from the excellent Requests Python library.
Requests enable you to send HEAD, GET, POST, PUT, DELETE, and PATCH HTTP requests.
With the help of Requests, you can add headers, form data, multipart files, and parameters with simple arrays, and access the response data in the same way.
Requests is ISC Licensed.
International Domains and URLs.
Browser-style SSL Verification.
Basic/Digest Authentication.
Automatic Decompression.
Connection Timeouts.
Requires PHP version 5. 2+
6. HTTPful
HTTPful is a pretty straightforward PHP library. It is good because it is chainable as well as readable. It is aimed at making HTTP readable.
Why it is considered useful is because it allows the developer to focus on interacting with APIs rather than having to navigate through curl set_opt pages. It is also great a PHP REST client.
HTTPful is licensed under the MIT license.
Readable HTTP Method Support (GET, PUT, POST, DELETE, HEAD, PATCH, and OPTIONS).
Custom Headers.
Automatic “Smart” Parsing.
Automatic Payload Serialization.
Basic Auth.
Client Side Certificate Auth.
Request “Templates. ”
Requires PHP version 5. 3+
7. Buzz
Buzz is useful as it is quite a light library and enables you to issue HTTP requests.
Moreover, Buzz is designed to be simple and it carries the characteristics of a web browser.
Buzz is licensed under the MIT license.
Simple API.
High performance.
Requires PHP version 7. 1.
8. Guzzle
Guzzle is useful because it is a PHP HTTP client which enables you to send HTTP requests in an easy manner. It is also easy to integrate with web services.
It has a simple interface which helps you build query strings, POST requests, streaming large uploads, stream large downloads, use HTTP cookies, upload JSON data, etc.
It can send both synchronous and asynchronous requests with the help of the same interface.
It makes use of PSR-7 interfaces for requests, responses, and streams. This enables you to utilize other PSR-7 compatible libraries with Guzzle.
It can abstract away the underlying HTTP transport, enabling you to write environment and transport agnostic code; i. e., no hard dependency on cURL, PHP streams, sockets, or non-blocking event loops.
Middleware system enables you to augment and compose client behavior.
Requires PHP version 5. 3. 3+.
Conclusion
As you can see, there are web scraping tool at your disposal and it will depend upon your web scraping needs as to what kind of tools will suit you.
However, a basic understanding of these PHP libraries can help you navigate through the maze of many libraries that exist and arrive at something useful.
I hope that you liked reading this post. Feel free to share your feedback and comments!
Topics:
php libraries,
web dev,
web scraping,
web parsing
Opinions expressed by DZone contributors are their own.
Goutte, a simple PHP Web Scraper – GitHub
Goutte is a screen scraping and web crawling library for PHP.
Goutte provides a nice API to crawl websites and extract data from the HTML/XML
responses.
Requirements
Goutte depends on PHP 7. 1+.
Installation
Add fabpot/goutte as a require dependency in your file:
composer require fabpot/goutte
Usage
Create a Goutte Client instance (which extends
Symfony\Component\BrowserKit\HttpBrowser):
use Goutte\Client;
$client = new Client();
Make requests with the request() method:
// Go to the website
$crawler = $client->request(‘GET’, ”);
The method returns a Crawler object
(Symfony\Component\DomCrawler\Crawler).
To use your own HTTP settings, you may create and pass an HttpClient
instance to Goutte. For example, to add a 60 second request timeout:
use Symfony\Component\HttpClient\HttpClient;
$client = new Client(HttpClient::create([‘timeout’ => 60]));
Click on links:
// Click on the “Security Advisories” link
$link = $crawler->selectLink(‘Security Advisories’)->link();
$crawler = $client->click($link);
Extract data:
// Get the latest post in this category and display the titles
$crawler->filter(‘h2 > a’)->each(function ($node) {
print $node->text(). “\n”;});
Submit forms:
$crawler = $client->click($crawler->selectLink(‘Sign in’)->link());
$form = $crawler->selectButton(‘Sign in’)->form();
$crawler = $client->submit($form, [‘login’ => ‘fabpot’, ‘password’ => ‘xxxxxx’]);
$crawler->filter(”)->each(function ($node) {
More Information
Read the documentation of the BrowserKit, DomCrawler, and HttpClient
Symfony Components for more information about what you can do with Goutte.
Pronunciation
Goutte is pronounced goot i. e. it rhymes with boot and not out.
Technical Information
Goutte is a thin wrapper around the following Symfony Components:
BrowserKit, CssSelector, DomCrawler, and HttpClient.
License
Goutte is licensed under the MIT license.
PHP Scraper – An opinionated web-scraping library for PHP
by Peter Thaleikis (opens new window) Web scraping using PHP can done easier. This is an opinionated wrapper around some great PHP libraries to make accessing the web easier. The examples tell the story much better. Have a look! # The Idea ️ Accessing websites and collecting basic information of the web is too complex. This wrapper around Goutte (opens new window) makes it easier. It saves you from XPath and co., giving you direct access to everything you need. Web scraping with PHP re-imagined. # Supporters ️ This project is sponsored by: Want to sponsor this project? Contact me (opens new window). # Examples Here are some examples of what the web scraping library can do at this point: # Scrape Meta Information: Most other information can be accessed directly – either as string or an array. # Scrape Content, such as Images: Some information optionally is returned as an array with details. For this example, a simple list of images is available using $web->images too. This should make your web scraping easier. More example code can be found in the sidebar or the tests. # Installation As usual, done via composer: This automatically ensures the package is loaded and you can start to scrape the web. You can now use any of the noted examples. # Contributing Awesome, if you would like contribute please check the guidelines before getting started. # Tests The code is roughly covered with end-to-end tests. For this, simple web-pages are hosted under, loaded and parsed using PHPUnit (opens new window). These tests are also suitable as examples – see tests/! This being said, there are probably edge cases which aren’t working and may cause trouble. If you find one, please raise a bug on GitHub.
