Jsdom Web Scraping
Web Scraping and Parsing HTML in Node.js with jsdom – Twilio
The internet has a wide variety of information for human consumption. But this data is often difficult to access programmatically if it doesn’t come in the form of a dedicated REST API. With tools like jsdom, you can scrape and parse this data directly from web pages to use for your projects and applications.
Let’s use the example of needing MIDI data to train a neural network that can generate classic Nintendo-sounding music. In order to do this, we’ll need a set of MIDI music from old Nintendo games. Using jsdom we can scrape this data from the Video Game Music Archive.
Getting started and setting up dependencies
Before moving on, you will need to make sure you have an up to date version of and npm installed.
Navigate to the directory where you want this code to live and run the following command in your terminal to create a package for this project:
The –yes argument runs through all of the prompts that you would otherwise have to fill out or skip. Now we have a for our app.
For making HTTP requests to get data from the web page we will use the Got library, and for parsing through the HTML we’ll use Cheerio.
Run the following command in your terminal to install these libraries:
npm install got@10. 4. 0 jsdom@16. 2. 2
jsdom is a pure-JavaScript implementation of many web standards, making it a familiar tool to use for lots of JavaScript developers. Let’s dive into how to use it.
Using Got to retrieve data to use with jsdom
First let’s write some code to grab the HTML from the web page, and look at how we can start parsing through it. The following code will send a GET request to the web page we want, and will create a jsdom object with the HTML from that page, which we’ll name dom:
const fs = require(‘fs’);
const got = require(‘got’);
const jsdom = require(“jsdom”);
const { JSDOM} = jsdom;
const vgmUrl= ”;
got(vgmUrl)(response => {
const dom = new JSDOM();
((‘title’). textContent);})(err => {
(err);});
When you pass the JSDOM constructor a string, you will get back a JSDOM object, from which you can access a number of usable properties such as window. As seen in this code, you can navigate through the HTML and retrieve DOM elements for the data you want using a query selector.
For example, querySelector(‘title’). textContent will get you the text inside of the