Parser Html Python
Web Scraping and Parsing HTML in Python with Beautiful Soup
The internet has an amazingly wide variety of information for human consumption. But this data is often difficult to access programmatically if it doesn’t come in the form of a dedicated REST API. With Python tools like Beautiful Soup, you can scrape and parse this data directly from web pages to use for your projects and applications.
Let’s use the example of scraping MIDI data from the internet to train a neural network with Magenta that can generate classic Nintendo-sounding music. In order to do this, we’ll need a set of MIDI music from old Nintendo games. Using Beautiful Soup we can get this data from the Video Game Music Archive.
Getting started and setting up dependencies
Before moving on, you will need to make sure you have an up to date version of Python 3 and pip installed. Make sure you create and activate a virtual environment before installing any dependencies.
You’ll need to install the Requests library for making HTTP requests to get data from the web page, and Beautiful Soup for parsing through the HTML.
With your virtual environment activated, run the following command in your terminal:
pip install requests==2. 22. 0 beautifulsoup4==4. 8. 1
We’re using Beautiful Soup 4 because it’s the latest version and Beautiful Soup 3 is no longer being developed or supported.
Using Requests to scrape data for Beautiful Soup to parse
First let’s write some code to grab the HTML from the web page, and look at how we can start parsing through it. The following code will send a GET request to the web page we want, and create a BeautifulSoup object with the HTML from that page:
import requests
from bs4 import BeautifulSoup
vgm_url = ”
html_text = (vgm_url)
soup = BeautifulSoup(html_text, ”)
With this soup object, you can navigate and search through the HTML for data that you want. For example, if you run after the previous code in a Python shell you’ll get the title of the web page. If you run print(t_text()), you will see all of the text on the page.
Getting familiar with Beautiful Soup
The find() and find_all() methods are among the most powerful weapons in your arsenal. () is great for cases where you know there is only one element you’re looking for, such as the body tag. On this page, (id=’banner_ad’) will get you the text from the HTML element for the banner advertisement.
nd_all() is the most common method you will be using in your web scraping adventures. Using this you can iterate through all of the hyperlinks on the page and print their URLs:
for link in nd_all(‘a’):
print((‘href’))
You can also provide different arguments to find_all, such as regular expressions or tag attributes to filter your search as specifically as you want. You can find lots of cool features in the documentation.
Parsing and navigating HTML with BeautifulSoup
Before writing more code to parse the content that we want, let’s first take a look at the HTML that’s rendered by the browser. Every web page is different, and sometimes getting the right data out of them requires a bit of creativity, pattern recognition, and experimentation.
Our goal is to download a bunch of MIDI files, but there are a lot of duplicate tracks on this webpage as well as remixes of songs. We only want one of each song, and because we ultimately want to use this data to train a neural network to generate accurate Nintendo music, we won’t want to train it on user-created remixes.
When you’re writing code to parse through a web page, it’s usually helpful to use the developer tools available to you in most modern browsers. If you right-click on the element you’re interested in, you can inspect the HTML behind that element to figure out how you can programmatically access the data you want.
Let’s use the find_all method to go through all of the links on the page, but use regular expressions to filter through them so we are only getting links that contain MIDI files whose text has no parentheses, which will allow us to exclude all of the duplicates and remixes.
Create a file called and add the following code to it:
import re
if __name__ == ‘__main__’:
attrs = {
‘href’: mpile(r’\$’)}
tracks = nd_all(‘a’, attrs=attrs, mpile(r’^((?! \(). )*$’))
count = 0
for track in tracks:
print(track)
count += 1
print(len(tracks))
This will filter through all of the MIDI files that we want on the page, print out the link tag corresponding to them, and then print how many files we filtered.
Run the code in your terminal with the command python
Downloading the MIDI files we want from the webpage
Now that we have working code to iterate through every MIDI file that we want, we have to write code to download all of them.
In, add a function to your code called download_track, and call that function for each track in the loop iterating through them:
def download_track(count, track_element):
# Get the title of the track from the HTML element
track_title = (). replace(‘/’, ‘-‘)
download_url = ‘{}{}'(vgm_url, track_element[‘href’])
file_name = ‘{}_{}'(count, track_title)
# Download the track
r = (download_url, allow_redirects=True)
with open(file_name, ‘wb’) as f:
(ntent)
# Print to the console to keep track of how the scraping is coming along.
print(‘Downloaded: {}'(track_title, download_url))
download_track(count, track)
In this download_track function, we’re passing the Beautiful Soup object representing the HTML element of the link to the MIDI file, along with a unique number to use in the filename to avoid possible naming collisions.
Run this code from a directory where you want to save all of the MIDI files, and watch your terminal screen display all 2230 MIDIs that you downloaded (at the time of writing this). This is just one specific practical example of what you can do with Beautiful Soup.
The vast expanse of the World Wide Web
Now that you can programmatically grab things from web pages, you have access to a huge source of data for whatever your projects need. One thing to keep in mind is that changes to a web page’s HTML might break your code, so make sure to keep everything up to date if you’re building applications on top of this.
If you’re looking for something to do with the data you just grabbed from the Video Game Music Archive, you can try using Python libraries like Mido to work with MIDI data to clean it up, or use Magenta to train a neural network with it or have fun building a phone number people can call to hear Nintendo music.
I’m looking forward to seeing what you build. Feel free to reach out and share your experiences or ask any questions.
Email:
Twitter: @Sagnewshreds
Github: Sagnew
Twitch (streaming live code): Sagnewshreds
html.parser — Simple HTML and XHTML parser — Python …
Source code: Lib/html/
This module defines a class HTMLParser which serves as the basis for
parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.
class (*, convert_charrefs=True)¶
Create a parser instance able to parse invalid markup.
If convert_charrefs is True (the default), all character
references (except the ones in script/style elements) are
automatically converted to the corresponding Unicode characters.
An HTMLParser instance is fed HTML data and calls handler methods
when start tags, end tags, text, comments, and other markup elements are
encountered. The user should subclass HTMLParser and override its
methods to implement the desired behavior.
This parser does not check that end tags match start tags or call the end-tag
handler for elements which are closed implicitly by closing an outer element.
Changed in version 3. 4: convert_charrefs keyword argument added.
Changed in version 3. 5: The default value for argument convert_charrefs is now True.
Example HTML Parser Application¶
As a basic example, below is a simple HTML parser that uses the
HTMLParser class to print out start tags, end tags, and data
as they are encountered:
from import HTMLParser
class MyHTMLParser(HTMLParser):
def handle_starttag(self, tag, attrs):
print(“Encountered a start tag:”, tag)
def handle_endtag(self, tag):
print(“Encountered an end tag:”, tag)
def handle_data(self, data):
print(“Encountered some data:”, data)
parser = MyHTMLParser()
(‘
‘
Parse me!
‘)
The output will then be:
Encountered a start tag: html
Encountered a start tag: head
Encountered a start tag: title
Encountered some data: Test
Encountered an end tag: title
Encountered an end tag: head
Encountered a start tag: body
Encountered a start tag: h1
Encountered some data: Parse me!
Encountered an end tag: h1
Encountered an end tag: body
Encountered an end tag: html
HTMLParser Methods¶
HTMLParser instances have the following methods:
(data)¶
Feed some text to the parser. It is processed insofar as it consists of
complete elements; incomplete data is buffered until more data is fed or
close() is called. data must be str.
()¶
Force processing of all buffered data as if it were followed by an end-of-file
mark. This method may be redefined by a derived class to define additional
processing at the end of the input, but the redefined version should always call
the HTMLParser base class method close().
Reset the instance. Loses all unprocessed data. This is called implicitly at
instantiation time.
Return current line number and offset.
t_starttag_text()¶
Return the text of the most recently opened start tag. This should not normally
be needed for structured processing, but may be useful in dealing with HTML “as
deployed” or for re-generating input with minimal changes (whitespace between
attributes can be preserved, etc. ).
The following methods are called when data or markup elements are encountered
and they are meant to be overridden in a subclass. The base class
implementations do nothing (except for handle_startendtag()):
HTMLParser. handle_starttag(tag, attrs)¶
This method is called to handle the start of a tag (e. g.
The tag argument is the name of the tag converted to lower case. The attrs
argument is a list of (name, value) pairs containing the attributes found
inside the tag’s <> brackets. The name will be translated to lower case,
and quotes in the value have been removed, and character and entity references
have been replaced.
For instance, for the tag ‘)
Decl: DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4. 01//EN” ”
Parsing an element with a few attributes and a title:
>>> (‘
Start tag: img
attr: (‘src’, ”)
attr: (‘alt’, ‘The Python logo’)
>>>
>>> (‘
Python
‘)
Start tag: h1
Data: Python
End tag: h1
The content of script and style elements is returned as is, without
further parsing:
>>> (‘
‘)
Start tag: style
attr: (‘type’, ‘text/css’)
Data: #python { color: green}
End tag: style
>>> (‘‘)
Start tag: script
attr: (‘type’, ‘text/javascript’)
Data: alert(“hello! “);
End tag: script
Parsing comments:
>>> (‘‘… ‘IE-specific content‘)
Comment: a comment
Comment: [if IE 9]>IE-specific content‘):
>>> (‘>>>’)
Named ent: >
Num ent: >
Feeding incomplete chunks to feed() works, but
handle_data() might be called more than once
(unless convert_charrefs is set to True):
>>> for chunk in [‘
Start tag: span
Data: buff
Data: ered
Data: text
End tag: span
Parsing invalid HTML (e. unquoted attributes) also works:
>>> (‘
Start tag: p
Start tag: a
attr: (‘class’, ‘link’)
attr: (‘href’, ‘#main’)
Data: tag soup
End tag: p
End tag: a
Parsing HTML using Python – Stack Overflow
I’m looking for an HTML Parser module for Python that can help me get the tags in the form of Python lists/dictionaries/objects.
If I have a document of the form:
then it should give me a way to access the nested tags via the name or id of the HTML tag so that I can basically ask it to get me the content/text in the div tag with class=’container’ contained within the body tag, or something similar.
If you’ve used Firefox’s “Inspect element” feature (view HTML) you would know that it gives you all the tags in a nice nested manner like a tree.
I’d prefer a built-in module but that might be asking a little too much.
I went through a lot of questions on Stack Overflow and a few blogs on the internet and most of them suggest BeautifulSoup or lxml or HTMLParser but few of these detail the functionality and simply end as a debate over which one is faster/more efficent.
the Tin Man152k39 gold badges201 silver badges282 bronze badges
asked Jul 29 ’12 at 12:00
1
So that I can ask it to get me the content/text in the div tag with class=’container’ contained within the body tag, Or something similar.
try:
from BeautifulSoup import BeautifulSoup
except ImportError:
from bs4 import BeautifulSoup
html = #the HTML code you’ve written above
parsed_html = BeautifulSoup(html)
print((‘div’, attrs={‘class’:’container’}))
You don’t need performance descriptions I guess – just read how BeautifulSoup works. Look at its official documentation.
Edward6252 gold badges10 silver badges29 bronze badges
answered Jul 29 ’12 at 12:12
AadaamAadaam3, 1591 gold badge12 silver badges9 bronze badges
9
I guess what you’re looking for is pyquery:
pyquery: a jquery-like library for python.
An example of what you want may be like:
from pyquery import PyQuery
html = # Your HTML CODE
pq = PyQuery(html)
tag = pq(‘div#id’) # or tag = pq(”)
print ()
And it uses the same selectors as Firefox’s or Chrome’s inspect element. For example:
The inspected element selector is ‘print’. So in pyquery, you just need to pass this selector:
pq(‘print’)
chris Frisina17. 9k20 gold badges75 silver badges157 bronze badges
answered Jul 29 ’12 at 12:47
YusuMishiYusuMishi2, 15716 silver badges7 bronze badges
Here you can read more about different HTML parsers in Python and their performance. Even though the article is a bit dated it still gives you a good overview.
Python HTML parser performance
I’d recommend BeautifulSoup even though it isn’t built in. Just because it’s so easy to work with for those kinds of tasks. Eg:
import urllib2
page = urllib2. urlopen(”)
soup = BeautifulSoup(page)
x = (‘div’, attrs={‘class’: ‘container’})
Matt Ellen9, 9834 gold badges62 silver badges81 bronze badges
answered Jul 29 ’12 at 12:07
QiauQiau5, 1283 gold badges27 silver badges40 bronze badges
3
I recommend lxml for parsing HTML. See “Parsing HTML” (on the lxml site).
In my experience Beautiful Soup messes up on some complex HTML. I believe that is because Beautiful Soup is not a parser, rather a very good string analyzer.
answered Oct 25 ’14 at 18:50
2
I recommend using justext library:
Usage:
Python2:
import requests
import justext
response = (“)
paragraphs = justext. justext(ntent, t_stoplist(“English”))
for paragraph in paragraphs:
print
Python3:
answered Jul 15 ’16 at 15:51
Wesam NaWesam Na1, 75922 silver badges20 bronze badges
I would use EHP
Here it is:
from ehp import *
doc = ”’
”’
html = Html()
dom = (doc)
for ind in (‘div’, (‘class’, ‘container’)):
Output:
Something here
Something else
answered Mar 20 ’16 at 9:44
Not the answer you’re looking for? Browse other questions tagged python xml-parsing html-parsing or ask your own question.
Frequently Asked Questions about parser html python
How do I parse HTML in Python?
Examplefrom html. parser import HTMLParser.class Parser(HTMLParser):# method to append the start tag to the list start_tags.def handle_starttag(self, tag, attrs):global start_tags.start_tags. append(tag)# method to append the end tag to the list end_tags.def handle_endtag(self, tag):More items…
Which Python package can you use to parse HTML?
Beautiful Soup (bs4) is a Python library that is used to parse information out of HTML or XML files. It parses its input into an object on which you can run a variety of searches.Jan 7, 2021
Can Python read HTML?
library known as beautifulsoup. Using this library, we can search for the values of html tags and get specific data like title of the page and the list of headers in the page.
