So as you see it's quite a simple one and intentionally I tried to find a
simple web page for you so here we go. I didn't want to distract you with lots of
content for now. Later you'll be able to grab information from a big website
with lots of data so for now let's try to grab, let's say we want to extract the
names of the cities from this page, so if you want to follow me please typing this
address on your address bar so with the dot HTML at the end, and so we've got
only three cities here that we will be extracting by the code that we will
write will work with any number of rows here. I'll be using the iPython notebook
or the Jupiter notebook as it is called now so it was renamed to Jupiter
notebook, so right right click and open your command line.
Jupiter notebook, and I'll create a Python 3, a notebook, great, so the first thing you
want to do is you want to load this source code in Python, and the way to do that
is by using the requests library. So if you don't have that installed you can
just go ahead and install it with pip install requests. Just like that. I have it
already so already satisfied, but the process is very easy so you already know
how to install packages with pip. You'll also need the BeautifulSoup
library, so to install that you need to say pip install again and not BeautifulSoup,
but bs4 so which stands for BeautifulSoup 4, so that's the latest version of
BeautifulSoup, and then so you want to import quest and so the first thing
you want to do is load the source code. And then we start looking for HTML tags
and extracting elements from those tags. Now let me import BeautifulSoup as well.
So from bs4 import BeautifulSoup,
so that's the syntax you're importing the BeautifulSoup cass from bs4.
If you are on Python 2 this should be slightly different, so you want to import
BeautifulSoup directly like this. Okay, alt enter and go to the next line.
So to load a webpage it's good to create a variable so you can load the web page
source code to this variable so R equals to requests.get, so the get method.
So you point to the library and then to the get method and all you need to pass
here is the URL of the web page that you want to load, so in this case
pythonhow.com.example.html, so don't forget the HTML. This is just a static
web page so you should pass HTML there. Now this should create a requests object
so we're still not there and what you want to do is grab the content from this
requests data type and maybe store it in another variable, so the content store it in
in the C variable like that and if you want to check now what this is. You'll
see that this is a bytes data type, and you can print it if you want
even though this one doesn't look very nice this is actually the source code
that you see in here, so we have the head tags, and the HTML tags and everything else there.
And now here is where the BeautifulSoup comes into play.
So all the request does is it loaded with the source code of a web page but
in a very scrambled form as you see here. Now if you want to make this beautiful
and extract the elements, and the text, and everything out of this source
code, you want to use BeautifulSoup, so all BeautifulSoup does is parsing
this source code and giving you what you want, so giving you the elements of the
HTML tags your interesting about, so you have already loaded this content and now
what you want to do is maybe create a variable and call it soup. And that would
be equal to the beautiful soup and guess what you want to pass here? Well that
would be the content and maybe an another argument so you want to specify the parser
you want to use for parsing these data, and that is normally the
HTML parser, so this assault you want to use almost always. If you don't specify
this, you'll get a warning but still things will work, so I normally pass it there.
And once you've done that so I execute that cell, if you know print soup
dot prettify with empty brackets there, you'll see the source code of the web
page in an organized form, so BeautifulSoup is trained to actually recognize
these tags and then render them in a visual way for the human eye.
However this is just for demonstration. Normally you will not have to actually
use the prettify method a lot because a better method to see this code as I
already mentioned before is to… Let me delete this cell. We don't need that,
so a better way to see that source code is to go to your web page and go to inspect
and here you see a more, a better syntax of the HTML code. So here you'll see that
we have three divisions here with the cities class, we have some more divisions here
but this is what we're interested about. So and the body is everything and if you
expand one of these divisions, you'll see that we have an h2 tag, so a heading tag
and also a paragraph tag so P tag and h2 tags, and also the other division which
is this one here has this h2 tag and the paragraph tag, and Tokyo also has the
same thing. So our duty now is to extract the names of these elements so that
would be the h2, the text of the h2 tags inside the cities tags.
So naturally you'll start thinking about iterating through these boxes which are
actually divisions. So you want to go through here, here, and here and extract
what you want to extract. So we go back to the code and what you want to do is
perform a method called find all, and what you want to find is divs, so divs
but there may be lots of divs in the web page so for instance we have two
more divs here and we don't want these to be found, we only one these three.
But these three as you see they have a common class attribute
which is equal to cities. So we want to make use of that and we
pass here a dictionary which would be class equals to cities. Okay, let me create
a variable here and call it all and execute it. Now if you print all, you'll
see that the divisions have been extracted from the source code so from
the soup which was the entire source code and I'd like you to actually see
closely here. You can see that the first division is divided by comma here
and then the second division starts. So for Paris, Paris is the second and it
ends here, and then Tokyo starts here so we've got a list with three elements, one
for each division. Now if you want to find only the first element with this
class attribute, so cities you'd want to use the find method. All, and in this case
you don't get a list, but you get the code for the division, for the first
division only which happens to be a tag element of BeautifulSoup, so it's not
a plain string but it's a special, let's say a special BeautifulSoup
string so that BeautifulSoup knows its structure, so it knows what are elements,
where the tags are, and where the text is and so on, so that BeautifulSoup is
able to give you the information that you are looking for, so all again.
So you extract the first element. Now an alternative way to extract the
first element is logically, so you have all elements here, is to use
the list indexing. So this object that I just showed you, the tag object of
BeautifulSoup supports indexing. So you execute that and in this
case as you see you extracted the first item of the tag object or you could do
it like this. So you grab all of them so here you have all of them and 0 is the
first one, you get the idea. Ok, but what if you want only the h2 tags
from this div class? Well in that case what you'd want to do this refer to the
all objects and then apply the find all method again and this time you'd want to
get the h2 element. And in this case you don't have a class attribute so you'll
have to leave it like that, and you get an error because what I did here is I
didn't point to this division but I pointed to actually the list containing
all these divisions, so Python is trying to get the h2 but this result set method
doesn't have this h2 element, so what you want to do is you want to point to the
first element of the list and that gives you the h2 element with the tags
and the text which is like a list so you want to perform a zero indexing there.
And if you want London only you apply text and you get London. So this is what we
wanted, right? To extract the cities. So we extracted London. Now how about
extracting Paris and Tokyo? Well as you might guess we need to use a for
loop, but first let me summarize what we did here.
So we loaded the content up here which this this one here, and then we loaded this
content in the BeautifulSoup method. And BeautifulSoup makes
this soup beautiful so that it recognizes these tags and so what we did
then is we found, we extracted from this content, we extracted all the division
elements, so together with the tags and the attributes, and the text inside them.
So everything inside these divisions with a class equals to cities.
Then we can perform for each of these elements of this list we can
perform again a find all method, so we can find subtags of these division tags.
In this case we found the h2 tags and then we grab the first item of the list
which in this case happens to be a list with only one item, so each of these
divisions have one h2 tags or alternatively you could just use find
here and without using this indexing but this is a general method and then we
apply the text attribute there so to extract the text out of this element.
So we got London. Now we need to do the same, but not in this case by iterating so for
let's say item in all, you want to print out, so item here is this one here,
so this is this would be the first item, so you want to print out the item dot find
all and you want to find the h2 tags from this first item for example.
So h2 tags and then you need to apply this zero indexing there and you
want to grab the text from this and that's it. Here are the data.
Alternatively you could just pass P here and you get the paragraphs, so these ones
here, the text. And so that's the idea of loading web pages in Python and
parsing them with BeautifulSoup and extracting text out of the web page.
So sorry if I was a bit repetitive in explaining this stuff, but I really
want to make sure you understand the core concepts, on the other hand if you
found these very basic, I would say let's move on to the next lectures where we
will be extracting some information from a more advanced website and
we'll be extracting links and not only text, so that's a real world program and
a very interesting one. So I'll talk to you later.