So how does web scrapping work anyway? How is Python able to grab the
information from the webpage and store it as text so that you can analyze it in your
convenience? Well this text that you see here, so this
is an example website and actually this domain is reserved for examples so but
it's a normal web page built with HTML, and CSS, and other word development tools.
So HTML is what renders the elements, the text, and everything on the browser.
Now luckily you can see the code of every web page by going to view page
source as I did here, so this is the code, the HTML code as you see it opens
here with HTML tags and it closes here. And so this has the title of the web page
which is this one here example domain and so on. And most importantly for you
to know is that the HTML script is made of HTML elements, so this line here is an
HTML element so it's a meta element and then we have division elements which is
this one here. We have paragraph elements, these here and these are called tags so
the div tag and the division closing tag, so opening division
tag, closing division tag and then we have the body tag, so here is where the
visible part of the HTML page, of a web page is placed so everything you place
inside the body tags you'll see there so if you have example domain here, this
domain established to be usable etc., here is where you see that text.
This domain is established to be used for etc. So the HTML elements are key for web
scrapping, so basically what you'll do is that
let's say you want to extract the text of h1 tags out of all the divs, all the
divisions, so what you tell Python to do is you tell Python to go to all the h1
tags and extract the text of these tags and Python will do that, but first of all
though you need to load this entire script in Python and a way to do that is
by using the request library, so request allows you to give Python a URL so like
example.com and Python will grab all this text, and then once you have this
text, you'll use the beautiful soup library to extract all the elements from the tags.
So for instance the example, the text inside h1 tag, and then
you can store the extracted text in variables, or in Python dictionaries,
or Pandas dataframes and wherever you find it's useful for
your needs. So that's the concept and you can see the same source code as you
might already know from the inspect, so right-click inspect and you see this.
So here is like this source code but you see it's more organized so for instance if
you're opening your mouse over the body tags the corresponding element in the
webpage will be highlighted as you see it, so if you expand this now you are on
the h1 tags and so on. So this allows you to actually see the names of all the
tags for the elements you want to extract, so we'll use the inspect
window for understanding the source code of our web pages. So that's about the
concept of web scrubbing and I'll see you in the next lecture where I'll load
a web page in Python and then we will extract some simple data from the web
page, so just to get you started with the requests library
and the BeautifulSoup library as well. So let's move on.