There's more...

This manner of navigating the DOM is relatively inflexible and is highly dependent upon the structure. It is possible that this structure can change over time as web pages are updated by their creator(s). The pages could even look identical, but have a completely different structure that breaks your scraping code.

So how can we deal with this? As we will see, there are several ways of searching for elements that are much better than defining explicit paths. In general, we can do this using XPath and by using the find methods of beautiful soup. We will examine both in recipes later in this chapter.