python爬虫练手，爬取名言，实现英语词典

要爬取的网站 http://quotes.toscrape.com/

**爬取名言，作者，标签。***

她们的Html为，通过beautiful库的html.parser解析，通过id,class选择器，提取我们需要的东西。

<span class="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>

<span class="text">“It is our choices, Harry, that show what we truly are, far more than our abilities.”</span>

<div class="tags">
            Tags:
            <a class="tag" href="/tag/change/page/1/">change</a>

            <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>

            <a class="tag" href="/tag/thinking/page/1/">thinking</a>

            <a class="tag" href="/tag/world/page/1/">world</a>

        </div>

<div class="tags">
            Tags:
            <a class="tag" href="/tag/abilities/page/1/">abilities</a>

            <a class="tag" href="/tag/choices/page/1/">choices</a>

        </div>

<small class="author">J.K. Rowling</small>

相关的code如下

import requests
import os
from bs4 import BeautifulSoup

def get_html(url):
    try:
        header={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331', }
        ht=requests.get(url,headers=header,verify=True)
        ht.raise_for_status
        ht.encoding=ht.apparent_encoding
        return ht
    except Exception as e:
        print("error:",e)

def writeHtml(url):
    text=get_html(url).content
    path="C:\\Users\\Administrator\\Desktop\\python\\baidu.html"
    with open(path,"wb") as f:
        f.write(text)
print("success")

def getInfor(url):
    finalSay=[]
    html=get_html(url).text
    soup=BeautifulSoup(html,"html.parser")
    print(" the saying is:")
    say_list=soup.select("span.text")
    print("-------名言--------------")
    print("the length=",len(say_list))
   # print(say_list)
    for elem in say_list:
        te=elem.text
        finalSay.append(te);
    print(finalSay)
    print("-------标签--------------")
    finalTag=[]
    tag_list=soup.select("div.tags")
    print("the length of tag=",len(tag_list))
    #print(tag_list)
    for tag in tag_list:
        a_tag=tag.select("a.tag")
        #print(a_tag)

       # for elem in a_tag:
       #    print(elem.text)
        tag_list=[elem.text for elem in a_tag]
        #print(tag_list)
        finalTag.append(tag_list)
    print(finalTag)
    print("-------作者--------------")
    finalAuthor=[]
    author_list=soup.select("small.author")
    print("the length of authoer=",len(author_list))
    for elem in author_list:
        finalAuthor.append(elem.text)
    print(finalAuthor)
    for i in range(len(author_list)):
        print("the saying:",finalSay[i],"\t","the author:",finalAuthor[i],"\t","the tag:",finalTag[i])

结果如下

the saying: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”          the author: Albert Einstein     the tag: ['change', 'deep-thoughts', 'thinking', 'world']
the saying: “It is our choices, Harry, that show what we truly are, far more than our abilities.”        the author: J.K. Rowling        the tag: ['abilities', 'choices']
the saying: “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”          the author: Albert Einstein     the tag: ['inspirational', 'life', 'live', 'miracle', 'miracles']
the saying: “The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”     the author: Jane Austen         the tag: ['aliteracy', 'books', 'classic', 'humor']
the saying: “Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”      the author: Marilyn Monroe      the tag: ['be-yourself', 'inspirational']
the saying: “Try not to become a man of success. Rather become a man of value.”          the author: Albert Einstein     the tag: ['adulthood', 'success', 'value']
the saying: “It is better to be hated for what you are than to be loved for what you are not.”   the author: André Gide          the tag: ['life', 'love']
the saying: “I have not failed. I've just found 10,000 ways that won't work.”    the author: Thomas A. Edison    the tag: ['edison', 'failure', 'inspirational', 'paraphrased']
the saying: “A woman is like a tea bag; you never know how strong it is until it's in hot water.”        the author: Eleanor Roosevelt   the tag: ['misattributed-eleanor-roosevelt']
the saying: “A day without sunshine is like, you know, night.”   the author: Steve Martin        the tag: ['humor', 'obvious', 'simile']

完整 https://cn.bing.com/dict/search?q=snow

其翻译的信息主要在

<span class="pos">n.</span>
<span class="pos">v.</span>
<span class="pos">adj.</span>
<span class="pos web">网络</span>

<span class="def"><span>书；书籍；著作；部</span></span>
<span class="def"><span>书籍的；书本上的；账面上的</span></span>

对应的正则表达式为

 re0=r'<span class="(pos|pos web)">(.*?)</span>'
 re1=r'<span class="def"><span>(.*?)</span></span>'

相关code
主要是正则表达式的使用，已经写文件操作。

def getDictionary():
    word=input("亲输入要翻译的词语:")
    url="https://cn.bing.com/dict/search?q="
    hurl=url+word
    trant=[]
    html=get_html(hurl).text
    re0=r'<span class="(pos|pos web)">(.*?)</span>'
    flag_list=[]
    nav_list=re.findall(re0,html)
    #print(hurl)
    #print(html)
    if len(nav_list)==0 and len(nav_list[0])&lt;1:
        return 
    for elem in nav_list:
        flag_list.append(elem[1])  
    print(flag_list)
    re1=r'<span class="def"><span>(.*?)</span></span>'
    tran_list=re.findall(re1,html)
    #print(tran_list)
    for i in range(len(nav_list)):
        tra="\t".join([flag_list[i],tran_list[i]])
        print(tra)
        trant.append(tra)
        #trant.append("\n")
    return trant

def writeTet(t):
    path="E:\\infor.txt"
    with open(path,"a+") as f:
        for txt in t:
             f.write(txt)
             f.write("\n")
    print("success")

结果为

亲输入要翻译的词语:Python
['n.', '网络']
n.      蟒；蚺蛇
网络      蟒蛇；巨蟒；派森
success

参考文章
beautifulsoup之CSS选择器
Beautiful Soup Documentation
Python 文件I/O
python对文件的操作读写追加等演示