你都怎样看知乎日报?

原创
2017/03/07 09:19
阅读数 247

1.分析知乎日报网页(http://daily.zhihu.com/)

<span class="title">这样一分析,你就明白该在哪里投广告了</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269183" class="link-button"><img src="http://pic2.zhimg.com/05c67496e38f662958a141847a734ffd.jpg" class="preview-image"><span class="title">有些热闹的「共享经济」,恐怕只是一个美好的童话</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9268452" class="link-button"><img src="http://pic3.zhimg.com/da8d8d3bf282170379c51ea0cf1ae4a6.jpg" class="preview-image"><span class="title">让孩子拥有属于自己的无聊时光,到底有多重要?</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269792" class="link-button"><img src="http://pic2.zhimg.com/245f3cf8dd4bfdd1bf0911ba4b486295.jpg" class="preview-image"><span class="title">看不懂,说人话,不然公司就亏大发了</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269818" class="link-button"><img src="http://pic3.zhimg.com/8b1588e23187bb05d160d599bbfe1752.jpg" class="preview-image"><span class="title">《金刚狼 3》中有哪些隐藏的彩蛋和有趣的细节?</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9266807" class="link-button"><img src="http://pic1.zhimg.com/4cb4d5dec4a68553e41dcbd483010e84.jpg" class="preview-image"><span class="title">瞎扯 · 如何正确地吐槽</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9259222" class="link-button"><img src="http://pic4.zhimg.com/16378a8129349aface9694cd27c71e2f.jpg" class="preview-image"><span class="title">小事 · 爱无能</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9267167" class="link-button"><img src="http://pic1.zhimg.com/fdf5e0ff47de69d615f706559a260168.jpg" class="preview-image">

每一个话题和图片都在一个span标签里

<span class="title">这样一分析,你就明白该在哪里投广告了</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269183" class="link-button"><img src="http://pic2.zhimg.com/05c67496e38f662958a141847a734ffd.jpg" class="preview-image">

很简单构造一个正则表达式去匹配上面(获取标题,图片,链接)

pattern=re.compile(u'<span class="title">(.*?)</span>.*?'+
        u'<a href="(.*?)".*?'+
        u'<img src="(.*?)".*?'
        ,re.S)

2.下载页面匹配正则表达式

先创建一个请求头

 self.header={
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'Accept-Encoding':'gzip, deflate, sdch',
            'Accept-Language':'zh-CN,zh;q=0.8',
            'Cache-Control':'max-age=0',
            'Connection':'keep-alive',
            'Host':'daily.zhihu.com',
            'Referer':'https://www.baidu.com/link?url=Eh6CKs72Buyf0LEjPd1795QSL8ZK74kwItBvzaybausT6proAZIr3UkkmMPSDfk7&wd=&eqid=d1bd8fd9004118c90000000258bd8149',
            'Upgrade-Insecure-Requests':'1',
            'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/56.0.2924.76 Chrome/56.0.2924.76 Safari/537.36'
        }

使用下载页面使用正则匹配

html=self.req.get(url=self.url,headers=self.header)
        pattern=re.compile(u'<span class="title">(.*?)</span>.*?'+
        u'<a href="(.*?)".*?'+
        u'<img src="(.*?)".*?'
        ,re.S)
        T=list()
        self.l=re.findall(pattern,html.text)

3.使用多线程下载图片

T=list()
        self.l=re.findall(pattern,html.text)
        for i in self.l:
            self.w.write(str(i)+'\n')
            #if(self.n<29):
            t=Thread(target=self.getimg,args=(i[2],self.n))
                #self.getimg(i[2],self.n)
            T.append(t)
            t.start()
            #print(i)
            self.n+=1
            #time.sleep(1)
        for tt in T:
            tt.join()
def getimg(self,src,n):
        try:
            h=self.req.get(url=src)
            s=open(str(n)+'.jpg','wb')
            s.write(h.content)
            s.close()
        except requests.exceptions.MissingSchema:
            print('这个url无效',n)

4.结果

全部代码:

#!/usr/bin/python3
#coding:utf8
import requests
import re
import time
from threading import Thread
class main(object):
    def __init__(self):
        self.url='http://daily.zhihu.com/'
        self.l=list()
        self.n=0
        self.req=requests.Session()
        self.header={
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'Accept-Encoding':'gzip, deflate, sdch',
            'Accept-Language':'zh-CN,zh;q=0.8',
            'Cache-Control':'max-age=0',
            'Connection':'keep-alive',
            'Host':'daily.zhihu.com',
            'Referer':'https://www.baidu.com/link?url=Eh6CKs72Buyf0LEjPd1795QSL8ZK74kwItBvzaybausT6proAZIr3UkkmMPSDfk7&wd=&eqid=d1bd8fd9004118c90000000258bd8149',
            'Upgrade-Insecure-Requests':'1',
            'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/56.0.2924.76 Chrome/56.0.2924.76 Safari/537.36'
        }
    def getpage(self):
        html=self.req.get(url=self.url,headers=self.header)
        pattern=re.compile(u'<span class="title">(.*?)</span>.*?'+
        u'<a href="(.*?)".*?'+
        u'<img src="(.*?)".*?'
        ,re.S)
        T=list()
        self.l=re.findall(pattern,html.text)
        for i in self.l:
            self.w.write(str(i)+'\n')
            #if(self.n<29):
            t=Thread(target=self.getimg,args=(i[2],self.n))
                #self.getimg(i[2],self.n)
            T.append(t)
            t.start()
            #print(i)
            self.n+=1
            #time.sleep(1)
        for tt in T:
            tt.join()
    def getimg(self,src,n):
        try:
            h=self.req.get(url=src)
            s=open(str(n)+'.jpg','wb')
            s.write(h.content)
            s.close()
        except requests.exceptions.MissingSchema:
            print('这个url无效',n)


if __name__=='__main__':
    p=main()
    p.w=open('zh.txt','w')
    p.getpage()
    p.w.close()

项目地址:https://git.oschina.net/nanxun/zhihuribao

 

 

 

展开阅读全文
打赏
0
1 收藏
分享
加载中
打赏
0 评论
1 收藏
0
分享
返回顶部
顶部