Python爬虫-小测验
一、 使用scrapy.Selector或BeautifulSoup，实现以下需求（30分） （1）读取给定的dangdang.html页面内容，注：编码为gbk（5分） （2）获取页面中所有图书的名称，价格，作者，出版社及图书图片的url地址（20分） （3）将获取的信息保存至文件（excel、csv、json、txt格式均可）（5分） 网页文件dangdang.html文件下载链接: https://pan.baidu.com/s/1awbG5zqOMdnWzXee7TZm6A 密码: 3urs 1.1使用BeautifulSoup解决 from bs4 import BeautifulSoup as bs import pandas as pd def cssFind(book,cssSelector,nth=1): if len(book.select(cssSelector)) >= nth: return book.select(cssSelector)[nth-1].text.strip() else: return '' if __name__ == "__main__": with open("dangdang.html",encoding='gbk') as file: html = file.read() soup = bs(html,'lxml') book_list = soup.select("div ul.bigimg li") result_list = [] for book in book_list: item = {} item['name'] = book.select("a.pic")[0]['title'] item['now_price'] = cssFind(book,"span.search_now_price") item['pre_price'] = cssFind(book,"span.search_pre_price") item['author'] = book.select("p.search_book_author a")[0]['title'] item['publisher'] = book.select("p.search_book_author span a")[-1].text item['detailUrl'] = book.select("p.name a")[0]['href'] item['imageUrl'] = book.select("a.pic img")[0]['src'] if item['imageUrl'] == "images/model/guan/url_none.png": item['imageUrl'] = book.select("a.pic img")[0]['data-original'] result_list.append(item) df = pd.DataFrame(result_list,columns=result_list[0].keys()) df.to_excel("当当图书信息.xlsx") 1.2使用scrapy.selector解决 from scrapy.selector import Selector import pandas as pd if __name__ == "__main__": with open("dangdang.html",encoding='gbk') as file: response = Selector(text=file.read()) book_list = response.xpath("//ul[@class='bigimg']/li") result_list = [] for book in book_list: item = {} item['name'] = book.xpath("a[@class='pic']/@title").extract_first() item['now_price'] = book.xpath(".//span[@class='search_now_price']/text()").extract_first() item['pre_price'] = book.xpath(".//span[@class='search_pre_price']/text()").extract_first() item['author'] = book.xpath("p[@class='search_book_author']//a/@title").extract_first() item['publisher'] = book.xpath("p[@class='search_book_author']//a/@title").extract()[-1] item['detailUrl'] = book.xpath(".//p[@class='name']/a/@href").extract_first() item['imageUrl'] = book.xpath("a[@class='pic']/img/@src").extract_first() if item['imageUrl'] == "images/model/guan/url_none.png": item['imageUrl'] = book.xpath("a[@class='pic']/img/@data-original").extract_first() result_list.append(item) df = pd.DataFrame(result_list,columns=result_list[0].keys()) df.to_excel("当当图书信息.xlsx") 二、 需求：抓取天猫三只松鼠旗舰店超级满减商品信息（55分） 网站地址如下https://sanzhisongshu.tmall.com/p/rd523844.htm?spm=a1z10.1-b-s.w5001-14855767631.8.19ad32fdW6UhfO&scene=taobao_shop 评分标准如下： 1、创建函数获取页面所有内容，代码无误（5分） 2、得到页面内容后解析信息，获取页面中图片链接，并将图片下载至本地photo文件夹。（10分） 3、获取页面中每个商品信息的商品名称、价格以及商品图片url信息（20分） 4、创建数据库product，及表格productinfo，包含（商品名称、价格及图片地址三个字段）（5分） 5、将第（3）步获取的结果写入数据库（10分） 6、代码规范，有注释（5分） import requests from bs4 import BeautifulSoup as bs import urllib import os import pymysql #获取实例化BeautifulSoup对象 def getSoup(url, encoding="gbk", **params): reponse = requests.get(url, **params) reponse.encoding = encoding soup = bs(reponse.text, 'lxml') return soup #下载单个图片函数 def downloadImage(imgUrl, imgName): imgDir = "photo" if not os.path.isdir(imgDir): os.mkdir(imgDir) imgPath = "%s/%s" %(imgDir,imgName) urllib.request.urlretrieve(imgUrl,imgPath) #下载所有图片函数 def downloadAllImages(soup): image_list = soup.select("img") count = 0 for image in image_list: try: srcStr = image['data-ks-lazyload'] imgFormat = srcStr[-3:] if imgFormat == 'gif': continue count += 1 imgName = "%d.%s" % (count, imgFormat) imgUrl = "http:" + srcStr downloadImage(imgUrl, imgName) except Exception as e: print(str(e)) #通过css选择器语法选择出标签 def cssFind(movie,cssSelector,nth=1): if len(movie.select(cssSelector)) >= nth: return movie.select(cssSelector)[nth-1].text.strip() else: return '' #获取数据库连接函数 def getConn(database ="product"): args = dict( host = 'localhost', user = 'root', passwd = '.... your password', charset = 'utf8', db = database ) return pymysql.connect(**args) if __name__ == "__main__": soup = getSoup("https://sanzhisongshu.tmall.com/p/rd523844.htm" \ "?spm=a1z10.1-b-s.w5001-14855767631.8.19ad32fdW6UhfO&scene=taobao_shop") #下载所有图片 downloadAllImages(soup) #获取数据库连接 conn = getConn() cursor = conn.cursor() #新建数据库中的表productinfo sql_list = [] sql_list.append("drop table if exists productinfo") sql_list.append("create table productinfo(name varchar(200)," \ "price varchar(20),imageUrl varchar(500))") for sql in sql_list: cursor.execute(sql) conn.commit() #获取商品信息并插入数据库 item_list = soup.select("div.item4line1 dl.item") for item in item_list: name = cssFind(item,"dd.detail a") price = cssFind(item,"dd.detail span.c-price") imageUrl = item.select("dt img")[0]['data-ks-lazyload'] insert_sql = 'insert into productinfo values("%s","%s","%s")' %(name,price,imageUrl) cursor.execute(insert_sql) conn.commit() 三、请以你的理解尽可能准确的描述出scrapy运行的原理图（15分） scrapy框架原理图.png 在实际编写代码的过程一种，一般按照下列顺序编写代码文件： 1.编写item.py文件；2.编写爬虫文件;3.编写pipelines.py文件；4.编写settings.py文件 在Scrapy框架理解上： 1.爬虫Spiders发送请求Requests给调度器Scheduler 2.调度器Scheduler发送下载网页的请求Requests给下载器Downloader 3.下载器Downloader获取网页相应response交给爬虫Spiders 4.爬虫Spiders对response进行解析形成Item 5.Item传送给管道，管道对数据进行相应处理，数据持久化。 6.Middelwares分为三种：调度中间件Scheduler middlewares、爬虫中间件spider Middlewares、下载中间件Download Middlewares。在编写scrapy-redis分布式爬虫时，redis就相当于调度中间件Scheduler middlewares；对爬虫进行伪装，设置用户代理User-agent和代理Ip,是在爬虫中间件spider Middlewares中进行设置，下载中间件Download Middlewares可以对下载进行相应设置。
