一、简介
网络爬虫（又被称为网页蜘蛛，网络机器人，在FOAF社区中间，更经常的称为网页追逐者），是一种按照一定的规则，自动地抓取万维网信息的程序或者脚本。另外一些不常使用的名字还有蚂蚁、自动索引、模拟程序或者蠕虫。
二、requests
python标准库中提供了：urllib、urllib2、httplib等模块以供Http请求，但是，它的 API 太渣了。它是为另一个时代、另一个互联网所创建的。它需要巨量的工作，甚至包括各种方法覆盖，来完成最简单的任务。
import urllib2 import json import cookielib def urllib2_request(url, method="GET", cookie="", headers={}, data=None): """ :param url: 要请求的url :param cookie: 请求方式，GET、POST、DELETE、PUT.. :param cookie: 要传入的cookie，cookie= 'k1=v1;k1=v2' :param headers: 发送数据时携带的请求头，headers = {'ContentType':'application/json; charset=UTF-8'} :param data: 要发送的数据GET方式需要传入参数，data={'d1': 'v1'} :return: 返回元祖，响应的字符串内容 和 cookiejar对象 对于cookiejar对象，可以使用for循环访问： for item in cookiejar: print item.name,item.value """ if data: data = json.dumps(data) cookie_jar = cookielib.CookieJar() handler = urllib2.HTTPCookieProcessor(cookie_jar) opener = urllib2.build_opener(handler) opener.addheaders.append(['Cookie', 'k1=v1;k1=v2']) request = urllib2.Request(url=url, data=data, headers=headers) request.get_method = lambda: method response = opener.open(request) origin = response.read() return origin, cookie_jar # GET result = urllib2_request('http://127.0.0.1:8001/index/', method="GET") # POST result = urllib2_request('http://127.0.0.1:8001/index/', method="POST", data= {'k1': 'v1'}) # PUT result = urllib2_request('http://127.0.0.1:8001/index/', method="PUT", data= {'k1': 'v1'})Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库，其在Python内置模块的基础上进行了高度的封装，从而使得Pythoner进行网络请求时，变得美好了许多，使用Requests可以轻而易举的完成浏览器可有的任何操作。
1、get请求
# 1、无参数实例 import requests ret = requests.get('https://github.com/timeline.json') print ret.url print ret.text # 2、有参数实例 import requests payload = {'key1': 'value1', 'key2': 'value2'} ret = requests.get("http://httpbin.org/get", params=payload) print ret.url print ret.text向 https://github.com/timeline.json 发送一个GET请求，将请求和响应相关均封装在 ret 对象中。
2、post
# 1、基本POST实例 import requests payload = {'key1': 'value1', 'key2': 'value2'} ret = requests.post("http://httpbin.org/post", data=payload) print ret.text # 2、发送请求头和数据实例 import requests import json url = 'https://api.github.com/some/endpoint' payload = {'some': 'data'} headers = {'content-type': 'application/json'} ret = requests.post(url, data=json.dumps(payload), headers=headers) print ret.text print ret.cookies3、其他请求
requests.get(url, params=None, **kwargs) requests.post(url, data=None, json=None, **kwargs) requests.put(url, data=None, **kwargs) requests.head(url, **kwargs) requests.delete(url, **kwargs) requests.patch(url, data=None, **kwargs) requests.options(url, **kwargs) # 以上方法均是在此方法的基础上构建 requests.request(method, url, **kwargs)requests模块已经将常用的Http请求方法为用户封装完成，用户直接调用其提供的相应方法即可，其中方法的所有参数有：
更多参数
def request(method, url, **kwargs): """Constructs and sends a :class:`Request <Request>`. :param method: method for the new :class:`Request` object. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`. :param json: (optional) json data to send in the body of the :class:`Request`. :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload. :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. :param timeout: (optional) How long to wait for the server to send data before giving up, as a float, or a :ref:`(connect timeout, read timeout) <timeouts>` tuple. :type timeout: float or tuple :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed. :type allow_redirects: bool :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``. :param stream: (optional) if ``False``, the response content will be immediately downloaded. :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair. :return: :class:`Response <Response>` object :rtype: requests.Response Usage:: >>> import requests >>> req = requests.request('GET', 'http://httpbin.org/get') <Response [200]> """ # By using the 'with' statement we are sure the session is closed, thus we # avoid leaving sockets open which can trigger a ResourceWarning in some # cases, and look like a memory leak in others. with sessions.Session() as session: return session.request(method=method, url=url, **kwargs)更多requests模块相关的文档见：http://cn.python-requests.org/zh_CN/latest/
自动登录抽屉并点赞
### 1、首先登陆任何页面，获取cookie i1 = requests.get(url= "http://dig.chouti.com/help/service") ### 2、用户登陆，携带上一次的cookie，后台对cookie中的 gpsd 进行授权 i2 = requests.post( url= "http://dig.chouti.com/login", data= { 'phone': "86手机号", 'password': "密码", 'oneMonth': "" }, cookies = i1.cookies.get_dict() ) ### 3、点赞（只需要携带已经被授权的gpsd即可） gpsd = i1.cookies.get_dict()['gpsd'] i3 = requests.post( url="http://dig.chouti.com/link/vote?linksId=8589523", cookies={'gpsd': gpsd} ) print(i3.text)“破解”微信公众号
“破解”微信公众号其实就是使用Python代码自动实现【登陆公众号】->【获取观众用户】-> 【向关注用户发送消息】。
注：只能向48小时内有互动的粉丝主动推送消息
a、自动登录
分析对于Web登陆页面，用户登陆验证时仅做了如下操作：
登陆的URL：https://mp.weixin.qq.com/cgi-bin/login?lang=zh_CN POST的数据为：{
'username': 用户名,
'pwd': 密码的MD5值,
'imgcode': "",
'f': 'json'
}
注：imgcode是需要提供的验证码，默认无需验证码，只有在多次登陆未成功时，才需要用户提供验证码才能登陆
POST的请求头的Referer值，微信后台用次来检查是谁发送来的请求 请求发送并登陆成功后，获取用户响应的cookie，以后操作其他页面时需要携带此cookie 请求发送并登陆成功后，获取用户相应的内容中的token登录代码
# -*- coding:utf-8 -*- import requests import time import hashlib def _password(pwd): ha = hashlib.md5() ha.update(pwd) return ha.hexdigest() def login(): login_dict = { 'username': "用户名", 'pwd': _password("密码"), 'imgcode': "", 'f': 'json' } login_res = requests.post( url= "https://mp.weixin.qq.com/cgi-bin/login?lang=zh_CN", data=login_dict, headers={'Referer': 'https://mp.weixin.qq.com/cgi-bin/login?lang=zh_CN'}) # 登陆成功之后获取服务器响应的cookie resp_cookies_dict = login_res.cookies.get_dict() # 登陆成功后，获取服务器响应的内容 resp_text = login_res.text # 登陆成功后，获取token token = re.findall(".*token=(\d+)", resp_text)[0] print resp_text print token print resp_cookies_dict login()登陆成功获取的相应内容如下：
响应内容： {"base_resp":{"ret":0,"err_msg":"ok"},"redirect_url":"\/cgi-bin\/home?t=home\/index=zh_CN&token=537908795"} 响应cookie： {'data_bizuin': '3016804678', 'bizuin': '3016804678', 'data_ticket': 'CaoX+QA0ZA9LRZ4YM3zZkvedyCY8mZi0XlLonPwvBGkX0/jY/FZgmGTq6xGuQk4H', 'slave_user': 'gh_5abeaed48d10', 'slave_sid': 'elNLbU1TZHRPWDNXSWdNc2FjckUxalM0Y000amtTamlJOUliSnRnWGRCdjFseV9uQkl5cUpHYkxqaGJNcERtYnM2WjdFT1pQckNwMFNfUW5fUzVZZnFlWGpSRFlVRF9obThtZlBwYnRIVGt6cnNGbUJsNTNIdTlIc2JJU29QM2FPaHZjcTcya0F6UWRhQkhO'