问题描述

我正在尝试使用 python 抓取页面

I'm trying to scrape a page using python

问题是，我不断收到对等方重置 Errno54 连接.

The problem is, I keep getting Errno54 Connection reset by peer.

运行此代码时出现错误 -

The error comes when I run this code -

urllib2.urlopen("http://www.bkstr.com/webapp/wcs/stores/servlet/CourseMaterialsResultsView?catalogId=10001&categoryId=9604&storeId=10161&langId=-1&programId=562&termId=100020629&divisionDisplayName=Stanford&departmentDisplayName=ILAC&courseDisplayName=126&sectionDisplayName=01&demoKey=d&purpose=browse")

此页面上的所有网址都会发生这种情况 - 有什么问题?

this happens for all the urls on this pag- what is the issue?

推荐答案

$> telnet www.bkstr.com 80
Trying 64.37.224.85...
Connected to www.bkstr.com.
Escape character is '^]'.
GET /webapp/wcs/stores/servlet/CourseMaterialsResultsView?catalogId=10001&categoryId=9604&storeId=10161&langId=-1&programId=562&termId=100020629&divisionDisplayName=Stanford&departmentDisplayName=ILAC&courseDisplayName=126&sectionDisplayName=01&demoKey=d&purpose=browse HTTP/1.0

Connection closed by foreign host.

从 python 或其他任何地方获取该 URL 不会有任何乐趣.如果它在您的浏览器中工作，那么肯定有其他事情发生，例如 cookie 或身份验证等.或者，可能是服务器坏了，或者他们改变了配置.

You're not going to have any joy fetching that URL from python, or anywhere else. If it works in your browser then there must be something else going on, like cookies or authentication or some such. Or, possibly, the server's broken or they've changed their configuration.

尝试在您以前从未访问过该网站的浏览器中打开它以进行检查.然后登录并重试.

Try opening it in a browser that you've never accessed that site in before to check. Then log in and try it again.

毕竟是 cookie:

It was cookies after all:

import cookielib, urllib2

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
#Need to set a cookie
opener.open("http://www.bkstr.com/")
#Now open the page we want
data = opener.open("http://www.bkstr.com/webapp/wcs/stores/servlet/CourseMaterialsResultsView?catalogId=10001&categoryId=9604&storeId=10161&langId=-1&programId=562&termId=100020629&divisionDisplayName=Stanford&departmentDisplayName=ILAC&courseDisplayName=126&sectionDisplayName=01&demoKey=d&purpose=browse").read()

输出看起来不错，但您必须检查它是否符合您的要求:)

The output looks ok, but you'll have to check that it does what you want :)

这篇关于Python urllib2.open 连接由对等错误重置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！