文档章节

CentOS系统Python2.7.x安装Scrapy

L
 Listen_ing
发布于 2016/11/22 10:56
字数 1998
阅读 517
收藏 0

Python2和Python3的取舍

包没有安装,安装之

最近想研究一下Python,看了3和2,也敲了代码,开始敲的3,后来敲的2,感觉2和3区别不大,而且2的后续版本已经在尽力向3靠近了,而且2也更成熟,所以最终决定使用2来做研究和开发。这篇文章写的是我在安装Scrapy框架时遇到的问题。

检测Python版本为2.7,后面的版本号不做纠结,区别不大,想升级的可自行升级。

[listen@localhost ~]$ python
Python 2.7.5 (default, Sep 15 2016, 22:37:39) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

安装Scrapy

切换到root用户,因为有些目录普通用户没有权限

[root@localhost listen]# pip install Scrapy

如果pip工具没有安装,请先安装,可用yum install pip 进行安装,在此不表

安装到最后会报这样的错误,搞了半天也没有找到原因,就在国外的网站上找到类似的解答,就试试,没想到还真解决了

    running build_ext
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c conftest.c -o conftest.o
    building 'twisted.test.raiser' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/src
    creating build/temp.linux-x86_64-2.7/src/twisted
    creating build/temp.linux-x86_64-2.7/src/twisted/test
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c src/twisted/test/raiser.c -o build/temp.linux-x86_64-2.7/src/twisted/test/raiser.o
    src/twisted/test/raiser.c:4:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ovYzET/Twisted/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-jZG5jR-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-ovYzET/Twisted/

[root@localhost listen]# pip install python-dev
Collecting python-dev
  Could not find a version that satisfies the requirement python-dev (from versions: )
No matching distribution found for python-dev
[root@localhost listen]# yum install python-devel
Loaded plugins: fastestmirror, langpacks
base                                                                                                                                                   | 3.6 kB  00:00:00     
extras                                                                                                                                                 | 3.4 kB  00:00:00     
updates                                                                                                                                                | 3.4 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package python-devel.x86_64 0:2.7.5-39.el7_2 will be installed
--> Processing Dependency: python(x86-64) = 2.7.5-39.el7_2 for package: python-devel-2.7.5-39.el7_2.x86_64
--> Running transaction check
---> Package python.x86_64 0:2.7.5-34.el7 will be updated
---> Package python.x86_64 0:2.7.5-39.el7_2 will be an update
--> Processing Dependency: python-libs(x86-64) = 2.7.5-39.el7_2 for package: python-2.7.5-39.el7_2.x86_64
--> Running transaction check
---> Package python-libs.x86_64 0:2.7.5-34.el7 will be updated
---> Package python-libs.x86_64 0:2.7.5-39.el7_2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================
 Package                                     Arch                                  Version                                       Repository                              Size
==============================================================================================================================================================================
Installing:
 python-devel                                x86_64                                2.7.5-39.el7_2                                updates                                392 k
Updating for dependencies:
 python                                      x86_64                                2.7.5-39.el7_2                                updates                                 89 k
 python-libs                                 x86_64                                2.7.5-39.el7_2                                updates                                5.6 M

Transaction Summary
==============================================================================================================================================================================
Install  1 Package
Upgrade             ( 2 Dependent packages)

Total size: 6.1 M
Total download size: 392 k
Is this ok [y/d/N]: y
Downloading packages:
python-devel-2.7.5-39.el7_2.x86_64.rpm                                                                                                                 | 392 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : python-libs-2.7.5-39.el7_2.x86_64                                                                                                                          1/5 
  Updating   : python-2.7.5-39.el7_2.x86_64                                                                                                                               2/5 
  Installing : python-devel-2.7.5-39.el7_2.x86_64                                                                                                                         3/5 
  Cleanup    : python-2.7.5-34.el7.x86_64                                                                                                                                 4/5 
  Cleanup    : python-libs-2.7.5-34.el7.x86_64                                                                                                                            5/5 
  Verifying  : python-devel-2.7.5-39.el7_2.x86_64                                                                                                                         1/5 
  Verifying  : python-2.7.5-39.el7_2.x86_64                                                                                                                               2/5 
  Verifying  : python-libs-2.7.5-39.el7_2.x86_64                                                                                                                          3/5 
  Verifying  : python-libs-2.7.5-34.el7.x86_64                                                                                                                            4/5 
  Verifying  : python-2.7.5-34.el7.x86_64                                                                                                                                 5/5 

Installed:
  python-devel.x86_64 0:2.7.5-39.el7_2                                                                                                                                        

Dependency Updated:
  python.x86_64 0:2.7.5-39.el7_2                                                      python-libs.x86_64 0:2.7.5-39.el7_2                                                     

Complete!
[root@localhost listen]# yum install python-dev
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
No package python-dev available.
Error: Nothing to do
[root@localhost listen]# pip install Scrapy
Requirement already satisfied: Scrapy in /usr/lib/python2.7/site-packages/Scrapy-1.2.1-py2.7.egg
Collecting Twisted>=10.0.0 (from Scrapy)
  Using cached Twisted-16.5.0.tar.bz2
Collecting w3lib>=1.15.0 (from Scrapy)
  Using cached w3lib-1.16.0-py2.py3-none-any.whl
Collecting queuelib (from Scrapy)
  Using cached queuelib-1.4.2-py2.py3-none-any.whl
Requirement already satisfied: lxml in /usr/lib64/python2.7/site-packages (from Scrapy)
Requirement already satisfied: pyOpenSSL in /usr/lib64/python2.7/site-packages (from Scrapy)
Collecting cssselect>=0.9 (from Scrapy)
  Using cached cssselect-1.0.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in /usr/lib/python2.7/site-packages (from Scrapy)
Collecting parsel>=0.9.3 (from Scrapy)
  Using cached parsel-1.0.3-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy)
  Using cached PyDispatcher-2.0.5.tar.gz
Collecting service_identity (from Scrapy)
  Using cached service_identity-16.0.0-py2.py3-none-any.whl
Requirement already satisfied: zope.interface>=3.6.0 in /usr/lib64/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied: constantly>=15.1 in /usr/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied: incremental>=16.10.1 in /usr/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Collecting attrs (from service_identity->Scrapy)
  Using cached attrs-16.2.0-py2.py3-none-any.whl
Collecting pyasn1-modules (from service_identity->Scrapy)
  Using cached pyasn1_modules-0.0.8-py2.py3-none-any.whl
Requirement already satisfied: pyasn1 in /usr/lib/python2.7/site-packages (from service_identity->Scrapy)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Twisted, w3lib, queuelib, cssselect, parsel, PyDispatcher, attrs, pyasn1-modules, service-identity
  Running setup.py install for Twisted ... done
  Running setup.py install for PyDispatcher ... done
Successfully installed PyDispatcher-2.0.5 Twisted-16.5.0 attrs-16.2.0 cssselect-1.0.0 parsel-1.0.3 pyasn1-modules-0.0.8 queuelib-1.4.2 service-identity-16.0.0 w3lib-1.16.0
[root@localhost listen]# python
Python 2.7.5 (default, Sep 15 2016, 22:37:39) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twisted
>>> 
KeyboardInterrupt
>>> 
KeyboardInterrupt
>>> 
[3]+  Stopped                 python
[root@localhost listen]# 

安装Twisted库的时候比较费劲,下载速度特别慢,还容易超时,安装好几次才成功,安装的时候多试几次,本来就3M的包,硬是下载了几十分钟,除了Twisted包其他的包都很容易就下载安装,这就是我在安装的过程中遇到的问题,可以给大家参考一下。

测试Scrapy是否安装成功

[root@localhost listen]# scrapy startproject tutorial
Traceback (most recent call last):
  File "/usr/bin/scrapy", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3011, in <module>
    parse_requirements(__requires__), Environment()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (pyasn1 0.1.6 (/usr/lib/python2.7/site-packages), Requirement.parse('pyasn1>=0.1.8'))

报错了,说明没有安装成功,看提示说有依赖包没有安装,安装之

[root@localhost listen]# pip install parse
Collecting parse
  Downloading parse-1.6.6-py2-none-any.whl
Installing collected packages: parse
Successfully installed parse-1.6.6
[root@localhost listen]# 
[root@localhost listen]# scrapy startproject tutorial
Traceback (most recent call last):
  File "/usr/bin/scrapy", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3011, in <module>
    parse_requirements(__requires__), Environment()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (pyasn1 0.1.6 (/usr/lib/python2.7/site-packages), Requirement.parse('pyasn1>=0.1.8'))
[root@localhost listen]# pip install pyasn1 --upgrade
Collecting pyasn1
  Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Installing collected packages: pyasn1
  Found existing installation: pyasn1 0.1.6
    Uninstalling pyasn1-0.1.6:
      Successfully uninstalled pyasn1-0.1.6
Successfully installed pyasn1-0.1.9
[root@localhost listen]# scrapy startproject tutorial
New Scrapy project 'tutorial', using template directory '/usr/lib/python2.7/site-packages/Scrapy-1.2.1-py2.7.egg/scrapy/templates/project', created in:
    /home/listen/tutorial

You can start your first spider with:
    cd tutorial
    scrapy genspider example example.com
[root@localhost listen]# 

最后从网上找到,是需要安装pip install pyasn1 --upgrade这个东东,安装好之后就成功了。

在开发的时候肯定会需要调试,所以最好把调试工具也安装上

[root@localhost turorial]# pip install ipython
Collecting ipython
  Downloading ipython-5.1.0-py2-none-any.whl (747kB)
    100% |████████████████████████████████| 747kB 11kB/s 
Collecting pygments (from ipython)
  Downloading Pygments-2.1.3-py2.py3-none-any.whl (755kB)
    100% |████████████████████████████████| 757kB 87kB/s 
Collecting setuptools>=18.5 (from ipython)
  Downloading setuptools-28.8.0-py2.py3-none-any.whl (472kB)
    100% |████████████████████████████████| 481kB 60kB/s 
Collecting backports.shutil-get-terminal-size; python_version == "2.7" (from ipython)
  Downloading backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl
Collecting traitlets>=4.2 (from ipython)
  Downloading traitlets-4.3.1-py2.py3-none-any.whl (74kB)
    100% |████████████████████████████████| 81kB 26kB/s 
Collecting simplegeneric>0.8 (from ipython)
  Downloading simplegeneric-0.8.1.zip
Collecting pickleshare (from ipython)
  Downloading pickleshare-0.7.4-py2.py3-none-any.whl
Requirement already satisfied: decorator in /usr/lib/python2.7/site-packages (from ipython)
Collecting pexpect; sys_platform != "win32" (from ipython)
  Downloading pexpect-4.2.1-py2.py3-none-any.whl (55kB)
    100% |████████████████████████████████| 61kB 18kB/s 
Collecting prompt-toolkit<2.0.0,>=1.0.3 (from ipython)
  Downloading prompt_toolkit-1.0.9-py2-none-any.whl (241kB)
    100% |████████████████████████████████| 245kB 24kB/s 
Collecting pathlib2; python_version == "2.7" or python_version == "3.3" (from ipython)
  Downloading pathlib2-2.1.0-py2.py3-none-any.whl
Collecting ipython-genutils (from traitlets>=4.2->ipython)
  Downloading ipython_genutils-0.1.0-py2.py3-none-any.whl
Requirement already satisfied: enum34; python_version == "2.7" in /usr/lib/python2.7/site-packages (from traitlets>=4.2->ipython)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from traitlets>=4.2->ipython)
Collecting ptyprocess>=0.5 (from pexpect; sys_platform != "win32"->ipython)
  Downloading ptyprocess-0.5.1-py2.py3-none-any.whl
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.3->ipython)
  Downloading wcwidth-0.1.7-py2.py3-none-any.whl
Installing collected packages: pygments, setuptools, backports.shutil-get-terminal-size, ipython-genutils, traitlets, simplegeneric, pathlib2, pickleshare, ptyprocess, pexpect, wcwidth, prompt-toolkit, ipython
  Found existing installation: setuptools 0.9.8
    Uninstalling setuptools-0.9.8:
      Successfully uninstalled setuptools-0.9.8
  Running setup.py install for simplegeneric ... done
Successfully installed backports.shutil-get-terminal-size-1.0.0 ipython-5.1.0 ipython-genutils-0.1.0 pathlib2-2.1.0 pexpect-4.2.1 pickleshare-0.7.4 prompt-toolkit-1.0.9 ptyprocess-0.5.1 pygments-2.1.3 setuptools-28.8.0 simplegeneric-0.8.1 traitlets-4.3.1 wcwidth-0.1.7

这个安装过程比较心累,发现了一个好的安装工具,可以下载试用一下Anaconda,这个工具的依赖比较全,最终的效果和手动安装一样,既然有工具为什么不用呢

© 著作权归作者所有

L
粉丝 13
博文 28
码字总数 53225
作品 0
宝山
程序员
私信 提问
安装Scrapy开发环境

强烈建议在Linux环境下进行开发,Windows环境会出现许多莫名其妙的问题 CentOS7安装Scrapy 首先安装libxml依赖libxml2,libxml2主要提供解析xpath的组件: 然后安装scrapy: 执行如下: Linu...

楠木楠
2016/09/21
708
0
ImportError: No module named RedisCrawlSpider

问题: 出现的问题是在centos7 安装scrapy的时候出现的,当时使用的python3.6 环境 当这个问题出现的时候,我们可能没有安装上去scrapy和scrapy_redis,所以尽量要再重新安装一次 解决 命令:...

徐代龙
2018/01/14
0
0
scrapy-1 安装scrapy

scrapy 2.2.0版本 环境 python2.7 可以使用pip install scrapy安装,建议在Ubuntu上安装。 Ubuntu 安装方法 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7;......

jastme
2014/06/13
126
0
Python Scrapy 爬虫(三):部署环境搭建

当我们的爬虫程序开发完成后,应该考虑的事情就是如何将我们的爬虫程序部署到我们的服务器上来运行了。 下面就以 CentOS 作为我们的爬虫程序的部署环境来介绍部署爬虫的第一步:部署环境搭建...

雨林_a1d6
2018/05/29
0
0
CentOS6.8安装scrapy爬虫框架

背景 最近想搞一个个人项目,需要用到爬虫,所以接触了scrapy。 环境参数 OS环境 CentOS6.8 x86_64 OS默认python版本 python2.6 安装问题 在默认python版本环境(python2.6)下安装了pip,通...

Baron聊聊技术
2017/11/23
0
0

没有更多内容

加载失败,请刷新页面

加载更多

OSChina 周二乱弹 —— 开发语言和语言开发的能一样么

Osc乱弹歌单(2019)请戳(这里) 【今日歌曲】 @花间小酌:#今日歌曲推荐# 分享The Score的单曲《Revolution》 《Revolution》- The Score 手机党少年们想听歌,请使劲儿戳(这里) @批判派...

小小编辑
今天
1K
15
oracle ORA-39700: database must be opened with UPGRADE option

ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-39700: database must be opened with UPGRADE option 进程 ID: 3650 会话 ID: 29......

Tank_shu
今天
3
0
分布式协调服务zookeeper

ps.本文为《从Paxos到Zookeeper 分布式一致性原理与实践》笔记之一 ZooKeeper ZooKeeper曾是Apache Hadoop的一个子项目,是一个典型的分布式数据一致性的解决方案,分布式应用程序可以基于它...

ls_cherish
今天
4
0
聊聊DubboDefaultPropertiesEnvironmentPostProcessor

序 本文主要研究一下DubboDefaultPropertiesEnvironmentPostProcessor DubboDefaultPropertiesEnvironmentPostProcessor dubbo-spring-boot-project-2.7.3/dubbo-spring-boot-compatible/au......

go4it
昨天
2
0
redis 学习2

网站 启动 服务端 启动redis 服务端 在redis 安装目录下 src 里面 ./redis-server & 可以指定 配置文件或者端口 客户端 在 redis 的安装目录里面的 src 里面 ./redis-cli 可以指定 指定 连接...

之渊
昨天
2
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部