example - Error de python de Scrapy-Falta el esquema en la URL de solicitud

scrapy vs beautifulsoup

Question

Estoy tratando de extraer un archivo de un servidor FTP protegido con contraseña. Este es el código que estoy usando:

import scrapy
from scrapy.contrib.spiders import XMLFeedSpider
from scrapy.http import Request
from crawler.items import CrawlerItem
class SiteSpider(XMLFeedSpider):
name = ''site''
allowed_domains = [''ftp.site.co.uk'']
itertag = ''item''
def start_requests(self):
yield Request(''ftp.site.co.uk/feed.xml'',
meta={''ftp_user'': ''test'', ''ftp_password'': ''test''})
def parse_node(self, response, selector):
item = CrawlerItem()
item[''title''] = (selector.xpath(''//title/text()'').extract() or [''''])[0]
return item

Este es el error de rastreo que obtengo:

 Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
self.mainLoop()
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
self.runUntilCurrent()
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilC
urrent
call.func(*call.args, **call.kw)
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/reactor.py", line 41, in __call__
return self._func(*self._a, **self._kw)
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/engine.py", line 112, in _next_reques
t
request = next(slot.start_requests)
File "/var/www/spider/crawler/spiders/site.py", line 13, in start_requests
meta={''ftp_user'': ''test'', ''ftp_password'': ''test''})
File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 26, in __i
nit__
self._set_url(url)
File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 61, in _se
t_url
raise ValueError(''Missing scheme in request url: %s'' % self._url)
exceptions.ValueError: Missing scheme in request url: ftp.site.co.uk/f
eed.xml 

Answer 1

Necesita agregar esquema para la URL:

ftp://ftp.site.co.uk

La sintaxis de FTP URL se define como:

ftp://[<user>[:<password>]@]<host>[:<port>]/<url-path>

Básicamente, haces esto:

yield Request(''ftp://ftp.site.co.uk/feed.xml'', ...)

Lea más sobre los esquemas en Wikipedia: http://en.wikipedia.org/wiki/URI_scheme