【发布时间】:2020-04-03 21:53:32
【问题描述】:
我是 docker 新手,并且已经构建了一个自定义容器来在我的云服务器上运行我的蜘蛛。我的爬虫是使用 python 3.6、scrapy 1.6、selenium 构建的,并使用 docker 在一个容器中运行所有内容。启动蜘蛛时,我有scrapy open_spider 方法,该方法在生成用于抓取scrapy 的url 的目录中运行另一个python 脚本。该脚本将链接保存在文本文件中但是,我收到 PermissionError: [Errno 13] Permission denied: 'tmp.
我尝试在 tmp 文件夹上运行 chmod 777 和 a+rw,这样我就可以创建文本文件,但我仍然收到权限被拒绝错误。我已经研究了好几天,不知道如何解决这个问题。
我笔记本电脑上的操作系统是 ubuntu 18.04。
下面是我的 docker 文件的链接
Dockerfile
FROM scrapinghub/scrapinghub-stack-scrapy:1.6-py3
RUN apt-get -y --no-install-recommends install zip unzip jq libxml2 libxml2-dev
RUN printf "deb http://archive.debian.org/debian/ jessie main\ndeb-src http://archive.debian.org/debian/ jessie main\ndeb http://security.debian.org jessie/updates main\ndeb-src http://security.debian.org jessie/updates main" > /etc/apt/sources.list
#============================================
# Google Chrome
#============================================
# can specify versions by CHROME_VERSION;
# e.g. google-chrome-stable=53.0.2785.101-1
# google-chrome-beta=53.0.2785.92-1
# google-chrome-unstable=54.0.2840.14-1
# latest (equivalent to google-chrome-stable)
# google-chrome-beta (pull latest beta)
#============================================
ARG CHROME_VERSION="google-chrome-stable"
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \
&& apt-get update -qqy \
&& apt-get -qqy install \
${CHROME_VERSION:-google-chrome-stable} \
&& rm /etc/apt/sources.list.d/google-chrome.list \
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/*
#============================================
# Chrome Webdriver
#============================================
# can specify versions by CHROME_DRIVER_VERSION
# Latest released version will be used by default
#============================================
ARG CHROME_DRIVER_VERSION
RUN CHROME_STRING=$(google-chrome --version) \
&& CHROME_VERSION_STRING=$(echo "${CHROME_STRING}" | grep -oP "\d+\.\d+\.\d+\.\d+") \
&& CHROME_MAYOR_VERSION=$(echo "${CHROME_VERSION_STRING%%.*}") \
&& wget --no-verbose -O /tmp/LATEST_RELEASE "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROME_MAYOR_VERSION}" \
&& CD_VERSION=$(cat "/tmp/LATEST_RELEASE") \
&& rm /tmp/LATEST_RELEASE \
&& if [ -z "$CHROME_DRIVER_VERSION" ]; \
then CHROME_DRIVER_VERSION="${CD_VERSION}"; \
fi \
&& CD_VERSION=$(echo $CHROME_DRIVER_VERSION) \
&& echo "Using chromedriver version: "$CD_VERSION \
&& wget --no-verbose -O /tmp/chromedriver_linux64.zip https://chromedriver.storage.googleapis.com/$CD_VERSION/chromedriver_linux64.zip \
&& rm -rf /opt/selenium/chromedriver \
&& unzip /tmp/chromedriver_linux64.zip -d /opt/selenium \
&& rm /tmp/chromedriver_linux64.zip \
&& mv /opt/selenium/chromedriver /opt/selenium/chromedriver-$CD_VERSION \
&& chmod 755 /opt/selenium/chromedriver-$CD_VERSION \
&& sudo ln -fs /opt/selenium/chromedriver-$CD_VERSION /usr/bin/chromedriver
#============================================
# crawlera-headless-proxy
#============================================
RUN curl -L https://github.com/scrapinghub/crawlera-headless-proxy/releases/download/1.1.1/crawlera-headless-proxy-linux-amd64 -o /usr/local/bin/crawlera-headless-proxy \
&& chmod +x /usr/local/bin/crawlera-headless-proxy
RUN chmod a+rw app/cars/spiders
RUN chmod a+rw app/cars/tmp
COPY ./start-crawl /usr/local/bin/start-crawl
ENV TERM xterm
ENV SCRAPY_SETTINGS_MODULE cars.settings
RUN pip install --upgrade pip
RUN mkdir -p /app
WORKDIR /app
COPY ./requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app
RUN python setup.py install
RUN chmod a+rw app/cars/tmp
这是我的 setup.py 文件的链接
# Automatically created by: shub deploy
from setuptools import setup, find_packages
setup(
name='cars',
version='1.0',
packages=find_packages(),
entry_points={'scrapy': ['settings = cars.settings']},
)
【问题讨论】:
-
也许更新你的 Docker 版本?
-
您发布的 dockerfile 没有任何问题(我是在我的机器上构建的),所以要么您没有使用 docker 的权限,要么您的 setup.py 脚本有问题。如果你发布它,我可以看看
-
@MicheleLambertucci 我的猜测是我没有权限,但不能 100% 确定,因为我是 docker 新手并将其部署在云中。这是我的 setup.py 文件的链接pastiebin.com/5defd230e2546
-
@dcarlo56ave 不要放置外部链接,因为它们可能会使您的问题对未来的读者毫无用处而被破坏。将所有代码放入您的问题中
-
@eyllanesc 对此感到抱歉,并且会这样做。
标签: python docker scrapy dockerfile