【发布时间】:2021-09-25 21:10:30
【问题描述】:
尝试在 AWS 上部署爬虫。我在压缩后将以下结构作为层导入lambda
lambdalayer (folder)
:- python ( folder)
:- covid.py (file to run)
:- lib
:- python3.8
:-site-pakcages
:- All dependencies
我已经尝试了很多变体并尝试将文件移动到不同的文件夹,但lambda 无法导入covid.py。
这是lambda 代码。
import boto3
import json
from covid import scrapeGlobalCase
def lambda_handler(event,context):
print(os.environ['PYTHONPATH'])
s3 =boto3.resource('s3')
print("Request Covid Data..")
cov_case = scrapeGolbalCase()
BUCKET_NAME='seleniumbucket43'
DATE=f"{cov_case['date']}"
OUTPUT_NAME = f'dataKeyTest{DATE}.json'
OUTPUT_BODY = json.dumps(cov_case)
print("Sending to S3")
S3.bucket(BUCKET_NAME).put_object(Key=OUTPUT_NAME,Body=OUTPUT_BODY)
print(f'Job done at {DATE}')
错误:-
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'covid'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
}
covid.py文件
import requests,datetime
from bs4 import BeautifulSoup
def scrapeGlobalCase():
try:
url = "https://www.worldometers.info/coronavirus/"
req = requests.get(url)
bsObj = BeautifulSoup(req.text, "html.parser")
data = bsObj.find_all("div",class_ = "maincounter-number")
NumConfirmed = int(data[0].text.strip().replace(',', ''))
NumDeaths = int(data[1].text.strip().replace(',', ''))
NumRecovered = int(data[2].text.strip().replace(',', ''))
NumActive = NumConfirmed - NumDeaths - NumRecovered
TimeNow = datetime.datetime.now()
return {
'date': str(TimeNow),
'ConfirmedCases': NumConfirmed,
'ActiveCases': NumActive,
'RecoveredCases': NumRecovered,
'Deaths': NumDeaths
}
except Exception as e:
print(e)
p = scrapeGlobalCase()
print(p)
【问题讨论】:
标签: python aws-lambda aws-lambda-layers