在 Azure Functions 中安装 snowflake-connector-python[pandas]

Question

我正在尝试使用带有熊猫附加功能的 python-snowflake 连接器将 python 代码部署到使用 VS Code 扩展的天蓝色函数。本地运行该功能有效，部署本身也有效

部署期间已安装依赖项的 requirements.txt 如下所示：

azure-functions
xlrd
numpy
pandas
azure-storage-blob
snowflake-connector-python

代码中的导入如下：

import numpy as np
import pandas as pd
import azure.storage.blob
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import snowflake.connector
from snowflake.connector import DictCursor
#from snowflake.connector.pandas_tools import write_pandas

有了这些要求，一切正常。

问题是，一旦我将 requirements.txt 更改为包含雪花连接器的 pandas 附加功能（代码中需要）：

azure-functions
xlrd
numpy
pandas
azure-storage-blob
snowflake-connector-python[pandas]

我在尝试执行该函数时收到以下错误消息：

Result: Failure Exception: KeyError: 'snowflake-connector-python' Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 262, in _handle__function_load_request func = loader.load_function( File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 32, in call return func(*args, **kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/loader.py", line 76, in load_function mod = importlib.import_module(fullmodname) File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/site/wwwroot/HttpTrigger1/__init__.py", line 23, in <module> import snowflake.connector File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/__init__.py", line 17, in <module> from .connection import SnowflakeConnection File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/connection.py", line 47, in <module> from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/cursor.py", line 48, in <module> from .arrow_result import ArrowResult File "arrow_result.pyx", line 16, in init snowflake.connector.arrow_result File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/options.py", line 35, in <module> _pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']

关于如何使函数正确运行的任何帮助或想法？谢谢！

Answer 1

这似乎是熊猫工具中的一个错误，我也遇到了同样的问题。根据您计划使用的 pandas 工具，您可能能够绕过它。

当我查询 Snowflake 时，我的函数依赖于 fetch_pandas_all() 函数，但我能够通过仅安装基本的 snowflake-connector-python 包并使用直接安装 pandas 来解决它，然后将我的代码更改为使用以下内容：

import logging
import snowflake.connector
import pandas as pd

class SnowflakeReader(object):
    def __init__(self, accnt: str, warehouse: str, dbuser: str, pw: str, db: str, dbschema: str):
    self.accnt = accnt
    self.warehouse = warehouse
    self.db = db
    self.dbschema = dbschema
    self.dbuser = dbuser
    self.pw = pw


    def form_connection(self):
        try:
            ctx = snowflake.connector.connect(
                user = self.dbuser,
                password = self.pw,
                account = self.accnt,
                warehouse = self.warehouse,
                database = self.db,
                schema = self.dbschema
            )
        except Exception as e:
            logging.error('Failed to connect to Snowflake: ' + str(e))
        return ctx

    def read_all(self, query: str, conn) -> pd.DataFrame:
        self.query = query
        self.conn = conn

        try:
            cs = conn.cursor()
            sql_cmd = self.query
            cs.execute(sql_cmd)
            result = cs.fetchall()
            result_df = pd.DataFrame(result)
        except Exception as e:
            logging.error('Failed to read from Snowflake: ' + str(e))

        return result_df

看起来您正在尝试导入 write_pandas，这让我认为您希望将 DataFrame 直接写入 Snowflake 中的表。如果是这种情况，您可以使用旧的基于 SQLAlchemy 的方法构建类似的解决方法，或者只是将 DataFrame 转换为 dict 并使用 execute() 函数传递给插入语句。