将 json 对象插入数据湖

Question

我有几个 python api 端点可以在请求正文中获取数据。每次 api 调用任何想法时，我都想将此数据插入/添加到 azure datalake？

示例 api 端点

 @main.route("/order/add", methods=["POST"])
    def post_add_new_order():
    data = request.json
    for key in data:
        if not typesModule.key_type_and_value_type_are_equal(key, data[key]):
            return {"err": "One of the value types is incorrect"}

想要将此数据插入 Azure 数据湖

Answer 1

如果你想在python包中向Azure Data Lake Storage Gen1添加数据，我们可以使用包azure-datalake-store来实现。

例如

1. 创建服务主体

az login
az ad sp create-for-rbac -n 'Myapp' --skip-assignment 

1. 将服务主体分配给 Azure Data Lake Storage Gen1 帐户文件或文件夹访问控制。

Azure 数据湖 gen1 的 ACL 具有三个权限。有读取、写入和执行。请根据您的需要进行配置。更多详情请参考here和here

1. 代码

import json

import azure.datalake.store.lib as lib
from azure.datalake.store.core import AzureDLFileSystem

RESOURCE = 'https://datalake.azure.net/'
client_id = '42e0d***c4c522d988c4'
client_secret  = 'Gbx2eK6****ClJDfQpIjoae:'
tenant = 'e4c9ab4e-bd27-40d5-8459-230ba2a757fb'

 @main.route("/order/add", methods=["POST"])
    def post_add_new_order():
        data = request.get_json()
        json_data = json.dumps(data).encode('utf-8')
        adlCreds = lib.auth(tenant_id = tenant,
                client_secret = client_secret,
                client_id = client_id,
                resource=RESOURCE)
        
        adlsFileSystemClient = AzureDLFileSystem(adlCreds, store_name='testbowman')
        # check if the file exist 
        if adlsFileSystemClient.access('/test/data.json'):
            #append content
            with adlsFileSystemClient.open(path='/test/data.json', mode='ab') as f:
                f.write(json_data)
                f.write(b'\r\n')
        else:
            #create file and write
            with adlsFileSystemClient.open(path='/test/data.json', mode='wb') as f:
                f.write(json_data)
                f.write(b'\r\n')
        return {'you sent' : data}