Cloud Storage 触发 Cloud Function 总是超时并超过最后期限答案

【问题标题】：Cloud Storage triggered Cloud Function always timed out and exceeds deadlineCloud Storage 触发 Cloud Function 总是超时并超过最后期限
【发布时间】：2020-08-06 02:39:44
【问题描述】：

我正在使用 Google Cloud Storage 事件 来触发将上传的 CSV 写入 Cloud Datastore 的云功能。问题是 csv 文件有多个 8000 rows 并且函数的最大超时 9mins 不够。

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

我也尝试过批处理操作，但仍然是同样的超时问题。 在不涉及太多重新架构的情况下，我可以使用替代解决方案吗？

const {Datastore} = require('@google-cloud/datastore');
const {Storage} = require('@google-cloud/storage');
const db = new Datastore();
const storage = new Storage();

const path = require('path');
const os = require('os');
const fs = require('fs');
const csv = require('csv-parser');

exports.updateMasterlist = async (object, context, callback) => {

    const fileBucket = object.bucket;
    const filePath = object.name; 
    const bucket = storage.bucket(fileBucket);
    const fileName = path.basename(filePath);
    const tempFilePath = path.join(os.tmpdir(), fileName);

    await bucket.file(filePath).download({destination: tempFilePath})
    
    var total = 0; 
    var max = 0;
    var employees = [];
    var batch = 1;
    const kind = 'masterlist';

    fs.createReadStream(tempFilePath)
        .pipe(csv())
        .on('data', (record) => {
            let key = record['EmployeeID'];
            var empKey = db.key([kind, key]);

            const employee = {
                emp_id: record['EmployeeID'],
                full_name: `${record['Firstname']} ${record['MiddleName']} ${record['Lastname']}`,
                group: record['GroupName'],
                division: record['Division'],
                department: record['Department'],
                is_id: record['SupervisorID'],
                email_address: record['Email']
            };

            const emp_entity = {
                key: empKey,
                data: employee,
            };

            employees.push(emp_entity);
            total++; max++;

            if (max >= 499){
                try {
                    db.upsert(employees);
                    console.log(`Uploading batch ${batch}`);
                    batch++;
                }
                catch (e) {
                    console.error(e);
                    process.exit(1);
                }
                employees.length = 0;
                max=0; 
            }

        })
        .on('end', async () => {
            try {
                await db.upsert(employees);
                console.log(`Uploading batch ${batch}`);
            }
            catch (e) {
                console.error(e);
                process.exit(1);
            }
            console.log("End of CSV file read!");
            console.log(`BATCH INFORMATION: `);
            console.log(`number of employees: ${total}`);
        });

    callback();
};

【问题讨论】：

请您提供一些代码。我怀疑它需要这么长时间。即使是 8000 行也不应该花太长时间 - csv 的大小是多少？有多个吗？
您的代码可能有问题，请编辑问题以分享。
编辑：我已经添加了代码
我的第一印象是你没有正确处理承诺。
函数将 8000 行写入您的本地环境需要多长时间？

标签： javascript google-cloud-platform google-cloud-functions google-cloud-storage

【解决方案1】：

这是使用 Python 3.7 的解决方案

from google.cloud import datastore
import csv
import time
start_time = time.time()
db = datastore.Client()
employees = []
count=1

with open('masterlist.csv', 'rt') as f:
    reader = csv.DictReader(f)
    for record in reader:
        # Datastore Entities
        key = record['EmployeeID']
        empKey = db.key('masterlist', key)
        employee = datastore.Entity(key=empKey)

        employee.update({
            'emp_id': record['EmployeeID'],
            'full_name': str(record['Firstname']+' '+record['MiddleName']+' '+record['Lastname']),
            'group': record['GroupName'],
            'division': record['Division'],
            'department': record['Department'],
            'is_id': record['SupervisorID'],
            'email_address': record['Email']
        })
        employees.append(employee)

        if(len(employees) >= 499):
            db.put_multi(employees)
            employees.clear()
            print('Batch: '+str(count)+' added')
            count += 1

db.put_multi(employees)
print('Batch: '+str(count)+' added')
print("--- %s seconds ---" % (time.time() - start_time))

【讨论】：