以编程方式创建没有源或目标的数据流任务

Question

我有一个 SSIS 包，可以将数据从 Microsoft Access 复制到 SQL Server。两组表几乎相同。

背景：不出所料，我们的表架构随着我们产品的开发而增长。因此，我们一直需要使用新列更新 SSIS 包。这是一项非常枯燥的任务，因此我尝试在 C# 中以编程方式创建 SSIS 包。进展顺利，但我想让这个过程更简单。

问题： 包生成过程要求源 (Access) 和目标 (SQL Server) 在 C# 运行时存在。这不适用于我们当前的流程。因此，我想：

• 以编程方式提供表和列（SQL 和 Access 表是从 XML 元数据生成的，因此我们可以从相同的元数据向 SSIS 提供表和列）
• 在加载包的时候，运行一些C#来修改包，即ReinitializeMetaData等

我非常喜欢这两种解决方案中的第一种。但是我不知道如何使它们中的任何一个起作用。

更多详情：我一直在使用以下 SSIS 代码（来自 Microsoft 的示例）。我想我想要一些无需执行 AcquireConnections 或 ReinitializeMetaData 即可运行的东西——我想自己提供元数据。显然，我提供的内容必须与验证和运行包时实际存在的内容完全匹配。

public IDTSComponentMetaData100 AddDestAdapter(IDTSPipeline100 pipeline, ConnectionManager destConnMgr, out IDTSDesigntimeComponent100 destDesignTimeComp)
{
    IDTSComponentMetaData100 destComp = pipeline.ComponentMetaDataCollection.New();
    destComp.ComponentClassID = OLEDB_DEST_GUID;
    destComp.ValidateExternalMetadata = true;
    destDesignTimeComp = destComp.Instantiate();
    destDesignTimeComp.ProvideComponentProperties();
    destComp.Name = "OleDB Destination - Sql Server";
    destDesignTimeComp.SetComponentProperty("AccessMode", 0);
    destDesignTimeComp.SetComponentProperty("OpenRowset", quotedTableName);

    // set connection
    destComp.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(destConnMgr);
    destComp.RuntimeConnectionCollection[0].ConnectionManagerID = destConnMgr.ID;

    // get metadata
    destDesignTimeComp.AcquireConnections(null);
    destDesignTimeComp.ReinitializeMetaData();
    destDesignTimeComp.ReleaseConnections();

    extCols = destComp.InputCollection[0].ExternalMetadataColumnCollection;

    return destComp;
}

public void AddPathsAndConnectColumns()
{
    IDTSOutput100 srcOutput = srcComp.OutputCollection[0];
    IDTSOutputColumnCollection100 srcOutputCols = srcOutput.OutputColumnCollection;
    IDTSInput100 destInput = destComp.InputCollection[0];
    IDTSInputColumnCollection100 destInputCols = destInput.InputColumnCollection;
    IDTSExternalMetadataColumnCollection100 destExtCols = destInput.ExternalMetadataColumnCollection;

    Hashtable destColtable = new Hashtable(destExtCols.Count);
    foreach (IDTSExternalMetadataColumn100 extCol in destExtCols)
    {
        destColtable.Add(extCol.Name, extCol);
    }

    // colConvertTable stores a pair of columns which need a type conversion
    // colConnectTable stores a pair of columns which dont need a type conversion and can be connected directly.
    Hashtable colConvertTable = new Hashtable(srcOutputCols.Count);
    Hashtable colConnectTable = new Hashtable(srcOutputCols.Count);
    foreach (IDTSOutputColumn100 outputCol in srcOutputCols)
    {
        // Get the column name to look for in the destination.
        // Match column by name if match table is not used.
        String colNameToLookfor = outputCol.Name;

        IDTSExternalMetadataColumn100 extCol = (String.IsNullOrEmpty(colNameToLookfor)) ? null : (IDTSExternalMetadataColumn100)destColtable[colNameToLookfor];
        // Does the destination column exist?
        if (extCol != null)
        {
                colConnectTable.Add(outputCol.ID, extCol);
        }
    }

    // Convert transform not needed. Connect src and destination directly.
    pipeline.PathCollection.New().AttachPathAndPropagateNotifications(srcOutput, destInput);

    IDTSVirtualInput100 destVirInput = destInput.GetVirtualInput();

    foreach (object key in colConnectTable.Keys)
    {
        int colID = (int)key;
        IDTSExternalMetadataColumn100 extCol = (IDTSExternalMetadataColumn100)colConnectTable[key];
        // Create an input column from an output col of previous component.
        destVirInput.SetUsageType(colID, DTSUsageType.UT_READONLY);
        IDTSInputColumn100 inputCol = destInputCols.GetInputColumnByLineageID(colID);
        if (inputCol != null)
        {
            // map the input column with an external metadata column
            destDesignTimeComp.MapInputColumn(destInput.ID, inputCol.ID, extCol.ID);
        }
    }
}

可能的探索路线：

• This MSDN thread 有人有同样的问题（虽然没有具体的解决方案）
• 根据对 BIML 的一些调查，似乎 BIML struggles with the same issues（OfflineSchema 似乎可以工作，虽然我有点担心，因为我在 Internet 上找不到一个示例）。李>

Answer 1

IDTSExternalMetadataColumn100 和 IDTSExternalMetadataColumnCollection100 是您正在寻找的用例的朋友。当对 AcquireConnections、ReinitializeMetaData、ReleaseConnections 的调用成功时，有关外部列的元数据将从外部源中获取并填充到集合中，并且每个外部列都映射到 IDTSInputColumn100/IDTSOutputColumn100 列（取决于组件的类型）。这些调用负责获取元数据并创建所需的 IDTSExternalMetadataColumn100 和 IDTSInputColumn100/IDTSOutputColumn100 列集合。

不幸的是，（至少据我所知），我们无法覆盖 ReinitializeMetaData 方法（此外，覆盖可能会导致其他问题，因此最好不要覆盖它们）。但是我们仍然可以实现最终结果，尽管方式更详细。这意味着我们将自己创建所需的 IDTSExternalMetadataColumn100 和 IDTSInputColumn100/IDTSOutputColumn100 列（以编程方式）。

查看下面的代码，了解涉及源组件的示例。您将对 Destination 组件执行相同的操作，但不是使用 IDTSOutputColumn，而是在 Destination 组件的情况下使用 IDTSInputColumn。

下面的代码直接使用了 SSIS 对象模型库；如您所见，您需要做很多事情。我写了一个库来简化其中的一些事情。该库包含在类似用例中创建目标组件的示例。看here；特别是 GenerateProjectToLoadTextFilesToSqlServerDatabase 方法末尾的代码。

namespace ConsoleApplication5
{
// A struct ot represent an external column
public struct Column
{
    public String Name;
    public String SSISDataType;
    public int Length;
    public int Precision;
    public int Scale;
    public int CodePage;

    public Column(String name, String ssisDataType, int length, int precision, int scale, int codePage)
    {
        Name = name;
        SSISDataType = ssisDataType;
        Length = length;
        Precision = precision;
        Scale = scale;
        CodePage = codePage;
    }
}

public class Packager
{
    public Packager()
    {
        build();
    }

    private void build()
    {
        #region Package Related
        // Package related
        Package package = new Package();
        Executable e = package.Executables.Add("STOCK:PipelineTask");
        TaskHost thMainPipe = e as TaskHost;
        MainPipe dataFlowTask = thMainPipe.InnerObject as MainPipe;
        thMainPipe.Name = "MyDFT";
        thMainPipe.DelayValidation = true;
        #endregion

        #region Add Connection Manager
        // Add Connection Manager
        ConnectionManager cm = package.Connections.Add("OLEDB");
        cm.Name = "OLEDB ConnectionManager";
        cm.ConnectionString = "Data Source=(local);" +
          "Initial Catalog=AdventureWorks;Provider=SQLOLEDB.1;" +
          "Integrated Security=SSPI;";
        #endregion

        #region Add a OleDB Source and set up basic properties
        // Add an OLE DB source to the data flow.  
        IDTSComponentMetaData100 component = dataFlowTask.ComponentMetaDataCollection.New();
        component.Name = "OLEDBSource";
        component.ComponentClassID = "Microsoft.OLEDBSource"; // check for the exact component class ID on your machine

        // Get the design time instance of the component.  
        CManagedComponentWrapper instance = component.Instantiate();

        // Initialize the component  
        instance.ProvideComponentProperties();

        // Specify the connection manager.  
        if (component.RuntimeConnectionCollection.Count > 0)
        {
            component.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(package.Connections[0]);
            component.RuntimeConnectionCollection[0].ConnectionManagerID = package.Connections[0].ID;
        }

        // Set the custom properties.  
        instance.SetComponentProperty("AccessMode", 2);
        instance.SetComponentProperty("SqlCommand", "Select * from Production.Product");
        #endregion

        #region Core example showcasing use of IDTSExternalMetadataColumn when external data source is not available.

        // Typically here we call acquireconnection, reinitmetadata etc to get the metadata from a data source that exists.
        // Instead we will populate the metadata ourselves

        #region Get External Columns Metadata
        // Get the collection of external columns
        List<Column> externalColumns = new List<Column>();

        // Hard Coding Here. But grab them from your metadata source programmatically.
        Column columnA = new Column("col_a", "DT_STR", 24, 0, 0, 1252);
        Column columnB = new Column("col_b", "DT_STR", 36, 0, 0, 1252);
        Column columnC = new Column("col_c", "DT_STR", 48, 0, 0, 1252);
        externalColumns.Add(columnA);
        externalColumns.Add(columnB);
        externalColumns.Add(columnC);
        #endregion

        #region Add External Columns to our required IDTSOutput100
        // Grab the appropriate output as needed. We will be adding ExternalColumns to this Output
        IDTSOutput100 output = component.OutputCollection[0];

        // Add each external column to the above IDTSOutPut
        foreach (Column extCol in externalColumns)
        {
            IDTSExternalMetadataColumn100 col = output.ExternalMetadataColumnCollection.New();
            col.Name = extCol.Name;
            col.Scale = extCol.Scale;
            col.Precision = extCol.Precision;
            col.Length = extCol.Length;
            col.CodePage = extCol.CodePage;
            col.DataType = (Wrapper.DataType)Enum.Parse(typeof(Wrapper.DataType), extCol.SSISDataType);
        }
        #endregion

        #region Create OutputColumn if it does not exist/or grab the output column if it Exists. Then associate it to the External Column

        // Now associate the External Column to an Output Column.
        // Here, we will simply associate the external column to an output column if the name matches (because of our use case)
        foreach (IDTSExternalMetadataColumn100 extCol in output.ExternalMetadataColumnCollection)
        {
            bool outputColExists = false;

            // Set DataTypes and Associate with external column if output col exists
            foreach (IDTSOutputColumn100 outputCol in output.OutputColumnCollection)
            {
                if (outputCol.Name == extCol.Name) // is map based on name
                {
                    // Set the data type properties
                    outputCol.SetDataTypeProperties(extCol.DataType, extCol.Length, extCol.Precision, extCol.Scale, extCol.CodePage);

                    // Associate the external column and the output column
                    outputCol.ExternalMetadataColumnID = extCol.ID;

                    outputColExists = true;
                    break;
                }
            }

            // Create an IDTSOutputColumn if not exists. 
            if (!(outputColExists))
            {
                IDTSOutputColumn100 outputCol = output.OutputColumnCollection.New();
                outputCol.Name = extCol.Name;  // map is based on name

                // Set the data type properties
                outputCol.SetDataTypeProperties(extCol.DataType, extCol.Length, extCol.Precision, extCol.Scale, extCol.CodePage);

                // Associate the external column and the output column
                outputCol.ExternalMetadataColumnID = extCol.ID;
            }
        }
        #endregion

        #endregion

        #region Save the Package to disk

        new Application().SaveToXml(@"C:\Temp\Pkg.dtsx", package, null);

        #endregion
    }
}

}

Answer 2

只需使用 BimlExpress，它是 Visual Studio 的免费插件。它提供了极大的灵活性，而无需与 DTS API 进行无聊且容易出错的交互。