如何将我的表转换为规范化结构？

Question

对于我的游戏应用，我有一个 CSV 格式的数据表（其中一个）。 它目前有 40 多列，每个职业（战士、盗贼、巫师等）和技能名称各一列。

每行 (40+) 包含技能名称，以及一个字符串，表示每个职业该技能的成本。

来自原始 .csv 的 sn-p 包含如下所示的数据：

Skill,Fighter,Thief,Rogue
Armor-Heavy,2,4,3
Armor-Light,1,2,1
Armor-Medium,2,3,2

要将其输入标准化数据库，我应该有一个职业列表表、一个技能列表表和一个技能成本表。

我已经建立并填充了技能列表和职业列表：

CREATE TABLE "skills" ("_id" INTEGER, "skill" TEXT)

CREATE TABLE "professions" ("_id" INTEGER, "profession" TEXT)

我可以创建一个表来保存标准化数据：

CREATE TABLE "skillCostByProfession" ("skill_id" INTEGER, "profession_id" INTEGER, "cost" INTEGER)

skillCostByProfession 将整理数据：

profession_id, skill_id, cost
1            |   1     |   2         (Fighter, Armor-Heavy, cost (2))
1            |   2     |   1         (Fighter, Armor-Light, cost (1))
1            |   3     |   2         (Fighter, Armor-Medium, cost (2))
2            |   1     |   4         (Thief, Armor-Heavy, cost (4))
2            |   2     |   2         (Thief, Armour-Light, cost (2))
2            |   3     |   3         (Thief, Armour-Medium, cost (3))
3            |   1     |   3         (Rogue, Armor-Heavy, cost (3))
3            |   2     |   1         (Rogue, Armour-Light, cost (1))
3            |   3     |   2         (Rogue, Armour-Medium, cost (2))

等

我正在努力寻找一种方法将原始 .csv 中的数据放入新的 SkillCostByProfession 表中

我正在使用 LibreOffice Calc 和 DB Browser (SQLite) 的组合来为我的 Android 应用程序准备数据库。

如何在不手写 1600 行的情况下将我拥有的数据表转换为我需要的格式？

Answer 1

我把它当作个人的挑战，并写了这个伪代码，以便你可以修改它：

public Dictionary professionsToID = new Dictionary<string, string>();

public  Dictionary skillsToID = new Dictionary<string, string>();

public void Main(string[] args)
{
        string newcsvfile = "path\tempfilename.csv";
        Dictionary d = prepare();
        WriteDictionaryToFile(newcsvfile,d);


        AddSkillsToDatabase("source.csv", CreateConnection())
        skillsToID = GetSkillsId();

        AddProfessionToDatabase("source.csv", CreateConnection())
        professionsToID = GetProfessionId();


        TextToReferences("source.csv", "destination.csv");
}

public Dictionary prepare()
{

    // Reading textfile line by line
    https://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java

    file = csv;

    List<String[]> actualCSVList = new List<String[]>
    String[][] actualCSVArr = new String[][]();


    try (BufferedReader br = new BufferedReader(new FileReader(file))) {
        String line;
        while ((line = br.readLine()) != null)
        {
            // process the line.

            //fill columnvalues in an array wich represents a row
            String[] arrOfRow = line.split(",");

            //Add row to List
            actualCSVList.Add(arrOfRow);
        }

        //convert list to multidimensional array
        actualCSVArr = actualCSVList.toArray();


        for(int d1 = 0; d1 <= actualCSVList.count-1, d1++)
        {
            //loop lines

            String[] actualLine = actualCSVArr[d1];
            Dictionary <String, String> rows = new Dictionary <String, String>();

            string skill = "";
            string prof = "";


            for(int i = 0; i <= actualLine.lenght; i++)
            {
                //replace index with profession (or skill)

                switch(i)
                {
                    case 0:
                        skill = actualLine[i].ToString();
                    break;

                    case 1:
                        prof = "Fighter";
                    break;

                    case 2:
                        prof = "Thief";
                    break;

                    case 3:
                        prof = "Rogue";
                    break;
                }

                //is the case when i = 0
                if(!prof.Equals(""))
                {
                    // Add: profession, skill, cost => "profession|arrayindex","skill:cost"
                    rows.Add(prof + "|" + i.ToString(), skill + ":" + actualLine[i].ToString());
                }
            }
        }

    return rows;
}

void WriteDictionaryToFile(string filename, Dictionary dic)
{
    //Write to the new CSV
    foreach (KeyValuePair<string,string>  kvp in dic)
    {
        string prof = kvp.Key.Split("|")[0];

        string[] skillAndCost = kvp.Value.Split(":");

        string skill = skillAndCost[0];
        string cost = skillAndCost[1];

        string nline = prof + "," + skill + "," + cost;

        System.IO.File.WriteNewLine(filename, nline);
    }
}

void TextToReferences(string sourcefile, string destinationfile)
{
    // Reading textfile line by line
    https://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java

    try (BufferedReader br = new BufferedReader(new FileReader(sourcefile)))
    {
        String line;
        while ((line = br.readLine()) != null)
        {
            // process the line.

            // 0=prof 1=skill 2=cost
            string[] arrline = line.Split(",");
            string prof = arrline[0];
            string skill = arrline[1];
            string cost = arrline[2];

            string nline = professionsToID[prof] + "," + skillsToID[skill] + "," + cost;

            System.IO.File.WriteNewLine(destinationfile, nline));
        }
     }
}

Dictionary GetSkillsId()
{
    //https://www.codeguru.com/csharp/.net/net_data/using-sqlite-in-a-c-application.html

    Dictionary dic = new Dictionary();
    SQLiteConnection sqlite_conn;
    sqlite_conn = CreateConnection();
    List<string> columnnames = ReadColumnnames(sqlite_conn);


    SQLiteDataReader sqlite_datareader;
    SQLiteCommand sqlite_cmd;

    foreach (string column in columnnames)
    {
        sqlite_cmd = conn.CreateCommand();
        sqlite_cmd.CommandText = "SELECT id FROM skills WHERE skill = column";

        sqlite_datareader = sqlite_cmd.ExecuteReader();
        while (sqlite_datareader.Read())
        {
            string id = sqlite_datareader.GetString(0);
            dic.Add(column,id);
        }
    }
    conn.Close();


    return dic;
}

Dictionary GetProfessionsId()
{
    //https://www.codeguru.com/csharp/.net/net_data/using-sqlite-in-a-c-application.html

    Dictionary dic = new Dictionary();
    SQLiteConnection sqlite_conn;
    sqlite_conn = CreateConnection();
    List<string> columnnames = ReadColumnnames(sqlite_conn);


    SQLiteDataReader sqlite_datareader;
    SQLiteCommand sqlite_cmd;

    foreach (string column in columnnames)
    {
        sqlite_cmd = conn.CreateCommand();
        sqlite_cmd.CommandText = "SELECT id, profession FROM professions";

        sqlite_datareader = sqlite_cmd.ExecuteReader();
        while (sqlite_datareader.Read())
        {
            string id = sqlite_datareader.GetString(0);
            string pro = sqlite_datareader.GetString(1);
            dic.Add(pro,id);
        }
    }
    conn.Close();


    return dic;
}

static SQLiteConnection CreateConnection()
{

    SQLiteConnection sqlite_conn;
    // Create a new database connection:
    sqlite_conn = new SQLiteConnection("Data Source=
    database.db;Version=3;New=True;Compress=True;");
    // Open the connection:
    try
    {
        sqlite_conn.Open();
    }
    catch (Exception ex)
    {

    }

    return sqlite_conn;
}

static List<string> ReadColumnnames(SQLiteConnection conn)
{
    //https://stackoverflow.com/questions/685206/how-to-get-a-list-of-column-names/685212

    List<string> columnnames = new List<string>();

    SQLiteDataReader sqlite_datareader;
    SQLiteCommand sqlite_cmd;
    sqlite_cmd = conn.CreateCommand();
    sqlite_cmd.CommandText = "SELECT sql FROM sqlite_master WHERE tbl_name = 'table_name' AND type = 'table'";

    sqlite_datareader = sqlite_cmd.ExecuteReader();
    while (sqlite_datareader.Read())
    {
    string myreader = sqlite_datareader.GetString(0);
    columnnames.Add(myreader);
    }
    conn.Close();

    return columnnames;
}

static SQLiteConnection CreateConnection()
{

    SQLiteConnection sqlite_conn;
    // Create a new database connection:
    sqlite_conn = new SQLiteConnection("Data Source=
    database.db;Version=3;New=True;Compress=True;");
    // Open the connection:
    try
    {
    sqlite_conn.Open();
    }
    catch (Exception ex)
    {

    }
    return sqlite_conn;
}

static void AddSkillsToDatabase(string sourcefile, SQLiteConnection conn)
{
    // Reading textfile line by line
    https://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java

    list<string> skills = new list<string>();

    try (BufferedReader br = new BufferedReader(new FileReader(sourcefile)))
    {
        String line;
        while ((line = br.readLine()) != null)
        {
            // process the line.

            // 1=skill
            string[] arrline = line.Split(",");
            string skill = arrline[0];

            if(!skills.Contains(skill))
            {
                skills.Add(skill);
            }
        }
     }


    SQLiteCommand sqlite_cmd;
    sqlite_cmd = conn.CreateCommand();

    foreach(string s in skills)
    {
        sqlite_cmd.CommandText = "INSERT INTO skills(skill) VALUES (s);";
        sqlite_cmd.ExecuteNonQuery();
    }

}

static void AddProfessionToDatabase(string sourcefile, SQLiteConnection conn)
{
    // Reading textfile line by line
    https://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java

    list<string> professions = new list<string>();

    try (BufferedReader br = new BufferedReader(new FileReader(sourcefile)))
    {
        String line;
        while ((line = br.readLine()) != null)
        {
            // process the line. Firstline are the professions

            string[] arrline = line.Split(",");

            foreach(string p = arrline)
            {
                professions.Add(p);
            }

            //TODO: remove firts entry

            break;
        }
     }


    SQLiteCommand sqlite_cmd;
    sqlite_cmd = conn.CreateCommand();

    foreach(string p in professions)
    {
        sqlite_cmd.CommandText = "INSERT INTO professions(profession) VALUES (p);";
        sqlite_cmd.ExecuteNonQuery();
    }

}

一个好的数据库的解决方案是一个好的概念。 为确保这一点，您应该始终在开始编写数据库之前对其进行规划（我自己并没有每次都这样做：D）。

1. 规划功能、窗口等
2. 规划数据库并确保您拥有步骤 1 中对象所需的所有值
3. 创建数据库
4. 为步骤 1 中的平面对象创建所需的查询
5. 实现第 1 步的对象并检查数据库和对象是否“匹配”

作为规划数据库的重要起点，您可以创建图表/方案

对于上面的方案，我使用了 Microsoft Visio（在模板搜索框中键入“UML”并选择“UML-Datanotation”）。 您可以找到很多工具，只需搜索“UML 工具”，然后您就可以搜索与您的规格相匹配的工具。 您应该寻找一种支持您的编程语言的工具，因为这些工具大多具有适合该语言的不错功能（例如设计数据库图 => 创建程序代码）。

一些工具是：

• Microsoft Visio（仅用于绘图）
• 型号
• 视觉范式

（@所有读者：你能添加一些工具吗？）

也许您的 IDE 也有一些图表功能。

但你应该先看看 UML（Unified Modeling Language）。 然后，您可以决定哪个程序以您最喜欢的方式绘制它（或者在最坏的情况下手动绘制图表 - 有时它在旅行和有想法时很有用）。

Answer 2

一个朋友给我写了一些 python 代码，它（几乎）可以满足我的需要。

filein=open("skills.csv","r")
fileout=open("out.txt","w")
professionline = filein.readline()
professions = professionline.rstrip().split(",")
for line in filein.readlines()[1:]:
    fields=line.rstrip().split(",")
    skill=fields[0]
    i=1
    for field in fields[1:]:
        cost=field
        if cost == "":
            cost = '0'      

        fileout.write("INSERT INTO skillCosts (skill_id, profession_id, cost) VALUES ((SELECT _id FROM skills WHERE category=\"{}\"), (SELECT _id FROM professions WHERE profession=\"{}\"), {});\n".format(skill, professions[i], cost))
        i+=1

这会加载 .csv，并将顶行（专业标题）转换为数组。
然后，它获取每一行成本并创建一个 SQL 查询。
这组查询被写入一个文件（此示例使用 .txt，我可以在其中复制/粘贴到 DB Browswer 的 sql-input 部分）
可以将单个查询格式化为 INSERT 数据，但我的表超过了 1,000 项限制。

Answer 3

首先，我想说这是您选择的最佳解决方案。

我想你可以这样解决：

（“轻”、“中”、“重”列用于检查“级别”是否可用——因此，如果每个技能都有这三个“级别”，则不必创建它们）

（轻、中、重 => 内容将是“级别”的成本）