【问题标题】:mongodb group values by multiple fieldsmongodb按多个字段分组值
【发布时间】:2014-05-20 20:57:44
【问题描述】:

例如,我有这些文件:

{
  "addr": "address1",
  "book": "book1"
},
{
  "addr": "address2",
  "book": "book1"
},
{
  "addr": "address1",
  "book": "book5"
},
{
  "addr": "address3",
  "book": "book9"
},
{
  "addr": "address2",
  "book": "book5"
},
{
  "addr": "address2",
  "book": "book1"
},
{
  "addr": "address1",
  "book": "book1"
},
{
  "addr": "address15",
  "book": "book1"
},
{
  "addr": "address9",
  "book": "book99"
},
{
  "addr": "address90",
  "book": "book33"
},
{
  "addr": "address4",
  "book": "book3"
},
{
  "addr": "address5",
  "book": "book1"
},
{
  "addr": "address77",
  "book": "book11"
},
{
  "addr": "address1",
  "book": "book1"
}

等等。


我怎样才能提出请求,它将描述前 N 个地址和每个地址前 M 本书?

预期结果示例:
地址1 | book_1: 5
| book_2: 10
| book_3: 50
|总计:65
______________________
地址2 | book_1: 10
| book_2: 10
|...
| book_M: 10
|总计:M*10
...
______________________
地址N | book_1: 20
| book_2: 20
|...
| book_M: 20
|总计:M*20

【问题讨论】:

    标签: mongodb aggregation-framework


    【解决方案1】:

    使用如下聚合函数:

    [
    {$group: {_id : {book : '$book',address:'$addr'}, total:{$sum :1}}},
    {$project : {book : '$_id.book', address : '$_id.address', total : '$total', _id : 0}}
    ]
    

    它会给你如下结果:

            {
                "total" : 1,
                "book" : "book33",
                "address" : "address90"
            }, 
            {
                "total" : 1,
                "book" : "book5",
                "address" : "address1"
            }, 
            {
                "total" : 1,
                "book" : "book99",
                "address" : "address9"
            }, 
            {
                "total" : 1,
                "book" : "book1",
                "address" : "address5"
            }, 
            {
                "total" : 1,
                "book" : "book5",
                "address" : "address2"
            }, 
            {
                "total" : 1,
                "book" : "book3",
                "address" : "address4"
            }, 
            {
                "total" : 1,
                "book" : "book11",
                "address" : "address77"
            }, 
            {
                "total" : 1,
                "book" : "book9",
                "address" : "address3"
            }, 
            {
                "total" : 1,
                "book" : "book1",
                "address" : "address15"
            }, 
            {
                "total" : 2,
                "book" : "book1",
                "address" : "address2"
            }, 
            {
                "total" : 3,
                "book" : "book1",
                "address" : "address1"
            }
    

    我没有完全得到您预期的结果格式,因此请随意将其修改为您需要的格式。

    【讨论】:

    • 那只解决了部分问题,并没有对两个分组做“顶”。
    • 此外,关于@WiredPrairie 的评论,我看不出这甚至解决了所提出问题的任何部分。 “每个地址的前 N ​​个地址和前 N 个书”。
    • 如果您可以解决 mongoDB 中的相关问题,请提供帮助 - stackoverflow.com/questions/61067856/…
    【解决方案2】:

    TLDR 总结

    在现代 MongoDB 版本中,您可以在基本聚合结果之外使用 $slice 强制执行此操作。对于“大”结果,请为每个分组运行并行查询(演示列表位于答案末尾),或等待SERVER-9377 解决,这将允许将项目数“限制”为@987654330 @ 到一个数组。

    db.books.aggregate([
        { "$group": {
            "_id": {
                "addr": "$addr",
                "book": "$book"
            },
            "bookCount": { "$sum": 1 }
        }},
        { "$group": {
            "_id": "$_id.addr",
            "books": { 
                "$push": { 
                    "book": "$_id.book",
                    "count": "$bookCount"
                },
            },
            "count": { "$sum": "$bookCount" }
        }},
        { "$sort": { "count": -1 } },
        { "$limit": 2 },
        { "$project": {
            "books": { "$slice": [ "$books", 2 ] },
            "count": 1
        }}
    ])
    

    MongoDB 3.6 预览版

    仍然无法解析SERVER-9377,但在此版本中$lookup 允许一个新的“非相关”选项,该选项将"pipeline" 表达式作为参数而不是"localFields""foreignFields" 选项。然后,这允许与另一个管道表达式“自连接”,我们可以在其中应用 $limit 以返回“top-n”结果。

    db.books.aggregate([
      { "$group": {
        "_id": "$addr",
        "count": { "$sum": 1 }
      }},
      { "$sort": { "count": -1 } },
      { "$limit": 2 },
      { "$lookup": {
        "from": "books",
        "let": {
          "addr": "$_id"
        },
        "pipeline": [
          { "$match": { 
            "$expr": { "$eq": [ "$addr", "$$addr"] }
          }},
          { "$group": {
            "_id": "$book",
            "count": { "$sum": 1 }
          }},
          { "$sort": { "count": -1  } },
          { "$limit": 2 }
        ],
        "as": "books"
      }}
    ])
    

    这里的另外一个补充当然是可以通过$expr 插值变量使用$match 来选择“join”中的匹配项,但一般前提是“管道中的管道”,其中内内容可以通过父项的匹配项进行过滤。由于它们本身都是“管道”,因此我们可以分别$limit 每个结果。

    这将是运行并行查询的下一个最佳选择,如果$match 被允许并且能够在“子管道”处理中使用索引,实际上会更好。因此,正如引用的问题所要求的那样,它不使用“限制为$push”,它实际上提供了一些应该更好地工作的东西。


    原创内容

    您似乎偶然发现了前“N”个问题。在某种程度上,您的问题很容易解决,尽管没有您要求的确切限制:

    db.books.aggregate([
        { "$group": {
            "_id": {
                "addr": "$addr",
                "book": "$book"
            },
            "bookCount": { "$sum": 1 }
        }},
        { "$group": {
            "_id": "$_id.addr",
            "books": { 
                "$push": { 
                    "book": "$_id.book",
                    "count": "$bookCount"
                },
            },
            "count": { "$sum": "$bookCount" }
        }},
        { "$sort": { "count": -1 } },
        { "$limit": 2 }
    ])
    

    现在你会得到这样的结果:

    {
        "result" : [
                {
                        "_id" : "address1",
                        "books" : [
                                {
                                        "book" : "book4",
                                        "count" : 1
                                },
                                {
                                        "book" : "book5",
                                        "count" : 1
                                },
                                {
                                        "book" : "book1",
                                        "count" : 3
                                }
                        ],
                        "count" : 5
                },
                {
                        "_id" : "address2",
                        "books" : [
                                {
                                        "book" : "book5",
                                        "count" : 1
                                },
                                {
                                        "book" : "book1",
                                        "count" : 2
                                }
                        ],
                        "count" : 3
                }
        ],
        "ok" : 1
    }
    

    因此,这与您的要求不同,虽然我们确实获得了地址值的最佳结果,但基础“书籍”选择不仅限于所需数量的结果。

    事实证明,这很难做到,但可以做到,尽管复杂性会随着您需要匹配的项目数量的增加而增加。为了简单起见,我们最多可以保持 2 场比赛:

    db.books.aggregate([
        { "$group": {
            "_id": {
                "addr": "$addr",
                "book": "$book"
            },
            "bookCount": { "$sum": 1 }
        }},
        { "$group": {
            "_id": "$_id.addr",
            "books": { 
                "$push": { 
                    "book": "$_id.book",
                    "count": "$bookCount"
                },
            },
            "count": { "$sum": "$bookCount" }
        }},
        { "$sort": { "count": -1 } },
        { "$limit": 2 },
        { "$unwind": "$books" },
        { "$sort": { "count": 1, "books.count": -1 } },
        { "$group": {
            "_id": "$_id",
            "books": { "$push": "$books" },
            "count": { "$first": "$count" }
        }},
        { "$project": {
            "_id": {
                "_id": "$_id",
                "books": "$books",
                "count": "$count"
            },
            "newBooks": "$books"
        }},
        { "$unwind": "$newBooks" },
        { "$group": {
          "_id": "$_id",
          "num1": { "$first": "$newBooks" }
        }},
        { "$project": {
            "_id": "$_id",
            "newBooks": "$_id.books",
            "num1": 1
        }},
        { "$unwind": "$newBooks" },
        { "$project": {
            "_id": "$_id",
            "num1": 1,
            "newBooks": 1,
            "seen": { "$eq": [
                "$num1",
                "$newBooks"
            ]}
        }},
        { "$match": { "seen": false } },
        { "$group":{
            "_id": "$_id._id",
            "num1": { "$first": "$num1" },
            "num2": { "$first": "$newBooks" },
            "count": { "$first": "$_id.count" }
        }},
        { "$project": {
            "num1": 1,
            "num2": 1,
            "count": 1,
            "type": { "$cond": [ 1, [true,false],0 ] }
        }},
        { "$unwind": "$type" },
        { "$project": {
            "books": { "$cond": [
                "$type",
                "$num1",
                "$num2"
            ]},
            "count": 1
        }},
        { "$group": {
            "_id": "$_id",
            "count": { "$first": "$count" },
            "books": { "$push": "$books" }
        }},
        { "$sort": { "count": -1 } }
    ])
    

    所以这实际上会给你前两个“地址”条目中的前两个“书”。

    但是为了我的钱,请保留第一种形式,然后简单地“切片”返回的数组元素以获取前“N”个元素。


    演示代码

    演示代码适用于 v8.x 和 v10.x 版本的当前 LTS 版本的 NodeJS。这主要是针对async/await 语法,但在一般流程中并没有真正具有任何此类限制的东西,并且几乎不改变普通的承诺甚至回到普通的回调实现。

    index.js

    const { MongoClient } = require('mongodb');
    const fs = require('mz/fs');
    
    const uri = 'mongodb://localhost:27017';
    
    const log = data => console.log(JSON.stringify(data, undefined, 2));
    
    (async function() {
    
      try {
        const client = await MongoClient.connect(uri);
    
        const db = client.db('bookDemo');
        const books = db.collection('books');
    
        let { version } = await db.command({ buildInfo: 1 });
        version = parseFloat(version.match(new RegExp(/(?:(?!-).)*/))[0]);
    
        // Clear and load books
        await books.deleteMany({});
    
        await books.insertMany(
          (await fs.readFile('books.json'))
            .toString()
            .replace(/\n$/,"")
            .split("\n")
            .map(JSON.parse)
        );
    
        if ( version >= 3.6 ) {
    
        // Non-correlated pipeline with limits
          let result = await books.aggregate([
            { "$group": {
              "_id": "$addr",
              "count": { "$sum": 1 }
            }},
            { "$sort": { "count": -1 } },
            { "$limit": 2 },
            { "$lookup": {
              "from": "books",
              "as": "books",
              "let": { "addr": "$_id" },
              "pipeline": [
                { "$match": {
                  "$expr": { "$eq": [ "$addr", "$$addr" ] }
                }},
                { "$group": {
                  "_id": "$book",
                  "count": { "$sum": 1 },
                }},
                { "$sort": { "count": -1 } },
                { "$limit": 2 }
              ]
            }}
          ]).toArray();
    
          log({ result });
        }
    
        // Serial result procesing with parallel fetch
    
        // First get top addr items
        let topaddr = await books.aggregate([
          { "$group": {
            "_id": "$addr",
            "count": { "$sum": 1 }
          }},
          { "$sort": { "count": -1 } },
          { "$limit": 2 }
        ]).toArray();
    
        // Run parallel top books for each addr
        let topbooks = await Promise.all(
          topaddr.map(({ _id: addr }) =>
            books.aggregate([
              { "$match": { addr } },
              { "$group": {
                "_id": "$book",
                "count": { "$sum": 1 }
              }},
              { "$sort": { "count": -1 } },
              { "$limit": 2 }
            ]).toArray()
          )
        );
    
        // Merge output
        topaddr = topaddr.map((d,i) => ({ ...d, books: topbooks[i] }));
        log({ topaddr });
    
        client.close();
    
      } catch(e) {
        console.error(e)
      } finally {
        process.exit()
      }
    
    })()
    

    books.json

    { "addr": "address1",  "book": "book1"  }
    { "addr": "address2",  "book": "book1"  }
    { "addr": "address1",  "book": "book5"  }
    { "addr": "address3",  "book": "book9"  }
    { "addr": "address2",  "book": "book5"  }
    { "addr": "address2",  "book": "book1"  }
    { "addr": "address1",  "book": "book1"  }
    { "addr": "address15", "book": "book1"  }
    { "addr": "address9",  "book": "book99" }
    { "addr": "address90", "book": "book33" }
    { "addr": "address4",  "book": "book3"  }
    { "addr": "address5",  "book": "book1"  }
    { "addr": "address77", "book": "book11" }
    { "addr": "address1",  "book": "book1"  }
    

    【讨论】:

    • 显然在 MongoDB 5.0 中,$lookup 中的子管道可以在某些条件下使用索引进行匹配($eq/$lt/$lte/$gt/$gte 运算符;没有多键索引;可以t 与数组或未定义比较;不超过一个字段路径)。
    【解决方案3】:

    以下查询将提供与所需响应中给出的完全相同的结果:

    db.books.aggregate([
        {
            $group: {
                _id: { addresses: "$addr", books: "$book" },
                num: { $sum :1 }
            }
        },
        {
            $group: {
                _id: "$_id.addresses",
                bookCounts: { $push: { bookName: "$_id.books",count: "$num" } }
            }
        },
        {
            $project: {
                _id: 1,
                bookCounts:1,
                "totalBookAtAddress": {
                    "$sum": "$bookCounts.count"
                }
            }
        }
    
    ]) 
    

    响应将如下所示:

    /* 1 */
    {
        "_id" : "address4",
        "bookCounts" : [
            {
                "bookName" : "book3",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    },
    
    /* 2 */
    {
        "_id" : "address90",
        "bookCounts" : [
            {
                "bookName" : "book33",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    },
    
    /* 3 */
    {
        "_id" : "address15",
        "bookCounts" : [
            {
                "bookName" : "book1",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    },
    
    /* 4 */
    {
        "_id" : "address3",
        "bookCounts" : [
            {
                "bookName" : "book9",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    },
    
    /* 5 */
    {
        "_id" : "address5",
        "bookCounts" : [
            {
                "bookName" : "book1",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    },
    
    /* 6 */
    {
        "_id" : "address1",
        "bookCounts" : [
            {
                "bookName" : "book1",
                "count" : 3
            },
            {
                "bookName" : "book5",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 4
    },
    
    /* 7 */
    {
        "_id" : "address2",
        "bookCounts" : [
            {
                "bookName" : "book1",
                "count" : 2
            },
            {
                "bookName" : "book5",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 3
    },
    
    /* 8 */
    {
        "_id" : "address77",
        "bookCounts" : [
            {
                "bookName" : "book11",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    },
    
    /* 9 */
    {
        "_id" : "address9",
        "bookCounts" : [
            {
                "bookName" : "book99",
                "count" : 1
            }
        ],
        "totalBookAtAddress" : 1
    }
    

    【讨论】:

    • 是否可以对每个组的“booksCounts”列表中的元素进行排序?这个答案确实帮助我汇总了一些数据,但是我有日期而不是 bookName,我想按日期对每个组中的数据进行排序
    猜你喜欢
    • 2018-11-28
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多