将复杂的 GROUP BY 和 INNER JOIN SQL 语句转换为 Django ORM 级别

Question

我正在尝试将下面提到的 SQL 查询转换为 Django ORM 层查询，但我无法获得 SQL 语句提供的完美输出。 模型

class YearlyTable(models.Model):

   class Meta:
       db_table = 'yearlytable'
       managed = True

   user_id = models.IntegerField(db_index=True)
   rotations = models.IntegerField()
   calories = models.FloatField()
   distance = models.FloatField()
   duration = models.IntegerField(default=0)
   year = models.IntegerField()
   created = models.DateTimeField(auto_now_add=True)
   modified = models.DateTimeField(auto_now=True)

---

class User(AbstractBaseUser):

    class Meta:
        db_table = 'users'
        managed = True

    email = models.EmailField(max_length=255, unique=True)
    first_name = models.CharField(max_length=255, blank=True, null=True)
    city = models.CharField(max_length=200, blank=True, null=True)
    state = models.CharField(max_length=200, blank=True, null=True)
    postal_code = models.IntegerField(blank=True, null=True)
    country = models.CharField(max_length=200, blank=True, null=True)

---

SELECT 
users.state,
sum(yearlytable.rotations) as sum_rotations,
sum(yearlytable.calories) as sum_calories,
sum(yearlytable.distance) as sum_distance
    FROM yearlytable
    INNER JOIN users on (yearlytable.user_id = users.id)
    WHERE yearlytable.user_id in(SELECT id FROM users WHERE country LIKE 'United States%' and  NOT ("email" LIKE '%yopmail.com%'))
    GROUP BY users.state

然后我尝试使用 RAW Django 查询示例执行上述查询：

User.objects.raw('select users.state,sum(yearlytable.rotations) as sum_rotations,sum(yearlytable.calories) as sum_calories,sum(yearlytable.distance) as sum_distance from yearlytable inner join users on (yearlytable.user_id = users.id) where yearlytable.user_id in(select id from users where country like \'United States%\' and  NOT ("email" LIKE \'%yopmail.com%\')) group by users.state;')

但这也没有用。现在我不想为此使用 CURSOR，因为我害怕 SQL 注入问题。所以光标不在桌面上。

for u in User.objects.raw('select users.state,sum(yearlytable.rotations) as sum_rotations,sum(yearlytable.calories) as sum_calories,sum(yearlytable.distance) as sum_distance from yearlytable inner join users on (yearlytable.user_id = users.id) where yearlytable.user_id in(select id from users where country like \'United States%\' and  NOT ("email" LIKE \'%yopmail.com%\')) group by users.state;'):
            print u

下面是堆栈跟踪：

    Traceback:
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  57.         return view_func(*args, **kwargs)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
  407.             response = self.handle_exception(exc)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
  404.             response = handler(request, *args, **kwargs)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/rest_framework/decorators.py" in handler
  51.             return func(*args, **kwargs)
File "/home/akki/rest_api/widget/views.py" in heat_map
  18.         for u in User.objects.raw('select users.state,sum(yearlytable.rotations) as sum_rotations,sum(yearlytable.calories) as sum_calories,sum(yearlytable.distance) as sum_distance from yearlytable inner join users on (yearlytable.user_id = users.id) where yearlytable.user_id in(select id from users where country like \'United States%\' and  NOT ("email" LIKE \'%yopmail.com%\')) group by users.state;'):
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/db/models/query.py" in __iter__
  1535.         query = iter(self.query)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py" in __iter__
  76.         self._execute_query()
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py" in _execute_query
  90.         self.cursor.execute(self.sql, self.params)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
  81.             return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/akki/rest_api/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
  65.                 return self.cursor.execute(sql, params)

我尝试的 Django ORM 是：

YearlyTable.objects.annotate(r=Sum('rotations'))

如果能把这个 sql 查询转换成 django orm 级别就好了。

Answer 1

这似乎可以使用 Aggregation Framework 和以下 ORM 查询来完成：

1) 我们过滤 User 以找到与最里面的 SELECT 语句匹配的那些。这将返回User.id 的列表。

2) values() 首先在YearlyTable 上使用，然后在User.state 上执行GROUP BY。

3) distinct() 用于确保我们只考虑每个可能的User.state 一次。

4) annotate() 用于执行您想要的值的Sum。

5) 最后我们再次调用 values() 来制作包含您在顶级SELECT 查询中请求的信息的字典。

from django.db.models import Sum

YearlyTable.objects.filter(
    user_id__in=User.objects.filter(
        country__startswith='United States'
    ).exclude(
        email__contains='yopmail.com'
    ).values_list('id', flat=True)
).values('user__state').distinct().annotate(
    sum_rotations=Sum('rotations'),
    sum_calories=Sum('calories'),
    sum_distance=Sum('distance')
).values('user__state', 'sum_rotations', 'sum_calories', 'sum_distance')

Answer 2

假设：

1. 使用 django ORM 而不求助于原始 SQL
2. 以惯用方式设计 django 模型，这意味着相关表应使用模型 ForeignKey、OneonOne 或 ManytoMany 属性。
3. YearlyTable 假定与用户具有一对一的关系。

在 models.py 中：

from django.db import models
from django.contrib.auth.models import AbstractBaseUser

class User(AbstractBaseUser):
    email = models.EmailField(max_length=255, unique=True)
    first_name = models.CharField(max_length=255, blank=True, null=True)
    city = models.CharField(max_length=200, blank=True, null=True)
    state = models.CharField(max_length=200, blank=True, null=True)
    postal_code = models.IntegerField(blank=True, null=True)
    country = models.CharField(max_length=200, blank=True, null=True)


    def __unicode__(self):
        return self.email


class YearlyTable(models.Model):
    user =  models.OneToOneField('User', unique=True)
    rotations = models.IntegerField()
    calories = models.FloatField()
    distance = models.FloatField()
    duration = models.IntegerField(default=0)
    year = models.IntegerField()
    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return str(self.user)

我用以下示例数据填充了表格：

    u = User(email='a@w.com', first_name='ab', city='New York', state='New York', postal_code='12345', country='United States')
    y = YearlyTable(user=u, rotations=10, calories=10.8, distance=12.5, duration=20, year=2011)

    u = User(email='b@w.com', first_name='ac', city='Buffalo', state='New York', postal_code='67891', country='United States') 
    y = YearlyTable(user=u, rotations=8, calories=11.8, distance=11.5, duration=30, year=2012)

    u = User(email='c@w.com', first_name='ad', city='Rochester', state='New York', postal_code='13579', country='United States')
    y = YearlyTable(user=u, rotations=20, calories=15.8, distance=13.5, duration=40, year=2013)

    u = User(email='d@w.com', first_name='ae', city='Pittsburgh', state='Pennsylvania', postal_code='98765', country='United States')
    y = YearlyTable(user=u, rotations=30, calories=10.2, distance=12.5, duration=40, year=2012)

    u = User(email='e@w.com', first_name='af', city='Los Angeles', state='California', postal_code='97531', country='United States')
    y = YearlyTable(user=u, rotations=10, calories=14.8, distance=13.5, duration=10, year=2010)

检查物理表并直接查询

    psql -d 

    # select * from testapp_user;
     id | password | last_login |  email  | first_name |    city     |    state     | postal_code |    country
    ----+----------+------------+---------+------------+-------------+--------------+-------------+---------------
      1 |          |            | a@w.com | ab         | New York    | New York     |       12345 | United States
      2 |          |            | b@w.com | ac         | Buffalo     | New York     |       67891 | United States
      3 |          |            | c@w.com | ad         | Rochester   | New York     |       13579 | United States
      4 |          |            | d@w.com | ae         | Pittsburgh  | Pennsylvania |       98765 | United States
      5 |          |            | e@w.com | af         | Los Angeles | California   |       97531 | United States
    (5 rows)

    # select * from testapp_yearlytable;
     id | rotations | calories | distance | duration | year |            created            |           modified            | user_id
    ----+-----------+----------+----------+----------+------+-------------------------------+-------------------------------+---------
      1 |        10 |     10.8 |     12.5 |       20 | 2011 | 2016-05-17 16:23:46.39941+00  | 2016-05-17 16:23:46.399445+00 |       1
      3 |         8 |     11.8 |     11.5 |       30 | 2012 | 2016-05-17 16:24:26.264569+00 | 2016-05-17 16:24:26.264606+00 |       2
      4 |        20 |     15.8 |     13.5 |       40 | 2013 | 2016-05-17 16:24:51.200739+00 | 2016-05-17 16:24:51.200785+00 |       3
      5 |        30 |     10.2 |     12.5 |       40 | 2012 | 2016-05-17 16:25:08.187799+00 | 2016-05-17 16:25:08.187852+00 |       4
      6 |        10 |     14.8 |     13.5 |       10 | 2010 | 2016-05-17 16:25:24.846284+00 | 2016-05-17 16:25:24.846324+00 |       5
    (5 rows)


    # SELECT
    testapp_user.state,
    sum(testapp_yearlytable.rotations) as sum_rotations,
    sum(testapp_yearlytable.calories) as sum_calories,
    sum(testapp_yearlytable.distance) as sum_distance
    FROM testapp_yearlytable
    INNER JOIN  testapp_user on (testapp_yearlytable.user_id = testapp_user.id)
    WHERE testapp_yearlytable.user_id in
    (SELECT id FROM testapp_user
    WHERE country LIKE 'United States%' and
    NOT ("email" LIKE '%a@w.com%'))
    GROUP BY testapp_user.state;

        state     | sum_rotations | sum_calories | sum_distance
    --------------+---------------+--------------+--------------
     New York     |            28 |         27.6 |           25
     Pennsylvania |            30 |         10.2 |         12.5
     California   |            10 |         14.8 |         13.5

在 python shell 中运行

    > python manage.py shell
    Python 2.7.6 (default, Jun 22 2015, 18:00:18)
    [GCC 4.8.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    (InteractiveConsole)
    >>> from testapp.models import User, YearlyTable
    >>> from django.db.models import Q, Sum
    >>> User.objects.filter(~Q(email__icontains='a@w.com'), country__startswith='United States') \
    ... .values('state') \
    ... .annotate(sum_rotations = Sum('yearlytable__rotations'), \
    ... sum_calories = Sum('yearlytable__calories'), \
    ... sum_distance = Sum('yearlytable__distance'))
    [{'sum_rotations': 28, 'state': u'New York', 'sum_calories': 27.6, 'sum_distance': 25.0}, {'sum_rotations': 30, 'state': u'Pennsylvania', 'sum_calories': 10.2, 'sum_distance': 12.5}, {'sum_rotations': 10, 'state': u'California', 'sum_calories': 14.8, 'sum_distance': 13.5}]