Kylin作为一个极其优秀的MOLAP,提供了完整的Cube创建、更新流程。同时提供了Sql查询。功能上看没有问题,但是在提供查询服务的时候还是有些不友好。
sql查询需要常常需要关联Hive表,Cube的作用是对查询做优化,但是用户需要知道hive表结果——为什么不提供接口让用户直接对Cube模型查询呢?
比如,我们用kylin建立了一个Sales Cube,关于公司销售数据统计。维度包括:年/季度/天,以及部门site;统计值measure包括,销售金额,销量,销售员数量等。 这个Cube需要通过两个hive表join得到基础数据。 我们不想让用户关心底层的hive表结构,而是希望他们能够更直接地对Cube的数据结构查询。
多维表达式是OLAP的查询语言,查询对象是多维数据结构Cube,解析器(例如Mondrian)会吧MDX转换成SQL来查询关系数据库(可能是多条查询)。
从API调用者的角度提供一套OLAP操作的API可能更友好,例如我们的Sales Cube模型建立好之后,通过drilldown/rollup, slice/dice操作的组合就能得到最终的统计结果。这比用MDX或者Sql都更方便。Cubes能做到(https://pythonhosted.org/cubes/index.html) 某种意义上Cubes是多维模型的ORM。
Cubes支持多种数据源,只要有SqlAlchemy dialect就可以。kylinpy是kylin的sqlalchemy包。但是跟cubes对接时需要稍作修改:
diff --git a/kylinpy/kylindb.py b/kylinpy/kylindb.py index bd0562e..6d6f7c7 100644 --- a/kylinpy/kylindb.py +++ b/kylinpy/kylindb.py @@ -39,6 +39,10 @@ class Cursor(object): ] for c in self._column_metas) def execute(self, query, *params, **kwargs): + for param in params: + for k,v in param.items(): + query = query.replace('%('+k+')s', str(v)) +根据Kylin的模型建立对应的Cubes模型文件:
{ "dimensions": [ { "name":"year", "levels": [ { "name":"YEAR", "label":"YEAR", "attributes": ["YEAR_BEG_DT"] }, { "name":"QUATER", "label":"QUATER", "attributes": ["QTR_BEG_DT"] }, { "name":"PART_DT", "label":"PART_DT", "attributes": ["PART_DT"] } ] }, { "name":"site", "levels": [ { "name": "LSTG_SITE_ID", "label": "LSTG_SITE_ID", "attributes": ["LSTG_SITE_ID"] } ] } ], "cubes": [ { "name": "KYLIN_SALES", "dimensions": ["year", "site"], "joins": [ {"master":"PART_DT", "detail":"KYLIN_CAL_DT.CAL_DT","method": "match"} ], "measures": [ {"name": "PRICE", "label": "PRICE"}, {"name": "ITEM_COUNT", "label": "ITEM_COUNT"}, {"name": "SELLER_ID", "label": "SELLER_ID", "aggregates":["count_distinct"]} ], "aggregates": [ { "name": "TOTAL_SOLD", "function": "sum", "measure": "PRICE" }, { "name": "TOTAL_ITEMS", "function": "sum", "measure": "ITEM_COUNT" }, { "name": "_COUNT_", "function": "count" }, { "name": "DISTINC_SALLERS", "function": "count_distinct", "measure": "SELLER_ID" } ], "mappings": { "year.PART_DT": "PART_DT", "year.YEAR_BEG_DT": "KYLIN_CAL_DT.YEAR_BEG_DT", "year.QTR_BEG_DT": "KYLIN_CAL_DT.QTR_BEG_DT", "site.LSTG_SITE_ID": "LSTG_SITE_ID" }, "info": { "min_date": "2010-01-01", "max_date": "2010-12-31" } } ] }slicer.ini 文件
[workspace] log_level: debug [server] host: localhost port: 5000 reload: yes prettyprint: yes [store] type: sql url: kylin://ADMIN:KYLIN@localhost:7070/Tutorial?version=v1 schema=DEFAULT dimension_schema=DEFAULT [models] main: model.json启动 slicer serve slicer.ini
http查询示例:
-- 按季度下钻所有统计结果 http://localhost:5000/cube/KYLIN_SALES/aggregate?drilldown=year:QUATER -- 按年下钻所有统计结果 http://localhost:5000/cube/KYLIN_SALES/aggregate?drilldown=year:YEAR -- 按年下钻site0的所有统计结果 http://localhost:5000/cube/KYLIN_SALES/aggregate?drilldown=year:YEAR&cut=site:0 -- 对0-4这几个销售点,统计2012年每个季度的结果 http://localhost:5000/cube/KYLIN_SALES/aggregate?drilldown=year.QUATER|site&cut=year.YEAR_BEG_DT:date'2012-01-01'|site:0-4
转载于:https://www.cnblogs.com/luweiseu/p/10007014.html