[Kylin] 3.Curl方式调用api进行增量build、PUT、GET

参考自官方英文文档:http://kylin.apache.org/docs/howto/howto_use_restapi.html

1.curl方式login创建cookie文件

注:其中QURNSU46S1lMSU4=’的获取见文章 [Kylin] 1.Rest API和java获取model_Desc的例子

curl -c /home/ap/job_lw/cookiefile.txt -X POST -H "Authorization: Basic QURNSU46S1lMSU4='" -H 'Content-Type: application/json' http://128.196.116.75:7070/kylin/api/user/authentication

返回结果:

{“userDetails”:{“password”:null,”username”:”ADMIN”,”authorities”:[{“authority”:”ROLE_ADMIN”},{“authority”:”ROLE_ANALYST”},{“authority”:”ROLE_MODELER”}],”accountNonExpired”:true,”accountNonLocked”:true,”credentialsNonExpired”:true,”enabled”:true}}

另外,cat cookiefile.txt结果如下:其中最后一行末尾字段即为cookie
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_128.196.116.75 FALSE /kylin/ FALSE 0 JSESSIONID 80FB9C5F88187BE0630C2ADFB12BE5C8

 

2.GET示例:List cubes 列出所有的cubes。注意这里的是projectName而不是project。

$ curl -b /home/ap/job_lw/cookiefile.txt -X GET -H 'Content-Type: application/json' -d '{"offset":0 "limit":10, "projectName":"learn_kylin" }' http://128.196.116.75:7070/kylin/api/cubes

参数说明:-b表示获取cookiefile以通过认证
Request Parameters
offset – required int Offset used by pagination
limit – required int Cubes per page.
cubeName – optional string Keyword for cube names. To find cubes whose name contains this keyword.
projectName – optional string Project name.
返回结果:(略)

可以用google的json解析来获取自己想要的信息

 

3.POST示例:Query cube data

$ curl -b /home/ap/job_lw/cookiefile.txt -X POST -H 'Content-Type: application/json' -d '{     "sql":"select * from kylin_sales",   "offset":0,   "limit":5,   "acceptPartial":false,   "project":"learn_kylin" }' http://128.196.116.75:7070/kylin/api/query

其中: offset为sql中相对记录首行的偏移量,limit为限制记录条数
返回结果:(略)

 

4.PUT示例:rebuild

curl -b /home/ap/job_lw/cookiefile.txt -X PUT -H 'Content-Type: application/json' -d '{"startTime":'1323376000000', "endTime":'3153900000000', "buildType":"BUILD"}' http://128.196.116.75:7070/kylin/api/cubes/kylin_sales_cube/rebuild

其中的startTime、endTime都需要先转成毫秒级timestamp

返回结果:

{
“uuid”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596”,
“last_modified”: 1494493965912,
“version”: “1.6.0”,
“name”: “kylin_sales_cube – 20111208202640_20691210112000 – BUILD – GMT+08:00 2017-05-11 17:12:45”,
“type”: “BUILD”,
“duration”: 0,
“related_cube”: “kylin_sales_cube”,
“related_segment”: “bbae2d20-8998-4f1b-96b5-40cce5dd8b29”,
“exec_start_time”: 0,
“exec_end_time”: 0,
“mr_waiting”: 0,
“steps”: [
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-00”,
“name”: “Create Intermediate Flat Hive Table”,
“sequence_id”: 0,
“exec_cmd”: null,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-01”,
“name”: “Redistribute Flat Hive Table”,
“sequence_id”: 1,
“exec_cmd”: null,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-02”,
“name”: “Extract Fact Table Distinct Columns”,
“sequence_id”: 2,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/fact_distinct_columns -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -statisticsenabled true -statisticsoutput /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/statistics -statisticssamplingpercent 100 -jobname Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-03”,
“name”: “Build Dimension Dictionary”,
“sequence_id”: 3,
“exec_cmd”: ” -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/fact_distinct_columns”,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-04”,
“name”: “Save Cuboid Statistics”,
“sequence_id”: 4,
“exec_cmd”: null,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-05”,
“name”: “Create HTable”,
“sequence_id”: 5,
“exec_cmd”: ” -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -partitions hdfs://hacluster/kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/rowkey_stats/part-r-00000 -statisticsenabled true”,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-06”,
“name”: “Build Base Cuboid Data”,
“sequence_id”: 6,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input FLAT_TABLE -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/base_cuboid -jobname Kylin_Base_Cuboid_Builder_kylin_sales_cube -level 0 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-07”,
“name”: “Build N-Dimension Cuboid Data : 8-Dimension”,
“sequence_id”: 7,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/base_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/8d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 1 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-08”,
“name”: “Build N-Dimension Cuboid Data : 7-Dimension”,
“sequence_id”: 8,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/8d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/7d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 2 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-09”,
“name”: “Build N-Dimension Cuboid Data : 6-Dimension”,
“sequence_id”: 9,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/7d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/6d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 3 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-10”,
“name”: “Build N-Dimension Cuboid Data : 5-Dimension”,
“sequence_id”: 10,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/6d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/5d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 4 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-11”,
“name”: “Build N-Dimension Cuboid Data : 4-Dimension”,
“sequence_id”: 11,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/5d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/4d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 5 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-12”,
“name”: “Build N-Dimension Cuboid Data : 3-Dimension”,
“sequence_id”: 12,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/4d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/3d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 6 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-13”,
“name”: “Build N-Dimension Cuboid Data : 2-Dimension”,
“sequence_id”: 13,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/3d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/2d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 7 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-14”,
“name”: “Build N-Dimension Cuboid Data : 1-Dimension”,
“sequence_id”: 14,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/2d_cuboid -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/1d_cuboid -jobname Kylin_ND-Cuboid_Builder_kylin_sales_cube_Step -level 8 -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-15”,
“name”: “Build Cube”,
“sequence_id”: 15,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf_inmem.xml -cubename kylin_sales_cube -segmentid bbae2d20-8998-4f1b-96b5-40cce5dd8b29 -output /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/ -jobname Kylin_Cube_Builder_kylin_sales_cube -cubingJobId bbcde94c-fcec-4c70-ac01-4cf6fe5ce596″,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-16”,
“name”: “Convert Cuboid Data to HFile”,
“sequence_id”: 16,
“exec_cmd”: ” -conf /home/ap/dip/appjob/shelljob/dzyh/apache-kylin-1.6.0-bin/conf/kylin_job_conf.xml -cubename kylin_sales_cube -partitions hdfs://hacluster/kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/rowkey_stats/part-r-00000_hfile -input /kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/cuboid/* -output hdfs://hacluster/kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/hfile -htablename KYLIN_3QENF1CWZG -jobname Kylin_HFile_Generator_kylin_sales_cube_Step”,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-17”,
“name”: “Load HFile to HBase Table”,
“sequence_id”: 17,
“exec_cmd”: ” -input hdfs://hacluster/kylin/kylin_metadata/kylin-bbcde94c-fcec-4c70-ac01-4cf6fe5ce596/kylin_sales_cube/hfile -htablename KYLIN_3QENF1CWZG -cubename kylin_sales_cube”,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-18”,
“name”: “Update Cube Info”,
“sequence_id”: 18,
“exec_cmd”: null,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
},
{
“interruptCmd”: null,
“id”: “bbcde94c-fcec-4c70-ac01-4cf6fe5ce596-19”,
“name”: “Hive Cleanup”,
“sequence_id”: 19,
“exec_cmd”: null,
“interrupt_cmd”: null,
“exec_start_time”: 0,
“exec_end_time”: 0,
“exec_wait_time”: 0,
“step_status”: “PENDING”,
“cmd_type”: “SHELL_CMD_HADOOP”,
“info”: {},
“run_async”: false
}
],
“submitter”: “ADMIN”,
“job_status”: “PENDING”,
“progress”: 0
}

然后打开页面选择monitor可以看到已经开始build,对应的name就是”name”:”kylin_sales_cube – 20111208202640_20691210112000 – BUILD – GMT+08:00 2017-05-11 17:12:45″

分类上一篇:     分类下一篇:

Leave a Reply