如何采集级联数据(比如最新省市县)呢?

原创
2019/03/15 12:20
阅读数 526

概述

通常抓取级联数数据情况不多,但要是真需要时,确多了一些麻烦,比如抓取商品分类级别信息等。本内容将讲述如何采集无限级联联数据,并以GoldData来抓取2019年最新的省市县三级为示例。

创建数据集

在数据集管理里,添加数据集area。如下图所示:

dataset

数据集相当于数据库中的表,只是字段是灵活的,可以随着需要而添加和变更。

创建规则

在规则管理里,添加规则arearule,并将地址http://xzqh.mca.gov.cn/map 填为抓取入口地址。

我们通过分析可知,我们可从http://xzqh.mca.gov.cn/map 获取省级数据,然后通过级级数据请求http://xzqh.mca.gov.cn/selectJson 带有省级名称来获得市级数据,最后再通过市级数据分别请求http://xzqh.mca.gov.cn/selectJson 带有市级数据来获得县级数据。

并且发现请求 http://xzqh.mca.gov.cn/selectJson 是需要发POST请求的,因此我们需要将url加个前缀fake:,然后用规则里用JavaScript去请求URL。

我们在此编写的area数据集,有以下字段:

名称 说明
sn 取区域编码作为记录唯一字段
name 名称
code 取区域编码
abbr 省名简写
parent_code 取父区域编码

因此编写规则如下:

rule

(注:具体规则内容请见文章最后)

然后编写完成后,我们就可以启动抓取器进行抓取。

查看数据

打开数据管理,选择area数据集查看如下图所示:

area

导出数据

回到数据管理,选择条件,选择需要导出的字段,进行导出。像这里数据比较多,GoldData将会以打包excel文件并压缩为zip文件下载。解压到本地,然后打开excel就可以看到抓取的数据,如下图所示:

area_export

结尾

通过这节内容,可以了解了如何通过GoldData 抓取级联数据。但是下一个问题是如何将数据导入自关联列表呢,且看下一往篇将会讲述如何将级联数据融合到数据库自关联表当中。

咐录:

(抓取规则)

[
  {
    __sample: http://xzqh.mca.gov.cn/map
    match0: http\:\/\/xzqh\.mca\.gov\.cn\/map
    fields0:
    {
      __model: true
      __node: js
      __js:
        '''
        var exp11=/json\s=\s(.+)\s+\$\(doc/
        var ret=exp11.exec(html)
        var ss=eval(ret[1])
        for(var i=0;i<ss.length;i++){
           var ele=ss[i]
           var exp12=/\((.+)\)/.exec(ele.shengji)
        var exp13=/(.+)\(/.exec(ele.shengji)
           out.add({sn:ele.quHuaDaiMa,name:exp13[1],code:ele.quHuaDaiMa,abbr:exp12[1],
         parent_code:null,

        })
        }

        '''
      name:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      code:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      abbr:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      sn:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      __dataset: area
      parent_code:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
    }
    fields1:
    {
      __node: js
      __js:
        '''
        var exp11=/json\s=\s(.+)\s+\$\(doc/
        var ret=exp11.exec(html)
        var ss=eval(ret[1])
        for(var i=0;i<ss.length;i++){
           var ele=ss[i]

          var url='fake:http://xzqh.mca.gov.cn/selectJson?shengji='+ele.shengji+"&code="+ele.quHuaDaiMa
           //var $ajax(url,[__method:'POST',data:'shengji='+ele.shengji]).content
           
           out.add({href:url})
        }

        '''
      href:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
    }
  }
  {
    __sample: fake:http://xzqh.mca.gov.cn/selectJson?shengji=北京市(京)&code=110000
    match0: fake\:http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=.+\&code=\d+
    fields0:
    {
      __model: true
      __dataset: area
      sn:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      __node: js
      __js:
        '''
        var urlRet= /http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=(.+)\&code=(\d+)/.exec(baseUri);
        var shengji=urlRet[1]
        var sjCode=urlRet[2]

        var url='http://xzqh.mca.gov.cn/selectJson'
        var  content=$ajax(url,{__method:'POST',data:'shengji='+shengji }).content

        var arr=eval(content);
        for(var i=0;i<arr.length;i++){
           var dj=arr[i];
           
            var area={
                 name:dj.diji,
                 code:dj.quHuaDaiMa,
                 parent_code:sjCode,
                 sn:dj.quHuaDaiMa,
            }
           out.add(area);
        }

        '''
      name:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      code:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      abbr:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      sn:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      parent_code:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
    }
    fields1:
    {
      __node: js
      __js:
        '''
        var urlRet= /http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=(.+)\&code=(\d+)/.exec(baseUri);
        var shengji=urlRet[1]
        var sjCode=urlRet[2]

        var url='http://xzqh.mca.gov.cn/selectJson'
        var  content=$ajax(url,{__method:'POST',data:'shengji='+shengji }).content

        var arr=eval(content);
        for(var i=0;i<arr.length;i++){
           var dj=arr[i];

          var url='fake:http://xzqh.mca.gov.cn/selectJson?shengji='+shengji+"&sjcode="+sjCode+"&diji="+dj.diji+"&djcode="+dj.quHuaDaiMa   
           out.add({href:url})
        }

        '''
      href:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
    }
  }
  {
    __sample: fake:href :fake:http://xzqh.mca.gov.cn/selectJson?shengji=北京市(京)&sjcode=110000&diji=北京市&djcode=110000
    match0: fake\:http\:\/\/xzqh\.mca\.gov\.cn/selectJson\?shengji=([^&]+)\&sjcode=([^&]+)\&diji=([^&]+)\&djcode=([^&]+)
    fields0:
    {
      __model: true
      __dataset: area
      __node: js
      sn:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      __js:
        '''
        var urlRet=/fake\:http\:\/\/xzqh\.mca\.gov\.cn\/selectJson\?shengji=([^\&]+)\&sjcode=([^\&]+)\&diji=([^\&]+)\&djcode=([^\&]+)/.exec(baseUri)

        var shengji=urlRet[1]
        var sjCode=urlRet[2]
        var dj=urlRet[3]
        var djCode=urlRet[4]


        var url='http://xzqh.mca.gov.cn/selectJson'
        var  content=$ajax(url,{__method:'POST',data:'shengji='+shengji+'&diji='+dj }).content

        var arr=eval(content);
        for(var i=0;i<arr.length;i++){
           var dj=arr[i];
           
            var area={
                 name:dj.xianji,
                 code:dj.quHuaDaiMa,
                 parent_code:djCode,
                 sn:dj.quHuaDaiMa,
            }
           out.add(area);
        }

        '''
      name:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      code:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      abbr:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      sn:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
      parent_code:
      {
        expr: ""
        attr: ""
        js: ""
        __label: ""
        __showOnList: false
        __type: ""
        down: "0"
        accessPathJs: ""
        uploadConf: ""
      }
    }
  }
]
展开阅读全文
加载中

作者的其它热门文章

打赏
0
0 收藏
分享
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部