案例|边缘智能:YiFUSION AI 推理应用之 EdgeX 集成 OpenVINO™

02/24 17:11
阅读数 74

EdgeX 与 OpenVINO™ 结合起来实现边缘智能案例,为 AI 应用场景提供一些解决方案。灵活切换 AI 模型(相对而言),动态处理推理请求。

  • EdgeXFoundry(V3.1.0)
  • OpenVINO™ 和 OVMS(OpenVINO™ Model Server)
  • Resnet 50 模型(Intel Model Zoo)
  • Device Services for OVMS
    • device-openvino-resnet(亿琪软件版权所有)
    • 其他模型微服务
  • Intel® CPU(内置 iGPU)
    • Intel® N100
      • CPU: Intel® N100, 4 Cores @ 3.40 GHz
      • iGPU: Intel® UHD Graphics 750MHz
    • Intel® i7-1165G7
      • CPU: 11th Gen Intel® Core®, 8 Cores @ 2.80 GHz[Max: 4.70 GHz]
      • iGPU: Intel® Iris® Xe Graphics 1.3GHz

以上信息除了 device-openvino-resnet 以外,都可以在网络上查询到,请自行查阅相关详细内容。

设计架构图

系统架构

arc

本系统设计硬件设计采用 Intel® CPU(iGPU) + GPU[可选] ,CPU 必须为 Intel® 架构,独立 GPU 可根据实际需要灵活扩展。

本系统全部采用 Docker 微服务运行,描述如下:

  • 图例中,① 作为 AI 推理服务器,运行 OpenVINO™ Model Server 容器;
  • 图例中,② 作为流媒体服务器,负责流媒体的编解码,实时查看视频流,运行亿琪软件产品 YiMEDIA,同样也采用容器运行;
  • 图例中,③ 作为边缘计算,运行 EdgeXFoundry 容器,负责整个系统的协调和业务驱动;

业务场景描述

客户需求

有边缘智能需求的大部分客户已经对 AI 推理和边缘计算有一定的了解,都希望可以将 边缘计算和 AI 结合在一起,实现硬件资源的充分利用,完成更高层次的业务结合。

  • AI Box: 已经兴起一段时间,各种业务场景使用的推理模型不计其数,调优和再训练已经成了当前的热门工作内容;
  • 边缘数采:对传统的传感器进行数据收集和处理,已经在各领域应用数十年;
  • 将上面两者充分结合,已经破在眉睫;
  • 客户需求:
    • 灵活切换推理模型,根据业务需要选择;
    • 与北向系统对接,完成云/企业服务融合;

解决方案:将 AI 推理集成到成熟的边缘计算框架中,那就是 EdgeXFoundry 所涉及的范围。

下面来具体介绍一下集成的过程和历史配置,并提供一些截图和效果。

准备工作

硬件环境

准备一台可以作为整个系统运行的硬件设备,笔者使用了开头提到的两种类型的硬件设备分别进行了测试。

Intel® N100 口袋型小主机

minipc

  • 电脑棒迷你主机
  • 作为基础型验证或少量摄像头推理;
  • <100 fps(Frame Per Second) 推理性能;

Intel® i7-1165G7 桌面型小主机

采用深圳市铂盛科技有限公司生产的 AIPC,型号:PZ21_2L2S。

aipc

  • 桌面型小主机,可带挂架
  • 作为商业级应用,<10台摄像头推理;
  • <200 fps(Frame Per Second) 推理性能;

软件环境

操作系统和硬件配置

采用 Ubuntu 22.04 作为 Docker 宿主主机,并且已经成功完成 Docker 运行环境的安装和测试。

# uname -a
Linux YiFUSION-N100 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb  9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy
采用 CPU 内置 iGPU 作为推理
# ll /dev/dri/
total 0
drwxr-xr-x   3 root root        100  2月 23 17:34 ./
drwxr-xr-x  20 root root       4620  2月 23 20:19 ../
crw-rw----+  1 root video  226,   0  2月 23 20:58 card0
crw-rw----+  1 root render 226, 128  2月 23 17:39 renderD128

① OpenVINO™ 服务

OpenVINO™ Model Server 运行环境

OpenVINO™ 相关配置 Configuration JSON

模型库配置:

  • name: 模型名称
  • base_path: 模型文件路径
  • target_device: 目标推理设备
  • layout: 需要的图片格式
# more models/config.json
{
  "model_config_list":[
      {
        "config":{
            "name":"resnet50-tf",
            "base_path":"/models/resnet-50-tf/",
            "target_device": "AUTO",
            "layout": "NHWC:NCHW"
        }
      },
      {
        "config":{
            "name":"resnet",
            "base_path":"/models/resnet50/",
            "target_device": "AUTO",
            "layout": "NHWC:NCHW"
        }
      },
      {
        "config":{
            "name":"dummy",
            "base_path":"/models/dummy",
            "target_device": "AUTO",
            "batch_size": "auto"
        }
      },
      {
        "config":{
            "name":"age",
            "base_path":"/models/age",
            "target_device": "AUTO",
            "batch_size": "auto"
        }
      }
  ]
}

OpenVINO™ 相关配置 Local Model Repository

既可以用云存储作为模型库存放位置,也可以使用本地磁盘作为模型库存放位置(本例子使用)。

  • 一级目录是对应的模型名称
  • 二级目录是模型的版本,比如:0,1,2...
  • xml 是 OpenVINO™ 模型库相关配置,bin 是 OpenVINO™ 模型 IR 文件

models 目录结构如下:

models
├── age
│   └── 1       ├── age-gender-recognition-retail-0013.bin
│       └── age-gender-recognition-retail-0013.xml
├── config.json
├── dummy
│   └── 1       ├── dummy.bin
│       └── dummy.xml
├── resnet
│   └── 1       ├── resnet50-binary-0001.bin
│       └── resnet50-binary-0001.xml
├── resnet-50-tf
│   ├── 1      ├── resnet-50-tf.bin
│      └── resnet-50-tf.xml
│   └── resnet_v1-50.pb
└── resnet50
    └── 1
        ├── resnet50-binary-0001.bin
        └── resnet50-binary-0001.xml

12 directories, 14 files

启动 Model Server 容器

Model Server 容器

docker run -d \
--name model_server \
--network edgex_edgex-network \
--device /dev/dri \
-v $(pwd)/models:/models \
-p 9000:9000 \
-p 8000:8000 \
openvino/model_server:latest-gpu \
--config_path /models/config.json \
--port 9000 \
--rest_port 8000 \
--log_level DEBUG

Model Server 服务测试

验证 Model Server 运行状态

通过 Kserve RESTful API 访问 OVMS 工作状态,这里访问已经启动的两个算法:dummy 和 resnet。

curl http://192.168.123.15:8000/v2/models/dummy

{
  "name": "dummy",
  "versions": [
    "1"
  ],
  "platform": "OpenVINO",
  "inputs": [
    {
      "name": "b",
      "datatype": "FP32",
      "shape": [
        1,
        10
      ]
    }
  ],
  "outputs": [
    {
      "name": "a",
      "datatype": "FP32",
      "shape": [
        1,
        10
      ]
    }
  ]
}

curl http://192.168.123.15:8000/v2/models/resnet

{
  "name": "resnet",
  "versions": [
    "1"
  ],
  "platform": "OpenVINO",
  "inputs": [
    {
      "name": "0",
      "datatype": "FP32",
      "shape": [
        1,
        224,
        224,
        3
      ]
    }
  ],
  "outputs": [
    {
      "name": "1463",
      "datatype": "FP32",
      "shape": [
        1,
        1000
      ]
    }
  ]
}

② YiMEDIA 流媒体服务

笔者采用亿琪软件公司自己的流媒体服务器软件:YiMEDIA,来实现视频流的编解码。

当然,用户也可以使用一些其他框架来支撑,比如:

③ EdgeXFoundry 服务

部署 EdgeXFoundry Docker 容器

建议

根据 EdgeXFoundry edgex-compose 仓库手册,创建自己的 EdgeX 运行环境。本测试案例,只需要核心服务和 mqtt-broker 服务即可。

Device Service for ONVIF Camera

注意

Device Service for ONVIF Camera 是否需要使用,取决于是否由 EdgeX 来管理网络摄像头(IPC),本例中暂时未启用此服务,采用手动设置。

Device Services for OVMS

测试

这是一个系列微服务,包含了各种算法库,支持算法调用和推理。本例子中使用 dummy  Resnet50 作为验证测试。

  • dummy:返回 10 个数列;
  • Resnet50:提交一张图片,返回目标识别结果;

EdgeX Device Service: device-openvino-resnet

此服务代码未开源,仅供商业服务。

设备服务配置参考

deviceList:
  - name: OpenVINO-Device01
    profileName: OpenVINO-Device
    description: Example of OpenVINO Device
    labels: [AI]
    protocols:
      ovms:
        Host: 192.168.199.201
        Port: 9000
        Model: resnet
        Version: 1
        Uri: rtsp://192.168.123.12:18554/test
        Snapshot: false
        Record: false
    autoEvents:
      - interval: 1s
        onChange: false
        sourceName: AllResource

这是一个设备服务基础配置例子,用户可根据实际业务需求配置:

  • Host: OVMS 服务地址
  • Port: OVMS 服务端口
  • Model: 模型名称
  • Version: 模型版本
  • Uri: IPC 摄像头流地址
  • Snapshot: 是否建立快照
  • Record: 是否记录推理视频
  • interval: 可修改此参数来配置推理 fps,如:100ms, 10fps = 1000s / 100ms

以下是完整的 EdgeX metadata:

# curl http://192.168.123.15:59881/api/v3/device/all

{
  "apiVersion": "v3",
  "statusCode": 200,
  "totalCount": 1,
  "devices": [
    {
      "created": 1708691437289,
      "modified": 1708693207326,
      "id": "6b02c540-eeae-4b61-b741-cdefcf93fd09",
      "name": "OpenVINO-Device01",
      "description": "Example of OpenVINO Device",
      "adminState": "UNLOCKED",
      "operatingState": "UP",
      "labels": [
        "AI"
      ],
      "serviceName": "device-openvino-resnet",
      "profileName": "OpenVINO-Device",
      "autoEvents": [
        {
          "interval": "500ms",
          "onChange": false,
          "sourceName": "predict"
        }
      ],
      "protocols": {
        "ovms": {
          "Host": "model_server",
          "Model": "resnet",
          "Port": 9000,
          "Record": false,
          "Snapshot": false,
          "Uri": "rtsp://192.168.123.12:18554/test",
          "Version": 1
        }
      }
    }
  ]
}

eKuiper 规则引擎

eKuiper 是 EdgeXFoundry 默认的规则引擎,我们可以简单的将结果导出到企业/云平台,这里,我们使用两种方式:导出到亿琪云,或 MQTT Broker。

{
  "triggered": true,
  "id": "Rule-demo.yiqisoft.cn",
  "sql": "SELECT * , meta(deviceName) as deviceName, tstamp() as ts FROM  EdgeXStream",
  "actions": [
    {
      "mqtt": {
        "dataTemplate": "{\"{{.deviceName}}\":[{\"ts\":{{.ts}}, \"values\":{{json .}}}]}",
        "insecureSkipVerify": true,
        "protocolVersion": "3.1",
        "qos": 0,
        "retained": false,
        "sendSingle": true,
        "server": "tcp://demo.yiqisoft.cn:1883",
        "topic": "v1/gateway/telemetry",
        "username": "***"
      }
    },
    {
      "mqtt": {
        "dataTemplate": "{\"{{.deviceName}}\":[{\"ts\":{{.ts}}, \"values\":{{json .}}}]}",
        "insecureSkipVerify": true,
        "protocolVersion": "3.1",
        "qos": 0,
        "retained": false,
        "sendSingle": true,
        "server": "tcp://edgex-mqtt-broker:1883",
        "topic": "topic"
      }
    }
  ]
}

验证工作

容器微服务状况

容器

所有的容器都运行成功后,可看到类似以下的结果:

# docker ps --format 'table {{.Image}}\t{{.Names}}'

IMAGE                                                NAMES
①
openvino/model_server:latest-gpu                     model_server

②
yiqisoft/YiMEDIA                                     yimedia

③
edgexfoundry/device-openvino-resnet:0.0.0-dev        edgex-device-openvino-resnet
edgexfoundry/app-service-configurable:3.1.0          edgex-app-rules-engine
edgexfoundry/core-data:3.1.0                         edgex-core-data
edgexfoundry/core-command:3.1.0                      edgex-core-command
lfedge/ekuiper:1.11.4-alpine                         edgex-kuiper
edgexfoundry/support-scheduler:3.1.0                 edgex-support-scheduler
edgexfoundry/core-common-config-bootstrapper:3.1.0   edgex-core-common-config-bootstrapper
edgexfoundry/support-notifications:3.1.0             edgex-support-notifications
edgexfoundry/core-metadata:3.1.0                     edgex-core-metadata
eclipse-mosquitto:2.0.18                             edgex-mqtt-broker
edgexfoundry/edgex-ui:3.1.0                          edgex-ui-go
redis:7.0.14-alpine                                  edgex-redis
hashicorp/consul:1.16.2                              edgex-core-consul

容器启动后,自动实现 1秒1次的推理请求。

device-openvino-resnet 日志

日志

# docker logs -f edgex-device-openvino-resnet

level=DEBUG ts=2024-02-24T07:07:57.469908Z app=device-openvino-resnet source=executor.go:52 msg="AutoEvent - reading predict"
level=DEBUG ts=2024-02-24T07:07:57.470141Z app=device-openvino-resnet source=driver.go:107 msg="Driver.HandleReadCommands: protocols: map[ovms:map[Host:192.168.123.15 Model:resnet Port:9000 Record:false Snapshot:false Uri:rtsp://192.168.123.12:18554/test Version:1]], resource: predict, attributes: map[]"
level=DEBUG ts=2024-02-24T07:07:57.480294Z app=device-openvino-resnet source=driver.go:130 msg="Image Size: [%!s(int=240) %!s(int=320)]"
level=DEBUG ts=2024-02-24T07:07:57.511554Z app=device-openvino-resnet source=driver.go:167 msg="Infer result: remote control, remote"
level=DEBUG ts=2024-02-24T07:07:57.511585Z app=device-openvino-resnet source=driver.go:174 msg="CommandValues: [DeviceResource: predict, String: remote control, remote]"
level=DEBUG ts=2024-02-24T07:07:57.511615Z app=device-openvino-resnet source=transform.go:123 msg="device: OpenVINO-Device01 DeviceResource: predict reading: {Id:b0032e5b-a66a-4751-967e-cdcec75d3fad Origin:1708758477511592000 DeviceName:OpenVINO-Device01 ResourceName:predict ProfileName:OpenVINO-Device ValueType:String Units: Tags:map[] BinaryReading:{BinaryValue:[] MediaType:} SimpleReading:{Value:remote control, remote} ObjectReading:{ObjectValue:<nil>}}"
level=DEBUG ts=2024-02-24T07:07:57.511624Z app=device-openvino-resnet source=command.go:65 msg="GET Device Command successfully. Device: OpenVINO-Device01, Source: predict, X-Correlation-ID: "
level=DEBUG ts=2024-02-24T07:07:57.518218Z app=device-openvino-resnet source=utils.go:82 msg="Event(profileName: OpenVINO-Device, deviceName: OpenVINO-Device01, sourceName: predict, id: 7aa2f9c9-f551-4214-8c6a-bede178e7c3a) published to MessageBus on topic: edgex/events/device/device-openvino-resnet/OpenVINO-Device/OpenVINO-Device01/predict"

可以看到推理结果:msg="Infer result: remote control, remote"

OpenVINO™ Model Server 日志

日志

# docker logs -f model_server

[2024-02-23 16:10:42.037][629][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: resnet; version: 1
[2024-02-23 16:10:42.037][629][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: resnet, version: 1
[2024-02-23 16:10:42.037][629][serving][debug][modelmanager.cpp:1537] Requesting model: resnet; version: 1.
[2024-02-23 16:10:42.037][629][serving][debug][modelinstance.cpp:1054] Model: resnet, version: 1 already loaded
[2024-02-23 16:10:42.037][629][serving][debug][predict_request_validation_utils.cpp:999] [servable name: resnet version: 1] Validating request containing binary image input: name: 0
[2024-02-23 16:10:42.037][629][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model resnet, version 1, nireq 0: 0.004 ms
[2024-02-23 16:10:42.037][629][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model resnet, version 1, nireq 0: 0.000 ms
[2024-02-23 16:10:42.037][629][serving][debug][deserialization.hpp:449] Request contains input in native file format: 0
[2024-02-23 16:10:42.037][629][serving][debug][modelinstance.cpp:1252] Deserialization duration in model resnet, version 1, nireq 0: 0.562 ms
[2024-02-23 16:10:42.057][629][serving][debug][modelinstance.cpp:1260] Prediction duration in model resnet, version 1, nireq 0: 20.265 ms
[2024-02-23 16:10:42.058][629][serving][debug][modelinstance.cpp:1269] Serialization duration in model resnet, version 1, nireq 0: 0.025 ms
[2024-02-23 16:10:42.058][629][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model resnet, version 1, nireq 0: 0.003 ms
[2024-02-23 16:10:42.058][629][serving][debug][modelinstance.cpp:1281] Used device: CPU
[2024-02-23 16:10:42.058][629][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 20.959 ms
  • 可以看到推理使用设备:Used device: CPU,也可以在 config.json 配置文件中修改为: AUTO  GPU
  • 整个请求所花的时间:Total gRPC request processing time: 20.959 ms

亿琪云接收推理结果

云服务

cloud

MQTT Borker

MQTT Broker 接收

mqtt_sub -h 192.168.123.15 -p 1883 -t #
{"predict":"remote control, remote"}
{"predict":"remote control, remote"}
{"predict":"remote control, remote"}
{"predict":"remote control, remote"}
{"predict":"hand-held computer, hand-held microcomputer"}
{"predict":"remote control, remote"}
{"predict":"cellular telephone, cellular phone, cellphone, cell, mobile phone"}
{"predict":"remote control, remote"}
{"predict":"cellular telephone, cellular phone, cellphone, cell, mobile phone"}

视频/图片信息

视频/图片

本例子使用的图片识别算法,并未对推理结果视频进行编码输出。

# ffmpeg -i "rtsp://192.168.123.12:18554/test"

Input #0, rtsp, from 'rtsp://192.168.123.12:18554/test':
Metadata:
  title           : YiMEDIA/1.1.0
Duration: N/A, start: 0.033333, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 320x240, 30 fps, 30 tbr, 90k tbn

样本图片参考:

pic

展开阅读全文
加载中
点击引领话题📣 发布并加入讨论🔥
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部