基于 Scrcpy 的远程调试方案

原创
2019/12/26 08:30
阅读数 865

基于Scrcpy的远程调试方案

前言

感谢STF的开源,让Android设备远程控制变得简单,STF通过minicap和minitouch实现设备的显示和控制。
STF在实际使用中会发现一些棘手的问题
1.电视不支持minitouch
2.新手机比如mi8 mi9不支持minicap
3.Android发布新版本需要适配minicap

分享一个新的方法,来弥补这些不足
演示效果如下,由于图片较大(14MB),手机党请谨慎点击如下链接,建议完全加载完在播放,否则会卡,电脑的录屏软件很不给力,渣渣画质。。。
https://github.com/wenxiaomao1023/scrcpy/blob/master/assets/out.gif

Scrcpy

app目录 运行在PC端,对于web远程控制,这部分是不需要的
server目录 运行在手机端,提供屏幕数据,接收并响应控制事件

Scrcpy对比minicap

1.获取frame数据方式是一致的(sdk19以上)
2.Scrcpy将frame编码h264
3.minicap将frame编码jpeg

Scrcpy处理方式看起来会更好,但是有一个问题,他的设计是将屏幕数据直接发给PC,然后在PC上解码显示,这种方式在网页上却很不好展示。

调研与尝试

1.Broadway在前端解码h264并显示
2.wfs.js在前端将h264转成mp4送给h5 MSE实现播放,这种类似直播,B站flv.js那种
以上两种尝试都获得了图像,但个人感觉,以上两个方案感觉都有坑,还需要大量优化才能脱坑

解决方法

当前摸索出的解决方法,Scrcpy将frame编码jpeg发给前端然后通过画布展示,浏览器兼容好,可行性高,minicap也是这么做的,修改方式见如下
https://github.com/wenxiaomao1023/scrcpy/commit/dce39887f562cd33ad75e12b95778be00955011a

当前已实现的功能

1.使用ImageReader获取frame数据,通过libjpeg-turbo编码jpeg
2.控制帧率,压缩率,缩放比例,可以减少带宽占用,提高流畅性
3.考虑到当前大多是minicap的方案,所以scrcpy返回的屏幕数据格式兼容了minicap的数据格式(banner+jpegsize+jpegdata),移植改动会很小

优点

1.德芙般丝滑,手机播放视频一点不卡,web端展示也很流畅(30 - 50 FPS)
2.支持电视touch
3.支持mi8,mi9等图像展示,不必在适配minicap.so啦,耶!

缺点

1.最低支持android5.0,由于还依赖android.system.Os,若想兼容低版本设备需要配合minicap使用

编译libjpeg-turbo

我已经编好了ARMv7 (32-bit)和ARMv8 (64-bit)
https://github.com/wenxiaomao1023/scrcpy/tree/master/server/libs/libturbojpeg/prebuilt
如果你需要其他平台,可参考此文档Building libjpeg-turbo for Android部分
https://github.com/libjpeg-turbo/libjpeg-turbo/blob/master/BUILDING.md
如果不需要,可跳过此步骤

编译Scrcpy代码

ninja编译方式

android sdk里有ninja,如Android/Sdk/cmake/3.6.4111459/bin/ninja,加到环境变量里即可,meson需要安装
如果不想安装这些,可以往下看,用gradle编译

git clone https://github.com/wenxiaomao1023/scrcpy.git
cd scrcpy
meson x --buildtype release --strip -Db_lto=true
ninja -Cx

编译后会在scrcpy目录下生成
x/server/scrcpy-server.jar
server/jniLibs/armeabi-v7a/libcompress.so
server/jniLibs/arm64-v8a/libcompress.so

gradle编译方式

git clone https://github.com/wenxiaomao1023/scrcpy.git
cd scrcpy/server
../gradlew assembleDebug

编译后会在scrcpy目录下生成
server/build/outputs/apk/debug/server-debug.apk
server/jniLibs/armeabi-v7a/libcompress.so
server/jniLibs/arm64-v8a/libcompress.so

server/build/outputs/apk/debug/server-debug.apk 和 x/server/scrcpy-server.jar 是一样的,下文中都按scrcpy-server.jar命名方式进行说明

启动scrcpy-server.jar

# 先看下设备的abi
adb shell getprop ro.product.cpu.abi
# armeabi-v7a
adb push scrcpy/server/jniLibs/armeabi-v7a/libcompress.so /data/local/tmp/
adb push scrcpy/server/libs/libturbojpeg/prebuilt/armeabi-v7a/libturbojpeg.so /data/local/tmp/
adb push scrcpy/x/server/scrcpy-server.jar /data/local/tmp/
adb shell chmod 777 /data/local/tmp/scrcpy-server.jar
adb shell LD_LIBRARY_PATH=/system/lib:/vendor/lib:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
# arm64-v8a
adb push server/jniLibs/arm64-v8a/libcompress.so /data/local/tmp/
adb push server/libs/libturbojpeg/prebuilt/arm64-v8a/libturbojpeg.so /data/local/tmp/
adb push scrcpy/x/server/scrcpy-server.jar /data/local/tmp/
adb shell chmod 777 /data/local/tmp/scrcpy-server.jar
adb shell LD_LIBRARY_PATH=/system/lib64:/vendor/lib64:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server

app_process / com.genymobile.scrcpy.Server 这个命令可以设置如下参数,建议使用命令如下,压缩质量60,最高24帧,缩放为屏幕长宽除以2
app_process / com.genymobile.scrcpy.Server -Q 60 -r 24 -P 2

Usage: %s [-h]
-Q <value>: JPEG quality (0-100).
-r <value>: Frame rate (frames/s).
-P <value>: Display projection (scale 1,2,4...).
-h: Show help.

启动app.js

scrcpy-server.jar兼容了minicap数据格式,可以直接用minicap的demo app.js看效果
https://github.com/openstf/minicap/tree/master/example
https://github.com/openstf/minicap/blob/master/example/app.js

需要把app.js改一下,多一个连接,修改如下

// 原始代码默认的图像socket
var stream = net.connect({
port: 1717
})

// 修改1 加一个控制socket
var controlStream = net.connect({
port: 1717
})
git clone https://github.com/openstf/minicap.git
cd minicap/example
npm install
# 注意这里要改为localabstract:scrcpy
adb forward tcp:1717 localabstract:scrcpy
node app.js

访问 http://127.0.0.1:9002

Scrcpy touch

Scrcpy touch的实现可以参考如下实现,当前实现常用的三种事件消息
// 键值 HOME,BACK,MENU等
CONTROL_MSG_TYPE_INJECT_KEYCODE
// 点击和滑动
CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
// 鼠标滚轮滚动
CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT

后端提供json格式接口

package main

import (
"errors"
"net"
"github.com/qiniu/log"
"bytes"
"encoding/binary"
)

type MessageType int8
const (
CONTROL_MSG_TYPE_INJECT_KEYCODE MessageType = iota
CONTROL_MSG_TYPE_INJECT_TEXT
CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON
CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL
CONTROL_MSG_TYPE_COLLAPSE_NOTIFICATION_PANEL
CONTROL_MSG_TYPE_GET_CLIPBOARD
CONTROL_MSG_TYPE_SET_CLIPBOARD
CONTROL_MSG_TYPE_SET_SCREEN_POWER_MODE
)

type PositionType struct {
X int32 `json:"x"`
Y int32 `json:"y"`
Width int16 `json:"width"`
Height int16 `json:"height"`
}

type Message struct {
Msg_type MessageType `json:"msg_type"`
// CONTROL_MSG_TYPE_INJECT_KEYCODE
Msg_inject_keycode_action int8 `json:"msg_inject_keycode_action"`
Msg_inject_keycode_keycode int32 `json:"msg_inject_keycode_keycode"`
Msg_inject_keycode_metastate int32 `json:"msg_inject_keycode_metastate"`
// CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
Msg_inject_touch_action int8 `json:"msg_inject_touch_action"`
Msg_inject_touch_pointerid int64 `json:"msg_inject_touch_pointerid"`
Msg_inject_touch_position PositionType `json:"msg_inject_touch_position"`
Msg_inject_touch_pressure uint16 `json:"msg_inject_touch_pressure"`
Msg_inject_touch_buttons int32 `json:"msg_inject_touch_buttons"`
// CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
Msg_inject_scroll_position PositionType `json:"msg_inject_scroll_position"`
Msg_inject_scroll_horizontal int32 `json:"msg_inject_scroll_horizontal"`
Msg_inject_scroll_vertical int32 `json:"msg_inject_scroll_vertical"`
}

type KeycodeMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_keycode_action int8 `json:"msg_inject_keycode_action"`
Msg_inject_keycode_keycode int32 `json:"msg_inject_keycode_keycode"`
Msg_inject_keycode_metastate int32 `json:"msg_inject_keycode_metastate"`
}

type TouchMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_touch_action int8 `json:"msg_inject_touch_action"`
Msg_inject_touch_pointerid int64 `json:"msg_inject_touch_pointerid"`
Msg_inject_touch_position PositionType `json:"msg_inject_touch_position"`
Msg_inject_touch_pressure uint16 `json:"msg_inject_touch_pressure"`
Msg_inject_touch_buttons int32 `json:"msg_inject_touch_buttons"`
}

type ScrollMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_scroll_position PositionType `json:"msg_inject_scroll_position"`
Msg_inject_scroll_horizontal int32 `json:"msg_inject_scroll_horizontal"`
Msg_inject_scroll_vertical int32 `json:"msg_inject_scroll_vertical"`
}

func drainScrcpyRequests(conn net.Conn, reqC chan Message) error {
for req := range reqC {
var err error
switch req.Msg_type {
case CONTROL_MSG_TYPE_INJECT_KEYCODE:
t := KeycodeMessage{
Msg_type: req.Msg_type,
Msg_inject_keycode_action: req.Msg_inject_keycode_action,
Msg_inject_keycode_keycode: req.Msg_inject_keycode_keycode,
Msg_inject_keycode_metastate: req.Msg_inject_keycode_metastate,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_KEYCODE error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_INJECT_TEXT:
case CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT:
var pointerid int64 = -1
var pressure uint16 = 65535
var buttons int32 = 1
req.Msg_inject_touch_pointerid = pointerid
req.Msg_inject_touch_pressure = pressure
req.Msg_inject_touch_buttons = buttons
t := TouchMessage{
Msg_type: req.Msg_type,
Msg_inject_touch_action: req.Msg_inject_touch_action,
Msg_inject_touch_pointerid: req.Msg_inject_touch_pointerid,
Msg_inject_touch_position: PositionType{
X: req.Msg_inject_touch_position.X,
Y: req.Msg_inject_touch_position.Y,
Width: req.Msg_inject_touch_position.Width,
Height: req.Msg_inject_touch_position.Height,
},
Msg_inject_touch_pressure: req.Msg_inject_touch_pressure,
Msg_inject_touch_buttons: req.Msg_inject_touch_buttons,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT:
t := ScrollMessage{
Msg_type: req.Msg_type,
Msg_inject_scroll_position: PositionType{
X: req.Msg_inject_scroll_position.X,
Y: req.Msg_inject_scroll_position.Y,
Width: req.Msg_inject_scroll_position.Width,
Height: req.Msg_inject_scroll_position.Height,
},
Msg_inject_scroll_horizontal: req.Msg_inject_scroll_horizontal,
Msg_inject_scroll_vertical: req.Msg_inject_scroll_vertical,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON:
case CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL:
case CONTROL_MSG_TYPE_COLLAPSE_NOTIFICATION_PANEL:
case CONTROL_MSG_TYPE_GET_CLIPBOARD:
case CONTROL_MSG_TYPE_SET_CLIPBOARD:
case CONTROL_MSG_TYPE_SET_SCREEN_POWER_MODE:
default:
err = errors.New("unsupported msg type")
}
if err != nil {
return err
}
}
return nil
}

前端调用

let scrcpyKey = (key) => {
ws.send(JSON.stringify({
"msg_type": 0,
"msg_inject_keycode_action": 0,
"msg_inject_keycode_keycode": key,
"msg_inject_keycode_metastate": 0
}))
ws.send(JSON.stringify({
"msg_type": 0,
"msg_inject_keycode_action": 1,
"msg_inject_keycode_keycode": key,
"msg_inject_keycode_metastate": 0
}))
}
let scrcpyTouchDown = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 0,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}}));
}
let scrcpyTouchMove = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 2,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}
}));
}
let scrcpyTouchUp = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 1,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}
}));
}
//向下滚动
let scrcpyScrollDown = (touch) => {
ws.send(JSON.stringify({
"msg_type": 3,
"msg_inject_scroll_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
},
"msg_inject_scroll_horizontal": 0,
"msg_inject_scroll_vertical": -1,
}));
}
//向上滚动
let scrcpyScrollUp = (touch) => {
ws.send(JSON.stringify({
"msg_type": 3,
"msg_inject_scroll_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
},
"msg_inject_scroll_horizontal": 0,
"msg_inject_scroll_vertical": 1,
}));
}

项目还在开发阶段,欢迎反馈问题 : )

收工~


↙↙↙阅读原文可查看相关链接,并与作者交流


本文分享自微信公众号 - TesterHome(AskTesterHome)。
如有侵权,请联系 support@oschina.cn 删除。
本文参与“OSC源创计划”,欢迎正在阅读的你也加入,一起分享。

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部