diff --git a/docs/hardware/zh/metasense/assets/tof-1.12.jpg b/docs/hardware/zh/metasense/assets/tof-1.12.jpg new file mode 100644 index 00000000..a20dd630 Binary files /dev/null and b/docs/hardware/zh/metasense/assets/tof-1.12.jpg differ diff --git a/docs/hardware/zh/metasense/assets/tof-1.13.jpg b/docs/hardware/zh/metasense/assets/tof-1.13.jpg new file mode 100644 index 00000000..8fd6e40c Binary files /dev/null and b/docs/hardware/zh/metasense/assets/tof-1.13.jpg differ diff --git a/docs/hardware/zh/metasense/assets/tof-1.7.jpg b/docs/hardware/zh/metasense/assets/tof-1.7.jpg deleted file mode 100644 index 796962e1..00000000 Binary files a/docs/hardware/zh/metasense/assets/tof-1.7.jpg and /dev/null differ diff --git a/docs/hardware/zh/metasense/assets/tof-1.9.jpg b/docs/hardware/zh/metasense/assets/tof-1.9.jpg deleted file mode 100644 index c526d409..00000000 Binary files a/docs/hardware/zh/metasense/assets/tof-1.9.jpg and /dev/null differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-13.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/a010-13.jpg deleted file mode 100644 index 5294deae..00000000 Binary files a/docs/hardware/zh/metasense/metasense-a010/assets/a010-13.jpg and /dev/null differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-14.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/a010-14.jpg deleted file mode 100644 index 53e1f3f2..00000000 Binary files a/docs/hardware/zh/metasense/metasense-a010/assets/a010-14.jpg and /dev/null differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-6.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/a010-6.jpg deleted file mode 100644 index a8ce8441..00000000 Binary files a/docs/hardware/zh/metasense/metasense-a010/assets/a010-6.jpg and /dev/null differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-7.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/a010-7.jpg deleted file mode 100644 index e7258f85..00000000 Binary files a/docs/hardware/zh/metasense/metasense-a010/assets/a010-7.jpg and /dev/null differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-10.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/ms-RVIZ.jpg similarity index 100% rename from docs/hardware/zh/metasense/metasense-a010/assets/a010-10.jpg rename to docs/hardware/zh/metasense/metasense-a010/assets/ms-RVIZ.jpg diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/ms-lamp.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/ms-lamp.jpg new file mode 100644 index 00000000..d4dfce60 Binary files /dev/null and b/docs/hardware/zh/metasense/metasense-a010/assets/ms-lamp.jpg differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/ms-mcu.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/ms-mcu.jpg new file mode 100644 index 00000000..3875452a Binary files /dev/null and b/docs/hardware/zh/metasense/metasense-a010/assets/ms-mcu.jpg differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/ms-people.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/ms-people.jpg new file mode 100644 index 00000000..4e8f35f7 Binary files /dev/null and b/docs/hardware/zh/metasense/metasense-a010/assets/ms-people.jpg differ diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-8.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/ms-ros.jpg similarity index 100% rename from docs/hardware/zh/metasense/metasense-a010/assets/a010-8.jpg rename to docs/hardware/zh/metasense/metasense-a010/assets/ms-ros.jpg diff --git a/docs/hardware/zh/metasense/metasense-a010/assets/a010-9.jpg b/docs/hardware/zh/metasense/metasense-a010/assets/ms-rqt.jpg similarity index 100% rename from docs/hardware/zh/metasense/metasense-a010/assets/a010-9.jpg rename to docs/hardware/zh/metasense/metasense-a010/assets/ms-rqt.jpg diff --git a/docs/hardware/zh/metasense/metasense-a010/at_command.md b/docs/hardware/zh/metasense/metasense-a010/at_command.md new file mode 100644 index 00000000..68350b08 --- /dev/null +++ b/docs/hardware/zh/metasense/metasense-a010/at_command.md @@ -0,0 +1,169 @@ +# MetaSense-A010 二次开发手册 + +## AT 指令表 + +| AT                             |                                                                                                                                                                                                                | +| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| +ISP
Image Signal Processor | :0: turn ISP off
=1: turn ISP on                                                                                                                                                                            | +| +BINN
full binning          | =1: output 100x100 pixel frame
=2: output 50x50 pixel frame
=4: output 25x25 pixel frame
                                                                                                            | +| +DISP
display mux           | =0: all off
=1: lcd display on
=2: usb display on
=3: lcd and usb display on
=4: uart display on
=5: lcd and uart display on
=6: usb and uart display on
=7: lcd, usb and uart display on | +| +BAUD
uart baudrate         | =0: 9600
=1: 57600
=2: 115200
=3: 230400
=4: 460800
=5: 921600
=6: 1000000
=7: 2000000
=8: 3000000                                                                                     | +| +UNIT
quantization unit     | =0: auto
=1-10: quantizated by unit(mm)                                                                                                                                                                     | +| +FPS
frame per second       | =1-19: set frame per second                                                                                                                                                                                    | +| +Save
save config           | : save current configuration                                 | + + +|输入|执行|注释| +|---|---|---| +|AT+ISP? |\r|返回当前ISP状态| +|AT+ISP=? |\r|返回所有支持的ISP状态| +|AT+ISP=< MODE >|\r|选择ISP状态| + +参数： +|< MODE > | 含义 | +|----|----| +|0 "STOP ISP" |立即关闭模组ISP，停止IR发射器| +|1 "LAUNCH ISP" |计划启动模组ISP，实际出图需等待1～2秒| + +BINN指令 +句法： +| 输入 | 执行 | 注释 | +|----|----|----| +| AT+BINN? | \r | 返回当前BINN状态 | +| AT+BINN=? | \r | 返回所有支持的BINN状态 | +| AT+BINN= < MODE > | \r | 选择BINN状态 | + +参数： +| < MODE > | 含义 | +|----------|------| +| 1 "1x1 BINN" | 1x1相当于无binning，实际出图分辨率为100x100。 | +| 2 "2x2 BINN" | 2×2binning，4个像素点合并成1个，实际出图分辨率为50×50计划启动模组ISP，实际出图需等待1～2秒。| +| 4 "4x4 BINN" | 4×4binning，16个像素点合并成1个，实际出图分辨率为25×25。 | + + +DISP指令 +请按需开启，避免资源过度占用 +句法： + +| 输入 | 执行 | 注释 | +|------|------|-----| +| AT+DISP? | \r | 返回当前DISP状态 | +| AT+DISP=? | \r | 返回所有支持的DISP状态 | +| AT+DISP=< MODE > | \r | 选择DISP状态 | + +参数： + + +| < MODE > | 含义 | +|----------|------| +| 0 | all off | +| 1 | lcd display on | +| 2 | usb display on | +| 3 | lcd and usb display on | +| 4 | uart display on | +| 5 | lcd and uart display on | +| 6 | usb and uart display on | +| 7 | lcd, usb and uart display on | + +BAUD指令 +句法： + +| 输入 | 执行 | 注释 | +|------|-----|------| +| AT+BAUD? | \r | 返回当前BAUD状态 | +| AT+BAUD=? | \r | 返回所有支持的BAUD状态 | +| AT+BAUD=< MODE > | \r | 选择BAUD状态 | + +参数： + +| < MODE > | 含义 | +|----------|------| +| 0 | 9600 | +| 1 | 57600 | +| 2 | 115200 | +| 3 | 230400 | +| 4 | 460800 | +| 5 | 921600 | +| 6 | 1000000 | +| 7 | 2000000 | +| 8 | 3000000 | + +UNIT指令 +句法： + +| 输入 | 执行 | 注释 | +|------|------|------| +| AT+UNIT? | \r | 返回当前UNIT值 | +| AT+UNIT=? | \r | 返回所有支持的UNIT值 | +| AT+UNIT=< UINT > | \r | 选择UNIT值 | + +参数： +| < UINT > | 含义 | +|----------|------| +| 0 "DEFAULT UNIT" | 采用默认量化策略，因tof特性导致成像近处精度优于远距离处，故放大近距离处差异，采用5.1*sqrt(x)将16bit的原数据量化为8bit +1...9 "QUANTIZE UNIT"s + +代表以x mm为单位进行量化，取值越小细节越多，同时可视距离越短，请合理设置 +FPS指令 +句法： +输入 +执行 +注释 +AT+FPS? +\r +返回当前FPS值 +AT+FPS=? +\r +返回所有支持的FPS值 +AT+FPS= +\r +选择FPS值 +参数： +< FPS > +含义 +1...19 "frame per second" + +tof出图帧率，越大越流畅 +SAVE指令 +句法： +输入 +执行 +注释 +AT+SAVE +\r +固化TOF摄像头当前配置，事后需要复位 + +多机 和 AE 指令建议加入 +ANTIMMI指令 +句法： +输入 +执行 +注释 +AT+ANTIMMI? +\r +返回当前ANTIMMI状态 +AT+ANTIMMI=? +\r +返回所有支持的ANTIMMI状态 +AT+ANTIMMI= +\r +选择ANTIMMI状态 +参数： + +含义 +-1 +disable anti-mmi +0 +auto anti-mmi +1-41 +manual anti-mmi usb display on + +图像数据包说明 +上电默认启动ISP并在显示屏显示图像，同时输出图像数据到uart和usb +图像数据封装成包（未稳定）： +1. 包头2字节：0X00、0XFF +2. 包长度2字节：当前包剩余数据的字节数 +3. 其他内容16字节：包括包序号、包长度、分辨率等等 +4. 图像帧 +5. 校验1字节：之前所有字节的“和”低八位 +6. 包尾1字节：0XDD \ No newline at end of file diff --git a/docs/hardware/zh/metasense/metasense-a010/code.md b/docs/hardware/zh/metasense/metasense-a010/code.md new file mode 100644 index 00000000..b9692bc7 --- /dev/null +++ b/docs/hardware/zh/metasense/metasense-a010/code.md @@ -0,0 +1,921 @@ +# 源码总览 + +- [源码总览](#源码总览) + - [tof_mainpy](#tof_mainpy) + - [streampy](#streampy) + - [calvolumespy](#calvolumespy) + - [calibratepy](#calibratepy) + +## tof_mainpy + +```python +from fpioa_manager import fm +from machine import UART +import lcd, image + +# lcd.init(invert=True) +lcd.init() +img = image.Image() + +fm.register(24, fm.fpioa.UART1_TX, force=True) +fm.register(25, fm.fpioa.UART1_RX, force=True) + +uart_A = UART(UART.UART1, 115200, 8, 0, 0, timeout=1000, read_buf_len=4096) + +def uart_readBytes(): + return uart_A.read() + + +def uart_hasData(): + return uart_A.any() + + +def uart_sendCmd(cmd): + uart_A.write(cmd) + +uart_sendCmd(b"AT+BAUD=5\r") + +uart_A.deinit() + +uart_A = UART(UART.UART1, 921600, 8, 0, 0, timeout=1000, read_buf_len=4096) + +jetcolors = [ + (128, 0, 0), (132, 0, 0), (136, 0, 0), (140, 0, 0), (144, 0, 0), (148, 0, 0), (152, 0, 0), (156, 0, 0), (160, 0, 0), (164, 0, 0), (168, 0, 0), (172, 0, 0), (176, 0, 0), (180, 0, 0), (184, 0, 0), (188, 0, 0), (192, 0, 0), (196, 0, 0), (200, 0, 0), (204, 0, 0), (208, 0, 0), (212, 0, 0), (216, 0, 0), (220, 0, 0), (224, 0, 0), (228, 0, 0), (232, 0, 0), (236, 0, 0), (240, 0, 0), (244, 0, 0), (248, 0, 0), (252, 0, 0), (255, 0, 0), (255, 4, 0), (255, 8, 0), (255, 12, 0), (255, 16, 0), (255, 20, 0), (255, 24, 0), (255, 28, 0), (255, 32, 0), (255, 36, 0), (255, 40, 0), (255, 44, 0), (255, 48, 0), (255, 52, 0), (255, 56, 0), (255, 60, 0), (255, 64, 0), (255, 68, 0), (255, 72, 0), (255, 76, 0), (255, 80, 0), (255, 84, 0), (255, 88, 0), (255, 92, 0), (255, 96, 0), (255, 100, 0), (255, 104, 0), (255, 108, 0), (255, 112, 0), (255, 116, 0), (255, 120, 0), (255, 124, 0), (255, 128, 0), (255, 132, 0), (255, 136, 0), (255, 140, 0), (255, 144, 0), (255, 148, 0), (255, 152, 0), (255, 156, 0), (255, 160, 0), (255, 164, 0), (255, 168, 0), (255, 172, 0), (255, 176, 0), (255, 180, 0), (255, 184, 0), (255, 188, 0), (255, 192, 0), (255, 196, 0), (255, 200, 0), (255, 204, 0), (255, 208, 0), (255, 212, 0), (255, 216, 0), (255, 220, 0), (255, 224, 0), (255, 228, 0), (255, 232, 0), (255, 236, 0), (255, 240, 0), (255, 244, 0), (255, 248, 0), (255, 252, 0), (254, 255, 1), (250, 255, 6), (246, 255, 10), (242, 255, 14), (238, 255, 18), (234, 255, 22), (230, 255, 26), (226, 255, 30), (222, 255, 34), (218, 255, 38), (214, 255, 42), (210, 255, 46), (206, 255, 50), (202, 255, 54), (198, 255, 58), (194, 255, 62), (190, 255, 66), (186, 255, 70), (182, 255, 74), (178, 255, 78), (174, 255, 82), (170, 255, 86), (166, 255, 90), (162, 255, 94), (158, 255, 98), (154, 255, 102), (150, 255, 106), (146, 255, 110), (142, 255, 114), (138, 255, 118), (134, 255, 122), (130, 255, 126), + (126, 255, 130), (122, 255, 134), (118, 255, 138), (114, 255, 142), (110, 255, 146), (106, 255, 150), (102, 255, 154), (98, 255, 158), (94, 255, 162), (90, 255, 166), (86, 255, 170), (82, 255, 174), (78, 255, 178), (74, 255, 182), (70, 255, 186), (66, 255, 190), (62, 255, 194), (58, 255, 198), (54, 255, 202), (50, 255, 206), (46, 255, 210), (42, 255, 214), (38, 255, 218), (34, 255, 222), (30, 255, 226), (26, 255, 230), (22, 255, 234), (18, 255, 238), (14, 255, 242), (10, 255, 246), (6, 255, 250), (2, 255, 254), (0, 252, 255), (0, 248, 255), (0, 244, 255), (0, 240, 255), (0, 236, 255), (0, 232, 255), (0, 228, 255), (0, 224, 255), (0, 220, 255), (0, 216, 255), (0, 212, 255), (0, 208, 255), (0, 204, 255), (0, 200, 255), (0, 196, 255), (0, 192, 255), (0, 188, 255), (0, 184, 255), (0, 180, 255), (0, 176, 255), (0, 172, 255), (0, 168, 255), (0, 164, 255), (0, 160, 255), (0, 156, 255), (0, 152, 255), (0, 148, 255), (0, 144, 255), (0, 140, 255), (0, 136, 255), (0, 132, 255), (0, 128, 255), (0, 124, 255), (0, 120, 255), (0, 116, 255), (0, 112, 255), (0, 108, 255), (0, 104, 255), (0, 100, 255), (0, 96, 255), (0, 92, 255), (0, 88, 255), (0, 84, 255), (0, 80, 255), (0, 76, 255), (0, 72, 255), (0, 68, 255), (0, 64, 255), (0, 60, 255), (0, 56, 255), (0, 52, 255), (0, 48, 255), (0, 44, 255), (0, 40, 255), (0, 36, 255), (0, 32, 255), (0, 28, 255), (0, 24, 255), (0, 20, 255), (0, 16, 255), (0, 12, 255), (0, 8, 255), (0, 4, 255), (0, 0, 255), (0, 0, 252), (0, 0, 248), (0, 0, 244), (0, 0, 240), (0, 0, 236), (0, 0, 232), (0, 0, 228), (0, 0, 224), (0, 0, 220), (0, 0, 216), (0, 0, 212), (0, 0, 208), (0, 0, 204), (0, 0, 200), (0, 0, 196), (0, 0, 192), (0, 0, 188), (0, 0, 184), (0, 0, 180), (0, 0, 176), (0, 0, 172), (0, 0, 168), (0, 0, 164), (0, 0, 160), (0, 0, 156), (0, 0, 152), (0, 0, 148), (0, 0, 144), (0, 0, 140), (0, 0, 136), (0, 0, 132), (0, 0, 128) +] + +def show(frameData, res): + resR = res[0] + resC = res[1] + for y in range(resR): + for x in range(resC): + pixel_cmap_rgb = jetcolors[frameData[y*resR + x]] + img.set_pixel(110 + x, 70 + y, pixel_cmap_rgb) + lcd.display(img) + img.clear() + +FRAME_HEAD = b"\x00\xFF" +FRAME_TAIL = b"\xCC" + +from struct import unpack +# send_cmd("AT+BINN=2\r") +uart_sendCmd(b"AT+DISP=5\r") +uart_sendCmd(b"AT+FPS=10\r") + +# while True: +# if uart_hasData(): +# print(uart_readBytes()) + +rawData = b'' +while True: + if not uart_hasData(): + continue + rawData += uart_readBytes() + idx = rawData.find(FRAME_HEAD) + if idx < 0: + continue + rawData = rawData[idx:] + # print(rawData) + # check data length 2Byte + dataLen = unpack("H", rawData[2: 4])[0] + # print("len: "+str(dataLen)) + frameLen = len(FRAME_HEAD) + 2 + dataLen + 2 + frameDataLen = dataLen - 16 + + if len(rawData) < frameLen: + continue + # get data + frame = rawData[:frameLen] + # print(frame.hex()) + rawData = rawData[frameLen:] + + frameTail = frame[-1] + # print("tail: "+str(hex(frameTail))) + _sum = frame[-2] + # print("checksum: "+str(hex(_sum))) + # check sum + # spi has no checksum but i add one + if frameTail != 0xdd and _sum != sum(frame[:frameLen - 2]) % 256: + continue + + frameID = unpack("H", frame[16:18])[0] + # print("frame ID: "+str(frameID)) + + resR = unpack("B", frame[14:15])[0] + resC = unpack("B", frame[15:16])[0] + res = (resR, resC) + # print(res) + # frameData=[ unpack("H", frame[20+i:22+i])[0] for i in range(0, frameDataLen, 2) ] + frameData = [unpack("B", frame[20+i:21+i])[0] + for i in range(0, frameDataLen, 1)] + + show(frameData, res) + + del frameData +``` + +## streampy + +```python +from PIL import Image +import requests +import matplotlib.pyplot as plt +import struct +import numpy as np +import cv2 + + +def frame_config_decode(frame_config): + ''' + @frame_config bytes + + @return fields, tuple (trigger_mode, deep_mode, deep_shift, ir_mode, status_mode, status_mask, rgb_mode, rgb_res, expose_time) + ''' + return struct.unpack("> with_config[1] + deepth_img = struct.unpack("<%us" % deepth_size, frame_payload[:deepth_size])[ + 0] if 0 != deepth_size else None + frame_payload = frame_payload[deepth_size:] + + # 0:16bit 1:8bit, resolution: 320*240 + ir_size = (320*240*2) >> with_config[3] + ir_img = struct.unpack("<%us" % ir_size, frame_payload[:ir_size])[ + 0] if 0 != ir_size else None + frame_payload = frame_payload[ir_size:] + + status_size = (320*240//8) * (16 if 0 == with_config[4] else + 2 if 1 == with_config[4] else 8 if 2 == with_config[4] else 1) + status_img = struct.unpack("<%us" % status_size, frame_payload[:status_size])[ + 0] if 0 != status_size else None + frame_payload = frame_payload[status_size:] + + assert(deep_data_size == deepth_size+ir_size+status_size) + + rgb_size = len(frame_payload) + assert(rgb_data_size == rgb_size) + rgb_img = struct.unpack("<%us" % rgb_size, frame_payload[:rgb_size])[ + 0] if 0 != rgb_size else None + + if (not rgb_img is None) and (1 == with_config[6]): + jpeg = cv2.imdecode(np.frombuffer( + rgb_img, 'uint8', rgb_size), cv2.IMREAD_COLOR) + if not jpeg is None: + rgb = cv2.cvtColor(jpeg, cv2.COLOR_BGR2RGB) + rgb_img = rgb.tobytes() + else: + rgb_img = None + + return (deepth_img, ir_img, status_img, rgb_img) + + +HOST = '192.168.233.1' +PORT = 80 + + +def post_encode_config(config=frame_config_encode(), host=HOST, port=PORT): + r = requests.post('http://{}:{}/set_cfg'.format(host, port), config) + if(r.status_code == requests.codes.ok): + return True + return False + + +def get_frame_from_http(host=HOST, port=PORT): + r = requests.get('http://{}:{}/getdeep'.format(host, port)) + if(r.status_code == requests.codes.ok): + # print('Get deep image') + deepimg = r.content + # print('Length={}'.format(len(deepimg))) + (frameid, stamp_msec) = struct.unpack(' 65535)] = 65535 + + # depth = depth_f.astype(depth.dtype) + + # depth[240//2, 320//2 - 5:320//2+5] = 0x00 + # depth[240//2-5:240//2+5, 320//2] = 0x00 + ax1.imshow(depth, cmap='jet_r') + ax2 = fig.add_subplot(222) + if not ir is None: + ax2.imshow(ir, cmap='gray') + ax3 = fig.add_subplot(223) + if not status is None: + ax3.imshow(status) + ax4 = fig.add_subplot(224) + if not rgb is None: + ax4.imshow(rgb) + + +if post_encode_config(frame_config_encode(1, 1, 255, 0, 2, 7, 1, 0, 0)): + # 打开交互模式 + plt.ion() + figsize = (12, 12) + fig = plt.figure('2D frame', figsize=figsize) + while True: + p = get_frame_from_http() + show_frame(fig, p) + # 停顿时间 + plt.pause(0.001) + # 清除当前画布 + fig.clf() + + plt.ioff() +``` + +## calvolumespy + +```python +from PIL import Image, ImageDraw +import requests +import matplotlib.pyplot as plt +import struct +import numpy as np +import cv2 + +HOST = '192.168.233.1' +PORT = 80 + +def depth2xyz(xp, yp, z, fx, fy, cx, cy, depth_scale=1000): + # h,w=np.mgrid[0:depth_map.shape[0],0:depth_map.shape[1]] + z = z/depth_scale + x = (xp-cx)*z/fx + y = (yp-cy)*z/fy + # xyz=np.dstack((x,y,z)) + # xyz=cv2.rgbd.depthTo3d(depth_map,depth_cam_matrix) + return [x, y, z] + + +def polygon_area(polygon): + area = 0 + q = polygon[-1] + for p in polygon: + area += p[0] * q[1] - p[1] * q[0] + q = p + return abs(area) / 2.0 + +def get_lenscoeff(host=HOST, port=PORT): + r = requests.get('http://{}:{}/getinfo'.format(host, port)) + if(r.status_code == requests.codes.ok): + lenscoeff_bin = r.content + (_fx,_fy,_cx,_cy) = struct.unpack(' diff_high, 0, diff1) + diff1 = (np.where(diff1 > 0, 1, 0)*255).astype(np.uint8) + # plt.imshow(diff1) + + # print(d_bk.shape) (240, 320) + output = np.zeros((img_h, img_w, 3), np.uint8) + + num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats( + diff1, connectivity=8) + # print('num_labels = ',num_labels) + # 连通域的信息：对应各个轮廓的x、y、width、height和面积 + # print('stats = ',stats) + res = list() + max_stats = list() + for i in range(5): + max_label = 1+np.argmax(stats[1:, 4]) + # print('stats[max_label] = ', stats[max_label]) + if i > 0 and stats[max_label][4] < 700: + break + max_stat = stats[max_label] + max_stats.append(max_stat) + stats[max_label][4] = 0 + + mask = (labels == max_label) + # (np.random.rand(3)*255).astype(np.uint8) + output[:, :, :][mask] = [200, 0, 0] + # plt.imshow(output) + + # kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)) + # eroded = cv2.erode(output, kernel) + # dilated = cv2.dilate(output, kernel) + + kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5, 5)) + # output = dilated + output=cv2.morphologyEx(output, cv2.MORPH_OPEN, kernel) + output=cv2.morphologyEx(output, cv2.MORPH_CLOSE, kernel) + + + volumes = [] + # points = [] + # areas = [] + for yp in range(img_h): + for xp in range(img_w): + if mask[yp, xp]: + x1, y1, z1 = depth2xyz( + xp, yp, d_bk[yp, xp], fx, fy, cx, cy, depth_scale=1) + x0, y0, z0 = depth2xyz( + xp, yp, d_bg[yp, xp], fx, fy, cx, cy, depth_scale=1) + + x = xp-1 + if x < 0: + x = 0 + y = yp + xl, yl, zl = depth2xyz( + x, y, d_bk[y, x], fx, fy, cx, cy, depth_scale=1) + x = xp+1 + if x >= img_w: + x = img_w-1 + y = yp + xr, yr, zr = depth2xyz( + x, y, d_bk[y, x], fx, fy, cx, cy, depth_scale=1) + x = xp + y = yp-1 + if y < 0: + y = 0 + xt, yt, zt = depth2xyz( + x, y, d_bk[y, x], fx, fy, cx, cy, depth_scale=1) + x = xp + y = yp+1 + if y >= img_h: + y = img_h-1 + xb, yb, zb = depth2xyz( + x, y, d_bk[y, x], fx, fy, cx, cy, depth_scale=1) + + area_a = polygon_area( + [[xt, yt], [xl, yl], [xb, yb], [xr, yr]])/2 + + dz = z0-z1 + dx = z1/fx + dy = z1/fy + area_b = dx*dy*2/2 + + area = (area_a+area_b)/2 # avg get better acc + volume = area*dz + # areas.append(area) + volumes.append(volume) + # points.append((x1, y1, dz)) + # areas = np.array(areas) + volumes = np.array(volumes) + # points = np.array(points) + + res.append("{}:{} cm3".format(i, int(np.sum(volumes)/1000))) + # print(res) + + img_pil = Image.fromarray(output) + draw = ImageDraw.Draw(img_pil) + for i in range(len(max_stats)): + max_stat = max_stats[i] + draw.rectangle([(max_stat[0], max_stat[1]), + (max_stat[0] + max_stat[2], max_stat[1] + max_stat[3])], outline="red") + draw.text((max_stat[0], max_stat[1]), res[i], fill=(255, 255, 255)) + output = np.array(img_pil) + + return output + + +def frame_config_decode(frame_config): + ''' + @frame_config bytes + + @return fields, tuple (trigger_mode, deep_mode, deep_shift, ir_mode, status_mode, status_mask, rgb_mode, rgb_res, expose_time) + ''' + return struct.unpack("> with_config[1] + deepth_img = struct.unpack("<%us" % deepth_size, frame_payload[:deepth_size])[ + 0] if 0 != deepth_size else None + frame_payload = frame_payload[deepth_size:] + + # 0:16bit 1:8bit, resolution: 320*240 + ir_size = (320*240*2) >> with_config[3] + ir_img = struct.unpack("<%us" % ir_size, frame_payload[:ir_size])[ + 0] if 0 != ir_size else None + frame_payload = frame_payload[ir_size:] + + status_size = (320*240//8) * (16 if 0 == with_config[4] else + 2 if 1 == with_config[4] else 8 if 2 == with_config[4] else 1) + status_img = struct.unpack("<%us" % status_size, frame_payload[:status_size])[ + 0] if 0 != status_size else None + frame_payload = frame_payload[status_size:] + + assert(deep_data_size == deepth_size+ir_size+status_size) + + rgb_size = len(frame_payload) + assert(rgb_data_size == rgb_size) + rgb_img = struct.unpack("<%us" % rgb_size, frame_payload[:rgb_size])[ + 0] if 0 != rgb_size else None + + if (not rgb_img is None) and (1 == with_config[6]): + jpeg = cv2.imdecode(np.frombuffer( + rgb_img, 'uint8', rgb_size), cv2.IMREAD_COLOR) + if not jpeg is None: + rgb = cv2.cvtColor(jpeg, cv2.COLOR_BGR2RGB) + rgb_img = rgb.tobytes() + else: + rgb_img = None + + return (deepth_img, ir_img, status_img, rgb_img) + + +def post_encode_config(config=frame_config_encode(), host=HOST, port=PORT): + r = requests.post('http://{}:{}/set_cfg'.format(host, port), config) + if(r.status_code == requests.codes.ok): + return True + return False + + +def get_frame_from_http(host=HOST, port=PORT): + r = requests.get('http://{}:{}/getdeep'.format(host, port)) + if(r.status_code == requests.codes.ok): + # print('Get deep image') + deepimg = r.content + # print('Length={}'.format(len(deepimg))) + (frameid, stamp_msec) = struct.unpack(' 65535)] = 65535 + + # depth = depth_f.astype(depth.dtype) + + # depth[240//2, 320//2 - 5:320//2+5] = 0x00 + # depth[240//2-5:240//2+5, 320//2] = 0x00 + + if not UPDATE_BG[1] is None: + ax1.imshow(cal_volume(depth, UPDATE_BG[1])) + else: + ax1.imshow(depth) + + if UPDATE_BG[0]: + UPDATE_BG[1] = depth + + # ax2 = fig.add_subplot(222) + # if not ir is None: + # ax2.imshow(ir, cmap='gray') + # ax3 = fig.add_subplot(223) + # if not status is None: + # ax3.imshow(status) + ax4 = fig.add_subplot(121) + if not rgb is None: + ax4.imshow(rgb) + + +UPDATE_BG = [False, None] + +if post_encode_config(frame_config_encode(1, 0, 255, 0, 2, 7, 1, 0, 0)): + # 打开交互模式 + def on_key_press(event): + if event.key == ' ': + UPDATE_BG[0] = True + elif event.key == 'c': + UPDATE_BG[1] = None + + plt.ion() + figsize = (12, 12) + fig = plt.figure('2D frame', figsize=figsize) + fig.canvas.mpl_connect('key_press_event', on_key_press) + + print("按下空格键更新背景图，按下c键清空背景图") + while True: + p = get_frame_from_http() + show_frame(fig, p) + if UPDATE_BG[0]: + UPDATE_BG[0] = False + print("update bg success!") + # 停顿时间 + plt.pause(0.001) + # 清除当前画布 + fig.clf() + + plt.ioff() +``` + +## calibratepy + +```python +# -*- coding:utf-8 -*- + +import requests +import struct +import shutil +import logging + + +from PIL import Image +import matplotlib.pyplot as plt +import numpy as np +import cv2 +import os +import json + +# 默认的warning级别，只输出warning以上的 +# 使用basicConfig()来指定日志级别和相关信息 + +logging.basicConfig(level=logging.INFO # 设置日志输出格式 + , filename="cali.log" # log日志输出的文件位置和文件名 + , filemode="w" # 文件的写入格式，w为重新写入文件，默认是追加 + # 日志输出的格式 + # -8表示占位符，让输出左对齐，输出长度都为8位 + # 时间输出的格式 + , format="%(asctime)s - %(name)s - %(levelname)-9s - %(filename)-8s : %(lineno)s line - %(message)s", datefmt="%Y-%m-%d %H:%M:%S" + ) + +logging.info("Starting Cali...") + +HOST = '192.168.233.233' +PORT = 80 + +rgb_dir = 'RGB/' +tof_dir = 'TOF/' + + +logging.info("Clear {} and {} dir...".format(rgb_dir, tof_dir)) + +if os.path.exists(rgb_dir): + shutil.rmtree(rgb_dir) + +if os.path.exists(tof_dir): + shutil.rmtree(tof_dir) + +os.makedirs(rgb_dir) +os.makedirs(tof_dir) + +logging.info("Clear directories success!") + +IMAGECOUNT = 0 + + +IMG_WIDTH = 800 # Do not change this +IMG_HEIGHT = 600 # and this +patternSize = (8, 6) # modifly this for your Chessboard Corners +squareSize = 28.3 # modifly this for chessboard box size (in mm) + + +def frame_config_decode(frame_config): + ''' + @frame_config bytes + + @return fields, tuple (trigger_mode, deep_mode, deep_shift, ir_mode, status_mode, status_mask, rgb_mode, rgb_res, expose_time) + ''' + return struct.unpack("> with_config[1] + deepth_img = struct.unpack("<%us" % deepth_size, frame_payload[:deepth_size])[ + 0] if 0 != deepth_size else None + frame_payload = frame_payload[deepth_size:] + + # 0:16bit 1:8bit, resolution: 320*240 + ir_size = (320*240*2) >> with_config[3] + ir_img = struct.unpack("<%us" % ir_size, frame_payload[:ir_size])[ + 0] if 0 != ir_size else None + frame_payload = frame_payload[ir_size:] + + status_size = (320*240//8) * (16 if 0 == with_config[4] else + 2 if 1 == with_config[4] else 8 if 2 == with_config[4] else 1) + status_img = struct.unpack("<%us" % status_size, frame_payload[:status_size])[ + 0] if 0 != status_size else None + frame_payload = frame_payload[status_size:] + + assert(deep_data_size == deepth_size+ir_size+status_size) + + rgb_size = len(frame_payload) + assert(rgb_data_size == rgb_size) + rgb_img = struct.unpack("<%us" % rgb_size, frame_payload[:rgb_size])[ + 0] if 0 != rgb_size else None + + if (not rgb_img is None) and (1 == with_config[6]): + jpeg = cv2.imdecode(np.frombuffer( + rgb_img, 'uint8', rgb_size), cv2.IMREAD_COLOR) + if not jpeg is None: + rgb = cv2.cvtColor(jpeg, cv2.COLOR_BGR2RGB) + rgb_img = rgb.tobytes() + else: + rgb_img = None + + return (deepth_img, ir_img, status_img, rgb_img) + + +def post_encode_config(config=frame_config_encode(), host=HOST, port=PORT): + r = requests.post('http://{}:{}/set_cfg'.format(host, port), config) + if(r.status_code == requests.codes.ok): + return True + return False + + +def get_frame_from_http(host=HOST, port=PORT): + r = requests.get('http://{}:{}/getdeep'.format(host, port)) + if(r.status_code == requests.codes.ok): + # print('Get deep image') + deepimg = r.content + # print('Length={}'.format(len(deepimg))) + (frameid, stamp_msec) = struct.unpack(' 65535)] = 65535 + + # depth = depth_f.astype(depth.dtype) + + # depth[240//2, 320//2 - 5:320//2+5] = 0x00 + # depth[240//2-5:240//2+5, 320//2] = 0x00 + ax1.imshow(depth, cmap='jet_r') + ax2 = fig.add_subplot(222) + if not ir is None: + ax2.imshow(ir, cmap='gray') + if save_flag: + Image.fromarray(ir).resize((800, 600)).save( + tof_dir+'IR-{}.png'.format(IMAGECOUNT)) + ax3 = fig.add_subplot(223) + if not status is None: + ax3.imshow(status) + ax4 = fig.add_subplot(224) + if not rgb is None: + ax4.imshow(rgb) + if save_flag: + Image.fromarray(rgb).resize((800, 600)).save( + rgb_dir+'RGB-{}.png'.format(IMAGECOUNT)) + + +def main(): + global IMAGECOUNT + if post_encode_config(frame_config_encode(1, 1, 255, 0, 2, 7, 1, 0, 0)): + KEEP_STREAM = [True] + save_once = [False] + + def on_key_press(event): + if event.key == ' ': + save_once[0] = True + elif event.key == 'y': + KEEP_STREAM[0] = False + + # 打开交互模式 + plt.ion() + figsize = (12, 12) + fig = plt.figure('2D frame', figsize=figsize) + fig.canvas.mpl_connect('key_press_event', on_key_press) + + print("按下空格键抓取图片，一次一张，保持稳定后才按，请不要长按连续拍摄，否则程序会识别出错退出，从而需要重新开始！") + print("抓取不同角度图片50张左右，之后按y键程序将进入下一阶段，自动完成等待正常退出即可，图片过少程序也会识别出错退出，从而需要重新开始！") + + while KEEP_STREAM[0]: + p = get_frame_from_http() + + show_frame(fig, p, save_once) + if save_once[0]: + save_once[0] = False + IMAGECOUNT += 1 + tmp = "saved {} pictures".format(IMAGECOUNT) + logging.info(tmp) + print(tmp) + # 停顿时间 + plt.pause(0.001) + # 清除当前画布 + fig.clf() + + plt.ioff() + + rgblist = [a for a in os.listdir( + rgb_dir) if a[-3:] == 'png' or a[-3:] == 'jpg'] + toflist = [a for a in os.listdir( + tof_dir) if a[-3:] == 'png' or a[-3:] == 'jpg'] + if(len(rgblist) != len(toflist)): + print("error pic count not match") + else: + print("there are ", len(rgblist), ' pairs') + # for r,t in zip(rgblist,toflist): + # print(r,t) + rgbimg = [] + tofimg = [] + for r, t in zip(rgblist, toflist): + rimg = cv2.imread(rgb_dir+r) + rimg = cv2.cvtColor(rimg, cv2.COLOR_BGR2RGB) + timg = cv2.imread(tof_dir+t) # , cv2.IMREAD_GRAYSCALE) + rimg = cv2.resize(rimg, (IMG_WIDTH, IMG_HEIGHT)) + timg = cv2.resize(timg, (IMG_WIDTH, IMG_HEIGHT)) + rgbimg.append(rimg) + tofimg.append(timg) + + criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.001) + # pt_found_t,ptst = cv2.findChessboardCorners(tofimg[0], patternSize); + # print(pt_found_t,ptst) + # PLT.figure() + # PLT.imshow(tofimg[0]) + # PLT.show() + reject_nums = 0 + imgp_rgb = [] + imgp_tof = [] + for r, t in zip(rgbimg, tofimg): + pt_found_r, ptsr = cv2.findChessboardCornersSB( + r, patternSize, flags=cv2.CALIB_CB_NORMALIZE_IMAGE) + pt_found_t, ptst = cv2.findChessboardCornersSB( + t, patternSize, flags=cv2.CALIB_CB_NORMALIZE_IMAGE) + if(not (pt_found_r and pt_found_t)): + reject_nums += 1 + # print(pt_found_r,pt_found_t) + continue + ptsr = cv2.cornerSubPix(cv2.cvtColor( + r, cv2.COLOR_RGB2GRAY), ptsr, (3, 3), (-1, -1), criteria) + ptst = cv2.cornerSubPix(cv2.cvtColor( + t, cv2.COLOR_RGB2GRAY), ptst, (3, 3), (-1, -1), criteria) + imgp_rgb.append(ptsr) + imgp_tof.append(ptst) + perview1 = cv2.drawChessboardCorners(r, patternSize, ptsr, pt_found_r) + perview2 = cv2.drawChessboardCorners(t, patternSize, ptst, pt_found_t) + # PLT.figure() + # PLT.imshow(perview1) + # PLT.show() + # PLT.figure() + # PLT.imshow(perview2) + # PLT.show() + tmp = "reject: {}, total: {}".format(reject_nums, len(rgbimg)) + logging.warning(tmp) + print(tmp) + + if reject_nums == len(rgbimg): + print("失败，请重新运行该程序，请保持标定板所有方格一直在画面中!") + return None + + xi = np.linspace(0, patternSize[0]-1, patternSize[0]) + yi = np.linspace(0, patternSize[1]-1, patternSize[1]) + zpos = np.zeros((patternSize[1], patternSize[0])) + boardpos = np.array(np.meshgrid(xi, yi))*squareSize + boardpos = np.concatenate(([boardpos[0]], [boardpos[1]], [zpos])) + boardpos = np.transpose(boardpos, (1, 2, 0)) + # print(boardpos) + cameraMatrix1 = np.zeros((3, 4)) + cameraMatrix2 = np.zeros((3, 4)) + distCoeffs1 = np.zeros((1, 5)) + distCoeffs2 = np.zeros((1, 5)) + R = np.zeros((3, 3)), + T = np.zeros((3, 1)), + # cbs=np.array([boardpos]*len(imgp_rgb)) + # cbs=cbs.reshape((cbs.shape[0],cbs.shape[1]*cbs.shape[2],cbs.shape[3])) + cbs = np.array([boardpos.astype(np.float32)]*len(imgp_rgb)) + cbs = cbs.reshape((cbs.shape[0], cbs.shape[1]*cbs.shape[2], cbs.shape[3])) + # print(cbs) + ret, cameraMatrix1, distCoeffs1, rvecs, tvecs = cv2.calibrateCamera( + cbs, imgp_rgb, (IMG_WIDTH, IMG_HEIGHT), cameraMatrix1, distCoeffs1) + ret, cameraMatrix2, distCoeffs2, rvecs, tvecs = cv2.calibrateCamera( + cbs, imgp_tof, (IMG_WIDTH, IMG_HEIGHT), cameraMatrix2, distCoeffs2) + flags = cv2.CALIB_FIX_INTRINSIC+cv2.CALIB_USE_INTRINSIC_GUESS + \ + cv2.CALIB_FIX_PRINCIPAL_POINT+cv2.CALIB_FIX_FOCAL_LENGTH+cv2.CALIB_FIX_ASPECT_RATIO + ret, K1, D1, K2, D2, R, T, E, F = cv2.stereoCalibrate( + cbs, imgp_rgb, imgp_tof, cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, (IMG_WIDTH, IMG_HEIGHT), flags=flags) + print("ret", ret) + print("K1", K1) + print("D1", D1) + print("K2", K2) + print("D2", D2) + print("R", R) + print("T", T) + print("E", E) + print("F", F) + param = { + "R_Matrix_data": R.reshape(-1).tolist(), + "T_Vec_data": T.reshape(-1).tolist(), + "Camera_Matrix_data": cameraMatrix1.reshape(-1).tolist(), + "Distortion_Parm_data": distCoeffs1.reshape(-1).tolist(), + } + + j = json.dumps(param, indent=4) + logging.info(j) + print(j) + with open('CameraParms.json', 'w') as fp: + fp.write(j) + fp.close() + + +if __name__ == '__main__': + main() +``` diff --git a/docs/hardware/zh/metasense/metasense-a010/metasense-a010.md b/docs/hardware/zh/metasense/metasense-a010/metasense-a010.md index 2fdbffb6..df518603 100644 --- a/docs/hardware/zh/metasense/metasense-a010/metasense-a010.md +++ b/docs/hardware/zh/metasense/metasense-a010/metasense-a010.md @@ -52,11 +52,11 @@ MetaSense-A010 是由 Sipeed 所推出的一款由 BL702 + 炬佑 100x100 TOF **COMTOOL 软件包** Windows 系统连接：下载压缩包后解压安装即可。 -Linux 系统：不提供软件压缩包，需用户自行编译。 +Linux 系统：不提供软件压缩包，需用户自行编译 [获取跳转](https://github.com/sipeed/COMTool)。 注意：Win 7 及以下系统需装驱动，可自行前往 FTDI 官网下载。 ### 上电互动预览 -将设备通电后，可在设备上自带 LCD 屏实时预览 color map 后的深度伪彩图 +将设备通电后，可在设备上自带 LCD 屏实时预览 color map 后的深度伪彩图。 ![010-3](assets/a010-3.jpg) ### PC 预览和微调 @@ -84,45 +84,33 @@ COMTOOL 上位机的配置控件说明 - FPS 设置出图帧率（不宜过高，根据对接设备的性能合理设置即可，减小帧率可以减少传输数据量） - Ev 曝光间隙控制（最左代表 AE，其他是固定曝光时间） +## 案例：远近中物体实拍 +物体摆放距离形成差异，模组通过捕捉到的深度值的差异显示冷暖色调，更直观了解 `TOF` 技术。 +![](./assets/../../assets/tof-1.10.jpg) ## 案例：检测人流 +高精度，大分辨率的实时监测人流走动的情况并统计。 -高精度，大分辨率的实时监测人流走动的情况统计。 - -![a010-14](assets/a010-14.jpg) - - +![msone-people](./assets/ms-people.jpg) ## 案例：键盘灯跟随 - 实现超酷炫的键盘灯跟随，实时跟踪手部的位置，再根据手部的位置映射键盘灯。 -![tof-1.9](./../assets/tof-1.9.jpg) +![](./assets/ms-larm.jpg) ## 案例：接入 MCU - MS-A010 拥有强大的兼容性，可基于串口协议外接 K210 bit 这样的单片机开发板或树莓派之类的 linux 开发板来进行二次开发。 +[MS-A010 外接 K210 bit 源码获取](./metasense-a010/../code.html#tof_mainpy) +![ms-mcu](./assets/ms-mcu.jpg) -![a010-13](asstes/../assets/a010-13.jpg) -获取链接： ## 二次开发：串口协议 可参考上方的案例：**K210 bit 外接 MCU** -| AT                             |                                                                                                                                                                                                                | -| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| +ISP
Image Signal Processor | :0: turn ISP off
=1: turn ISP on                                                                                                                                                                            | -| +BINN
full binning          | =1: output 100x100 pixel frame
=2: output 50x50 pixel frame
=4: output 25x25 pixel frame
                                                                                                            | -| +DISP
display mux           | =0: all off
=1: lcd display on
=2: usb display on
=3: lcd and usb display on
=4: uart display on
=5: lcd and uart display on
=6: usb and uart display on
=7: lcd, usb and uart display on | -| +BAUD
uart baudrate         | =0: 9600
=1: 57600
=2: 115200
=3: 230400
=4: 460800
=5: 921600
=6: 1000000
=7: 2000000
=8: 3000000                                                                                     | -| +UNIT
quantization unit     | =0: auto
=1-10: quantizated by unit(mm)                                                                                                                                                                     | -| +FPS
frame per second       | =1-19: set frame per second                                                                                                                                                                                    | -| +Save
save config           | : save current configuration                                 | - @@ -130,7 +118,7 @@ MS-A010 拥有强大的兼容性，可基于串口协议外接 K210 bit 这样 ### 接入 ROS1 **1. 准备工作** -首先，准备适用的环境：Linux 系统 +首先，准备适用的环境：`Linux` 系统 可使用虚拟机 `virtual box` 或者 `vmware` 也可安装双系统，安装方法请自行查询。 **2. 安装运行** @@ -151,13 +139,13 @@ rosrun sipeed_tof_ms_a010 a010_publisher _device:="/dev/ttyUSB0" **4. RVIZ2 预览** 打开 `rviz2` 后，在界面左下角的 `Add`->`By topic`->`PointCloud2或/depth` ->`Image 添加` ->`Display/Global Options/Fixed Frame` 需要修改成 `tof`，才能正常显示点云，根据添加的内容，左侧会显示 `Image` 而中间则显示点云。 -![a010-8](asstes/../assets/a010-8.jpg) +![ms-rviz](asstes/../assets/ms-RVIZ.jpg) ### 接入 ROS2 **1. 准备工作** -首先，准备适用的环境：Linux 系统 +首先，准备适用的环境：`Linux` 系统 可使用虚拟机 `virtual box` 或者 `vmware` 也可安装双系统，安装方法请自行查询。 **2. 安装运行** @@ -175,9 +163,9 @@ source install/setup.sh **3. RQT 查看帧率** -![a010-9](asstes/../assets/a010-9.jpg) +![ms-rqt](asstes/../asstes/../assets/ms-rqt.jpg) **4. RVIZ2 预览** 打开 `rviz2` 后，在界面左下角的 `Add`->`By topic`->`PointCloud2或/depth` ->`Image 添加` ->`Display/Global Options/Fixed Frame` 需要修改成 `tof`，才能正常显示点云，根据添加的内容，左侧会显示 `Image` 而中间则显示点云。 -![a010-10](assets/a010-10.jpg) +![ms-ros](assets/ms-rqt.jpg) diff --git a/docs/hardware/zh/metasense/metasense-a075v/assets/tof-15.jpg b/docs/hardware/zh/metasense/metasense-a075v/assets/tof-15.jpg new file mode 100644 index 00000000..a21567f5 Binary files /dev/null and b/docs/hardware/zh/metasense/metasense-a075v/assets/tof-15.jpg differ diff --git a/docs/hardware/zh/metasense/metasense-a075v/assets/tof-9.png b/docs/hardware/zh/metasense/metasense-a075v/assets/tof-9.jpg similarity index 100% rename from docs/hardware/zh/metasense/metasense-a075v/assets/tof-9.png rename to docs/hardware/zh/metasense/metasense-a075v/assets/tof-9.jpg diff --git a/docs/hardware/zh/metasense/metasense-a075v/assets/tog-14.jpg b/docs/hardware/zh/metasense/metasense-a075v/assets/tog-14.jpg new file mode 100644 index 00000000..27737d73 Binary files /dev/null and b/docs/hardware/zh/metasense/metasense-a075v/assets/tog-14.jpg differ diff --git a/docs/hardware/zh/metasense/metasense-a075v/metasense-a075v.md b/docs/hardware/zh/metasense/metasense-a075v/metasense-a075v.md index 7d4ff22f..1095b41c 100644 --- a/docs/hardware/zh/metasense/metasense-a075v/metasense-a075v.md +++ b/docs/hardware/zh/metasense/metasense-a075v/metasense-a075v.md @@ -18,7 +18,9 @@ MS-A075V 是由 Sipeed 所推出的一款具有 RGB 功能的 3D TOF 摄像机 ## 产品开箱指南 ### 准备工作 -**接线说明**： +**接线说明** +![tof-07514](asstes/../assets/tog-14.jpg) + **安装驱动**：上电前一定要确保网络环境中没有使用 192.168.233.0/24 的地址段，MS-A075V 会使用 RNDIS 并设置自己的 ip 地址为 http://192.168.233.1 Windows 系统需安装驱动才可正常运行。 @@ -28,7 +30,7 @@ Windows 系统需安装驱动才可正常运行。 ### 网页上位机预览 1. 把设备使用 type-c 线与电脑链接，MS-A075V 的风扇会开始工作，产品正面镜头处就会闪烁红灯。 -2. 此时可打开浏览器输入 http://192.168.233.233 预览 3D 点云图，上电后有延迟需等待一段时间后，系统和程序才会启动完成。 +2. 此时可打开浏览器输入 http://192.168.233.1 预览 3D 点云图，上电后有延迟需等待一段时间后，系统和程序才会启动完成。 3. 使用网页上位机快速预览 演示图（正面和侧面）： @@ -67,6 +69,10 @@ Windows 系统需安装驱动才可正常运行。 - **SavePointCloud**：可保存一帧 3D 点云图，保存格式为 pcd ，同样可以通过上述提供的脚本预览。 注意：raw 数据可通过开放的接口获取，开发者进行解析即可基于此二次开发，但点云（pointcloud）是基于 raw 数据和相机内参进行计算得到的，无相应接口提供。 +## 案例：远中近点云实拍 +高精度的映射物品摆放距离的差异，点云图可直观清楚感受到更真实的可视化。 +![tof-a07514](assets/tof-15.jpg) + ## 案例：避障小车 模组可搭载小车或无人机来回移动获取障碍物的远近深度值，并通过差异判断画面中是否有障碍物，做出快速反应并精准规避障碍物（例程暂未开源，待整理公开)。 @@ -93,10 +99,11 @@ Windows 系统需安装驱动才可正常运行。 ### 解包推流 理解了上述 `python SDK` 数据获取和解码的逻辑后，我们可以尝试进阶版，连续获取解码并调用第三方 `python` 图像库，例如：matplotlib 进行实时显示。而 `toturial.py` 给出了获取一帧数据的逻辑实现，通过 plt 显示并外套循环即可做到实时显示。 -**解包推流**： +**解包推流**：`python stream.py` [点我查看stream.py内容](./../metasense-a010/code.html#streampy) **使用方式**：装好所有的依赖包后即可 `python stream.py` 运行。 ![tof-a0755](assets/tof-5.jpg) +[233](./../metasense-a010/code.html#calibratepy) ### 检测体积 基于第三方 `python` 包，理解了上述数据获取和解码的逻辑后，再次进阶，不但持续显示多帧并且再通过 SDK 获取相机内参后计算出初略的点云，做累加得到总体积。限制：要求俯视图可以看到除底面外的所有细节 @@ -111,7 +118,7 @@ Windows 系统需安装驱动才可正常运行。 ### 接入 ROS1 **1. 准备工作** -首先，准备适用的环境：Linux 系统 +首先，准备适用的环境：`Linux` 系统 可使用虚拟机 `virtual box` 或者 `vmware` 也可安装双系统，安装方法请自行查询。 **2. 安装运行** @@ -136,7 +143,7 @@ rosrun sipeed_tof_cpp publisher ### 接入 ROS2 **1. 准备工作** -首先，准备适用的环境：Linux 系统 +首先，准备适用的环境：`Linux` 系统 可使用虚拟机 `virtual box` 或者 `vmware` 也可安装双系统，安装方法请自行查询。 **2. 安装运行** diff --git a/docs/hardware/zh/metasense/readme.md b/docs/hardware/zh/metasense/readme.md index 0cd293a0..e8f03b40 100644 --- a/docs/hardware/zh/metasense/readme.md +++ b/docs/hardware/zh/metasense/readme.md @@ -4,9 +4,10 @@ title: MetaSense 系列 ## MetaSense 是什么? -MetaSense 系列摄像头模组分为 MetaSense-A075V 和 MetaSense-A010 两款产品。 -- MS-A010 是由 Sipeed 所推出的一款由 BL702 + 炬佑 100x100 TOF 模组所组成的极致性价比的 TOF 3D 传感器模组，最大支持 100x100 的分辨率和 8 位精度，并且带有 240×135 的 LCD 显示屏可实时预览 color map 后的深度图。 -- 而 MS-A075V 是一款具有 RGB 功能的 3D TOF 摄像机模组，该模组可以实现免驱的即插即用，实现实时彩色 3D 显示。 +MetaSense 系列产品搭载 TOF 深度摄像头，目前主要有 MetaSense-A010 和 MetaSense-A075VS 两款产品。 +- MS-A010 是一款由 BL702 + 炬佑 100x100 TOF 模组所组成的极致性价比的 TOF 3D 传感器模组，最大支持 100x100 的分辨率和 8 位精度，并且带有 240×135 的 LCD 显示屏可实时预览 color map 后的深度图。 +- 而 MS-A075V 是一款具有 RGB 功能的 3D TOF 摄像机模组，该模组可以实现 Linux 免驱的即插即用，实现实时彩色 3D 显示。 + summary @@ -15,7 +16,7 @@ MetaSense 系列摄像头模组分为 MetaSense-A075V 和 MetaSense-A010 两款 -|              |

MateSense-010

|

MateSense-A075V

| +|              |

MateSense-A010

|

MateSense-A075V

| | :----------- |:----------------------------------------------- | :------------------------------------------------- | |              |![tof-1.3](./assets/tof-1.3.jpg)                | ![tof-1.2](./assets/tof-1.2.jpg)                  | | 接口         | 1.25mm 串口连接器 \*1
Type-C USB2.0 \*1       | 1.25mm 串口连接器 \*1
Type-C USB2.0 \*1         | @@ -26,7 +27,6 @@ MetaSense 系列摄像头模组分为 MetaSense-A075V 和 MetaSense-A010 两款 | 测量范围     |0.2-2.5m                                         | 0.15-1.5m                                          | | 测量精度     |<=1%/cm                                       | <=1%/cm                                         | -相关交流社区及社群： @@ -34,56 +34,49 @@ MetaSense 系列摄像头模组分为 MetaSense-A075V 和 MetaSense-A010 两款 ### 案例：远中近物体实拍 - +高精度的映射物品摆放距离的差异，点云图可直观感受到更真实的可视化， ![tof-1.10](./assets/tof-1.10.jpg) ### 案例：人流统计 +可高精度、大分辨率的实时监测人流走动的情况并统计。 -高精度，大分辨率的实时监测人流走动的情况统计。 - -![tof-1.5](./assets/tof-1.5.jpg) +![ms-people](./metasense-a010/assets/ms-people.jpg) ### 案例：小车避障 - 可搭载于小车移动并判断画面是否有障碍物，模组自带 LCD 屏幕精准显示距离并做出反应规避障碍物。 ![tof-a0756](assets/../metasense-a075v/assets/tof-6.jpg) ### 案例：键盘灯跟随 - 实现超酷炫的键盘灯跟随，实时跟踪手部的位置，再根据手部的位置映射键盘灯。 -![tof-1.9](./assets/tof-1.9.jpg) +![,s-lamp](./assets/../metasense-a010/assets/ms-lamp.jpg) ### 案例：体积测量 - 通过 SDK 获取到的相机内参后计算出初略的点云，做累加得到全部的总体积，达到体积测量的效果。 - ![tof-a0757](assets/../metasense-a075v/assets/tof-7.png) -### 案例：外接 MCU +### [案例：外接 MCU](./metasense-a010/code.html#tof_mainpy) -MS-A010 拥有强大的兼容性，基于串口协议，可外接 K210 bit 这样的单片机开发板或树莓派之类的 linux 开发板来进行二次开发 - -![a010-13](assets/../metasense-a010/assets/a010-13.jpg) +MS-A010 拥有强大的兼容性，基于串口协议的数据传输，可外接 K210 bit 这样的单片机开发板或树莓派之类的 linux 开发板来进行二次开发。 +![a010-13](assets/../metasense-a010/assets/ms-mcu.jpg) ### 案例：接入 ROS1 + ROS2 - 双支持 ROS 系统，开放 ROS1+ROS2 接入功能包，可快速获得深度数据及深度图。 - - + + ## 快速了解 TOF 技术 1. TOF: Time of flight（飞行时间）它是一种测距的方法，通过测量超声波/微波/光等信号在发射器和反射器之间的“飞行时间”来计算出两者之间的距离。能够实现 TOF 测距的传感器就是 TOF 传感器。种类较多，使用较多的是通过红外或者激光进行测距的 TOF 传感器。 -2. RGBD：RGB 摄像头加 TOF 深度摄像头组成的一整个模组。图源自[百度](https://baike.baidu.com/item/TOF/19952376?fr=aladdin)。 -![tof-1.1](./assets/tof-1.1.jpg) +2. 物体摆放距离形成差异，模组通过捕捉到的深度值的差异显示冷暖色调，冷暖色随着对距离的映射发生变化，距离越近色调呈暖调（橘红）而越远色调呈冷调（蓝色） + ## 更多 -关于 MS-010 更详细的资料获取： -关于 MS-075 更详细的资料获取： \ No newline at end of file +关于 MS-010 更详细的资料获取：[点击跳转](https://wiki.sipeed.com/hardware/zh/metasense/metasense-a010/metasense-a010.html) +关于 MS-075 更详细的资料获取：[点击跳转](https://wiki.sipeed.com/hardware/zh/metasense/metasense-a075v/metasense-a075v.html) \ No newline at end of file diff --git a/pages/index/en/metasense.md b/pages/index/en/metasense.md new file mode 100644 index 00000000..070061ae --- /dev/null +++ b/pages/index/en/metasense.md @@ -0,0 +1,4 @@ +--- +layout: redirect +redirect_url: https://wiki.sipeed.com/hardware/en/metasense/index.html +--- \ No newline at end of file diff --git a/pages/index/en/metasense10.md b/pages/index/en/metasense10.md new file mode 100644 index 00000000..788d10d8 --- /dev/null +++ b/pages/index/en/metasense10.md @@ -0,0 +1,4 @@ +--- +layout: redirect +redirect_url: https://wiki.sipeed.com/hardware/en/metasense/metasense-a010/metasense-a010.html +--- \ No newline at end of file diff --git a/pages/index/en/metasense75.md b/pages/index/en/metasense75.md new file mode 100644 index 00000000..29734764 --- /dev/null +++ b/pages/index/en/metasense75.md @@ -0,0 +1,4 @@ +--- +layout: redirect +redirect_url: https://wiki.sipeed.com/hardware/en/metasense/metasense-a075v/metasense-a075v.html +--- \ No newline at end of file