在 r 中解码声纳二进制数据

【问题标题】：Decode sonar binary data in r在 r 中解码声纳二进制数据
【发布时间】：2018-09-11 16:53:28
【问题描述】：

作为一名水文学家而不是程序员，我一直在解决简单的任务——从 LOWRENCE Fish Finder 中提取数据。它以二进制格式SL2 写入轨迹、水深、温度等。这是一个 144 字节的长度，带有 10 字节的标头。有一堆用不同语言编写的解码实现（例如Java 或Ruby）。

基于wiki、Ruby code 和堆栈溢出，我试图从this example file 中至少提取深度。在.csv 中可以找到here。

我知道，depth 是从 60 开始的 4 字节浮点数。但是，this answer 建议的代码返回 zero：

 # Open binary file
 toread <-  file("Chart 09_07_2018 [0].sl2", "rb")
 # all data
 alldata <- readBin(toread, raw(), n = 144, size = 1, endian = "little")
 # read WaterDepth
 readBin(alldata[59:65], double(), size = 4) 
> [1] 0
 close(toread)

如果我使用seek 函数，同样的回报：

con <-  file("Chart 09_07_2018 [2].sl2", "rb")
seek(con, 60L)
readBin(con, double(), endian="little")
close(con)

文件中的数据存在——我可以通过Sonar Viewer (see .csv example) 提取它。

【问题讨论】：

    标签： r binary seek


    【解决方案1】：

    制作 javascript/node.js 版本的小伙子修复了字段转录中的一些错误，值得一两奖。

    以下内容应该很容易理解，但如果有任何需要'splainin.您需要处理经度/纬度编码（等）。

    请注意，您可以跳过阅读函数源代码而直接执行：

    devtools::install_git("https://gitlab.com/hrbrmstr/arabia")

library(arabia) # b/c I like puns way too much

read_sl2("your-sl2-file.sl2")

    您可以在此处或此处查看源代码：

    read_sl2 <- function(path, verbose=TRUE) {

  f <- file(path.expand(path), "rb")
  dat <- readBin(f, "raw", n = file.size(path.expand(path)), endian="little")
  close(f)

  # read in the header
  header <- readBin(dat, what = "raw", n = 10)

  format <- readBin(header[1:2], "int", size=2, endian="little", signed=FALSE)

  if (!(format %in% 1:3)) stop("Invalid 'format' in header; Likely not an slg/sl2/sl3 file")

  ok_formats <- c("slg", "sl2", "sl3")
  if (verbose) message("Format: ", ok_formats[format])

  version <- readBin(header[3:4], "int", size=2, endian="little", signed=FALSE)
  blockSize <- readBin(header[5:6], "int", size=2, endian="little", signed=FALSE)

  if (blockSize == 1970) {
    if (verbose) message("Block size: downscan")
  } else if (blockSize == 3200) {
    if (verbose) message("Block size: sidescan")
  } else {
    stop("Block size is not 'downscan' or 'sidescan'; Likely not an slg/sl2/sl3 file")
  }

  alwaysZero <- readBin(header[7:8], "int", size=2, endian="little", signed=FALSE)

  # yep, we're going to build a list the hard/slow way
  sl2_lst <- vector("list")
  idx <- 1
  pos <- 8 # keeping track of our place in the stream

  while (pos < length(dat)) {

    # if verbose mode echo a "." every 100 records
    if (verbose && ((idx %% 100) == 0)) cat(".")

    blockSize <- readBin(dat[(pos+29):(pos+30)], "int", size=2, endian="little", signed=FALSE)
    prevBlockSize <- readBin(dat[(pos+31):(pos+32)], "int", size=2, endian="little", signed=FALSE)
    packetSize <- readBin(dat[(pos+35):(pos+36)], "int", size=2, endian="little", signed=FALSE)
    frameIndex <- readBin(dat[(pos+37):(pos+40)], "int", size=4, endian="little")

    dplyr::data_frame(
      channel = readBin(dat[(pos+33):(pos+34)], "int", size=2,endian="little", signed=FALSE),
      upperLimit = readBin(dat[(pos+41):(pos+44)], "double", size=4, endian="little"),
      lowerLimit = readBin(dat[(pos+45):(pos+48)], "double", size=4, endian="little"),
      frequency = readBin(dat[(pos+51)], "int", size=1, endian="little", signed=FALSE),
      waterDepth = readBin(dat[(pos+65):(pos+68)], "double", size=4, endian="little"),
      keelDepth = readBin(dat[(pos+69):(pos+72)], "double", size=4, endian="little"),
      speedGps = readBin(dat[(pos+101):(pos+104)], "double", size=4, endian="little"),
      temperature = readBin(dat[(pos+105):(pos+108)], "double", size=4, endian="little"),
      lng_enc = readBin(dat[(pos+109):(pos+112)], "integer", size=4, endian="little"),
      lat_enc = readBin(dat[(pos+113):(pos+116)], "integer", size=4, endian="little"),
      speedWater = readBin(dat[(pos+117):(pos+120)], "double", size=4, endian="little"),
      track = readBin(dat[(pos+121):(pos+124)], "double", size=4, endian="little"),
      altitude = readBin(dat[(pos+125):(pos+128)], "double", size=4, endian="little"),
      heading = readBin(dat[(pos+129):(pos+132)], "double", size=4, endian="little"),
      timeOffset = readBin(dat[(pos+141):(pos+144)], "integer", size=4, endian="little"),
      flags = list(
        dat[(pos+133):(pos+134)] %>%
          rawToBits() %>%
          as.logical() %>%
          set_names(
            c(
              "headingValid", "altitudeValid", sprintf("unk%d", 1:7),
              "gpsSpeedValid", "waterTempValid", "unk8", "positionValid",
              "unk9", "waterSpeedValid", "trackValid"
            )
          ) %>%
          .[c(1:2, 10:11, 13, 15:16)] %>%
          as.list() %>%
          purrr::flatten_df()
      )
    ) -> sl2_lst[[idx]]

    idx <- idx + 1

    pos <- pos + (packetSize+145-1)

  }

  if (verbose) cat("\n")

  dplyr::bind_rows(sl2_lst) %>%
    dplyr::mutate(
      channel = dplyr::case_when(
        channel == 0 ~ "Primary",
        channel == 1 ~ "Secondary",
        channel == 2 ~ "DSI (Downscan)",
        channel == 3 ~ "Left (Sidescan)",
        channel == 4 ~ "Right (Sidescan)",
        channel == 5 ~ "Composite",
        TRUE ~ "Other/invalid"
      )
    ) %>%
    dplyr::mutate(
      frequency = dplyr::case_when(
        frequency == 0 ~ "200 KHz",
        frequency == 1 ~ "50 KHz",
        frequency == 2 ~ "83 KHz",
        frequency == 4 ~ "800 KHz",
        frequency == 5 ~ "38 KHz",
        frequency == 6 ~ "28 KHz",
        frequency == 7 ~ "130-210 KHz",
        frequency == 8 ~ "90-150 KHz",
        frequency == 9 ~ "40-60 KHz",
        frequency == 10~ "25-45 KHz",
        TRUE ~ "Other/invalid"
      )
    ) %>%
    tidyr::unnest(flags)

}

    试一试：

    xdf <- read_sl2("~/Downloads/Chart 09_07_2018 [2].sl2")
## Format: sl2
## Block size: downscan
## .............

xdf
## # A tibble: 1,308 x 22
##    channel      upperLimit lowerLimit frequency waterDepth keelDepth speedGps temperature lng_enc lat_enc
##    <chr>             <dbl>      <dbl> <chr>          <dbl>     <dbl>    <dbl>       <dbl>   <int>   <int>
##  1 Secondary             0       13.3 200 KHz         2.62     0.328      0.5        15.8 4433307 7003054
##  2 DSI (Downsc…          0       13.4 200 KHz         2.62     0.328      0.5        15.8 4433307 7003054
##  3 Primary               0       13.3 200 KHz         2.62     0.328      0.5        15.9 4433307 7003054
##  4 Secondary             0       13.3 200 KHz         2.62     0.328      0.5        15.9 4433307 7003054
##  5 DSI (Downsc…          0       13.4 200 KHz         2.59     0.328      0          15.8 4433307 7003054
##  6 Secondary             0       13.3 200 KHz         2.59     0.328      0          15.8 4433307 7003054
##  7 Secondary             0       13.3 200 KHz         2.52     0.328      0          15.9 4433307 7003054
##  8 DSI (Downsc…          0       13.4 200 KHz         2.52     0.328      0          15.9 4433307 7003054
##  9 Primary               0       13.3 200 KHz         2.52     0.328      0          15.8 4433307 7003054
## 10 DSI (Downsc…          0       13.4 200 KHz         2.52     0.328      0          15.8 4433307 7003054
## # ... with 1,298 more rows, and 12 more variables: speedWater <dbl>, track <dbl>, altitude <dbl>,
## #   heading <dbl>, timeOffset <int>, headingValid <lgl>, altitudeValid <lgl>, gpsSpeedValid <lgl>,
## #   waterTempValid <lgl>, positionValid <lgl>, waterSpeedValid <lgl>, trackValid <lgl>

    再看一眼：

    glimpse(xdf)
## Observations: 1,308
## Variables: 22
## $ channel         <chr> "Secondary", "DSI (Downscan)", "Primary", "Secondary", "DSI (Downscan)", "Sec...
## $ upperLimit      <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ lowerLimit      <dbl> 13.3, 13.4, 13.3, 13.3, 13.4, 13.3, 13.3, 13.4, 13.3, 13.4, 13.3, 13.4, 13.3,...
## $ frequency       <chr> "200 KHz", "200 KHz", "200 KHz", "200 KHz", "200 KHz", "200 KHz", "200 KHz", ...
## $ waterDepth      <dbl> 2.620, 2.620, 2.620, 2.620, 2.586, 2.586, 2.516, 2.516, 2.516, 2.516, 2.516, ...
## $ keelDepth       <dbl> 0.328084, 0.328084, 0.328084, 0.328084, 0.328084, 0.328084, 0.328084, 0.32808...
## $ speedGps        <dbl> 0.5, 0.5, 0.5, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0....
## $ temperature     <dbl> 15.84112, 15.84112, 15.86293, 15.86293, 15.79128, 15.79128, 15.86293, 15.8629...
## $ lng_enc         <int> 4433307, 4433307, 4433307, 4433307, 4433307, 4433307, 4433307, 4433307, 44333...
## $ lat_enc         <int> 7003054, 7003054, 7003054, 7003054, 7003054, 7003054, 7003054, 7003054, 70030...
## $ speedWater      <dbl> 0.5, 0.5, 0.5, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0....
## $ track           <dbl> 4.974188, 4.974188, 4.974188, 4.974188, 4.974188, 4.974188, 4.974188, 4.97418...
## $ altitude        <dbl> 324.7375, 324.7375, 324.7375, 324.7375, 324.8687, 324.8687, 324.8687, 324.868...
## $ heading         <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ timeOffset      <int> 1317703, 1317706, 1318036, 1318905, 1318946, 1318982, 1319130, 1319140, 13192...
## $ headingValid    <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ...
## $ altitudeValid   <lgl> TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, FAL...
## $ gpsSpeedValid   <lgl> TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, FAL...
## $ waterTempValid  <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ...
## $ positionValid   <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ...
## $ waterSpeedValid <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ...
## $ trackValid      <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ...

    您可能想与您创建的 CSV 进行比较，因为我只是观察了一下。请注意，当它们在标志中标记为“无效”时，您似乎使用了“结转”值的导出器。我只是把它们留在里面。

    如果您不是“tidyverse”那种人，那么“un-tidyverse”应该很容易（留给提问者的练习）。

    【讨论】：

    • 这太棒了！太感谢了。 read_sl2 工作正常，缓慢但很好。读取示例 sl2 需要 18 秒，读取 18 Mb sl2 文件需要 10 分钟 :(( 我不知道为什么我无法在我的 Windows 机器上的 R 3.5.1 下安装 arabia 包。安装甚至在rstudio.cloud 中都失败了。warning in system(cmd) : 'make' not found 都失败了。你知道为什么吗？
    • 这似乎是一个RTools 依赖问题，但find_rtools() 返回TRUE...
    • 我不再需要在 arabia 包中进行编译，并且还大大加快了基础 R 版本的速度。我也将更快的基本 R 版本功能复制到了答案中。我忘记将代码从不必要地复制大量数据切换到仅移动流位置指针。有机会就试试吧。
    • 超级！我仍然无法安装 arabia 软件包，但加快了基本 R read_sl2 的工作完美：18Mb sl2 文件需要 16 秒！
    • 没什么可添加的，但很高兴看到有人发现我的 nodejs 代码很有用。
    猜你喜欢
    • 1970-01-01
    • 2021-08-03
    • 2014-08-09
    • 1970-01-01
    • 2021-12-05
    • 2011-11-02
    • 1970-01-01
    • 2021-06-11
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode