【问题标题】:Why is the endian reversed after sending over TCP为什么通过 TCP 发送后字节序反转
【发布时间】:2018-03-23 12:22:35
【问题描述】:

我有一个用 C# 编写的客户端和一个用 python 编写的服务器。我通过套接字发送的消息是 8 个字节,后面是数据,这 8 个字节是数据长度。

在发送之前,我在C#中将8字节的数据长度转换为太大的字节序,如下所示:

public void Send(SSLMsg m)
{
    string json = m.Serialize();
    byte[] data = Encoding.ASCII.GetBytes(json);
    ulong dataLen = (ulong)data.Length;
    byte[] dataLenPacked = packIt(dataLen);

    Log("Sending " + dataLen + " " + json);

    sslStream.Write(dataLenPacked);
    sslStream.Write(data);
    sslStream.Flush();

}

private byte[] packIt(ulong n)
{
    byte[] bArr = BitConverter.GetBytes(n);
    if (BitConverter.IsLittleEndian)
        Array.Reverse(bArr, 0, 8);
    return bArr;
}

消息发送成功,我被python服务器代码捆绑了,因为这里的解包格式应该是正确的,不是吗?

(length,) = unpack('>Q', data)
# len(data) is 8 here
# length is 1658170187863248538

大端字符不是'>'吗?为什么我的长度这么长?

更新:

有一个错误,我解压了错误的 8 个字节,现已修复,现在我解压正确的数据我仍然有同样的问题。

(length,) = unpack('>Q', data)
# len(data) is 8 here
# length is 13330654897016668160L

只有当我使用 little endian 解包时才给出正确的长度,即使我使用 big-endian 将字节发送到服务器......所以我期待 >Q 工作,但是

(length,) = unpack('<Q', data)
# len(data) is 8 here
# length is 185

这是我在 python 中接收字节的方式:

while (True):
    r,w,e = select.select(...)
    for c in r:
        if (c == socket):
            connection_accept(c)
        else
           # c is SSL wrapped  at this point
           read = 0
           data = []
           while (read != 8):
               bytes = c.recv(min(8-read, 8))
               read += len(bytes)
               data.append(bytes)
           joinedData = ''.join(data)

           # the below length is 13330654897016668160L
           # I am expecting it to be 185
           (length,) = unpack('>Q', joinedData)

           # the below length is 185, it should not be however
           # since the bytes were sent in big-endian
           (length,) = unpack('<Q', joinedData)

【问题讨论】:

  • 有必要获取有关您在客户端和服务器端使用的应用程序堆栈的信息,以创建问题的工作重现,然后解决它。从我的角度来看,这可能是网络/应用程序堆栈中某处的字节序的双重转换。

标签: c# python sockets


【解决方案1】:

您的代码有问题:

长度为1658170187863248538

这是十六进制的 1703010020BB4E9A。这与长度 8 无关,无论涉及哪种字节序。相反,它看起来有点像 TLS 记录:

 17    - record type application data (decimal 23)
 03 01 - protocol version TLS 1.0 (aka SSL 3.1)
 00 20 - length of the following encrypted data (32 byte)
 .. 

由于根据您的代码,您正在执行 SSL,因此您的接收器可能有问题。我的猜测是您从普通套接字而不是 SSL 套接字读取,因此读取的是加密数据而不是解密数据。

【讨论】:

  • 是的,有一个类似的错误,现在已修复。使用 >Q 时长度现在为 13330654897016668160,但是当我使用
  • @user740521:请不要让您的问题成为移动目标。最初发现读取错误数据的问题。现在,您刚刚更改了问题以提供不同且不一致的值,您声称您的实际长度为 13330654897016668160,实际长度为 8,即使实际长度为 185,正如您在评论中正确显示但不是问题。
【解决方案2】:

根据https://msdn.microsoft.com/en-us/library/z78xtwts(v=vs.110).aspx,当您调用时,您正在反转 9 个字节:

if (BitConverter.IsLittleEndian)
    Array.Reverse(bArr, 0, 8);

根据https://www.displayfusion.com/Discussions/View/converting-c-data-types-to-c/?ID=38db6001-45e5-41a3-ab39-8004450204b3,C# 中的 ulong 只有 8 个字节。

我认为这不一定是一个答案,但也许这是一个线索?

【讨论】:

    【解决方案3】:

    在客户端,当您将数据写入流时,您会执行两次写入调用:

    sslStream.Write(dataLenPacked);
    sslStream.Write(data);
    sslStream.Flush();
    

    MSDN 关于NetworkStream.Write 的说法:Write 方法一直阻塞,直到发送了请求的字节数或抛出了 SocketException。 在服务器端,不能保证您会收到所有字节在一次接听电话中 - 它取决于操作系统、eth 驱动程序/配置等。因此,您必须处理这种情况。正如我所看到的,您通过读取 8 个或更少的字节来处理它,但socket.recv 说,最好以更大的部分接收。这是我在 Python 上实现的服务器。它在当前文件夹中创建带有接收字节的二进制文件 - 可能有助于分析问题所在。设置监听端口需要使用 -p/--port 参数:

    #!/usr/bin/env python
    
    import sys, socket, io
    import argparse
    import struct
    
    CHUNK_SIZE = 4096
    
    def read_payload(connection, payload_len):
        recv_bytes = 0
        total_data = ""
        while (recv_bytes < payload_len):
            data = connection.recv(CHUNK_SIZE)
            if not data:
                break
            total_data += data
            recv_bytes += len(data)
        if len(total_data) != payload_len:
            print >> sys.stderr, "-ERROR. Expected to read {0} bytes, but have read {0} bytes\n".format(payload_len, len(total_data))
        return total_data
    
    def handle_connection(connection, addr):
        total_received = 0
        addrAsStr = "{0}:{1}".format(addr[0], addr[1])
    
        # write receved bytes to file for analyzis
        filename = "{0}_{1}.bin".format(addr[0], addr[1])
        file = io.FileIO(filename, "w")
        print "Connection from {0}".format(addrAsStr)
    
        try:
            # loop for handling data transfering for particular connection
            while True:
                header = connection.recv(CHUNK_SIZE)
                header_len = len(header)
                total_received += header_len
                if header_len == 0:
                    break
                if header_len < 8:
                    print >> sys.stderr, "-ERROR. Received header with len {0} less than 8 bytes!\n".format(header_len)
                    break
    
                print("Header len is {0} bytes".format(len(header)))
    
                # extract payload length - it's first 8 bytes
                real_header = header[0:8]
                file.write(real_header)
    
                # more about unpack - https://docs.python.org/3/library/struct.html#module-struct
                # Byte order - network (= big-endian), type - unsigned long long (8 bytes)
                payload_len = struct.unpack("!Q", real_header)[0]
                print("Payload len is {0} bytes".format(payload_len))
    
                # extract payload from header
                payload_in_header = header[8:] if header_len > 8 else ""
                if len(payload_in_header) > 0:
                    print "Payload len in header is {0} bytes".format(len(payload_in_header))
                    file.write(payload_in_header)
    
                # calculate remains
                remains_payload_len = payload_len - len(payload_in_header)
                remains_payload = read_payload(connection, remains_payload_len)
                payload = payload_in_header + remains_payload
                print("Payload is '{0}'".format(payload))
    
                if remains_payload:
                    file.write(remains_payload)
                else:
                    break
                total_received += len(remains_payload)
        finally:
                file.close()
    
        return total_received
    
    def main():
        parser = argparse.ArgumentParser()
        parser.add_argument('-p', '--port', required=True)
        args = parser.parse_args()
    
        # listen tcp socket on all interfaces
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.bind(("0.0.0.0", int(args.port)))
        s.listen(1)
    
        # loop for handling incoming connection
        while True:
            print "Waiting for a connection..."
            (connection, addr) = s.accept()
            addrAsStr = "{0}:{1}".format(addr[0], addr[1])
    
            try:
                total_received = handle_connection(connection, addr)
                print "Handled connection from {0}. Received: {1} bytes\n".format(addrAsStr, total_received)
            finally:
                # Clean up the connection
                connection.close()
    
    if __name__ == "__main__":
        main()
    

    为了让这个例子更完整,这里是 C# 客户端。它使用一个外部库 - Newtonsoft.Json 进行序列化:

    using Newtonsoft.Json;
    using System;
    using System.Net;
    using System.Net.Sockets;
    using System.Text;
    using System.Threading;
    
    namespace SimpleTcpClient
    {
        class SimpleTcpClient : IDisposable
        {
            readonly TcpClient _client;
    
            public SimpleTcpClient(string host, int port)
            {
                _client = new TcpClient(host, port);
            }
            public void Send(byte[] payload)
            {
                // Get network order of array length
                ulong length = (ulong)IPAddress.HostToNetworkOrder(payload.LongLength);
                var stream = _client.GetStream();
                // Write length
                stream.Write(BitConverter.GetBytes(length), 0, sizeof(long));
                // Write payload
                stream.Write(payload, 0, payload.Length);
                stream.Flush();
                Console.WriteLine("Have sent {0} bytes", sizeof(long) + payload.Length);
            }
            public void Dispose()
            {
                try { _client.Close(); }
                catch { }
            }
        }
        class Program
        {
            class DTO
            {
                public string Name { get; set; }
                public int Age { get; set; }
                public double Weight { get; set; }
                public double Height { get; set; }
                public string RawBase64 { get; set; }
            }
    
            static void Main(string[] args)
            {
                // Set server name/ip-address
                string server = "192.168.1.101";
                // Set server port
                int port = 8080;
                string[] someNames = new string[]
                {
                    "James", "David",    "Christopher",  "George",   "Ronald",
                    "John", "Richard",  "Daniel",   "Kennet",  "Anthony",
                    "Robert","Charles", "Paul", "Steven",   "Kevin",
                    "Michae", "Joseph", "Mark", "Edward",   "Jason",
                    "Willia", "Thomas", "Donald",   "Brian",    "Jeff"
                };
                // Init random generator
                Random rnd = new Random(Environment.TickCount);
                int i = 1;
                while (true) {
                    try {
                        using (var c = new SimpleTcpClient(server, port)) {
                            byte[] rawData = new byte[rnd.Next(16, 129)];
                            rnd.NextBytes(rawData);
    
                            // Create random data transfer object
                            var d = new DTO() {
                                Name = someNames[rnd.Next(0, someNames.Length)],
                                Age = rnd.Next(10, 101),
                                Weight = rnd.Next(70, 101),
                                Height = rnd.Next(165, 200),
                                RawBase64 = Convert.ToBase64String(rawData)
                            };
    
                            // UTF-8 doesn't have endianness - so we can convert it to byte array and send it
                            // More about it - https://stackoverflow.com/questions/3833693/isn-t-on-big-endian-machines-utf-8s-byte-order-different-than-on-little-endian 
                            var bytes = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(d));
                            c.Send(bytes);
                        }
                    }
                    catch (Exception ex) {
                        Console.WriteLine("Get exception when send: {0}\n", ex);
                    }
                    Thread.Sleep(200);
                    i++;
                }
            }
        }
    }
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2018-06-13
      • 2023-03-23
      相关资源
      最近更新 更多