Re: pthreads question

[Malcolm Kavalsky]

Nadav Har'El wrote:

[EMAIL PROTECTED]">

On Sat, Mar 16, 2002, Malcolm Kavalsky wrote about "Re: pthreads question":

Before you attack me, try the following test:

1. Malloc a 100 Mb buffer, fill it with random data
2. Send the buffer over a Unix socket to another process
3. Time how long it takes to send the data ...

The result will be, that it takes almost nothing. Certainly faster than 
memory speed.
This is because the data is not actually copied, but the OS does clever 
pointer manipulation,
copy on write, etc to speed stuff like this up.

I didn't try this, but I'll bet you're wrong even without trying ;)
In Linux, sending on sockets (and it doesn't really matter what type)
is *not* zero-copy. It can't be, if you consider the fact that write()
takes one process's buffer and the other process's read() takes that
processes buffer - so the kernel MUST copy data from one buffer to another.
No amount of pointer manipulation in the world can do anything against that -
they'd need to change the socket API to do that... (the sendfile() system
call in Linux 2.4 is a start, but not relevant to this discussion).
In reality more copies may be (and probably are) done inside the kernel.

I attach a program benchmark.c that compares speed of memcpy versus data transfer
over unix sockets.

Compile:     gcc -o benchmark benchmark.c
Run:           ./benchmark
Output: (On my humble 667Mhz pentium III)

Memcpy'ed 2000 blocks of size 1048576 in 18 seconds => 111 Mbytes/second
Sent 2000 blocks of size 1048576 in 4 seconds over unix socket => 500 Mbytes/second


(If you run 'top' in a window, you will see two processes spawned when the socket benchmark
 is being run)

Malcolm

 pthread_exit(NULL)

#include <stdio.h>
#include <malloc.h>
#include <string.h>
#include <time.h>
#include <sys/socket.h>
#include <sys/un.h>

#define BUFSIZE 0x100000  /* 1 Megabyte */
#define NBLOCKS   2000
#define PORT_NAME    "/tmp/foo"

socket_benchmark()
{
  if ( fork() == 0 ) {
    server();
  } else {
    sleep(1); /* Dirty, but ensures client runs after server is ready */
    client();
  }
}

server()
{
  struct sockaddr_un sin,from;
  int s,g,len;
  char *buf;
  
  buf = malloc( BUFSIZE );
  /* Create an unbound socket */
  if( (s=socket( PF_UNIX, SOCK_STREAM, 0 )) < 0 ){
    printf( "Bad socket\n");
    return 0;
  }
  strcpy( sin.sun_path, PORT_NAME );
  sin.sun_family = PF_UNIX;
  if( bind( s, (struct sockaddr *)&sin, 
            strlen(sin.sun_path) + sizeof(sin.sun_family)) < 0){
    printf( "Bad bind\n");
    return 0;
  }
  listen( s, 5 );
  len = sizeof(from);
  g = accept( s, (struct sockaddr *)&from, &len );
  while( read( g, buf, BUFSIZE ) > 0 ); /* sink all data received */
  close(g);
  close(s);
  unlink( PORT_NAME );
}

client()
{
  struct sockaddr_un sin;
  int s;
  char *buf;
  time_t start_time, elapsed_time;
  int i;
  
  buf = malloc( BUFSIZE );
  
  if( (s=socket( PF_UNIX, SOCK_STREAM, 0 )) < 0 ){
    printf( "Bad socket\n");
    return -1;
  }
  strcpy( sin.sun_path, PORT_NAME );
  sin.sun_family = PF_UNIX;
  if( connect( s, (struct sockaddr *)&sin, sizeof(sin)) < 0 ){
    printf("Bad connect\n");
    close(s);
    return -1;
  }

  start_time = time(0);
  for( i=0; i< NBLOCKS && write(s, buf, BUFSIZE) == BUFSIZE ; i++ );
  elapsed_time = time(0) - start_time;
  close(s);
  printf( "Sent %d blocks of size %d in %d seconds over unix socket =>",
          i, BUFSIZE, elapsed_time );
  printf( " %d Mbytes/second \n", (NBLOCKS * BUFSIZE) / (0x100000 * elapsed_time) );

}

memcpy_benchmark()
{
  char *src, *dst;
  time_t start_time, elapsed_time;
  int i;

  src = malloc ( BUFSIZE );
  dst = malloc ( BUFSIZE );
  start_time = time(0);
  for( i=0; i< NBLOCKS; i++ )
    memcpy( dst, src, BUFSIZE );
  elapsed_time = time(0) - start_time;

  printf( "Memcpy'ed %d blocks of size %d in %d seconds =>",
          NBLOCKS, BUFSIZE, elapsed_time );
  printf( " %d Mbytes/second\n", (NBLOCKS * BUFSIZE) / (0x100000 * elapsed_time) );
}

main()
{
  memcpy_benchmark();
  socket_benchmark();
}