[OpenAFS-devel] Bug with OpenAFS memory-mapped file I/O under Linux

[Elam, Jeffrey S]

We've run into a rather annoying issue here while running tests on Openafs
1.2.2 on Redhat 7.0 (using the 2.4.9-12 kernel).  Maybe someone on the list
is familiar with the problem or has a suggestion.

Some of the tools we use occasionally do memory mapped file I/O in
MAP_SHARED mode.  It appears that anytime you use this kind of file I/O
under OpenAFS, the output file gets corrupted in a strange way:  The client
that wrote the file sees the correct output, and it stays that way only on
that client (in cache).  However, the fileserver, and every other client,
see a corrupted output file.  It's as if the file is closed on the network
before the final data is written to it, or the final write is not flushed to
the network, or something like that.

Below is a simple test case that duplicates the problem quite easily.  The
test case takes a single parameter - the name of an output file to create.
Create the output on an AFS partition and use "od -x outputfile" to see the
contents.  The correct output looks like this:

0000000 f00d baad f00d baad f00d 600d f00d 600d
0000020 f00d 600d f00d 600d f00d 600d f00d 600d
*
6543440 f00d 600d f00d 600d 0000 0000 0000 0000
6543460

The corrupted output is missing the modifications made to the beginning of
the file.  In this case, you'll see:

0000000 f00d 600d f00d 600d f00d 600d f00d 600d
*
6543440 f00d 600d f00d 600d 0000 0000 0000 0000
6543460

If anyone has an idea how to fix this problem, I'm all ears.  This is
hurting our ability to roll out OpenAFS on our Linux systems.

BTW, earlier Redhat kernels had a similar problem under NFS, but it's fixed
in 2.4.9-12 and later.

_____________
Jeffrey S. Elam
Engineering Computing Support
Intel Austin Design Center
1501 S. Mopac Expressway, Suite 400, Austin, TX 78746
M/S: AN1-4A13  |  Ph: (512) 314-0612


--------------------------------- cut here ---------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>

#define SZ 1754928
int main(int argc, char **argv)
{
	void *pa;
	int output_fd;
	FILE *fp;
	char *nm = argv[1];
	char *buf = (char*)malloc(SZ);
	int i;
	unsigned long *bp;
	char dummy = 0;

	memset(buf, 0, 8);
	for(bp=(unsigned long*)buf, i=8; i<SZ; i+=sizeof(long)) {
		*bp++ = 0x600df00d;
	}
	output_fd = open(nm, O_RDWR | O_CREAT | O_TRUNC, 0666);

	if (output_fd < 0) {
		perror("test");
		exit(-1);
	}

	if (write(output_fd, buf, SZ) < 0) {
		perror("write");
		exit(-2);
	}

	if (lseek(output_fd, SZ-1, SEEK_SET) < 0) {
		perror("lseek");
		exit(-3);
	}

	if (write(output_fd, &dummy, 1) < 0) {
		perror("write");
		exit(-4);
	}

	pa = mmap(NULL, SZ, PROT_READ | PROT_WRITE, MAP_SHARED, output_fd,
0);

	if ((long)pa == -1L) {
		perror("mmap");
		exit(-5);
	}


	bp = (unsigned long*)pa;

	*bp++ = 0xbaadf00d;
	*bp++ = 0xbaadf00d;

	if (munmap(pa, SZ) == -1) {
		perror("munmap");
		exit(-6);
	}

	if (ftruncate(output_fd, SZ)) {
		perror("ftruncate");
		exit(-7);
	}

	if ((fp = fdopen(output_fd, "w")) == NULL) {
		perror("fdopen");
		exit(-8);
	}

	if (fseek(fp, SZ-1, SEEK_SET) < 0) {
		perror("fseek");
		exit(-9);
	}

	if (fwrite(&dummy, sizeof(char), 1, fp) <= 0) {
		perror("fwrite");
		exit(-10);
	}

	if (fclose(fp)) {
		perror("fclose");
		exit(-11);
	}
}