best distributed file system for homebrew hedge funds

lolatency · Mar 14, 2009

I'm losing my mind managing the volumes and volumes of data I have.

What's the best distributed file system that can be used from both windows and linux that's not samba?

NFS? I know NFS is insecure, but it doesn't matter for my homebrew network. I just need to make a few sizable volumes available on all boxes.

speculatus · Mar 14, 2009

Here is one of the options:

http://www.openafs.org/

wenzi · Mar 14, 2009

Quote from lolatency:

I'm losing my mind managing the volumes and volumes of data I have.

What's the best distributed file system that can be used from both windows and linux that's not samba?

NFS? I know NFS is insecure, but it doesn't matter for my homebrew network. I just need to make a few sizable volumes available on all boxes.

'Best' for what? Just serving data to running applications? File storage ?

If you are dealing with Windows, networking is not as nice as Linux or OSX. Give NFS a try.

ATLien · Mar 14, 2009

Need more of a use case scenario to give you a good recommendation.

I like HDFS.

rosy2 · Mar 14, 2009

http://aws.amazon.com/s3/

lolatency · Mar 14, 2009

I'm trying to speed up data-mining. Right now, I have R-code that runs; however, R is not multi-threaded. So it runs really slow, or I have to manually cut the files and run multiple instances of R. Also, R is really bad with large files.

What I want to do is write the data to a network file system and then use something like Gearman (http://www.gearman.org/doku.php) to do the functions on the data from the file.

Samba can do what I want, but what kind of bugs me about samba is that when I am windows, it tries to reconnect those drives and if a box is down, I end up getting some really terrible boot ups. Maybe I can use samba?

I looked at AFS before. AFS looks hard to maintain and looks complex. Is there a decent tutorial? I -love- programming, but I hate system administration.

ATLien · Mar 14, 2009

Have you checked out Biocep?

http://biocep-distrib.r-forge.r-project.org/

Baywolf · Mar 14, 2009

If ur behind a firewall, NFS should be fine. Thats what I use on a daily basis. To be honest though, having to use Windows in a linux environment throws a monkey wrench in the way I would like to work. But you already know that.

http://technet.microsoft.com/en-us/interopmigration/bb380242.aspx

dcraig · Mar 14, 2009

Quote from lolatency:

Samba can do what I want, but what kind of bugs me about samba is that when I am windows, it tries to reconnect those drives and if a box is down, I end up getting some really terrible boot ups. Maybe I can use samba?

Can't say I enjoy system admin myself, but this is the sort of problem than can be solved by system admin. If slow boots because of unavailable file systems is the problem, then why don't you just mount whatever you want in parallel. (man page for mount says -F option will do the trick).

If smbmount won't do this trick, just fork it from the startup scripts. (/etc/init.d/mountall.sh is the place to start looking [Ubuntu 8.10]). Or create your own script to mount network fs.

Might be a quick and easy solution with few changes until you decide on a network fs to use permanently.

Baywolf · Mar 14, 2009

Quote from dcraig:

Can't say I enjoy system admin myself, but if slow boots because of unavailable file systems is the problem, then why don't you just mount whatever you want in parallel. (man page for mount says -F option will do the trick).

If smbmount won't do this trick, just fork it from the startup scripts. (/etc/init.d/mountall.sh is the place to start looking [Ubuntu 8.10]). Or create your own script to mount network fs.

Might be a quick and easy solution with few changes until you decide on a network fs to use permanently.

I also don't enjoy mucking with startup scripts. I noticed my installation of redhat was trying to mount a network fs before the networking was even established. stupid...

fyi, I use autofs to handle the auto mounting on startup.

from ubuntu: apt-get -y install autofs