• some files are more important than others
  • some files need redundancy, some others might not need it.
• several disks with different sizes
• no special hardware
• transfer speed is not very important
• possibility to verify the data integrity
• replace a small disk with a bigger disk to increase the total capacity

That means:

• Drobo is off the table. It should be possible to unplug the disks (or even a disk) and mount it on another computer
• a LVM solution would be to complicated and confusing (several partitions on every disk, combine them with LVM to volume groups with redundancy, …)
• zfs sounds like a good idea because of the focus on data integrity

Solution:

• no redundancy on block level due to the different sized disks and the fact that some files don’t need to be protected
• redundancy on file level
  • rsync some folders to several disks triggered by inotify?!?
    • not flexible and automated enough

After some time I found Greyhole, which provides a solution for that specific need. You can use every filesystem you want. Each file on the fileserver will be stored on different disks depending on the importance of the file. Greyhole will take care or me files and the storage. Adding a share, add disk, remove a disk, replace a disk, …

Trade-off:

• one disk goes down and some files are not available anymore. The symlinks must be recreated before you can access the files (not a Raid1)
• It does not speed up the transfer. New files are always written to one disk and always be read from one disk (no striping)
• A mysql DB is needed to track the jobs
• It is written php
• Only files that will be written through samba will be stored redundant.
  • Workaround: Mounting the share on the server from localhost and share that folder with nfs

It is not a perfect solution, but good enough for my use case.

Amahi is a distribution that tries to this whole setup easier. It is similar to FreeNAS. I’ve decided to do the setup in a manual way, because of the whole platform idea around Amahi (dyndns service, IP configuration inside the amahi website, …)

We only need to make sure that you have enough SATA ports and space for the disks (JBOD, no raid)
I used a HP Microserver N40L and boot it from an USB flash drive.

• 4 disk slots for the data disks
• eSATA with multiport support
• low power consumption, but still powerful
• relatively cheap – I paid about 200€ for the server itself (2GB RAM, 250GB disk)

Debian is used as OS for the new server.
I’ve prepared the USB stick inside a VM, just because I don’t want to wast a CD-R for that 🙂
Create a new VM, remove the virtual harddisk, attached the iso file and connect the usb drive to the VM. Boot and install Debian as usual.
Partitions: 7,5GB for / and 0,5 for swap. Time will tell if it was a bad idea or not 🙂
Software selection: SSH server, Standard system utilities
Install applications that you might want to have on the server afterwards. For example:

apt-get install curl sudo vim nmap iotop iftop htop

There are two options to install zfs. Fuse has less performance, supports only an older zfs version.
The other option is zfsonlinux. You can find a good performance comparison here: exitcode.de
I tried to use zfsonlinux but it produced 100% CPU load and it was not possible to copy any files to the server. I found some bug reports at that time and this issue should be fixed in a new release.
install zfs:

apt-get install zfs-fuse

verify disk layout and names:

fdisk -l | grep GB
[...]
Disk /dev/sda: 250.1 GB, 250059350016 bytes
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes

ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Apr 19 04:09 ata-Hitachi_abc... -> ../../sdb
lrwxrwxrwx 1 root root  9 Apr 19 04:09 ata-VB0250_def... -> ../../sda
...

create zfs volumes:

zpool create -o version=23 bay1 -m /data/disks/disk01 /dev/disk/by-id/ata-VB0250_abc...
zpool create -o version=23 bay2 -m /data/disks/disk02 /dev/disk/by-id/ata-Hitachi_def...
zpool create -o version=23 bay3 -m /data/disks/disk03 /dev/disk/by-id/ata-WDC__ghi...
zpool create -o version=23 bay4 -m /data/disks/disk04 /dev/disk/by-id/scsi-SATA_WDC-jkl...

version=23 means the version of zfs itself. zfs-fuse is only supporting v23, Zevo on mac os is at v28, Solaris is at v32, …
To be compatible I’ve decided to use v23 to be as compatible as possible.

verify:

zpool list
zfs list
zpool status
zpool iostat -v

automount during startup

update-rc.d zfs-fuse defaults

Here would be the best time to create additional volumes and enable compression or dedub. I have not done it. Those feature need RAM and right now I have only 2GB available. I plan to play around with this options later on.

echo "deb http://www.greyhole.net/releases/deb stable main" > /etc/apt/sources.list.d/greyhole.list
curl -s http://www.greyhole.net/releases/deb/greyhole-debsig.asc | apt-key add -
apt-get update
apt-get install greyhole

set the root password for mysql
set the Workgroup for samba

mysql_secure_installation

Answer the questions (remove anonymous mysql users, disallow root login remotely, …)

create user:

useradd -d /home/user1 -m -U -s /bin/bash user1
passwd user1
smbpasswd -a user1

vi /etc/samba/smb.conf

[Global]
...
#Settings for Greyhole 
unix extensions = no
wide links = yes

[minor]
    path = /data/greyhole-landing/minor
    create mask = 0770
    directory mask = 0770
    read only = no
    available = yes
    browseable = yes
    writable = yes
    guest ok = no
    printable = no
    valid users = user1
    dfree command = /usr/bin/greyhole-dfree
    vfs objects = greyhole

[important]
    path = /data/greyhole-landing/important
    create mask = 0770
    directory mask = 0770
    read only = no
    available = yes
    browseable = yes
    writable = yes
    guest ok = no
    printable = no
    valid users = user1
    dfree command = /usr/bin/greyhole-dfree
    vfs objects = greyhole

[critical]
    path = /data/greyhole-landing/critical
    create mask = 0770
    directory mask = 0770
    read only = no
    available = yes
    browseable = yes
    writable = yes
    guest ok = no
    printable = no
    valid users = user1
    dfree command = /usr/bin/greyhole-dfree
    vfs objects = greyhole

create the dirs and restart samba
I’ve created the landing directories on one disk, and some links in a separate folder and those are configured in the smb.conf
It should make it easier to replace one of the disk in the future by just changing the link instead of changing the samba config.

mkdir /data/greyhole-landing
mkdir /data/disks/disk01/landing

mkdir /data/disks/disk01/landing/minor 
mkdir /data/disks/disk01/landing/important 
mkdir /data/disks/disk01/landing/critical 
chown user1 /data/disks/disk01/landing/*

ln -s /data/disks/disk01/landing/minor /data/greyhole-landing/minor
ln -s /data/disks/disk01/landing/important /data/greyhole-landing/important 
ln -s /data/disks/disk01/landing/critical /data/greyhole-landing/critical 

chown user1 /data/greyhole-landing/* 
chmod 700 /data/greyhole-landing/* 

service samba restart 
update-rc.d samba defaults

create mysql database and user:

service mysql start

mysql -u root -p -e "create database greyhole; grant all on greyhole.* to greyhole_user@localhost identified by 's3cret_passw0rd';" 
# you need to enter the mysql root password you picked earlier

mysql -u greyhole_user -p greyhole < /usr/share/greyhole/schema-mysql.sql
# you need to enter the password you entered for the user greyhole_user in the last command (s3cret_passw0rd in this example)

create folders:

mkdir /data/disks/disk01/gh
mkdir /data/disks/disk02/gh
mkdir /data/disks/disk03/gh
mkdir /data/disks/disk04/gh

vi /etc/greyhole.conf

[...]
db_pass = s3cret_passw0rd
[...]
email_to = example@mydomain.com
[...]
num_copies[minor] = 1
num_copies[important] = 2
num_copies[critical] = 3
[...]
storage_pool_drive = /data/disks/disk01/gh, min_free: 10gb
storage_pool_drive = /data/disks/disk02/gh, min_free: 10gb
storage_pool_drive = /data/disks/disk03/gh, min_free: 10gb
[...]
sticky_files = minor
        stick_into = /data/disks/disk03/gh
sticky_files = important
        stick_into = /data/disks/disk01/gh
        stick_into = /data/disks/disk02/gh

greyhole start

service greyhole start \\
update-rc.d greyhole enable 3
greyhole --logs

If you create a file on the landing directory it would stay there, because Samba needs to inform Greyhole about the change.
To work with the file on the server self you need to mount the share on the server itself. Inside that mounted directory you can access the files like normal.
This is also useful if you want to make your data accessible through other protocols like nfs or ftp.
There is a script that automates the process for you. It is not multiuser friendly. Create one user that has access to all shares.

apt-get install cifs-utils 
#download script and replace /mnt/samba/ with /data/samba-local-mount/  
curl "https://raw.github.com/gist/1099419/mount_shares_locally" | sed 's/\/mnt\/samba\//\/data\/samba-local-mount\//' > /etc/init.d/mount_shares_locally  
chmod +x /etc/init.d/mount_shares_locally 
update-rc.d mount_shares_locally defaults

vi /etc/init.d/mount_shares_locally

   username="user1"

vi /home/user1/.smb_credentials

username=user1
password=sEcure_pAssworD
domain=HOME

Now you can start the new service:

service mount_shares_locally start 
update-rd.d mount_shares_locally defaults
df -h | grep /mnt/samba

I want that my fileserver shows up in the Mac OS Finder automatically. Avahi is a mDNS deamon that and can send out the broadcasts.

apt-get install avahi-daemon

vi /etc/avahi/services/samba.service

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
   <name replace-wildcards="yes">%h</name>
   <service>
       <type>_smb._tcp</type>
       <port>445</port>
   </service>
   <service>
      <type>_device-info._tcp</type>
      <port>0</port>
      <txt-record>model=Xserve</txt-record>
   </service>
</service-group>

/etc/init.d/avahi-daemon restart