Terabytes on a budget… 2U 14.5TB usable backup device

15th July, 2011 - Posted by Willott - 1 Comment

So, a while ago, our tape library broke, properly. The manufacturer had only given a 1yr warranty, on Enterprise hardware this seems beyond stupid, but hey (yes I could’ve bought extended, but as the rest of their kit, servers, switches etc have 3yr minimum on them was a little annoyed that the tape library didn’t have 3yrs). So, instead of forking out a shed load of money for a replacement drive for the tape library I started planning a storage server which could be stored offsite, and act as an always on data store to backup to, and would cost the same (or less as it turned out) than a replacement tape drive. I’ve now got it up and running, and here’s some info to do with it.

I was interested by Backblaze’s device but the size, and using consumer components didn’t sit all that well with me considering our device would be a secondary backup server, and wouldn’t be mirroring data multiple times across multiple servers. I also started considering what OS to use, looking at Freenas, OpenFiler (after suggestions on Edugeek) and after some other suggestions Nexenta. The bit that interested me most – ZFS. Compression, dedupe, snapshots, scrubbing, healing file system, very interesting! So what did I choose in the end…

Well, hardware wise:

  • Xcase 2U 12 drive case with redundant PSU
    • basic case, quite cheap, space for future and some redundancy
  • Tyan S7002WGM2NR
    • Onboard SAS and iKVM
  • Xeon 5620
    • Fairly beefy quad core, plenty enough for ZFS
    • Decent amount to assist ZFS
  • 8x 2TB Seagate LP Drives
    • 5900RPM drives, IOP requirement not significant, transfer rate good, cheap (to buy and replace)
  • 2x 8GB USB sticks
    • cheap, flexible, easily replaced

I’ll let you price up the above, would be wrong for me to say what we spend.

Software wise I chose NexentaStor Community Edition due to it’s zero cost. I had been using Nexenta Core, but thought that using GUI would be more friendly for my staff. NexentaStor also has an upgrade path to a paid version for support should it ever be needed here. I looked at Freenas and OpenFiler, but the ZFS versions within Nexenta are more advanced, due to it being based on OpenIndianna, related to OpenSolaris and Sun, the home of ZFS.

That gave me 14.5TB usable space in a single RAIDZ2 array, that should last me a while (1.5TB data to backup – dedupe will be bought in backup software as backups encrypted).

Comparing with one of our other devices, I got that lot for about the same as a QNAP (with drives) cost wise, but get the benefits of ZFS and some enterprise hardware as well.

Performance wise it’s quite good too, the following is from bonnie:

401MB/s   30%    108MB/s   9%     146MB/s   5%     206/sec
400MB/s   30%    110MB/s   10%    137MB/s   5%     147/sec
——— —-   ——— —-   ——— —-   ———
802MB/s   30%    219MB/s   9%     284MB/s   5%     176/sec

I’m a little bemused by the write result (possibly RAM having some effect here), so am basing my thoughts on the re-write and read speeds, which seem quite good, especially for the environment I’m aiming for. Over CIFS I’m hitting 60MB/s writes and 80MB/s reads – I need to do some more investigations on this – trying Jumbo frames across network possibly may help increase this, or trunking interfaces, but in all honesty, eventually this thing will be going the other end of a 100Mb pipe, so I’ll be looking at 12MB/s as an absolute max, and this performance investigation is just for my own interest.

So, thoughts? Well, cheap storage is easy to come by nowadays, but I think this is one step up from that, using a mix of consumer and enterprise hardware to maximise space, keep performance acceptable for the job in hand and minimise cost. So all in all a good experiment I think and it provides us with what we need (in this day and age, buy what suits your needs both current and future rather than overspec to silly proportions).

Next year’s project? A Unified Storage Server based on the same principle – maximise performance and space for minimal cost (ZFS to the rescue), using 6G SAS drives (7200RPM or higher… not made that decision yet, mirrored rather than RAIDZ), 6G SAS expanders, SSD SIL and L2ARC and 10GbE NICs.

Future projects? The 4u, 48 drive BlackHole Storage device. Just need to get funding, a PCB manufacturer who can throw together a SAS expander backplane for it (I have in mind what I want, just need PCB creating) and someone to CAD and manufacture case (again, have design in mind, just need it creating!).

Could we expand this to Petabytes? Couldn’t we put a large number of BlackHole Storage devices in a rack, have 2 uber spec’d head servers, both connected with the BlackHole Storage devices as iSCSI targets for the heads, and RAID-Z  the BlackHole Storage devices with some failover between the 2 heads? Possibly 640TB usable space in a rack (dependant on RAIDZ configs in devices/heads)? Something for you and me to ponder… or maybe you look to each of the BlackHoles to be an OpenStack storage node (or Gluster, or Lustre) and have inhouse cloud… the possibilities, as they say, are endless!

Exchange Routing Agent

15th September, 2010 - Posted by Willott - No Comments

So, one of my recent projects has been rolling out Exchange 2010 in our organisation, moving away from our ISP’s Exchange service. We’re still using the ISP’s mail scanning platform for incoming mail, and one thing I’d noticed with their service was that mail sent between our staff went through this mailscanning platform.

Due to the size of our organisation, Exchange 2010 is a single server implementation – therefore we only had the hub transport role to play with. I searched for days trying to find a ready built solution that would take internal mail and fire it out SMTP to the mailscanner. My idea was that we could rewrite the To address so that it left the Exchange organisation, the mailscanner would pick it up, scan it, then rewrite the address back to the correct domain and deliver back to the Exchange organisation:

  • ExchangeUser1@primaydomain sends email to ExchangeUser2@primarydomain
  • Exchange Server rewrites ExchangeUser2@primarydomain to ExchangeUser2@primarydomain.fakescandomain
  • Exchange server deliverys email to Mail scanner
  • Mail scanner scans
  • Mail scanner rewrites to ExchangeUser2@primarydomain
  • Mail scanner delivers to Exchange server
  • Exchange Server delivers to ExchangeUser2@primarydomain’s mailbox

Eventually after more searching I found a post on an MSDN blog and adapted my agent from the details posted there. I’m planning a few more changes to my routing agent (it will already allow system mails direct to mailboxes rather than being scanned), such as looking up the To domain in the accepted domains list, and a check that the mailscanner is alive before scanning, but I’ll try to put the code up shortly for anyone who’s interested.

Long time no post

15th September, 2010 - Posted by Willott - No Comments

It’s been a little while since posting anything – things have been hectic at work and at home, but in those things I have come across a few things I’m going to post about shortly (and a few ideas of things I’d like to do which I’ll post also).

Hopefully I’ll have a couple of posts up shortly!

VPN alternative for County Schools

22nd October, 2009 - Posted by Willott - No Comments

Well, a few months back there was discussion about being on a pilot for the VPN solution for County schools. Well, after waiting and not hearing much, I decided to look into doing our own version… and with my testing, we’re successful. The only part that my solution doesn’t meet is “anywhere access”, which I believe is probably another step to keeping things secure. I also have 2 factor authentication with factors specific to user and machine, so I can revoke on either part… a useful thing to do I think! Took a little while to get things pushed through EMBC, but I did use non-default ports, and it’s an unusual request, so it had to go through multiple approvals! Now I’ve got the boss testing it, and I’ll get a few others to before I roll it our to all staff – and external access for reports is here 🙂

Anyway, if you’re looking to be able to do something similar, feel free to get in contact, and I’ll see what I can do to help.

HP TX2 Laptop

22nd October, 2009 - Posted by Willott - No Comments

Well, we grabbed one of these for a dropped laptop at work, good price and very good spec (Dual core CPU, 4GB RAM, 500GB HDD, Touchscreen, Tablet/Laptop format). Must say I was very impressed with it, works beautifully with Windows 7 and has some very cool features linked to the touch screen. There’s some videos on youtube regarding them – worth a watch! Well, everything installed well, including SIMs (though some bits won’t work as they’re 16 bit – lets hope they update them sooner rather than later), so now just got to wait and see what the user thinks… and wait for the rest of the staff to decide they want one as well!


21st October, 2009 - Posted by Willott - No Comments

I’m quite looking forward to the next few weeks, moving forward with Frog – which is to be our new VLE. It looks very shiny from what I’ve seen of it so far, and communication from the guys that make it (Frogtrade) has been very good as well. So far of note on a negative side – the use of Dell servers, the lack of command line access to the box – as it’s a managed service, and the email functionality in general (not really configurable and the web interface is poor). I’ll hopefully post more when we’ve got the machine running.

On the interesting note, I’ve asked for a subdomain of “webbed” to be setup for access to this server (Frog, webbed, get it? Also web, and ed [education])

Oh well, off to bed!

Windows 7 64 bit

21st October, 2009 - Posted by Willott - No Comments

I’ve spent the whole of today on and off trying to get Windows 7 64-bit to install on a new laptop for School. I keep hitting a message that’s widely known (something about missing CD/DVD device drivers), and still have no luck with it with any of the work arounds. I’m hoping that it’s something to do with downloading while at School, so I’ve now downloaded both the Professional and Enterprise versions from Service Centre (I’m English – I’ll use the correct spelling) whilst at home, and we’ll see where we get!

Edit: Certainly was the image. Enterprise 64 bit is now on the Deputy Head’s new laptop (which incidently is shiny!), will have to update mine to 64 bit next week I think 🙂


21st October, 2009 - Posted by Willott - 2 Comments

I’ve been trying to decide for a while what to do with the webspace I have… turns out a blog may be what is best for it… with my job, I tend to find random useful/useless things and remedies to situations, so I’ll probably post most of them here!