My Giant Hard Drive: Building a Storage Box with FreeNAS

UPDATE 2/20/2015: This build failed after about 15 months, due to extensive drive failure. By extensive, I mean there were a total of 9 drive replacements, before three drives gave out over a weekend. This correlates closely to data recently published by Backblaze, which suggested 3 TB Seagate drives are exceptionally prone to failure. I’ve replaced these with 6 HGST Deskstar NAS 4TB drives, which were rated highly, and are better suited for NAS environments.

For many years, I’ve had a lot of hard drives being used for data storage. Movies, TV shows, music, apps, games, backups, documents, and other data have been moved between hard drives and stored in inconsistent places. This has always been the cheap and easy approach, but it has never been really satisfying. And with little to no redundancy, I’ve suffered a non-trivial amount of data loss as drives die and files get lost. Now, I’m not alone to have this problem, and others have figured out ways of solving it. One of the most interesting has been in the form of a computer dedicated to one thing: storing data, and lots of it. These computers are called network-attached storage, or NAS, computers. A NAS is a specialized computer that has lots of hard drives, a fast connection to the local network, and…that’s about it. It doesn’t need a high-end graphics card, or a 20-inch monitor, or other things we typically associate with computers. It just sits on the network and quietly serves and stores files. There are off-the-shelf boxes you can buy to do this, such as machines made by Synology or Drobo, and you can assemble one yourself for the job.

I’ve been considering making a NAS for myself for over a year, but kept putting it off due to expense and difficulty. But a short time ago, I finally pulled the trigger on a custom assembled machine for storing data. Lots of it; almost 11 terabytes of storage, in fact. This machine is made up of 6 hard drives, and is capable of withstanding a failure on two of them without losing a single file. If any drives do fail, I can replace them and keep on working. And these 11 terabytes act as one giant hard drive, not as 6 independent ones that have to be organized separately. It’s an investment in my storage needs that should grow as I need it to, and last several years.

Building a NAS took a lot of research, and other people have been equally interested in building their own NAS storage system, so I have condensed what I learned and built into this post. Doing this yourself is not for the faint of heart; it took at least 12 hours of work to assemble and setup the NAS to my needs, and required knowledge of how UNIX worked in order to make what I wanted. This post walks through a lot of that, but still requires skill in system administration (and no, I probably won’t be able to help you figure out why your system is not working). If you’ve never run your own server before, you may find this to be too overwhelming, and would be better suited with an off-the-shelf NAS solution. However, building the machine yourself is far more flexible and powerful, and offers some really useful automation and service-level tools that turn it from a dumb hard drive to an integral part of your data and media workflows.

Read More

2011 is coming to a close, so I’d like to take a moment to highlight a few apps and games on Mac and iPhone that have been invaluable to me. I broke this out into four categories, each with two apps. I have purposely omitted iPad, because frankly, I rarely use my iPad (and I prefer the TouchPad over the iPad), and don’t feel I’ve played with enough iPad apps to really give it a fair shake. So I’ve left that off to focus on iPhone and Mac apps and games. I hope you’ll check out all of these great apps.

Read More

Adobe is finally putting an end to Flash Player. They’ve announced they’re stopping development of the mobile Flash Player, which is where the future of tech innovation is heading, and the writing is on the wall for desktop Flash Player as well. This is a good thing for a myriad of reasons, both technical and political.

However, it is important to remember that Flash drove much of the innovation on the web as we know it today. When Flash was conceived over a decade ago, the web was a glimmer of what it is today. Creating something visually impressive and interactive was almost impossible. Flash brought the ability to do animation, sound, video, 3D graphics, and local storage in the browser when nothing else could.

Without Flash, MapQuest would not have been able to provide maps for years before Google did in JavaScript. The juggernaut YouTube would not have been possible until at least 2009, four years after its actual launch. Gaming on the web, which has been around as long as Flash, would only now be possible a decade later. Flash enabled developers to create rich user experiences in a market dominated by slow moving browser developers. Even in 2011 Flash exists to provide those more powerful apps to less tech-savvy people who still use old versions of Internet Explorer.

Flash Player itself seemed like a means to an end. Macromedia, and then Adobe who acquired them, sells the tool that you use to build Flash content. Thus, Adobe’s incentive was not to build a great Flash Player, but a pervasive one that would sell its tools. Its technical stagnation provided a market opportunity for browser developers to fill in the gaps that Flash provided. As a result it has a huge market dominance in tools for building rich apps for the web, tools HTML5 lacks.

This puts Adobe in a unique position. As HTML5 continues to negate the need for Flash Player, Adobe has the tools for implementing Flash within HTML5, and the market eager for those tools. Hopefully this move signals that Adobe will be moving in this direction. Because the web DOES need great HTML5 tools for people who aren’t savvy in JavaScript, especially for the people who used Flash to do it previously.

HTML5 offers developers the ability to build high-performance, low-power apps and experiences. Browser innovation has never been faster; Apple, Google, Microsoft, and Mozilla are all competing to bring the best new features to their browsers in compatible ways. But they’re just now filling in many features Flash Player has had for years. Adobe can harness this to help build a better web, and few others can. Hopefully they seize this moment.

iCloud looks like it will be an incredible technology for moving app data between devices. This is inherently a good thing, and it will open avenues for many new types of apps. But, there is a fundamental problem. Right now, the only way to access it is through Objective-C APIs embedded into iOS and Mac OS X. Under the hood, they are obviously talking to the network and doing the business of syncing data, but that networking layer is not exposed or documented, and would have to be reverse-engineered in order to understand and use. So the only way for developers to move their data through this system is through a pre-compiled bundle that gets referenced within an application.

This has a few interesting practical repercussions. If you build an application targeting iCloud, you can only ever put it on two platforms – Mac and iOS. You will never be able to port it to Android, WebOS, Windows Phone, or the web (mobile or desktop). If you sync data through iCloud, And, you will never be able to have a server component that can do things with your data all the time.

Here’s some examples of what I’m talking about. In my To Do list app, Todolicious, one thing I would love to be able to do is to push badges to your iPhone and Mac showing the number of To Dos you have left. When you tap a To Do to mark it as done, suddenly all your devices would show the correct number on the icon. With the sync server I was building, this was fairly trivial; wait for the user’s list to change, and send a signal to push the count everywhere. But if I back Todolicious with iCloud, I have no way of speaking between my server and iCloud (and I’d still need a server of some kind to send the notifications, after all).

Similarly, if I were to build a web app version of Todolicious (which I was planning on), I could not get access to that data within iCloud at all. I’d have to have either to sync to both iCloud and a custom solution (unwieldy, poor UX and network traffic, and otherwise gross), or not load existing data at all (completely negating the benefit of having such a web app).

So there is a serious ecosystem lock-in problem for apps that wish to target iCloud. All of these problems go away when iCloud is made available as a server-to-server API. A big benefit in the promise of cloud computing includes service interoperability, but right now iCloud is merely a data silo. I have filed this as a bug, rdar://9598555, for a server-to-server API (through which you could build code that speaks to iCloud on your server or on other platforms). I dearly hope Apple addresses it.

Such a server-to-server API would drastically decrease the friction of setting up cloud services to complement an iCloud-backed app, and would lead to better apps and more pleased users.

Two years ago, I began working on a new Twitter client for iPhone, named Streamlines. I hinted at it about a year ago, and has been a driving force in my development of MGTwitterEngine and a ton of open source projects. I’ve come to the conclusion that I won’t have time to finish and release it, as there’s still probably another 6 months of development needed to really ship it, and hostility from Twitter and from users of other Twitter clients make effort into building one unsustainable. However, I think there are UI concepts in there which are totally unique and have never been seen before, so I’d like to share them with you before this project is lost to the annals of dead projects.

Read More is a website that offers visitors the ability to jailbreak their iPhone without a computer-based tether. It does this by exploiting the system-wide ability for applications to read PDF files, where an incorrectly-formatted PDF file can lead a hacker to do anything they want to your system. While this bug CAN be used maliciously to steal all the personal data from your phone, the developers in this instance used it to enable jailbreaking.

Others will tell you why you should or should not jailbreak your iPhone. Others will decry the developers for bringing to light a serious vulnerability in the iPhone OS. In this blog post, I won’t do any of that, but will instead point out some things you should and shouldn’t do if you decide to jailbreak.

Read More

iPhone had the first two store UIs; the iTunes Store for content like music and movies, and the App Store for software. The iPad will add a third, the iBookstore, for buying eBooks. These stores all provide content for users to extend the utility of their device. But each has a pretty different user interaction model for accessing, purchasing, and consuming that content.

  • The iTunes Store is a separate app that is completely distinct from the iPod app. When you find something to buy, prompting you for your iTunes account password. It then adds the purchase to the app’s Downloads tab. Once you have purchased the content, you must then switch back to the iPod app to listen to or watch it.
  • The App Store is a separate app. When you purchase something, it prompts you for your iTunes password, and then exits to the home screen, switching to the screen where the app will live. The state of the download is reflected in the app icon. When the download is complete, you tap the icon on the home screen to use it
  • The iBookstore (the one word is the official name as used by Apple) is not a separate app, but lives within the iBooks app on the iPad. Purchasing content prompts for the iTunes password and downloads in-app, which can be directly accessed after it has finished downloading.

Each type of content follows a different workflow when going from access to purchase to use. If a goal of the iPad’s low price is to drive content sales through the three stores, as some speculate is the case, then the purchase model should be as streamlined for the different types of content. Forcing different workflows will only confuse users who can’t remember which type of content comes from where.

I could not be more pleased about this news. iTunes 8.1, which was released about an hour ago, renamed “Party Shuffle” to iTunes DJ. I’ve used Party Shuffle for a long time as a queue of music to play for day-to-day work, and have been frustrated with Apple’s Remote iPhone app and its lack of support for Party Shuffle. But their 1.2 App Store update, also released earlier today, has full support for it.

iTunes DJ divides iPhone users into two groups – guests and, for lack of a better term, the admin. The admin’s iPhone is paired to iTunes as before, and they get the iTunes DJ playlist added. You can skip through songs, cast a vote (addressed below), and request songs. While iTunes DJ is playing, selecting a song anywhere else throughout the app brings up a menu asking if you want to add to iTunes DJ or play on its own.

Guests don’t have to pair their iPhones to iTunes. They get to see a modified Now Playing screen, devoid of playback control. They can, however, request songs for iTunes DJ, and cast votes for songs. When they request a song, it automatically casts a vote for it.

Votes are pretty simple on the facet, but appear quite useful. Songs in the queue can be voted on by anybody. As they get voted on, they are automatically resorted in the queue by number of votes. However, you might get some guy at the party vote-clogging all night; fortunately, votes can be shut off by the admin.

All in all, this is the Big Deal feature in iTunes 8.1, and seems very well thought out.