For many years, I've had a lot of hard drives being used for data storage. Movies, TV shows, music, apps, games, backups, documents, and other data have been moved between hard drives and stored in inconsistent places. This has always been the cheap and easy approach, but it has never been really satisfying. And with little to no redundancy, I've suffered a non-trivial amount of data loss as drives die and files get lost. Now, I'm not alone to have this problem, and others have figured out ways of solving it. One of the most interesting has been in the form of a computer dedicated to one thing: storing data, and lots of it. These computers are called network-attached storage, or NAS, computers. A NAS is a specialized computer that has lots of hard drives, a fast connection to the local network, and...that's about it. It doesn't need a high-end graphics card, or a 20-inch monitor, or other things we typically associate with computers. It just sits on the network and quietly serves and stores files. There are off-the-shelf boxes you can buy to do this, such as machines made by Synology or Drobo, and you can assemble one yourself for the job.

I've been considering making a NAS for myself for over a year, but kept putting it off due to expense and difficulty. But a short time ago, I finally pulled the trigger on a custom assembled machine for storing data. Lots of it; almost 11 terabytes of storage, in fact. This machine is made up of 6 hard drives, and is capable of withstanding a failure on two of them without losing a single file. If any drives do fail, I can replace them and keep on working. And these 11 terabytes act as one giant hard drive, not as 6 independent ones that have to be organized separately. It's an investment in my storage needs that should grow as I need it to, and last several years.

Building a NAS took a lot of research, and other people have been equally interested in building their own NAS storage system, so I have condensed what I learned and built into this post. Doing this yourself is not for the faint of heart; it took at least 12 hours of work to assemble and setup the NAS to my needs, and required knowledge of how UNIX worked in order to make what I wanted. This post walks through a lot of that, but still requires skill in system administration (and no, I probably won't be able to help you figure out why your system is not working). If you've never run your own server before, you may find this to be too overwhelming, and would be better suited with an off-the-shelf NAS solution. However, building the machine yourself is far more flexible and powerful, and offers some really useful automation and service-level tools that turn it from a dumb hard drive to an integral part of your data and media workflows.

Before we begin, I'd like to talk about the concepts and terminology to be discussed as part of the assembly. Feel free to skip this section if you already understand RAID, ZFS, and computer assembly.

Data Storage for Newbies

At its core, a NAS is just a computer with a number of hard drives in it. Its only purpose is to store and load data, and make all that stuff available over the network. Since all it's ever doing is holding on to lots of data, you typically don't need a lot of the things that you'd put into a normal computer; stuff like a graphics card, keyboard, mouse, and monitor aren't needed very much. You instead buy parts that focus on a few key areas: number of hard drives you can connect, and how fast you can get data in and out. In this case, you need these parts:

  • a motherboard
  • a CPU
  • some RAM
  • a bunch of hard drives
  • a power supply
  • a case to put everything inside of

Your laptop has a hard drive in it. If you've ever plugged in an external drive or a Flash drive, you'd see that they're two separate places for you to store stuff. If one of them fails, you lose all of the data on it, but it doesn't affect the data on your other drives. And you have to organize everything yourself. Trying to scale up to 4 or 6 or 10 drives sounds like a disaster. What we really would like is to make all of those drives pretend like they're one giant hard drive. And we'd like to be resilient to a hard drive dying without losing data.

There's a tool for this, and it's called RAID, or "redundant array of independent disks". RAID is a set of technologies that takes multiple hard drives, called an array, and combines them under the hood to make them look and act like one giant hard drive. The way this works is complicated, but the basic idea is that RAID takes a file, chops it up into little pieces, and spreads them out across all your hard drives. Then, when you want the file, RAID will grab all those pieces from each hard drive and combine them back into the original file. (Please note: this is an overly simplified discussion of the technology, and is not technically accurate, but is adequate for our purposes of conceptualizing.) There are different strategies called "RAID levels" you can use that will change the specific behavior; some are more focused on redundancy, some are focused on speed.

The benefits you get with most RAID levels are: a bunch of hard drives that look like one storage place, improved speed when reading/writing data, the ability to survive a drive failing, and the ability to replace a dead drive with a new one. However, the downside is potentially a big one. Because the files are never stored as a whole on one drive, if you lose enough drives at once and don't replace them in time, you lose all the data, even on drives that haven't failed. Depending on your RAID level, you can survive zero, one, two, three, or more drives failing. But the more dead drives you want to be able to withstand, the more storage of those drives gets used for redundant data. So it's a balance of how much storage you want vs. how much protection you want from dying drives. You can calculate how much storage you'll have based on how many drives you buy using a RAID calculator. A healthy minimum is that for every 3 drives you buy, you want to be able to withstand one failing. So 2 or 3 drives should withstand 1 drive failing, 4-6 drives should withstand 2 failing, 7-9 should withstand 3, etc.

For this build, I set up my array as a form of RAID called RAID-Z2. RAID-Z and RAID-Z2 are based on a technology called ZFS, which is a modern file system that supports "storage pools". This gives us the "make a bunch of hard drives act like one giant hard drive" behavior, which RAID-Z builds on to give us the "survive a hard drive failure" behavior we want. RAID-Z lets you survive one drive failure, RAID-Z2 lets you survive two, RAID-Z3 lets you survive 3. The major downside to RAID-Z is that it requires all data to be processed by the CPU, so you'll want something reasonably fast to process your data. The more drives you add, the bigger the CPU will need to be.

Building the Computer

The part that was the most daunting for me to overcome was actually purchasing the pieces necessary to build the computer. I'm a software guy who's owned Macs all my life, so I've never actually assembled a computer before (I will take this opportunity to let all the nerds out there get a good laugh in before we move on). If the idea of building your own computer is scary, you may want to just go buy an off-the-shelf NAS, such as the Synology DS413j and stop reading. Keep in mind, though, that a preassembled NAS will be more expensive and far less flexible than building one yourself.

After waffling on this for months, I finally decided to go the custom build approach. I figured I could make it cheaper, quieter, and run whatever services I wanted directly on the machine by building it myself. After putting some pieces together, here's the parts I went with. Prices are what they cost as of September 30, 2012. All links to Amazon include affiliate links, so I get a tiny kickback. Feel free to search for the part names if you wish. You may be able to find these parts cheaper elsewhere on the Internet.

A few notes about this hardware configuration:

  • The case has 6 hard drive slots, so you can put up to 6 drives in it. You can, of course, put fewer in it.
  • The motherboard has 6 SATA ports, but only two are 6 Gbps, while the others are 3 Gbps.
  • The power supply has 5 SATA connections, so if you want to run 6 drives, you'll need a Molex to SATA power adapter.
  • Besides the Molex adapter, the parts mentioned all the cables necessary for internal setup. But you will need your own power cable.
  • The motherboard includes some onboard graphics, and you'll want to have a DVI monitor available for making sure the machine is booting correctly. You won't need to keep it plugged in beyond setup, however.
  • RAM is cheap, and if you're accessing the same files over and over, they can remain in RAM and be even faster than loading from disk. It's better not to skimp on this. Just make sure your CPU is 64-bit.
  • There's no Wi-Fi here, so you'll either need to get a wireless card or (ideally) plug an Ethernet cable into it connected to your network.

Installing the OS

For the operating system, I decided to use FreeNAS 8.2, a distro of FreeBSD that is designed to run ZFS-based RAID systems. It includes a web-based administration tool that lets you set up the array, monitor it, set up tests to detect failing drives, run services and plugins, and lots of other stuff. To run this, I copied it to a USB key (at least 2 GB necessary, you probably want 4 GB) and just leave that plugged in to the back of the machine all the time. Once you copy the image onto the key, you set the default boot drive to the USB key, and it will boot to it each time. You will also need a keyboard (and note, Apple's keyboards will not work with this setup, so have a USB or even a PS/2 Windows keyboard) to get into the BIOS settings. After you have the BIOS auto-boot set up, when you turn the computer on, it'll take a minute or two to set everything up, and then the web admin will be available on your local network. If you have a router that can tell you what's connected, you can get the IP there; otherwise, plug a monitor into the motherboard and it'll tell you the IP. If your router supports it, you should grab the MAC address and assign it to a static IP on your network so that your NAS is always available on the same IP address. Once this is all running automatically, you can disconnect the monitor and keyboard and just run the machine headless.

The web admin is divided into a few sections. Along the top are the sections/actions that are the most commonly used; System, Network, Storage, Sharing, Services, Account, Help, Alert Status, and Log Out. The absolute first thing you should do is click the Account button and change the username and password for the admin account (which you got logged into automatically). Once this is set, nobody will be able to log in to the web admin without these credentials, or without physical access to the machine (as you can disable the login from the console if you have a monitor/keyboard attached). You'll also want to click the Users tab in that section and create a user for yourself for connecting to the array. Make sure it's in the group "wheel", at the very least.

Once you have that out of the way, you can set up your storage array and actually get those hard drives to do something. Click Storage at the top to view the Active Volumes, which is empty, as we haven't set any up yet. Set one up by clicking the Volume Manager button; give the volume a name (I just called mine "Main"), select all the disks from your list, choose ZFS, then choose your RAID-Z level. Click Add, and after some processing, you'll have a giant hard drive. The amount of storage will be considerably less than the sum capacity of the hard drives you put in, as it is reporting the capacity after taking out the backup data it will eventually be storing. In my case, the 6x3TB drives have about 16.3 TB of raw capacity, but after the backup data in RAID-Z2 is accounted for, only 10.7 TB is available. Note: If you added 6 drives to the array, you should see 6 drives in the list when creating the volume; if you don't, you probably didn't connect something correctly inside the machine. Make sure you set the permissions on this new volume so your user can access it, and do this recursively.

ZFS has a cool feature called "datasets". A dataset is just a folder with special types of rules around how big those folders can be. You can set a quota, which is the maximum size a folder can grow to, and a reserved space amount, which (as the name implies) reserves a certain amount of space for use in that folder. You can customize permissions on these separately from the whole array. You can set certain compression levels based on if you're more concerned with speed vs space. All of these values can be changed later. You can also ignore all of this, and just use datasets for organization. So, for example, I have two primary datasets:

  • Media, which has no quota or reserved space, permissions set so that anyone can read but only I can write, and no compression so it can stream fast, and
  • Backups, for Time Machine, which has the maximum level of compression (as read/write speed doesn't matter), no access to anyone except my user, and a quota of 500 GB

Actually Getting Data In/Out

So now I have a ZFS volume running RAID-Z2, /mnt/Main, which has two datasets, /mnt/Main/Media and /mnt/Main/Backups. Now we need to actually make them available for use by other computers. To do this, we set up Shares. FreeNAS has three different types of shares - AFP (for Macs), CIFS (for Windows, also known as SMB or Samba), and NFS (for Unix/Linux/FreeBSD). For our purposes, I will be setting up two AFP shares, one for each of the two datasets.

Shares are a type of Service, which is a program that FreeNAS will run automatically for you. Besides Shares, FreeNAS has services for things like FTP, LDAP, Rsync, SSH, UPS integration, and plugins. At the top of the admin UI, click Services, and click the On/Off switch next to the AFP service to start it up. Feel free to turn on whatever else you like (except Plugins, which will not quite work out of the box, but I'll discuss Plugins at greater length below). You may be prompted for settings before a given service will start.

Now you can create your Shares. Click the Sharing tab at the top, and make sure "Apple (AFP)" is selected. Click the "Add Apple (AFP) Share" button, and you'll be prompted with a daunting form. You can leave some of the more confusing fields as their default. The fields you really need to worry about are:

  • Name, the displayed name of the share
  • Path, where you want the share to point
  • Share password, if you want to set a password
  • Allow/Deny list and Read-Only/Read-Write Access, to control who can do what on the share
  • Disk Discovery, which will allow the share to be seen if you just ask the server for a list of shares
  • Disk Discovery Mode, which will let you toggle between a normal Finder share and a Time Machine backup share
  • Permissions, which let you control who can read, write, and run programs on the share

Once you have this in place, click OK, and you'll have created the Share. If you enabled Disk Discovery mode, your NAS should appear in the Finder's sidebar. If you did not, you can connect to it by selecting "Connect To Server" from the Go menu in the Finder (⌘K), and typing afp://NAS_IP/SHARE_NAME and filling in the NAS_IP and SHARE_NAME as appropriate. Authenticate if you set it up, and you should be connected. Then you can drag stuff from your hard drive into the share and it will copy over. You can also use cp from the Terminal to copy data.

When I tried setting this up originally, I got permissions errors while doing this. My rules for setting the permissions up are:

  • Make sure the user you want to have read/write access is in both the allow list and the read-write access list
  • If you want read-only access available to everyone, add @nobody to the allow list and the read-only list
  • Set all file/directory permissions to on, with the exception of "other/write".
  • Set the owner of the ZFS dataset to your user, and set all the permissions there to on, with the exception of "other/write".

To test the permissions on the ZFS dataset, the easiest thing to do is enable the SSH service, SSH into the machine with your user account, cd into the dataset, and try to touch a file. If it fails, you can't write. If it does work, cat the file; if it fails, you can't read. If that succeeds, but trying to connect via AFP doesn't let you read/write files, the error is on the AFP share permissions.

Keeping Your NAS Healthy

If you have a system dedicated to making sure your data is reliably accessible, you want to know sooner rather than later if you're going to have hard drive problems. FreeNAS includes a drive testing system called S.M.A.R.T. which is a tool for testing your drives to determine if they are behaving abnormally (higher temperature, higher error rates when reading data, lower throughput, etc.). These can then be emailed to you on a schedule you decide for your analysis. These tests are not run on the array as a whole, but rather on individual disks within the array. These tests can be created and found on the sidebar, under System > S.M.A.R.T. Tests.

I rely primarily on the "short" S.M.A.R.T. test which runs once a day, and occasionally a "long" test which runs manually when I won't need the array for awhile. The short test scans electrical circuits and selected parts of the disk for errors, and these tests take only a couple of minutes. The long test scans every bit on the drive for failures; this takes a very long time, especially on high capacity disks, so it should be run infrequently. There's also a "conveyance" test, which is useful to run before/after moving the drives, to determine if they were damaged during transport. Set these up at your preference.

The easiest way to see this data is to have it emailed to you. Test reports are sent to the email address associated with the root user. To change this, select Account > Users > View Users from the sidebar. In the list that appears, the root user will be at the top of the second list. The last button lets you change the email address, so set this to your email address. You then have tell FreeNAS how to connect to an SMTP server with an account. You can use Gmail or iCloud for this. On the sidebar, select System > Settings and choose the Email tab. Fill out the fields as appropriate for your mail server. Once this is in place, you can send a test email. If you get it, you're all set up, and your S.M.A.R.T. tests will send their results to you when they run.

Extending with Plugins

Note: This is a more advanced topic, and to make this work you'll need an understanding of how SSH and shell access works, which is beyond the scope of this post.

FreeNAS 8.2 introduced a plugin system based on FreeBSD jails, which are sandboxed environments for programs to run in. Plugins are like other services that run automatically in the background, but instead of being services for managing the array themselves, they are apps that you might want to run directly on your storage array. As they are sandboxed, they will only be able to write to specific folders in your array. A number of services have been ported to the FreeNAS plugin format, and you can use these to extend your array's functionality and provide even more utility. I'll demonstrate how to set up Transmission, the BitTorrent client, to run natively on your NAS. You can find other plugins on the FreeNAS Forums, or even make them yourself if the app has been ported to FreeBSD.

To begin, we need a place on the array to store plugins, and to store the Jail. Create two ZFS datasets for this (I call them "Jail" and "Plugins"). You'll rarely need to go in here manually, but the plugin system needs a place for this stuff to live. All FreeNAS plugins are .pbi files, and in fact the service that runs the plugins is itself a pbi file, which is not installed by default. Once you have your datasets set up, go to the Services tab, and click the settings icon next to the Plugins service. There are three steps to the installation. First, it needs a temporary place to store the plugin while it installs (this will be the root of your ZFS volume). Next, it needs to know the path to your dataset for your jail and plugins folder, as well as the IP address you're going to use as the jail's IP (make this something unique, out of your DHCP range). Finally, it needs the plugin service PBI that is appropriate for the version of FreeNAS you're using and the architecture of your CPU.

If it installed successfully, you can then install plugins. Near the top is a tab called "Plugins". Here you can upload the pbi for whatever plugin you like. On the page where you downloaded the plugin service PBI, you can also download the pbi for Transmission. Download it from the site and upload it to your NAS. You'll have to set up the parameters before you can turn it on. Make note of the Download directory you specify, as we'll need it later (but you can leave it as the default). Then, you can turn it on and access it by going to http://JAIL_IP:9091/ in your browser.

Now, before we go on a download spree, we need to understand where those files will end up. They go into the Download directory specified in the settings, which for me was /usr/pbi/transmission-amd64/etc/transmission/home/Downloads. But there's a catch: since this is in a FreeBSD jail, that path is relative to the jail root, which is itself part of your array. Now, you can access that folder, but you probably will want to set up a nicer path for it, that doesn't go through your jail.

That's where Mount Points come in. A Mount Point is a way of making a folder available from the outside of your jail to inside of it. So you can set up a Downloads dataset at /mnt/Main/Downloads, and establish a Mount Point from that to the Transmission download folder, and suddenly everything Transmission downloads will appear in /mnt/Main/Downloads, even though Transmission itself is jailed. In the Plugins tab of Services, there is a "View Mount Points" button. If you add a mount point, it asks you what the route is you want to set up. So for the case above, we need a mount point that looks like this:

  • Source: /mnt/Main/Downloads
  • Destination: /mnt/Main/Plugins/Jail/usr/pbi/transmission-amd64/etc/transmission/home/Downloads

Once this is set up, turn it on, and it will just start writing data from the Transmission downloads folder into your Downloads dataset. You may have to fiddle with permissions; I found I had to make the folder within the jail writable by the user that was running the Transmission process. To enter a jail, SSH in to the NAS box as a user in the wheel group, su root, and run jexec 1 csh. To exit, just exit.

Result

The machine, named Holocron, sitting in its new home next to my media center. Given the level of nerdy this project entails, Twilight Sparkle is appropriate here. Forgive the wiring and other clutter; that's one of my next projects.

The case was larger than I expected, but not too large. It's about as tall and deep as my media center, so it sits nicely next to it (which is handy as that's where my Internet switch is). The case looks great, with off-black on all sides and no obnoxious branding on the front, and has some convenient USB ports on top. The only problem with the front is that the case has a power button on the top with a REALLY BRIGHT BLUE LED noting that the machine is on; I would love to figure out a way to turn that off (or at least knock the brightness down). But the real win here is that the case is very quiet. It has noise insulating material on the walls, which knock down the sound, and the hard drive trays have rubber grommets on the screw holes, which helps quiet the spinning of the hard drives. The case emits so little sound that, even with 6 hard drives and fans, the entire thing is less noisy than a single Western Digital MyBook (and I had 5 of those to replace). It blew away my expectations of noise.

The machine is quite fast. It handles reading and writing stuff like a champ, downloading and streaming at the same time with no problems. It's been running for weeks at a time with no uptime issues. Even with 7 plugin services running, it has all run very, very smoothly. I've run into one or two bugs in the FreeNAS web admin UI, mostly happening when you try to save an options form that includes a permissions field (when you aren't actually changing permissions). When this happens, a manual reboot of the machine fixes the problems, and since it's manual you can take down connections as you need to. But you really shouldn't have to change them once they're set up, so this is a problem of setup more than anything.

The permissions on the system remain the biggest single headache. I've definitely spent most of my time struggling to make sense of the permission model, which gets more complicated and difficult to track down when you introduce Shares and Mount Points into the mix. But once you have it figured out, you can build in the permissions you want to offer and it will stick. You can also SSH in to the system to see the permissions at the UNIX level, which is helpful if you're familiar with the shell.

The second biggest headache has been learning FreeBSD, which is starkly different from Linux or Mac OS X. There have been several times where I'll do some muscle-memory shell command, like sudo su transmission, and it will fail because FreeBSD does things a little differently (in this case, I've been doing su root followed by su transmission). These are probably just differently configured and there's ways to get it to do what I want, but it's not a big deal.

However, nits aside, once this system is running, it's providing a ton of value. As someone who has always cobbled together storage based on what I had and what was the easiest to get setup, this definitely took more discipline to configure and get working properly, but the value is paying off huge. Since everything is pooled together, I have more incentive to keep it organized and optimized for how I want to use it. The assumptions I set up for myself and through the plugins mean everything works as I want and everything ends up where I need it to be. The extra effort makes it a more useful system.

Building a NAS is not for the cheap or faint of heart. It requires money, time, and effort to build into a great storage system. It is also not a panacea of storage; you still want to back up critical stuff onto a different drive, ideally offsite or in the cloud, and you still need to worry about drives failing. But if you put that energy in, you'll end up with an indispensable tool that will be more reliable and more powerful than a glued-together system of disparate components and wonky services. It's an investment that I'm hoping will pay off for a number of years.

 

2011 is coming to a close, so I'd like to take a moment to highlight a few apps and games on Mac and iPhone that have been invaluable to me. I broke this out into four categories, each with two apps. I have purposely omitted iPad, because frankly, I rarely use my iPad (and I prefer the TouchPad over the iPad), and don't feel I've played with enough iPad apps to really give it a fair shake. So I've left that off to focus on iPhone and Mac apps and games. I hope you'll check out all of these great apps.

DISCLAIMER: I am friends with the guys at Tapbots (makers of Tweetbot) and the guys at TapTapTap (makers of Camera+). However the apps would not have made it onto the list if they were not of the highest quality, and have not influenced my reviews. I have deliberately excluded apps made by any company that I have worked for either now or in the past. I have also not included affiliate links.

Best iPhone Apps

Tweetbot

$2.99 - Tweetbot came out this year as a pretty full-featured Twitter client, but naturally everybody has their own pet features they would like. The guys at Tapbots have steadily improved the app over the year, adding support for push notifications, muting, Favstar integration, and plenty more. It has since become the best designed and most full featured Twitter client, far exceeding Twitter's iPhone app.

Camera+

$0.99 - The iPhone has the best camera of any mobile device (and I test a lot of mobile devices). Camera+ has many features that go beyond the included Camera app. The most important ones actually help you take better photos, such as the image stabilizer, which uses the iPhone's gyroscope and only captures a photo when your hands aren't not shaking. The touch up tools are very handy, and the filters look pretty good compared to other photo apps. And a suite of sharing tools help you share your moments with your Twitter, Facebook, and Flickr friends. It's the tool you should reach for when taking photos, and it shows how good a replacement the iPhone can be for a standalone camera.

Best iPhone Games

Super Stickman Golf

$0.99 - I'm a sucker for a good physics game, and Super Stickman Golf is a great one. Get the ball into the hole while navigating steep terrain, dodging death traps, and staying under par. With dozens of 9-hole courses and a grab bag of power ups, there is a ton of replayability here. It also includes an exhilarating multiplayer mode which completely changes the mechanics from precision to speed. One of the greatest physics games for iPhone that isn't just another Angry Birds clone.

Jetpack Joyride

$0.99 - The form factor of the iPhone lends itself to simpler games over more complex ones. One-tap games like Canabalt are great because of their simplicity. Jetpack Joyride takes this to great lengths, packing a ton of variety into a single tap. And it shows how to do free-to-play games right, in a way that doesn't give an unfair competitive advantage or make you feel forced into spending more money. Gorgeous graphics with a ton of tiny nuance help seal the deal.

Best Mac Apps

Spotify

Free w/ optional subscription - Spotify combines a cloud-powered streaming music service, a lovely native Mac app, and excellent sharing features into one great app. Though Spotify has been around for awhile, it only recently became available in the US. It lets you combine their huge streamable catalog with music in your iTunes library. It syncs your playlists in the cloud, and copies tracks to your iPhone via Wi-Fi. And it lets you share tracks and playlists, either to one person or the world, quickly and easily. I've ranted before about how I don't like iTunes, and have searched long and far to find a suitable replacement. This year, Spotify became that replacement.

Pixelmator

$29.99- Photo-editing tools typically come in one of two favors; way too complicated, or way too simple. Pixelmator bridges the gap by bringing much of the power of Photoshop to mere mortals at an affordable price. While pro designers probably won't drop the Adobe tools for it, it's a fantastic and flexible tool for editing photos. And it has great support for core Mac OS X technologies like Core Image and Auto Save. It's the perfect tool for going beyond red-eye reduction and one-click canned filters to make photos and images look fantastic.

Best Mac Games

Minecraft

$26.95 - Not many games have staying power for me; I typically lose interest in most games after a couple weeks. Not Minecraft. I started playing Minecraft in September of 2010 and have been continually coming back to it ever since. A Lego-style deformable terrain lets you create the world in your image, getting lost in massive caves, and dodging monsters who hunt you down and try to kill you. The game has incredible depth, forcing you to find raw materials and learn how to turn them into weapons, armor, building materials, and so much more. If you've got some friends who play, and one of them is technically savvy, you can set up a multiplayer server where everyone can contribute to the same world, helping each other survive and creating ever-cooler worlds. The 1.0 version, released in November, has a series of objectives and an end-game, though you can happily ignore it and just have fun making your world. You'll spend hours playing Minecraft, and actually have something to show for it after.

Galaxy on Fire 2

$19.99 - One dream that will probably go unrealized in our lifetimes is that of interstellar space travel. We've all wanted to live out battle scenes from Star Wars. Galaxy on Fire 2 is a gorgeous space action game that brings that to life. You can hunt pirates, trade minerals, go on side quests, and hunt down the mysterious race known only as the Voids. A large economy lets you customize your ship to give you the edge in battle, or help you carry more precious resources across the galaxy. A massive story, incredible graphics, great voice acting, tremendous depth, and perfectly-tuned controls make this a must play. It's available on iPhone and iPad, but with so much graphical detail, you'll want to check this one out on the big screen.

, , , , ,    Tags
 

Adobe is finally putting an end to Flash Player. They've announced they're stopping development of the mobile Flash Player, which is where the future of tech innovation is heading, and the writing is on the wall for desktop Flash Player as well. This is a good thing for a myriad of reasons, both technical and political.

However, it is important to remember that Flash drove much of the innovation on the web as we know it today. When Flash was conceived over a decade ago, the web was a glimmer of what it is today. Creating something visually impressive and interactive was almost impossible. Flash brought the ability to do animation, sound, video, 3D graphics, and local storage in the browser when nothing else could.

Without Flash, MapQuest would not have been able to provide maps for years before Google did in JavaScript. The juggernaut YouTube would not have been possible until at least 2009, four years after its actual launch. Gaming on the web, which has been around as long as Flash, would only now be possible a decade later. Flash enabled developers to create rich user experiences in a market dominated by slow moving browser developers. Even in 2011 Flash exists to provide those more powerful apps to less tech-savvy people who still use old versions of Internet Explorer.

Flash Player itself seemed like a means to an end. Macromedia, and then Adobe who acquired them, sells the tool that you use to build Flash content. Thus, Adobe's incentive was not to build a great Flash Player, but a pervasive one that would sell its tools. Its technical stagnation provided a market opportunity for browser developers to fill in the gaps that Flash provided. As a result it has a huge market dominance in tools for building rich apps for the web, tools HTML5 lacks.

This puts Adobe in a unique position. As HTML5 continues to negate the need for Flash Player, Adobe has the tools for implementing Flash within HTML5, and the market eager for those tools. Hopefully this move signals that Adobe will be moving in this direction. Because the web DOES need great HTML5 tools for people who aren't savvy in JavaScript, especially for the people who used Flash to do it previously.

HTML5 offers developers the ability to build high-performance, low-power apps and experiences. Browser innovation has never been faster; Apple, Google, Microsoft, and Mozilla are all competing to bring the best new features to their browsers in compatible ways. But they're just now filling in many features Flash Player has had for years. Adobe can harness this to help build a better web, and few others can. Hopefully they seize this moment.

 

I wrote a guest post for MacStories, covering the history of patent law surrounding patent trolls. While recent lawsuits from Lodsys and Kootol are causing panic and alarm from indie developers, it's not like this threat is suddenly new. Patent lawsuits have always been on the table, but they were ignored by the majority of small companies. Now it's clear that patent holders will pursue people who violate their patent. Whether ethical or not, they are legally required to defend their patents, and that means we will see more patent lawsuits pursued by trolls. Meanwhile, none of these small developers can afford to fight, so they settle, perpetuating the cycle.

 

iCloud looks like it will be an incredible technology for moving app data between devices. This is inherently a good thing, and it will open avenues for many new types of apps. But, there is a fundamental problem. Right now, the only way to access it is through Objective-C APIs embedded into iOS and Mac OS X. Under the hood, they are obviously talking to the network and doing the business of syncing data, but that networking layer is not exposed or documented, and would have to be reverse-engineered in order to understand and use. So the only way for developers to move their data through this system is through a pre-compiled bundle that gets referenced within an application.

This has a few interesting practical repercussions. If you build an application targeting iCloud, you can only ever put it on two platforms - Mac and iOS. You will never be able to port it to Android, WebOS, Windows Phone, or the web (mobile or desktop). If you sync data through iCloud, And, you will never be able to have a server component that can do things with your data all the time.

Here's some examples of what I'm talking about. In my To Do list app, Todolicious, one thing I would love to be able to do is to push badges to your iPhone and Mac showing the number of To Dos you have left. When you tap a To Do to mark it as done, suddenly all your devices would show the correct number on the icon. With the sync server I was building, this was fairly trivial; wait for the user's list to change, and send a signal to push the count everywhere. But if I back Todolicious with iCloud, I have no way of speaking between my server and iCloud (and I'd still need a server of some kind to send the notifications, after all).

Similarly, if I were to build a web app version of Todolicious (which I was planning on), I could not get access to that data within iCloud at all. I'd have to have either to sync to both iCloud and a custom solution (unwieldy, poor UX and network traffic, and otherwise gross), or not load existing data at all (completely negating the benefit of having such a web app).

So there is a serious ecosystem lock-in problem for apps that wish to target iCloud. All of these problems go away when iCloud is made available as a server-to-server API. A big benefit in the promise of cloud computing includes service interoperability, but right now iCloud is merely a data silo. I have filed this as a bug, rdar://9598555, for a server-to-server API (through which you could build code that speaks to iCloud on your server or on other platforms). I dearly hope Apple addresses it.

Such a server-to-server API would drastically decrease the friction of setting up cloud services to complement an iCloud-backed app, and would lead to better apps and more pleased users.

 

Two years ago, I began working on a new Twitter client for iPhone, named Streamlines. I hinted at it about a year ago, and has been a driving force in my development of MGTwitterEngine and a ton of open source projects. I've come to the conclusion that I won't have time to finish and release it, as there's still probably another 6 months of development needed to really ship it, and hostility from Twitter and from users of other Twitter clients make effort into building one unsustainable. However, I think there are UI concepts in there which are totally unique and have never been seen before, so I'd like to share them with you before this project is lost to the annals of dead projects.

Here's a video walking through some of the main UI concepts found in Streamlines.

Streamlines is a social networking client that is designed to show you what you want to see, and hide what you don't care about. To do this, it avoids using tab bars and navigation stacks, and instead uses a card interface with a horizontal swipe, similar to the iPhone's Weather app or WebOS' multitasking UI. You pick which timelines you want to see.

On top of that, you can merge multiple timelines together, across all types of timelines, accounts, and services. For example, if you use Lists on Twitter and Facebook to organize, say, your family members, you can create one contiguous timeline which combines both those lists and shows you what your family members are doing, regardless of where they posted it to. Or, you can combine your Twitter followers, mentions, and direct messages together, similar to how Twitterrific works. This saves you time, as it lets you create your own timelines which show you new perspectives on your social networks that you simply can't get with most Twitter clients.

Streamlines tries to make sure you always are looking at the best data, so for every tweet you see on screen, it will update the relative date in real time. So if a tweet is 6 seconds old, it'll update live to say "6s", "7s", "8s", etc. The time is always up to date, so you know how long ago someone actually posted something. Streamlines accounts for API rate limiting, using some advanced heuristics to schedule API requests so that you never run out. This was more a problem two years ago, when you had 30 API requests per hour shared across all your Twitter clients, but still handy. And it handles incoming and outgoing attachments, so if you or someone else embeds content from another site, Streamlines will replace the URL inline with a preview of the image or video.

Under the Hood

Streamlines is backed by two frameworks I wrote, BirdNest and BirdNestUI. They're frameworks because, well, I actually have about 8 apps which use the same Twitter source code, spanning iPhone, iPad, and Mac. This framework tries to encapsulate a lot of functionality - it includes multiple accounts (with credentials stored in the Keychain) spanning multiple services (Twitter and Facebook, with plans to expand into other timeline services like Google Buzz, Foursquare/Gowalla, Yelp, etc.), networking, persistent data, and lots more. It's powered with Core Data and has about 12 open source projects which make up various pieces. The UI framework contains views for showing and creating accounts and timelines, creating tweets or wall posts, and showing timelines in tables (and there's a corresponding UI framework for Mac).

Why Won't It Ship?

There are many reasons. First and foremost, it's at least 6 months out from being released, and that's optimistic. There are lots of bugs, crashes, and UI problems alone, not to mention whole views just not having been built yet. So there's a huge body of work still to be done. I haven't had much opportunity to work on it recently, and there's not much to suggest I'll have more soon.

On top of that, building a Twitter client has become far less appealing than it was two years ago. In those two years, Twitter has drastically increased their feature set to include a TON of things, including geolocation, native retweets, and lists to name a few. The only way to remain competitive is to iterate extremely fast to include every new feature, whenever Twitter announces them. At the same time, Twitter has made several very hostile moves to make it even less appealing to develop on their platform; the most egregious of which was the acquisition of Atebits and their Twitter client Tweetie. There's now such tremendous market saturation for Twitter clients, especially on iPhone. Releasing a new Twitter app now is difficult, as nobody really pays attention to new Twitter clients any more.

Competing with Twitter's free app is hard enough. Even if I could get it to market, there are tons of users who will demand that every feature Twitter offers be crammed into every new release. And everybody's list of must-haves is different; some people will only care about lists, and will rail on you if they can't edit lists. Some will only care about the geolocation feature and if you can put that on a map. Some will only care about native retweets and seeing a list of people who retweeted. Every one of those features, to these users, is a line item on a checklist that needs to exist. To a developer, every one of those features can take weeks or even months to build properly. But every one has to be in there. Oh, and all of those features need to coexist, cleanly, on a small screen, with a fantastic user interface. It is an insanely complex problem to solve.

All of which needs to be built for an app that will not receive much attention, that will be crowded out of the market and will need to be priced cheaply to compensate, and that will be overshadowed by Twitter's own app anyway. There just simply isn't much reason to build a Twitter client anymore. I would much rather spend my time building an app that users will judge for its own merits, not for its completeness in binding to another service.

Going Forward

I'm not sure what to do with Streamlines as it is today. I do think there is a market for niche apps which use Twitter, and I still have some intention of bringing those to market. But I can't say when, and if they ever come, they will not be full Twitter clients.

I'm considering open-sourcing the code, but I'm not convinced of its practicality yet. I'm hesitant to think that someone will want to adopt a monolithic framework for building their own Twitter apps. It's possible they do. But the code is not the cleanest, and there are surely lots of bugs.

That being said, if anyone wishes to build an app with a similar interface, they have my full blessing and encouragement to do it without any permission from or attribution to me needed. I'd like to see more variance in Twitter client UI in general, as the tab bar metaphor is pretty worn at this point.

I would love to hear any and all feedback, positive or negative, either by Twitter or by leaving comments on the YouTube video above. Thanks for watching and reading.

 

Twitter recently introduced a feature on its website called "Who To Follow". This feature presents you with a list of people you aren't following already, but who are active in your social graph. However, I happen to be very proactive in finding new people to follow through a variety of means, and have no need for Twitter to point it out to me. I thought it was a bit obnoxious to see, especially considering both of my first recommendations were people I had blocked.

This Safari extension removes that box from the Twitter homepage, whether you have it turned on for you or not. It's a simple CSS stylesheet that sets display:none on that box. You'll never have to see it again.

You can download it here. I'm still a bit new at Safari extensions, but it should auto-update in the future if I ever release an update.

Update 9/18/2010: Follow Freely 1.1 has been released, with support for the new Twitter web client. It also fixes the issue where Safari would constantly say there was an update available.

Status:

Released.

, , ,    Tags
 

JailbreakMe.com is a website that offers visitors the ability to jailbreak their iPhone without a computer-based tether. It does this by exploiting the system-wide ability for applications to read PDF files, where an incorrectly-formatted PDF file can lead a hacker to do anything they want to your system. While this bug CAN be used maliciously to steal all the personal data from your phone, the developers in this instance used it to enable jailbreaking.

Others will tell you why you should or should not jailbreak your iPhone. Others will decry the developers for bringing to light a serious vulnerability in the iPhone OS. In this blog post, I won't do any of that, but will instead point out some things you should and shouldn't do if you decide to jailbreak.

Backup first, and backup the backup

It should go without saying that, before you start mucking around with the internals of the software on your phone, you should back everything up with iTunes. Sync down all the data into iTunes, and explicitly backup by right-clicking the iPhone in the sidebar and choosing "Back Up". Once that is done, you should backup the actual backup files to somewhere safe. This way, if you ever want to go back to a vanilla iPhone, it's fairly straightforward. The files are located in ~/Library/Application Support/MobileSync/Backup.

Understand what you're doing

Jailbreaking lets you run apps on your iPhone that, for a variety of political and technical reasons, you could not run otherwise. Apple has gone to great lengths to prevent you from running unauthorized apps on your iPhone, and for several reasons; the most important is for security. Since jailbreaking is designed to let you run those apps, that means that in order for the jailbreak to work, several of those security measures are simply shut off and disabled. This does not mean that you'll automatically get viruses and have your data stolen, but it does open up more avenues for hackers to gain access to your data. You simply must be more vigilant and attentive about security when your phone is jailbroken.

Only add sources that you trust completely

When you jailbreak, you will notice a new app on your home screen, called "Cydia". You can think of this as the jailbroken App Store for your iPhone. You will be able to use this to install lots of apps; you can also install mods that change app icons and fonts, mods that change how apps behave, and mods that add new features system-wide. One way this differs from Apple's built-in App Store is that third parties can publish their own list of apps and mods at their own whim, and users can add those lists to Cydia. You can find lists of third-party sources available by doing some creative Googling.

Now, since you can add any third-party list you want, and those lists can contain mods which can access all of the data on your iPhone, you need to be extremely mindful of which sources you add. Seemingly innocuous apps, such as simple wallpaper lists, can contain code which subtly and sneakily siphons away your contacts, or worse. Since you don't have Apple vetting apps before they hit your phone, you won't be able to trust that an app isn't malicious if it's from an unknown source.

Only install what you need

Many of the apps and mods you can download through Cydia will not be things that you can technically do on the iPhone using Apple's published APIs. An example of this is the project which allows you to install a Growl-like UI for push notifications; it simply is not possible to do through the App Store. This means that you will have mods injecting code into the memory of other apps (sometimes into EVERY app). The more mods like this you have, the more they will start to clash with each other. This can lead to crashes, drained batteries, hangs, and system slowdown. You should consciously try to minimize the number of mods that you install, to preserve the experience of your iPhone.

Be mindful of OpenSSH

Packages in Cydia often times will require use of other libraries to achieve their goals. These needs are called dependencies in Cydia, and they will be listed when you try and install packages. There are packages which will blindly install a package called OpenSSH, which installs a server on your iPhone that allows you to log in via a Terminal. Now, this package uses a file on the iPhone to determine what the default password is, which happens to be 'alpine'. As you can imagine, many people don't change that password by default, and instead just let the default stick and never change it; this led to disaster last year when someone used the default password to extort lazy iPhone jailbreakers.

If you install this package, the absolute first thing you should do is change the root password.

Be wary of iOS software updates

In all likelihood, your iOS software updates will be far more involved than non-jailbreaking. The hacks used to enable jailbreaking are usually patched in the next update of the OS. This means that, if you want to keep your jailbreak mods, you will need to wait for the iPhone dev community to release an updated jailbreak procedure. Sometimes this takes hours, sometimes this takes weeks. Once the jailbreak is released, updating generally consists of backing up everything, restoring your iPhone to the new OS, re-jailbreaking, and reinstalling all of your jailbreak software. It is a far more involved process, on top of the already involved update process of the iOS. You will likely update the OS far less than you would if you were non-jailbroken.

 

Caboose is an app that loads notifications from the Boxcar service. It provides a reusable class for interacting with the Boxcar service for receiving push notifications. Currently it loads notifications for one account and dumps them to a Growl feed, but a full UI is planned.

Status:

In development.

 

A great app for viewing non-native video on the iPad.

Page 1 of 4 Next →