1. The Server
The first step for an open alternative is an Open Source server, which provides a user-friendly interface like Facebook. Now a protocol must be developed, so different instances of this server can connect with each other. Users at one server must be able to friend user profiles on another server. The biggest problem is probably to find a dummy-proof process for "inter-site friending". My current guess is: Bookmarklet.
A further requirement for this friending is privacy and security. Every user must have a public-private key pair. To friend somebody means to give her my public key, so she can send me encrypted status updates, which only i (more precisely, my server) can decrypt. She could even check the fingerprint of my public key with me in person to ensure the identity of my request. Another challenge for people with more knowledge in cryptography is the concept of groups. My quick and dirty suggestion is that a server hosts groups (or things like Facebook Pages) just like another user profile.
2. The Application
The obvious problem from a cryptographers point of view is that i need to trust the server, where my account and data is stored. While everybody is free to host its own server, this is not an option for most people. Therefore another protocol should be developed, which allows the user to keep its private key on his desktop. The server only needs to host a proxy profile, which acts as a status message store like an email server, because the user may not be online all the time. The user can fetch status updates from his friends from the server and decrypt them locally. For other servers there is no difference between a proxy and a real profile, so this second is step purely optional and can be deferred.
3. World domination
The heart of this project are the protocols between servers and applications. Once these are stable, alternative implementations should be written and encouraged. iPhone apps, Android apps, high-speed servers, and many more things come to my mind. To encourage developers to build these things, the protocols must be kept really simple.
The clever twist would be build a server that makes Facebook profiles available into the open network by offering an app to Facebook users.
More reading ...
There are already a lot of initiatives on the web. Foremost the Ostatus spec, which feels like an extended Twitter protocol, but it references and bundles other stuff, like PubSubHubbub, ActivityStreams, Salmon, Portable Contacts, and Webfinger. Transitively, also read about OAuth, Open Social, and DiSo. Personally, i would probably not use those protocols and go for a quick prototype instead. One should read through them, though, because they list a lot of subtle requirements.
Good luck, Diaspora!