This Statement Is False

Approach I started working on my file sharing approach. Sharing will not involve deltas or packs, just chunks, by Git id. The idea is to share Git commits, with all of their associated tree and blob objects. They'll be shared using a manifest for each commit that exhaustively lists all of the chunks in that commit. A manifest will be a binary XML document (using fastinfoset , same as Xus uses for its protocol), since that's a pretty efficient binary serialization format (using BER encoding for numbers in a lot of places and whatnot) and then using another XML doc to list chunks for the manifest: A commit manifest contains the serialized commit data, the commit id, and a list of tree object ids/crcs and blob ids with blob chunk ids/crcs A commit manifest chunk list contains a list of chunks for the commit manifest file, since a commit manifest file could get large I'll probably use an extra branch to store the chunks that a peer currently has for that branch. When it ge...

I play a lot of indie RPGs, like Spirit of the Century and Swashbucklers of the Seven Skies , but for the past couple months, we've been retrying D&D 4e (and I'll be using "DM" below a lot of times in the generic sense). I tried it for 5 months in 2008, running a Keep on the Shadowfell adventure. I have to say that I think that was a mistake for Wizards. It's a dungeon. A lot of the monsters aren't intelligent (rats, oozes, skeletons, zombies). Etc. It ran like a 5 month long miniatures game. Maybe it was me, I don't know. Nevertheless, we're giving 4e another go (this time, I even get to be a player!) D&D 4e has some great ideas and it has great resources for DMs. Since 2005, I've used techniques described in the "Group Storytelling" in the DMG2. You can find a lot of helpful hints for DMs out there on the net and I haven't read nearly all of them, but maybe I use some novel techniques. Some of these techniques wer...

I'm wondering if it would be a good thing to use a variant of the rsync algorithm with Xus file transfer. This would help save a lot of bandwidth for large files that only have small changes. One example is Sauerbraten maps, which can be 2M or more (after compression). When you move a tree, only a small part of the map file is changed, but without rsync behavior, pushing an update requires transferring the whole file. At this point, Plexus breaks files into chunks that are 50K (if I remember correctly) and keys them by SHA-1 hashes. If we include rsync's rolling CRC with the chunk ID, that would be enough (I think) for a peer to determine whether it could reuse a chunk from its current file and avoid downloading a chunk. I'll have to check on rsync to see its default chunk size. We chose 50K based on tuning with Pastry and Xus' behavior will probably be very different.

It's been a long time since my last post; various RL intrusions encroached on Xus. Now I'm starting the file sharing piece. I'm basing it on what I did in Plexus, but starting right away with Git integration instead of putting it off, since that seems to be a better way to actually have Git integration :). Xus has kind of a weird profile for a p2p system. It's made to handle a mass of small clusters, rather than a giant, monolithic cloud. It's really based on the needs of Plexus, but I suspect other systems out there can benefit from it. In a monolithic p2p storage system like bittorrent or FreePastry's PAST, the cloud can (theoretically) be so large that the data can "live in the cloud". In a game like Plexus (or Sauerbraten), however, the clusters are small and may often drop to 0 users (like when the players for a particular world are asleep), so there needs to be a way to restore cluster state (so a world with no players is still up-to-date w...

According to my only somewhat extensive automated tests, Milestone 1 (Basic Xus) is now complete. That includes layers 1 and 2 and the property service (transient and persistent properties). Plexus relies heavily on its property service (currently backed by FreePastry) and I think it will be useful to other programs as well. It's kind of a persistent broadcast; it's really just for relatively small databases, like if you have to track a few hundred things for the topic, like player locations in a game or account credit, for instance. Plexus uses properties for player presence (nickname, which world they're connected to, etc.) and object information (location, orientation, costume, etc.) If you need to store large amounts of data, it might be appropriate to use the file storage service (coming soon and featuring optional Git integration) or maybe to add a database service (and contributing it back to the project would be nifty, too). DHT message routing is useful for ...

Xus has a 2-pronged approach for integrating its file storage with Git: the basic file storage will just use ordinary file storage, with an approach similar to Git's, ensuring that Xus' model is "compatable" with Git's provide a plugin that uses JGit (http://www.eclipse.org/egit/) for people who want to maintain a local Git repository Each peer can be optionally Git-enabled, since the peers will only interact with a local Git repository. It doesn't use Git's protocol to exchange data, because we want Xus to be able to swarm downloads, but a Git-enabled peer could use Git-clone to seed its cache. Since the file storage protocol doesn't have to interact with Git, Xus just needs #1 to make sure that a peer can fetch everything it needs over the net. Since Git uses SHA-1 hashes as keys for almost every type of object, this shouldn't bloat the protocol too much. Note that when I say "protocol" here, I'm talking about the file storage...

I have layers 1 and 2 working now, with automated tests (it's on github). Layer 1 is simple packets and layer 2 is p2p messaging. That covers: strong authentication (challenge/response on connect with RSA keys & signatures) direct peer-to-peer messages broadcast messages unicast messages dht messages (delegated to the peer with the peer id closest to the dht key) delegated peer-to-peer messages (one peer sending to another peer through a topic) Right now, as long as your peer doesn't spoof its public key, it can connect to any topic (authorization is part of layer 3). I think one other thing I should add to layer 1 is a heart beat, so you can quickly detect dropped connections (TCP timeouts are notoriously unreliable). Layer 3 is the service layer and has these components: Properties Topic authorization and management (including peer "accounts") File sharing Port configuration testing Connect-back requests (for when only one peer can accept connections) A note...