Published: 2012-11-27 18:50:13 by Daniele Ricci

Kontalk is a collaborative network: it relies on multiple servers working together as a cluster. From a storage point of view, it's not a simple mirroring cluster; a Kontalk network can handle failover and load balance.

Failover

This is very simple: if a client can't connect to server A, it will simply connect to server B. It will be able to use its credentials with server B because of the recognized signature by server A. But what about user-related data? What if a user vCard is stored on server A, B wants to retrieve it but A is not available?

Load balance

This is also simple for client connections: if server A has too many connections, client will go to server B. Storage would be the same: if everyone upload files to server A, users will connect only to server A to download them. This way, server B will host no files.

Solution 1: replication by caching

All servers are pretty much always connected with each other in order to accomplish several tasks, such as forwarding messages, probing presence. Even when broadcasting presence, the net component will connect to every server for sending the presence stanza — because the router would broadcast to every route.
This solution is quite simple: whenever a change occurs, the server will broadcast it to the network. Of course there could be many optimizations to that, including:

  • cache expiration
  • broadcast only to some servers, based on user presence and subscriptions

Whenever a server needs information, it can always request it manually. The main con here is server unavailability: server A will not be able to obtain user info only stored on server B.

Solution 2: IRC-like full replication

On server join, a full presence situation and all user data are sent to the joining server by another one. This is an immense waste of resources, but I need to investigate if it can bring some benefits.

Solution 3: distributed hash table

This is actually my dream storage: real partitioned distribution. A DHT is ideally the perfect storage — at least on paper — because of the distributed nature of the Kontalk network. If DHT communication is secured with strong encryption and protected by access privileges, it could bring huge benefits, not to mention the reliability it could have.
But, as all things, it has its cons: synchronization. A presence system needs to be synchronized to the second. Even if with solution 1 we might have synchronization issues too, a DHT is a far more complex beast to handle.

Conclusion

Kontalk needs to store various kinds of data:

  • presence information
  • vCard data
  • privacy lists
  • media files metadata

I guess the solution I'll use will be #1, because it's relatively easy to implement and debug. I will try to optimize it to extreme, but, of course, I will fall into DHT eventually.

Published: 2012-11-23 15:52:11 by Daniele Ricci

Kontalk XMPP server is now under Gitorious hosting!!! You can find it in xmppserver repository:

http://gitorious.org/kontalk/xmppserver

Although it's still a bit rough to setup and run, it does work with any XMPP client. We'll publish further information — and possibly tutorials — soon.

Published: 2012-11-23 12:40:32 by Daniele Ricci

I'm studying how to implement message delivery receipts on the new Kontalk XMPP server. I hit these two extensions:

XEP-0184: Message Delivery Receipts
XEP-0079: Advanced Message Processing

The former is about client confirmation: receipt is sent voluntarily by the peer, so it's possible for a user to disable them — that could actually be a feature.
The latter is about server and inter-server delivery: it provides a more advanced way of knowing (especially between two servers) if the message has been stored, delivered, bounced, etc.

In a first release, I will implement XEP-0184 only, so the server would not be touched — it is a client feature. In privacy settings users will be able to decide if the client should send delivery receipts.
However, AMP would be mandatory if I want to make s2s fully functional and reliable.

If a client receives a message and before sending the receipt the connection is lost, clients can just persist the receipt and send it later.

Published: 2012-11-21 19:46:50 by Daniele Ricci

Today someone told me:

The admins of the server I use don't allow the installation of PHP due to the numerous and for years not decreasing number of security problems with the language itself and the majority of applications using it

That really doesn't surprise me: I know PHP can be a real pain when it comes to using some monster apps such as Wordpress, Drupal, Joomla, and so on. Actually I don't use them, but I was curious, so I googled:

php security issues

OMG! I knew PHP wasn't safe, but as long as you write your application in a safe way and correctly setup your hosting, you should be ok. At least I like to believe I should be ok. By the way, this checklist is very interesting:

https://www.owasp.org/index.php/PHP_Top_5

This blogging platform is written in PHP...... Should I get worried?

Published: 2012-11-20 12:26:13 by Daniele Ricci

I've been evaluating the XMPP protocol for a long time now. Before starting Kontalk I took a really good look to XMPP, thinking if it was worth implementing a server (Kontalk is different from a plain standard XMPP server, so I can't use any existing implementation). Let's say because of me being alone in such a decision, not having other point of views, etc. and also because (I admit it) of my laziness studying a new protocol, I decided to build my own. Actually there are some advantages with my protocol over XMPP, such as:

  • protocol buffers are very lightweight
  • fits very well in the Kontalk architecture design (because it's built right for it)

But: going over these months, I'm realizing that the community will not accept a new protocol standard for IM (even if it is open), unless it's already very wide adopted; being XMPP the de-facto standard today, a brand-new protocol, in order to replace that kind of beast, would have to bring tremendous benefits that justify not using XMPP in the first place. Implementing an XMPP server will allow Kontalk to:

  • chat with users on other XMPP servers (e.g. GTalk)
  • use exisiting protocol implementations: libraries implementing XMPP exist for almost every programming language on Earth
  • use an existing XMPP client (with a few modifications for buddy list integration, but works out of the box for just messaging)
  • easily debug traffic data (this is not really important, but it deserves to be in this list)

Of course XMPP is not 100% fitted for the Kontalk architecture as I designed it, so a Kontalk XMPP server will not be 100% standard XMPP:

  • there is no concept of roster (buddy) list: presence subscription is requested on-demand and will be discarded when client disconnects
  • no roster list means no authorization: a contact may send a message to any other without prior subscription approval (this is intended — I know: "what!? blocking users is not possible? I have a stalker sending me messages every minute!!!!" — blocking users will be available by other means, but it will be blacklisting, not the contrary)
  • registration and authentication are a bit different than a normal XMPP account registration

JIDs in Kontalk would be something like

e73ea3be23d0449597a82c62ed981f584a5c181b@kontalk.net/LO3L8CE4

SHA-1 hash of phone number, remember?

Servers within a Kontalk network will communicate with each other using a slightly different server-to-server protocol.

Though I haven't fully made up my mind yet, I've been working on a prototype of XMPP server for the last few weeks, which I will probably push to Gitorious soon.