This kioslave is a wrapper around KTorrent's DBus interface which will
allow you to transparently use torrent web shares inside of KDE. It is
also a reference implementation for a heavy extension of the magnet
URI scheme.

DISCUSSION
==========

Join the KDE mailing list (default list for ktorrent usage), if you
want to discuss this magnet-concept:
https://mail.kde.org/mailman/listinfo/kde/ For questions around the
BitTorrent implementation and stuff around the general status of
torrent support in KTorrent go to its forums:
http://www.kde.org/applications/internet/ktorrent/

Send bug reports to ch_weil AT web D0T de for now. Likely I will use
the KDE bugtracker in the future.

INTRODUCTION
============

This is a very powerful tool and most likely the heaviest stretch of
the KDE libraries involved. Although you might consider depending on a
GUI application (via DBus) like KTorrent for an ioslave hackish, it
works quite well and has some benefits (rough summary):

a) Quick prototyping. I wouldn't have had the time to get that going
with writing a whole new Torrent client, even if I'd reused some of
the awesome FOSS libraries, in this case especially libktorrent. If
the slave fails or you simply don't know anymore what is going on you
can simply use KTorrent's UI to manage anything that is going on and
you don't need another control UI.

b) Using a library would have been a bad idea actually, as this would
mean that you will setup a torrent client for almost each file access,
even to the same torrent. Not only would that mean that you would have
to forward many ports, you'd also have to keep the ioslaves in sync.
This would be an error-prone system. The right solution for the future
in my opinion is some headless daemon with an extensive DBus inteface,
which can then be managed by an UI (for example KTorrent's).  That way
all of KDE/the Freedesktop would get a torrent service which can be
used without having the GUI of KTorrent setup, but being able to use
it if you like to. This involves a very heavy redesign of KTorrent
though, much more than what I would have been able to achieve.

c) Memory usage. Using only a thin DBus inteface makes this solution
quite efficient. Although this is not the primary design concern it is
important to keep an eye on ressource usage. Especially if you want to
compete with http.

d) Can use any of KTorrents plugins to improve the BitTorrent(c)
experience. The best examples are the UPnP and the Zeroconf plugins,
which make setting up your router for port-forwarding or sharing
already loaded ressources (e.g. webvideo youtube(c)-like) inside your
lan a breeze.

The disadvantages are:

a) Depending on GUI. You cannot use this in a terminal without X with
kioclient for example.  Don't know if that would work anyway, have
never tried it. You would likely use rtorrent in that case anyway and
the benefits of kio_magnet are very limited here.

b) KTorrent's UI needs to be running. This costs some memory. KTorrent
is very well designed and stable though and the UI is well seperated
from the core functionality (which resides in libktorrent).

c) I have to use multithreading for the event loop of DBus. Sadly this
is not really easy to debug.

MAGNET URI/URLS
===============

The slave heavily extends the current magnet URI-scheme for
magnet-links, while keeping backward- and standards-compatible, where
possible.

New parameters introduced (all optional):
=========================================

1. to

"to" references a (http) ressource for the torrent file and therefore
speeds up loading the metadata.  This in fact makes linking seperate
torrent files unnecessary and makes magnet links a sane default.
Trackers of both the link and the file get merged. Note: Since we
define an ioslave here, we can even reference torrent files from
inside another torrent via a magnet-link. So you could simply ship
some archives of torrents to be hardly censorable even for barely
shared torrents.

2. pt

"pt" contains the path to a file/dir inside the torrent. This allows
to directly download and open files from a multifile torrent. It also
makes shipping collections of files in a single torrent more painless,
because you can still address them directly for the user.

3. sp

"sp" contains an integer which gives a target stream speed (kib/s)
which is necessary to consume the torrent.  kio_magnet currently
assumes 200 on some filetypes. While streaming is considered to be
suboptimal, because chunks don't get distributed most efficiently but
by progressively loading the file, it is the only way to make the
application of stream-downloading possible with torrents. So it is no
solution to simply drop the feature from p2p just because it has some
bad side-effects. Most of todays web-video (I don't know any other) is
streaming capable. Users do expect the video to playback nearly
immediatly.  Therefore KTorrent and many other commercial clients, in
opposition to Transmission for example, have decided to allow stream
downloading.  To soften the bad sideeffects of streaming (which can
actually dry out the last chunks of the streamed files and choke the
torrent as pointed out in the Transmission arguments against
streaming) it should only be applied if it can reach the target
streaming speed and it should not be applied above that speed. So
while you might streamload chunks to achieve a speed of 200 kib/s you
might as well keep loading other chunks for 100 kib/s more. Doing that
you in fact buffer as well, but all over the file. If you cannot
achieve your target stream speed the client should fall back to
standard loading and retry after a few minutes (notifying the user to
wait for a moment).  That should make streaming bearable for torrent
health, especially since upload bitrates should improve massively in
the next years.  Sadly streaming is a brand-new feature in KTorrent
(and therefore in kio_magnet), so for now it is really dumb and
streamloads whenever a sp paramter is defined. It will be implemented
correctly in the near future. It also is only possible to streamload
one file per torrent as streaming several files is useless. You likely
won't watch two videos at the same time. If it should become necessary
this could change in the future. Anyway, that should not effect the
definition of this parameter.

4. pf

"pf" defines prefetches in a comma seperated list. If it contains "all",
all files will be downloaded, when the torrent is added. Otherwise you
define a list of paths like in pt to only select some files that you
want to be fetched as well. This helps if you load linked ressources
like html pages or slideshows. The tradeoff for additional traffic and
disc space might be low compared to the extra time needed to
reestablish connections to fetch the next chunks. Take a webpage for
example. You might want to prefetch the css, some javascript and the
pictures. This can make your url very long depending on your
setup. "all" helps out here, since the files accessed through the
slave will still be higher priority and therefore finish quicker.  So
instead of prefetching everything one level away, you should prefetch
files needed to view the content and fetch the rest with "all". pf may
also include whole folders inside the torrent, so you don't have to
point to files selectively. Consider modularizing your torrents
appropriately, for example one torrent for each slideshow instead of
one for all slideshows and then build a metatorrent which puts in some
html glue. Files loaded directly always have the highest priority,
while files prefetched have a "high" priority set.
Note: Streamdownloading does not give you higher priority. It is best
to avoid it for all static files if they are not really stream
content, otherwise you will slow down the swarm.

Example:

magnet:?xt=urn:btih:022692d131d3a44d770b38498022dffc9769104d&dn=Sintel.2010.Theora.Ogv-VODO&tr=http%3A%2F%2Ftracker.vodo.net%3A6970%2Fannounce&to=http%3A%2F%2Fvodo.net%2Fmedia%2Ftorrents%2FSintel.2010.Theora.Ogv-VODO.torrent&pt=Sintel.2010.Theora-VODO.ogv&sp=200

LIMITATION OF AN URI
====================

While this already improves magnet-links a lot, it still is worse than
necessary. For some reason the inventors of magnet-uris have decided
to not define a URL-scheme. While basically you should be able to use
the host/path part in magnet, most clients (for example
[MICRO]torrent) don't load them anymore.  Even KTorrent didn't accept
that in the first place, because it didn't do URL parsing but rather
String parsing.  Now you can already reference any file inside a
torrent with this code, in fact kio_magnet can also load directories,
but you will quickly recognize that url aware applications like a
browser or an html-renderer won't be able to find out the path and
won't let you browse the ressource like you would do with http/ftp.
This is not a big issue on proprietary systems, since they won't let
you plugin new IO-URL systems anyway, but FOSS systems are mostly
build in a much more generic and flexible way. So what you need is
unique location (hash as host) and a path build on top of the
location. This is the new, not backward, but feature-compatible
[bijective] mapping of the magnet scheme:

magnet://HASH.TYPE[.HASH.TYPE]*/PATH?QUERY

Note: I can also insert "subdomain"-like identifiers before the hash
like wikileaks.HASH.TYPE

The query can stay the same, except that you don't need the xt
parameter anymore, since it has moved into the host part. You still
can reference as many hashes followed with their type as you want, so
this is not limiting the old scheme to BitTorrent(c) somehow, but it
allows us to use it the same way like http, ftp or file. The query
also is no longer necessary, since the only mandatory parameter was
xt, which is now in the host part.

Example:

magnet://e817b88024a258b9777fc198fc0e75a149600dc2.btih/wired-creative-commons-cd.txt
magnet://eac3b61d4c1cf1cbbf54fd6bf3c069eee2ce2abc.btih/index.html?dn=wikileaks+mirror+2010-12-20&to=http://dl.btjunkie.org/torrent/wikileaks-magnet-p2p-storage-kio-magnet/4122eac3b61d4c1cf1cbbf54fd6bf3c069eee2ce2abc/download.torrent&pf=all,IMG/favicon.ico,squelettes/css.css,squelettes/random.js,IMG/wlogo.png,IMG/jpg/ja-main.jpg

LIMITATIONS OF THE CONCEPT
==========================

a) FROZEN CONTENT The most obvious limitation is that you can't change
files. So if you want to dynamically create some content your are
screwed. While this does not lower the value of the concept and there
still can be other protocols like git integrated in the IO-layer in
the future, a slow update solution is possible. I am currently
thinking about simply allowing to define another parameter which
defines hashes of torrents to overlay in a stack like approach. So you
would fetch the torrent for the month and then a daily update torrent
with only the file changes included (read incremental backup).  You
can then at any time change the link to a new base torrent, if the
changes grow too large. It is also reasonable to split torrents
appropriately, so you could for example ship an updated sqlite db
everyday and write a quite static web app around it to access the
data.  NOTE: This also has a benefit, meaning you can actually treat
files like local files and don't have to check if they have changed,
because they can't.

b) WRITE ACCESS Well you can use the above (a) concept and a central
repository to allow changes and then distribute them once you need
them to.

c) LATENCY Obviously you have some latency especially when accessing
small files. Even if you use the to+pf parameters it still takes time
until the connection for the chunks get made. This improves within
healthy swarms, but still if you try the wrong peer in the first place
(e.g. a blocked or recently disconnected one), you will have to wait
for a timeout.  If you want to have low latency you can use webseeds
or simply stick to http for small files. The advantage of webseeds (in
KTorrent they are used very conservatively albeit barely) is that you
will get the advantage of http/ftp without any drawback.  This has to
be tested out though.

PERSPECTIVES OF THIS TOWARDS
============================

a) FOSS Although this is a standards proposal and I can imagine
Opera(c) and even some other proprietary systems implementing magnet://
links support, it is not very likely to get supported by Micro$oft or
App1e. In fact the overall future of P2P depends very much on
FOSS. This is most likely true since eMule and especially BitTorrent,
as it has been developped completely in the open, which made it the
most sophisticated and used P2P technology today.  Deep integration of
magnet-urls in FOSS can be expected, and I really hope for it, in
GIO/Transmission, Firefox/Gecko and hopefully curl/libtorrent as
well. In general it should be possible to use magnet wherever http can
be used, even mixed, to get the optimal performance.  This will be a
very important advantage over the proprietary competition.  They want
to deal with or even invest into the digital media companies.  They
don't want an open web here, so this might be a deadlock for them
which can be exploited to FOSS' benefits.

b) COMMERCIAL WEB At first this is only another possibility to store
static content. It won't outwigh http on www-sites likely soon, since
the latency is simply too high. With a snappy webseeding feature it
could be an easy and cheep scabillity/backup/uncensor bonus to use it
though.  Bigger files, which still cost you serious money and lock you
in, even if hidden by ad-financement(Sharinghosts) or leaking all your
private communication, e.g. 6oogle(c) services, are very interesting
now already though.  This is since you barely can access the
http-files directly from everywhere and you don't have the right to
publish (read host) yourself. You are always dependant on a web
service hoster and the money to be made from it. Though it will be
attractive for small highly frequented/censored sites now, too.

In the longer term though the files which are stored in P2P will
become interesting for metadata providers, especially social services.
But if they want to actually provide services for this part of the
web, they have to join and share their bandwidth to distribute as
well. Otherwise they will be banned from file access, as they are
leechers only.  This does not only make legal prosecution more
difficult, it will also force them to share in the network which can
seriously back it up.  Although it is often said that only with total
control metadata can be generated nicely for a social web, this is not
true here. You can still inform others about the resources you use,
but the control will be with you and not your service provider. This
will also distress node points of the underlying infrastracture and
solve the caching problems very elegantly (read near
perfect). Additionally it proves that if you want to be an equal user
of the web you need the same upstream as downstream for your
connection, which will heavily unchoke the infrastructure, as this is
BitTorrents overall goal.  Note: I don't say that commercial services
won't be possible with this technology. The opposite is true. By
opening and decentralizing the general communication infrastructure
you make many new innovative and profitable applications of the web
only possible. It also does not say that commercial providers in P2P
are not integratable or even discriminated against. The opposite is
true. This will be very interesting for commercial usage, if
understood correctly.

c) P2P AND ANONYMITY kio_magnet approaches the P2P future orthogonal
compared to things like Freenet, GNUnet or OneSwarm, which try to
create darknets. This is a fake battle you cannot win, because people
won't use it (it's always much slower than direct connections). You
have to develop P2P technology which gives benefits to people the same
way FOSS does for software. Give them a reliable, open, fast and
flexible system that is not disturbed from commercial interests and
cannot be monopolized and voila you are going to succeed in the long
term. We definetly need more like this, especially for metadata of the
web. The best safety for open communication is to make it more
economical. And to make it more economical you simply have to
socialize and maximize each others freedoms. That is the not so secret
secret of the success of FOSS. E.g. you can crack a commercial
software or you can write a free implementation. The crack will
quickly set you up, but the free one will outrule the commercial one
in the end (even if it takes decades). But then the crack will always
be, well, a crack. The same counts for networks.  If you can't provide
a successful open communicational infrastructure you get locked in, no
matter how much you try to hide.

Note: That does not mean that I am against darknets or anonymizing
functionality. I am running two TOR nodes on my servers.  Having only
a short look at our history, this is very important. The only point is
that this is not the general perspective. You have to be able to use
it every day legally for it to succeed. In fact darknet functionality
(especially encryption support) can really help the proposed usage
here a lot. It just should not split the network up, like OneSwarm for
example does, but has to make anonymizing seemlessly possible if
wanted and only then, not in general. Don't force users onto your
perspective of the web, but let them chose between commercial
providers with http, p2p ressources (magnet), central/decentral
web-applications with a mix of both, seperately allow public and
private (darknet) BitTorrent usage selectively and make all that
transparent to the application stack, so you can chose your channel of
communication freely everywhere.

d) BITTORRENT Bittorrent is still very limited to fixed GUI's even in
FOSS (like Miro or even KTorrent(+loaded plugins) itself), which make
todays torrent-software bloatware and very badly integrated into the
overall system. This is one reason why Joe the average user prefers to
follow some adblown webinterface (read youtube, porn sites, illegal
movie hosters,... ) to streamwatch the movie he wants, because that is
very likely to work without having to figure out another GUI. I don't
have anything against rich interfaces, but you should keep to the Unix
paradigma not for programs but at least for functionality on the
desktop as well. Every technology has its place and BitTorrent(c)
definetly belongs in the IO layer.  Note: I am not opposing rich
clients, with bloatware I mean, they need unnecessary bloat to
actually provide useful functionality. KTorrent for example include a
media-player plugin to stream watch movies. While this works the right
solution is definetly to load the stream in your default player. Miro
is really nice, but I don't like to depend on it. With kio_magnet you
can in fact transform it into a website which you can load in any
browser supporting magnet links.

e) THE WEB AND CLOUDCOMPUTING Today the web is mostly focused on http
connections and something else on the server side, which is when it is
distributed is now called cloud computing.  This has a lot to do with
the closed nature of web technologies, especially with the dominance
of 1nterne1 3xplorer. But it is losing ground massively to open
browsers because they make improving the web communication possible in
many ways. Still this has not changed the very old http-default. Now a
good solution, as shown by IP/TCP protocolls is not to reinvent the
wheel, but to find better technologies complementing the existing
infrastructure. With magnet ressources you can easily distribute your
storage to all users. At first I imagine a heavy switch of webvideo
here.  This is not very interesting if you build services to lock in
and monetize users, but it is very nice for free platforms and an open
web. You can for example distribute your web-app (written in GWT e.g.,
to take some openly perverted example) and ship it with some sqlite
database and make it only connect some service once in a time or not
at all. You can then handle the browser the same way as Google likes
to do it from the server/commercial side, but still keep all the
flexibility of an open web.  You can use it as a fallback VM for
platform agnostic apps, but you are not bound to a central server
anymore. In fact you can create package-management systems to deploy
website components freely to every browser on the web without hitting
unbearable bandwidth problems.

f) KDE/THE FOSS DESKTOP Well, this was and still is only possible on
KDE. This is not by accident. To be able to really create new
communication and interaction you must have a very flexible
infrastructure which is built with technology extension in mind.  I
don't want to blame anybody, I have used Gnome as well and also like
highly customized setups. I can even imagine to live in Emacs most of
the time, if what I do is primarily plaintext related. But I think KDE
(and Qt) has a very sensible perspective on the longtime future of
free GUI frameworks. So while others might have better usability
(Gnome) or stability at times, KDE can generate enormous synergies in
the long run and I'm curious about its future. Its universal
perspective for designing frameworks is what really makes it stand
out.  For the near future, projects like Owncloud promise a nice
integration with a private tracker for friends for example. One could
also autoupload torrent files there and set that as a default in
KTorrent/kio_magnet. I really hope for my Gnome friends out there that
the Transmission guys rethink their decission about streaming video
and rather try to implement it best instead of simply dropping
it. With the necessary DBus interface it shouldn't be hard to write a
similar module for GIO then.

This project is the result of a "green" brainstorm. Legal1ze 1t!
Watch "The Union, business behind getting high" if you think I am weird.
