Enter The Freenetrix
The Freenet Help Site
Enter The Freenetrix
Licences used on this wiki
Welcome admin to Freenet WikiServer
Edit Page - Diff - Revisions - Title Search - Preferences - Bottom
 
Freenet Client Protocol 2.0 Specification
    • Note that this specification is a work in progress, and is subject to change. Third-party application developers are encouraged to experiment with FCP 2.0, but should be prepared for backwards-incompatible changes. We suggest that these developers track changes to this page through [[1]].**
Purpose
The Freenet Client Protocol is a simple, text-based protocol designed to allow third-party applications to interact with [[2]]. Supported functionality includes:
- Inserting of data into Freenet
- Retrieval of data from Freenet
- Querying the status of Freenet

History
FCP 2.0 is was first supported in [[FreenetZeroPointSeven Freenet 0.7]], and is a non-backwards compatible replacement for the original Freenet Client Protocol supported by versions of Freenet prior to 0.7. The protocol is not backwards compatible because of the fundamental nature of the changes to Freenet 0.7 relative to previous versions of Freenet. Downloading large files with the old FCP API was very complex, and involved a great deal of code duplication between different client apps; the philosophy of FCPv2 is that it should be possible to write simple apps, but powerful options should be available when needed. The node does most of the work (e.g. reassembling splitfiles) for the client, unlike in Freenet 0.5's FCP. If you need additional functionality from FCP, please contact us, we may well be able to give you it.

Transport
FCP operates over a TCP connection, typically between Freenet and another application running on the same computer. By default, Freenet listens for FCP 2.0 connections on port 9481 (note that previous incompatible versions of FCP used port 8481).

Basic Protocol Syntax
The basic unit of FCP is the message, which consists of a message name, and 0 or more key-value pairs called "fields". Message and field names are typically specified in [[3]] using alphanumeric characters. Messages can be sent from client to server, or from server to client. Multiple messages can be sent in either direction one after the other.
%%
Message Name
Field 1=Value 1
Field 2=Value 2
...
End
%%

Some messages may be followed by a "payload" of binary data of a length that will be specified within the message itself.

For ease of debugging, blank lines between messages are ignored.

Messages

ClientHello

This must be the first message from the client on any given connection. The node will respond with a NodeHello.

%%
Client Hello
Name=<my name>
Expected Version=2.0
End Message
%%

The Name parameter is used by the node to uniquely identify a client (e.g. for persistence). Only one connection for each Name is allowed. We do not do anything with ExpectedVersion yet, but it may be verified in the future.

NodeHello

%%
Node Hello
FCPVersion=2.0
Version=Fred,0.7,1.0,399
Node=Fred
Testnet=true
Compression Codecs=1
End Message
%%

All fairly self-explanatory: the node tells us what version it is, what protocol version it's using, whether testnet mode is enabled, and how many compression codecs are currently supported (this is important with the StartedCompression message).

CloseConnectionDuplicateClientName

%%
Close Connection Duplicate Client Name
End
%%

Indicates that this connection is being closed because another connection has been opened with the same client Name. The latter takes precedence, because the reason it was opened might be that the first connection was broken.

ClientPut

%%
Client Put
URI=CHK@ // could as easily be an insertable SSK or KSK URI
Metadata.Content Type=text/html // MIME type; for text, if charset is not specified, node should auto-detect it and force the auto-detected version
Identifier=Insert-1 // identifier, as always
Verbosity=0 // just report when complete
Max Retries=999999 // lots of retries; -1 = retry forever
Priority Class=1 // fproxy priority level
Upload From=direct // attached directly to this message
Data Length=100 // 100 bytes decimal
Get CHK Only=false // true = don't insert the data, just return the key it would generate
Global=false // see Persistence section below
Dont Compress=true // hint to node: don't try to compress the data, it's already compressed
Client Token=Hello!!! // sent back to client on the Persistent Put if this is a persistent request
End
<data>

or
Upload From=disk // upload a file from disk
Filename=/home/toad/something.html
End

or

Upload From=redirect // create a redirect to another key
Target URI=KSK@gpl.txt // some other freenet URI
End

%%

Neither IgnoreDS nor DSOnly make sense for inserts.

Verbosity is a bitmask; at present, for inserts, two values are supported:
1 = splitfile progress (send SimpleProgress messages)
512 = compression start and end (send StartedCompression and FinishedCompression messages)

URIGenerated

%%
URIGenerated
Identifier=hello
URI=freenet:KSK@something-1-2
End Message
%%

This indicates the final URI of the data inserted; this may not be the same as the original URI, which could have been CHK@ ("insert this as a CHK and tell me what its CHK is"), or an SSK using a private key.

PutSuccessful

%%
Put Successful
Identifier=<identifier>
URI=<uri to fetch>
End Message
%%

Indicates a successful insert.

PutFailed

%%
Put Failed
Code=9 // error code
Identifier=1 // identifier
Expected URI=freenet:KSK@hello.toad // insert URI if it succeeded
Code Description=Insert collided with different, pre-existing data at the
same key // description of error
Extra Description=SSK collision // more info on the error
Fatal=true // No point retrying; permanent error
Short Code Description=Insert collision // short version of Code Description
End Message
%%

It is also possible to have a more complex PutFailed: (this is an example, the strings and codes returned may be different)

%%
Put Failed
Code=8
Identifier=7
Expected URI=KSK@hello.toad
Code Description=Too many retries in blocks
Extra Description=Splitfile failed because many blocks had too many retries
Fatal=false // can retry, but we ran out of retries for at least one block
Short Code Description=Too many retries in blocks
Errors.7.Description=Route not found // the number is that particular error code
Errors.7.Count=457 // 457 block inserts were abandoned due to RNF
Errors.4.Description=Rejected due to overload
Errors.4.Count=781 // 781 block inserts were abandoned due to overload
End
%%

ClientGet

%%
Client Get
Ignore DS=false // true = ignore the datastore (in old FCP this was Remove Local Key)
DSOnly=false // true = only check the datastore, don't route (~= htl 0)
URI=KSK@gpl.txt // key to fetch
Identifier=Request Number One
Verbosity=0 // no status, just tell us when it's done
Return Type=direct // return all at once over the FCP connection
Max Size=100 // maximum size of returned data (all numbers in DECIMAL)
Max Temp Size=1000 // maximum size of intermediary data
Max Retries=100 // automatic retry supported as an option; -1 means retry forever
Priority Class=1 // priority class 1 = interactive
Persistence=reboot // continue until node is restarted; report progress while client is
connected, including if it reconnects after losing connection
Client Token=hello // returned in Persistent Get, a hint to the client, so the client
doesn't need to maintain its own state
Global=false // see Persistence section below
End Message
%%

Several ReturnType's are supported:
- direct = return the data directly to the client via an AllData message, once we have all of it.
- none = don't return the data at all, just fetch it and tell the client when we have finished.
- disk = write the data to disk (set Filename=<where you want to put it>, and optionally TempFilename=<temporary file in same directory>).
In future, ReturnType=chunked may also be supported (return it in segments as they are ready), but this is not yet implemented.

Verbosity is a bitmask, as with ClientPut. At present only one optional value is supported:
1 - send SimpleProgress messages

DataFound

%%
Data Found
Identifier=Request Number One
Metadata.Content Type=text/plain;charset=utf-8
Data Length=37261
End Message
%%

This indicates a successful fetch of the key, but does not actually include the data.

AllData

%%
All Data
Identifier=Request Number One
Data Length=37261 // length of data
End Message
<37261 bytes of data>
%%

For a ClientGet with ReturnType=direct, the data is returned directly to the client, all at once, using the AllData message. Obviously in many situations this will not be desirable, hence the other ReturnType options.

GetFalied

%%
Get Failed
Code=<error code>
Code Description=<description of error code>
Extra Description=<extra description of this specific error, if available>
Fatal=<true |false; true means there isn't much point retrying as it's
probably an error with the data>
Short Code Description=<short code description>
Identifier=<identifier passed in by client>
End Message
%%

Complex GetFailed's are also possible; the format is identical to that of a PutFailed.

ProtocolError

%%
Protocol Error // indicates an FCP protocol error
Code=1 // error code
Code Description=Client Hello must be first message // description of error
Extra Description=Duh // more info on the error if available
Fatal=false // means the connection stays open
Identifier=<ident> // if we managed to parse one
End Message
%%

GenerateSSK

%%
Generate SSK
Identifier=<my identifier, sent back on SSKKeypair>
End
%%

Ask the node to generate us an SSK keypair.

SSKKeypair

%%
SSKKeypair
Insert URI=freenet:SSK@AKTTKG6YwjrHzWo67laRcoPqibyiTdyYufjVg54fBlWr,Aw USJG5 ZS-FDZTqnt6skTzhxQe08T-fbKXj8aEHZsXM/
Request URI=freenet:SSK@Bn HX Xv 3 Fa 43w~~iz1tNUd~cj4OpUuDjVouOWZ5XlpX0,Aw USJG5 ZS-FDZTqnt6skTzhxQe08T-fbKXj8aEHZsXM,AQABAAE/
Identifier=<my identifier from Generate SSK>
End Message
%%

The insert URI contains the private key, it is the URI you should use when inserting.
The request URI contains the public key, it is the URI you should distribute for people to request from.
It is possible to quickly go from the private key to the public key (by inserting something with Get CHK Only=true with the privkey, the node will tell you the corresponding pubkey request URI) but not vice versa.

IdentifierCollision

%%
Identifier Collision // client tried to reuse an Identifier
Identifier=1
End Message
%%

Identifiers are unique to the specific request. They can be reused after that request has completed and been forgotten.

StartedCompression

%%
Started Compression
Identifier=Insert-1
Codec=0 // first codec
End
%%

Indicates that the node has started to attempt to compress the data with the first codec.

FinishedCompression

%%
Finished Compression
Identifier=Insert-1
Codec=0 // first codec worked best
Orig Size=89000000 // was originally 89MB
Compressed Size=752 // compressed is 752 bytes; pretty good compression, maybe it was all zeros!
End
%%

Indicates that the node has finished trying to compress the data.

SimpleProgress

%%
Simple Progress
Total=12288 // 12,288 blocks we can fetch
Required=8192 // we only need 8,192 of them (because of splitfile redundancy)
Failed=452 // 452 of them have failed due to running out of retries
Fatally Failed=0 // none of them have encountered fatal errors
Succeeded=1027 // we have successfully fetched 1,027 blocks
Finalized Total=true // the Total will not increase any further (if this is false, it may increase; it will never decrease)
Identifier=Request Number One
End
%%

This indicates the progress of a large request (usually a splitfile, a big file which is split across a large number of blocks). Note that before FinalizedTotal=true, Total may vary wildly on a ClientGet, because we may follow redirects, have to fetch multi-level splitfiles and so on. However, once we are fetching the final splitfile, FinalizedTotal will be set to true. Whereas on a ClientPut, we can't generate the metadata until quite late on, so it takes a long time to set FinalizedTotal=true, but this doesn't matter as the Total will not increase very much (since the majority of the insert is the actual data and check blocks).

WatchGlobal

%%
Watch Global
Enabled=true // watch the global queue
Verbosity Mask=1 // only let through the basic completion messages, and Simple Progress; not any other progress messages
End
%%

See the section below on persistence.

ListPersistentRequests

%%
List Persistent Requests
End
%%

Ask the node to list all persistent requests for this node (and for the global queue too if watch global queue is enabled). Node will respond with a series of PersistentGet/PersistentPut's, possibly some DataFound/GetFailed/PutFailed/PutSuccessful's, and possibly some status messages (SimpleProgress/StartedCompression/FinishedCompression), and then an EndListPersistentRequests.

PersistentGet

%%
Identifier=Request Number Two
URI=KSK@gpl.txt
Verbosity=0
Return Type=disk
Filename=/home/toad/gpl.txt
Temp Filename=/home/toad/gpl.txt.freenet-temp
Client Token=hello again
Persistence Type=forever
Global=true
End Message
%%

This is very similar to the original ClientGet (which we haven't shown as we want to show Filename/TempFilename, which won't happen with ReturnType=direct). It is simply the node telling us about the request.

PersistentPut

%%
Identifier=Insert-1
URI=CHK@
Verbosity=0
Priority Class=1
Upload From=disk
Filename=/home/toad/blah.html
Metadata.Content Type=text/html
Global=false
End Message
%%

Similarly for inserts.

EndListPersistentRequests

%%
End List Persistent Requests
End Message
%%

Indicates the end of a list of persistent requests.

RemovePersistentRequest

%%
Remove Persistent Request
Global=false // if true, remove it from the global queue
Identifier=Insert-1
End
%%

Remove a persistent request. Note that you must specify Global=true if it is a global request - even if WatchGlobal is enabled.

ModifyPersistentRequest

%%
Modify Persistent Request
Global=false
Identifier=Insert-1
Client Token=new client token
Priority Class=5
End
%%

Modifies a persistent request.

ClientPutDiskDir

%%
Client Put Disk Dir

<standard headers for any kind of put>:
Identifier=<identifier>
Verbosity=<verbosity as Client Put>
Max Retries=<max retries as Client Put>
Priority Class=<priority class>
URI=<target URI>
Get CHK Only=<Get CHK Only as Client Put>
Dont Compress=<Dont Compress as Client Put>
Client Token=<Client Token as Client Put>
Persistence=<Persistence as Client Put>
Global=<Global as Client Put>

<headers unique to a Client Put Disk Dir>:
Filename=<filename of directory on disk to insert as a manifest>
Allow Unreadable Files=<unless true, any unreadable files cause the whole request to fail>
End
%%

Inserts an entire on-disk directory (including subdirs) as a manifest file, so that [final uri]/[filename] returns the file that had that filename.

ClientPutComplexDir

%%
Client Put Complex Dir
<boring headers as with Client Put Disk Dir - see above>
...
Files.0.Name=hello.txt
Files.0.Upload From=direct
Files.0.Metadata.Content Type=text/plain
Files.0.Data Length=6
Files.1.Name=something.pdf
Files.1.Upload From=disk
Files.1.Filename=something.pdf
Files.2.Name=gpl.txt
Files.2.Upload From=redirect
Files.2.Target URI=KSK@gpl.txt
End
hello
%%

Creates a manifest with two entries: <final uri>/hello.txt is a text/plain document containing "hello", and <final uri>/something.pdf is application/pdf (auto-detected) containing the contents of the file something.pdf. As you can see, ClientPutComplexDir is somewhat more complex, but also far more flexible, than ClientPutDiskDir. Note that the numbers under Files must start at zero and advance sequentially, and that the data for any UploadFrom=direct files will be read in that order at the end of the message.

PersistentPutDir

%%
Persistent Put Dir
Identifier=Insert-1
URI=CHK@
Verbosity=0
Priority Class=1
Global=false
Files.2.Filename=/root/something.txt
Files.2.Name=something.txt
Files.2.Data Length=5035
Files.2.Upload From=disk
Files.2.Metadata.Content Type=text/plain
Files.0.Name=hello.txt
Files.0.Data Length=6
Files.0.Upload From=direct
Files.0.Metadata.Content Type=text/plain
Files.1.Name=goodbye.txt
Files.1.Data Length=8
Files.1.Upload From=direct
Files.1.Metadata.Content Type=text/plain
End Message
%%

This is the message returned for a ClientPutDiskDir/ClientPutComplexDir when the client does ListPersistentRequests. It is similar to PersistentPut for obvious reasons.

Persistence and the Global Queue
ClientGet's and ClientPut's (collectively "requests") can have a Persistence header:

Persistence=connection // request is not persistent, will be dropped if the client connection is lost; default
Persistence=reboot // request will persist across client connection drops, but will be lost if the node is restarted
Persistence=forever // request will persist forever; the node will write it to a plaintext file on disk, and restart it if the node is restarted

If a request is persistent (i.e. if Persistence=reboot or forever), then the node will send an acknowledgement when it is started - either a PersistentPut or a PersistentGet. When the request completes, the client will be notified - if it is connected. If the client is not connected at the time when the request completes, then it will be notified when it next connects. The completed request will persist until it is explicitly removed by the client through a RemovePersistentRequest message. When the client connects, all completed persistent requests will be sent to it - in the form of the PersistentGet/PersistentPut, followed by the DataFound/GetFailed/PutSuccessful/PutFailed message, possibly followed by AllData for a ReturnType=direct request. If the client needs to know about all requests, including those still running, it can send a ListPersistentRequests message. The node will then send PersistentGet/PersistentPut's for each running and completed persistent request, and also any other pending messages for them, as above, possibly with progress messages as well (SimpleProgress, StartedCompression, FinishedCompression).

Requests can be assigned to the Global Queue, by setting Global=true when starting them. The idea with the global queue is to have a single queue which all interested FCP apps (e.g. FUQID) can watch, and which all FCP apps can assign downloads to (e.g. Frost or Fproxy queueing a download to the global queue so that the user gets the benefit of his preferred download manager interface). All apps have full control of all downloads on the global queue; they can remove them, list them, and receive notifications about them:

%%
Client Put
URI=CHK@
Upload From=disk
Filename=/home/toad/debian-sarge-dvd1.iso
Identifier=sarge-disk-1
Client Token=This is the debian sarge dvd number 1
Global=true
End
%%

This queues an insert on the global queue. Clients do not receive notifications about requests on the global queue, even if they started the request, unless they ask for them with a WatchGlobal message. Once watch-the-global-queue is enabled, they will receive status updates about all requests on the global queue, and if they do ListPersistentRequests, they will see both their own queued persistent requests and those queued to the global persistent requests queue.

Priority classes
Several priority classes are supported:
0 - Maximum priority: anything more important than Fproxy
1 - Interactive priority class: Fproxy
2 - Semi-interactive priority class: Fproxy immediate-mode (not to disk) large file downloads
3 - Updatable site checks
4 - Bulk offline splitfile fetches (usually to disk)
5 - Prefetch priority class
6 - Minimum priority class (anything less important than prefetch)

For ClientGet's, the default priority depends on the ReturnType: ReturnType=none defaults to 5, ReturnType=disk defaults to 4, and ReturnType=direct defaults to 2. For ClientPut, the default priority class is 2. In Freenet 0.7 at present, inserts and requests do not compete directly with one another, nor do SSKs with CHKs; they are scheduled separately because of their different performance characteristics.

Library Implementations
    • Note:** FCP library authors are encouraged to release their libraries under the Lesser GNU Public License.
- Java FCP Lib
- Python FCP Lib
- Ruby FCP Lib
- Perl FCP Lib
- Cpp FCP Lib

Relevant Links
- [[4]]
- [[5]]