Freenet Client Protocol 2.0 Specification
- Note that this specification is a work in progress, and is subject to change. Third-party application developers are encouraged to experiment with FCP 2.0, but should be prepared for backwards-incompatible changes. We suggest that these developers track changes to this page through [].**
The Freenet Client Protocol is a simple, text-based protocol designed to allow third-party applications to interact with [
]. Supported functionality includes:
- Inserting of data into Freenet
- Retrieval of data from Freenet
- Querying the status of Freenet
FCP 2.0 is was first supported in [[FreenetZeroPointSeven Freenet 0.7]], and is a non-backwards compatible replacement for the original Freenet Client Protocol supported by versions of Freenet prior to 0.7. The protocol is not backwards compatible because of the fundamental nature of the changes to Freenet 0.7 relative to previous versions of Freenet. Downloading large files with the old FCP API was very complex, and involved a great deal of code duplication between different client apps; the philosophy of FCPv2 is that it should be possible to write simple apps, but powerful options should be available when needed. The node does most of the work (e.g. reassembling splitfiles) for the client, unlike in Freenet 0.5's FCP. If you need additional functionality from FCP, please contact us, we may well be able to give you it.
FCP operates over a TCP connection, typically between Freenet and another application running on the same computer. By default, Freenet listens for FCP 2.0 connections on port 9481 (note that previous incompatible versions of FCP used port 8481).
Basic Protocol Syntax
The basic unit of FCP is the message, which consists of a message name, and 0 or more key-value pairs called "fields". Message and field names are typically specified in [
] using alphanumeric characters. Messages can be sent from client to server, or from server to client. Multiple messages can be sent in either direction one after the other.
Field 1=Value 1
Field 2=Value 2
Some messages may be followed by a "payload" of binary data of a length that will be specified within the message itself.
For ease of debugging, blank lines between messages are ignored.
This must be the first message from the client on any given connection. The node will respond with a NodeHello.
The Name parameter is used by the node to uniquely identify a client (e.g. for persistence). Only one connection for each Name is allowed. We do not do anything with ExpectedVersion yet, but it may be verified in the future.
All fairly self-explanatory: the node tells us what version it is, what protocol version it's using, whether testnet mode is enabled, and how many compression codecs are currently supported (this is important with the StartedCompression message).
Close Connection Duplicate Client Name
Indicates that this connection is being closed because another connection has been opened with the same client Name. The latter takes precedence, because the reason it was opened might be that the first connection was broken.
URI=CHK@ // could as easily be an insertable SSK or KSK URI
Metadata.Content Type=text/html // MIME type; for text, if charset is not specified, node should auto-detect it and force the auto-detected version
Identifier=Insert-1 // identifier, as always
Verbosity=0 // just report when complete
Max Retries=999999 // lots of retries; -1 = retry forever
Priority Class=1 // fproxy priority level
Upload From=direct // attached directly to this message
Data Length=100 // 100 bytes decimal
Get CHK Only=false // true = don't insert the data, just return the key it would generate
Global=false // see Persistence section below
Dont Compress=true // hint to node: don't try to compress the data, it's already compressed
Client Token=Hello!!! // sent back to client on the Persistent Put if this is a persistent request
Upload From=disk // upload a file from disk
Upload From=redirect // create a redirect to another key
Target URI=KSK@gpl.txt // some other freenet URI
Neither IgnoreDS nor DSOnly make sense for inserts.
Verbosity is a bitmask; at present, for inserts, two values are supported:
1 = splitfile progress (send SimpleProgress messages)
512 = compression start and end (send StartedCompression and FinishedCompression messages)
This indicates the final URI of the data inserted; this may not be the same as the original URI, which could have been CHK@ ("insert this as a CHK and tell me what its CHK is"), or an SSK using a private key.
URI=<uri to fetch>
Indicates a successful insert.
Code=9 // error code
Identifier=1 // identifier
@hello.toad // insert URI if it succeeded
Code Description=Insert collided with different, pre-existing data at the
same key // description of error
Extra Description=SSK collision // more info on the error
Fatal=true // No point retrying; permanent error
Short Code Description=Insert collision // short version of Code Description
It is also possible to have a more complex PutFailed: (this is an example, the strings and codes returned may be different)
Code Description=Too many retries in blocks
Extra Description=Splitfile failed because many blocks had too many retries
Fatal=false // can retry, but we ran out of retries for at least one block
Short Code Description=Too many retries in blocks
Errors.7.Description=Route not found // the number is that particular error code
Errors.7.Count=457 // 457 block inserts were abandoned due to RNF
Errors.4.Description=Rejected due to overload
Errors.4.Count=781 // 781 block inserts were abandoned due to overload
Ignore DS=false // true = ignore the datastore (in old FCP this was Remove Local Key)
DSOnly=false // true = only check the datastore, don't route (~= htl 0)
URI=KSK@gpl.txt // key to fetch
Identifier=Request Number One
Verbosity=0 // no status, just tell us when it's done
Return Type=direct // return all at once over the FCP connection
Max Size=100 // maximum size of returned data (all numbers in DECIMAL)
Max Temp Size=1000 // maximum size of intermediary data
Max Retries=100 // automatic retry supported as an option; -1 means retry forever
Priority Class=1 // priority class 1 = interactive
Persistence=reboot // continue until node is restarted; report progress while client is
connected, including if it reconnects after losing connection
Client Token=hello // returned in Persistent Get, a hint to the client, so the client
doesn't need to maintain its own state
Global=false // see Persistence section below
Several ReturnType's are supported:
- direct = return the data directly to the client via an AllData message, once we have all of it.
- none = don't return the data at all, just fetch it and tell the client when we have finished.
- disk = write the data to disk (set Filename=<where you want to put it>, and optionally TempFilename=<temporary file in same directory>).
In future, ReturnType=chunked may also be supported (return it in segments as they are ready), but this is not yet implemented.
Verbosity is a bitmask, as with ClientPut. At present only one optional value is supported:
1 - send SimpleProgress messages
Identifier=Request Number One
This indicates a successful fetch of the key, but does not actually include the data.
Identifier=Request Number One
Data Length=37261 // length of data
<37261 bytes of data>
For a ClientGet with ReturnType=direct, the data is returned directly to the client, all at once, using the AllData message. Obviously in many situations this will not be desirable, hence the other ReturnType options.
Code Description=<description of error code>
Extra Description=<extra description of this specific error, if available>
|false; true means there isn't much point retrying as it's
probably an error with the data>
Short Code Description=<short code description>
Identifier=<identifier passed in by client>
Complex GetFailed's are also possible; the format is identical to that of a PutFailed.
Protocol Error // indicates an FCP protocol error
Code=1 // error code
Code Description=Client Hello must be first message // description of error
Extra Description=Duh // more info on the error if available
Fatal=false // means the connection stays open
Identifier=<ident> // if we managed to parse one
Identifier=<my identifier, sent back on SSKKeypair>
Ask the node to generate us an SSK keypair.
@AKTTKG6YwjrHzWo67laRcoPqibyiTdyYufjVg54fBlWr,Aw USJG5 ZS-FDZTqnt6skTzhxQe08T-fbKXj8aEHZsXM/
@Bn HX Xv 3 Fa 43w~~iz1tNUd~cj4OpUuDjVouOWZ5XlpX0,Aw USJG5 ZS-FDZTqnt6skTzhxQe08T-fbKXj8aEHZsXM,AQABAAE/
Identifier=<my identifier from Generate SSK>
The insert URI contains the private key, it is the URI you should use when inserting.
The request URI contains the public key, it is the URI you should distribute for people to request from.
It is possible to quickly go from the private key to the public key (by inserting something with Get CHK Only=true with the privkey, the node will tell you the corresponding pubkey request URI) but not vice versa.
Identifier Collision // client tried to reuse an Identifier
Identifiers are unique to the specific request. They can be reused after that request has completed and been forgotten.
Codec=0 // first codec
Indicates that the node has started to attempt to compress the data with the first codec.
Codec=0 // first codec worked best
Orig Size=89000000 // was originally 89MB
Compressed Size=752 // compressed is 752 bytes; pretty good compression, maybe it was all zeros!
Indicates that the node has finished trying to compress the data.
Total=12288 // 12,288 blocks we can fetch
Required=8192 // we only need 8,192 of them (because of splitfile redundancy)
Failed=452 // 452 of them have failed due to running out of retries
Fatally Failed=0 // none of them have encountered fatal errors
Succeeded=1027 // we have successfully fetched 1,027 blocks
Finalized Total=true // the Total will not increase any further (if this is false, it may increase; it will never decrease)
Identifier=Request Number One
This indicates the progress of a large request (usually a splitfile, a big file which is split across a large number of blocks). Note that before FinalizedTotal=true, Total may vary wildly on a ClientGet, because we may follow redirects, have to fetch multi-level splitfiles and so on. However, once we are fetching the final splitfile, FinalizedTotal will be set to true. Whereas on a ClientPut, we can't generate the metadata until quite late on, so it takes a long time to set FinalizedTotal=true, but this doesn't matter as the Total will not increase very much (since the majority of the insert is the actual data and check blocks).
Enabled=true // watch the global queue
Verbosity Mask=1 // only let through the basic completion messages, and Simple Progress; not any other progress messages
See the section below on persistence.
List Persistent Requests
Ask the node to list all persistent requests for this node (and for the global queue too if watch global queue is enabled). Node will respond with a series of PersistentGet/PersistentPut's, possibly some DataFound/GetFailed/PutFailed/PutSuccessful's, and possibly some status messages (SimpleProgress/StartedCompression/FinishedCompression), and then an EndListPersistentRequests.
Identifier=Request Number Two
Client Token=hello again
This is very similar to the original ClientGet (which we haven't shown as we want to show Filename/TempFilename, which won't happen with ReturnType=direct). It is simply the node telling us about the request.
Similarly for inserts.
End List Persistent Requests
Indicates the end of a list of persistent requests.
Remove Persistent Request
Global=false // if true, remove it from the global queue
Remove a persistent request. Note that you must specify Global=true if it is a global request - even if WatchGlobal is enabled.
Modify Persistent Request
Client Token=new client token
Modifies a persistent request.
Client Put Disk Dir
<standard headers for any kind of put>:
Verbosity=<verbosity as Client Put>
Max Retries=<max retries as Client Put>
Priority Class=<priority class>
Get CHK Only=<Get CHK Only as Client Put>
Dont Compress=<Dont Compress as Client Put>
Client Token=<Client Token as Client Put>
Persistence=<Persistence as Client Put>
Global=<Global as Client Put>
<headers unique to a Client Put Disk Dir>:
Filename=<filename of directory on disk to insert as a manifest>
Allow Unreadable Files=<unless true, any unreadable files cause the whole request to fail>
Inserts an entire on-disk directory (including subdirs) as a manifest file, so that [final uri]/[filename]
returns the file that had that filename.
Client Put Complex Dir
<boring headers as with Client Put Disk Dir - see above>
Creates a manifest with two entries: <final uri>/hello.txt is a text/plain document containing "hello", and <final uri>/something.pdf is application/pdf (auto-detected) containing the contents of the file something.pdf. As you can see, ClientPutComplexDir is somewhat more complex, but also far more flexible, than ClientPutDiskDir. Note that the numbers under Files must start at zero and advance sequentially, and that the data for any UploadFrom=direct files will be read in that order at the end of the message.
Persistent Put Dir
This is the message returned for a ClientPutDiskDir/ClientPutComplexDir when the client does ListPersistentRequests. It is similar to PersistentPut for obvious reasons.
Persistence and the Global Queue
ClientGet's and ClientPut's (collectively "requests") can have a Persistence header:
Persistence=connection // request is not persistent, will be dropped if the client connection is lost; default
Persistence=reboot // request will persist across client connection drops, but will be lost if the node is restarted
Persistence=forever // request will persist forever; the node will write it to a plaintext file on disk, and restart it if the node is restarted
If a request is persistent (i.e. if Persistence=reboot or forever), then the node will send an acknowledgement when it is started - either a PersistentPut or a PersistentGet. When the request completes, the client will be notified - if it is connected. If the client is not connected at the time when the request completes, then it will be notified when it next connects. The completed request will persist until it is explicitly removed by the client through a RemovePersistentRequest message. When the client connects, all completed persistent requests will be sent to it - in the form of the PersistentGet/PersistentPut, followed by the DataFound/GetFailed/PutSuccessful/PutFailed message, possibly followed by AllData for a ReturnType=direct request. If the client needs to know about all requests, including those still running, it can send a ListPersistentRequests message. The node will then send PersistentGet/PersistentPut's for each running and completed persistent request, and also any other pending messages for them, as above, possibly with progress messages as well (SimpleProgress, StartedCompression, FinishedCompression).
Requests can be assigned to the Global Queue, by setting Global=true when starting them. The idea with the global queue is to have a single queue which all interested FCP apps (e.g. FUQID) can watch, and which all FCP apps can assign downloads to (e.g. Frost or Fproxy queueing a download to the global queue so that the user gets the benefit of his preferred download manager interface). All apps have full control of all downloads on the global queue; they can remove them, list them, and receive notifications about them:
Client Token=This is the debian sarge dvd number 1
This queues an insert on the global queue. Clients do not receive notifications about requests on the global queue, even if they started the request, unless they ask for them with a WatchGlobal message. Once watch-the-global-queue is enabled, they will receive status updates about all requests on the global queue, and if they do ListPersistentRequests, they will see both their own queued persistent requests and those queued to the global persistent requests queue.
Several priority classes are supported:
0 - Maximum priority: anything more important than Fproxy
1 - Interactive priority class: Fproxy
2 - Semi-interactive priority class: Fproxy immediate-mode (not to disk) large file downloads
3 - Updatable site checks
4 - Bulk offline splitfile fetches (usually to disk)
5 - Prefetch priority class
6 - Minimum priority class (anything less important than prefetch)
For ClientGet's, the default priority depends on the ReturnType: ReturnType=none defaults to 5, ReturnType=disk defaults to 4, and ReturnType=direct defaults to 2. For ClientPut, the default priority class is 2. In Freenet 0.7 at present, inserts and requests do not compete directly with one another, nor do SSKs with CHKs; they are scheduled separately because of their different performance characteristics.
- Note:** FCP library authors are encouraged to release their libraries under the Lesser GNU Public License.
- Java FCP Lib
- Python FCP Lib
- Ruby FCP Lib
- Perl FCP Lib
- Cpp FCP Lib