REST API

The QDB API follows RESTful principles and you don't need any driver libraries to communicate with it. HTTP POST does create or update, PUT does update only, DELETE deletes stuff and GET retrieves resources. All endpoints accept JSON data and/or URL-encoded form data and most return JSON. The examples in this documentation use curl, a utility for making HTTP requests from the command line. A Python wrapper (pyqdb) is also available.

# create or update a queue "foo" with max size of 100 gb
$ curl http://127.0.0.1:9554/q/foo -d maxSize=100g
# do the same but using a JSON document foo.json: { "maxSize": "100g" }
$ curl -H "Content-Type: application/json" http://127.0.0.1:9554/q/foo -d @foo.json

The usual HTTP status codes are used. POST requests that actually create something return 201 (created), those that just do an update return 200 (ok). POST and PUT requests both only update fields supplied in the request and leave others as is. A full JSON representation of the object is returned. Well-formed requests with validation errors return 422 (unprocessable entity). If a version number is supplied with a request and the version of the object currently on the server is different then a 409 (conflict) is returned.

QDB returns sizes as human readable strings using binary suffixes (1 MB = 1048576 bytes) and dates as ISO8601 with timezone. You can get number byte values and dates as milliseconds since 1970 by turning on borg mode:

$ curl http://127.0.0.1:9554/q/foo
{
    "id" : "foo",
    "version" : 1,
    "maxSize" : "100 GB",
    "maxPayloadSize" : "1 MB",
    "contentType" : "application/octet-stream"
}
$ curl http://127.0.0.1:9554/q/foo?borg=true
{
    ...
    "maxSize" : 107374182400,
    "maxPayloadSize" : 1048576,
}

QDB supports HTTP keep-alive so repeated calls to the server can be done using the same connection.

Server Status

# is all well in the world of qdb?
$ curl http://127.0.0.1:9554/status
{
    "uptime" : "15 days 8:22:04",
    "heapFreeMemory" : "10.86 MB",
    "heapMaxMemory" : "15.06 MB",
    "otherFreeMemory" : "60.01 MB",
    "otherMaxMemory" : "80 MB"
}
# use gc=true to garbage collect for more accurate free memory info
$ curl http://127.0.0.1:9554/status?gc=true

Queues

QDB queues are ring buffers with a configurable maximum size. When a queue is full the oldest messages are efficiently deleted in chunks. QDB supports fast retrieval of messages by ID and timestamp from any point in the buffer/queue. Queues can have outputs which automatically send messages somewhere else (e.g. to RabbitMQ).

# create queue "incoming" with capacity of 200 gb and max message size of 10 mb
$ curl http://127.0.0.1:9554/q/incoming -d maxSize=200g -d maxPayloadSize=10m
# the contentType of a queue is used when retrieving individual messages
$ curl http://127.0.0.1:9554/q/incoming -d contentType="application/json; charset=utf-8"
# list queues
$ curl http://127.0.0.1:9554/q

Appending Messages

Messages can be POSTed individually or in bulk. Each message may have an optional routing key string (max 255 characters), is timestamped and assigned a unique id. The message itself can be any binary or text data. Message id's are not consecutive but newer messages always have bigger id's than older messages. The first message appended has id 1.

# append a single message with payload from message.dat (the body of the POST)
$ curl http://127.0.0.1:9554/q/incoming/messages?routingKey=abc:def -d @message.dat
{
  "id" : 44,
  "timestamp" : "2013-07-07T23:13:29.277+0200",
  "payloadSize" : 100,
  "routingKey" : "abc:def"
}

POSTing individual mentions can be quite fast (2000-3000 messages/sec on an 2011 MacBook Pro, each message 0-4k in size) because QDB supports HTTP keep-alive. If you need more performance POST messages in batches by specifying multiple=true.

QBD uses a variation of netstrings to encode the routing key and payload of message in a batch. The routing key is prefixed by its size in bytes in decimal ASCII digits followed by a colon and the UTF8 encoded string and a linefeed (\n, ASCII 10). The payload is prefixed with its size in bytes followed by a colon and the actual byte data and a linefeed. Extra linefeeds are ignored. Example (batch.txt):

3:one
12:Hello world!
7:one:two
26:The payload is binary data

QDB returns an array of JSON objects, one for each message appended:

$ curl http://127.0.0.1:9554/q/incoming/messages?multiple=true -d @batch.txt
[ {
  "id" : 30,
  "timestamp" : "2013-07-14T19:47:21.241+0200",
  "payloadSize" : "12 bytes",
  "routingKey" : "one"
}, {
  "id" : 60,
  "timestamp" : "2013-07-14T19:47:21.241+0200",
  "payloadSize" : "26 bytes",
  "routingKey" : "one:two"
} ]

If there is an error then appending stops at the first failed message. The created field in the JSON error response identifies the successfully appended messages:

$ curl "http://127.0.0.1:9554/q/incoming/messages?multiple=true" -d @batch.txt
{
  "responseCode" : 422,
  "message" : "Expected 27 bytes, only read 26 while reading payload",
  "created" : [ {
    "id" : 0,
    "timestamp" : "2013-07-14T19:45:56.743+0200",
    "payloadSize" : "12 bytes",
    "routingKey" : "one"
  } ]
}

QDB can append 5000-6000 messages/sec, each 0-4k in size on a 2011 MBP using batches of 50.

Queue Status

This endpoint shows the current values of the configurable parameters of the queue as well as its current status:

$ curl http://127.0.0.1:9554/q/mentions
{
  "id" : "mentions",
  "version" : 2,
  "maxSize" : "20 GB",
  "maxPayloadSize" : "1 MB",
  "contentType" : "application/json; charset=utf-8",
  "status" : "OK",
  "size" : "5.14 GB",
  "messageCount" : 3392333,
  "duration" : "10 days 11:11:03",
  "oldestMessage" : "2013-07-11T07:42:20.274+0100",
  "newestMessage" : "2013-07-21T18:53:24.136+0100",
  "newestMessageReceived" : "0:00:02 ago",
  "oldestMessageId" : 1,
  "nextMessageId" : 5514576311
}
# duration = time between newest and oldest messages
# newestMessageReceived = how long ago was the most recent message appended?

Examining the newestMessageReceived field is an easy way to check that whatever application is supposed to be appending to the queue is actually doing so.

Queue Alerts

QDB can monitor newestMessageReceived and log warnings and errors when it exceeds configurable thresholds, warnAfter and/or errorAfter respectively:

# The current WARN or ERROR status is reflected in the queue status
$ curl http://127.0.0.1:9554/q/mentions -d warnAfter=60 -d errorAfter=5:00
{
"id" : "mentions",
...
"status" : "WARN: Last message appended 0:01:02 ago",
...
"newestMessageReceived" : "0:01:02 ago",
}

Changes in status are logged. The WARN and ERROR messages are repeated every 5 minutes (this is configurable):

23:31:56.539 [queue-status-monitor] WARN  ... - /q/mentions: Last message appended 0:01:00 ago
23:35:56.540 [queue-status-monitor] ERROR ... - /q/mentions: Last message appended 0:05:00 ago
23:40:56.540 [queue-status-monitor] ERROR ... - /q/mentions: Last message appended 0:10:00 ago
23:42:12.537 [queue-status-monitor] INFO  ... - /q/mentions has recovered

The duration can include days e.g. "2d 10:20:30" is 2 days 10 hours 20 minutes and 30 seconds.

Retrieving Messages

Old and incoming messages can be efficiently retrieved by id or timestamp, one at a time or streamed. Specify single=true to retrieve a single message as the body of the response. The Content-Length is the size in bytes of the message payload. The Content-Type is the contentType parameter of the queue:

# the timestamp, routing key and id are returned via QDB-* headers
$ curl -v "http://127.0.0.1:9554/q/incoming/messages?single=true&fromId=30"
...
< Content-Type: application/octet-stream
< QDB-Timestamp: 2013-07-14T19:47:21.241+0200
< QDB-RoutingKey: one
< Content-Length: 12
< QDB-Id: 30
...
Hello world!
# Fetch the first message received on or after 19h47 today
$ curl "http://127.0.0.1:9554/q/incoming/messages?single=true&from=19:47"
Hello world!
# Fetch the first message received on or after 15 Jul 2013 17h26:42.491 GMT+2
$ curl "http://127.0.0.1:9554/q/incoming/messages?single=true&from=2013-07-15T17:26:42.491+0200"
Hello world!

A number of different date/time formats are supported.

Multiple Messages can be streamed as they come in or from a past id or timestamp:

# Dump new messages to the console until ctrl-C
$ curl http://127.0.0.1:9554/q/incoming/messages
88:{"id":30,"timestamp":"2013-07-14T19:47:21.241+0200","payloadSize":12,"routingKey":"one"}
Hello world!
92:{"id":60,"timestamp":"2013-07-14T19:47:21.241+0200","payloadSize":26,"routingKey":"one:two"}
The payload is binary data

Each messages is prefixed with a single line JSON header preceded by its size in bytes in decimal ASCII digits (see netstrings) and a colon, followed by a linefeed (\n, ASCII 10). The binary payload immediately follows the header and is followed by a linefeed. QDB will write extra linefeeds every 29 seconds if there are no new messages to keep the connection alive. The Content-Type of the response is application/octet-stream.

Use the to (timestamp) or toId parameter to fetch messages until a particular time or id. These parameters are exclusive i.e. if there is a message with exactly the to/toId it is not returned.

# Fetch messages received between 19:47 and 20:00 (exclusive) today
$ curl "http://127.0.0.1:9554/q/incoming/messages?from=19:47&to=20:00"

This endpoint supports a number of other optional parameters shown with the default values:

limit unlimited Stop after returning this many messages
noHeaders false Don't send the header JSON
noPayload false Don't send the message payloads
noLengthPrefix false Don't prefix the header JSON with its length
timeoutMs 0 (forever) Stop if no message is received for this many milliseconds
keepAliveMs 29000 Send a keep alive (linefeed) every this many milliseconds if no messages are available
keepAlive \n The string sent to keep the connection alive (can be multiple characters)
separator \n The string sent immediately after each payload (can be multiple characters)

Filtering

QDB can filter messages on routing key and/or payload using the routingKey and grep parameters respectively:

# Fetch messages containing a line starting with a number
# The -g option to curl stops it from messing with [] and {}
# The %2b is the + character url encoded
$ curl -g "http://127.0.0.1:9554/q/incoming/messages?grep=^[0-9]%2b"
# Fetch messages with routingKey matching a RabbitMQ expression
# The hash (#) must be url encoded as %23
$ curl "http://127.0.0.1:9554/q/incoming/messages?routingKey=foo.*.%23"
# Fetch messages with routingKey matching a Java regular expression
$ curl -g "http://127.0.0.1:9554/q/incoming/messages?routingKey=/^[0-9]%2b/"

The routingKey expression is treated as Java regular expression if it starts with a slash. Otherwise it follows RabbitMQ conventions i.e. the string is split into words at '.' characters and * matches zero or more words, # matches exactly one word.

The grep and routingKey parameters can be used together in which case both must match.

When grep is used the message is converted from bytes to a string using the encoding specified in the contentType of the queue with UTF8 as the default.

Timelines

QBD maintains an efficient index of each queue by timestamp and message id. You can peek at this index using the timelines endpoint:

$ curl http://127.0.0.1:9554/q/foo/timeline
[ {
  "id" : 0,
  "timestamp" : "2013-07-15T17:49:38.271+0200",
  "bytes" : 50,
  "millis" : 20628,
  "count" : 2
}, {
  "id" : 50,
  "timestamp" : "2013-07-15T17:49:58.899+0200",
  "bytes" : 0,
  "millis" : 0,
  "count" : 0
} ]

Each bucket listed shows the first message id in the bucket, its timestamp, total bytes of messages in the bucket and the message count. The last bucket is an empty placeholder and always has an id equal to the id that the next message appended to the queue will be assigned. The number of buckets depends on the maxSize and maxPayloadSize of the queue and how full it is.

Each bucket listed can be displayed in more detail by looking it up by id:

$ curl http://127.0.0.1:9554/q/foo/timeline/0
[ ... ]

If the id requested is before the first bucket then the first bucket is listed.

Outputs

Outputs are attached to queues and do something with incoming or existing messages (e.g. publish them to a RabbitMQ exchange).

# output messages to foobar RabbitMQ exchange
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d type=rabbitmq \
    -d url=amqp://127.0.0.1 -d queues=foo,bar -d exchange=foobar

The exchange and queues are created if they do not exist and bound together. Likewise if the output already exists it is updated. More output types will be added in future releases.

By default newly created outputs only output new messages. Use the from or fromId parameters to change this (at creation time or later):

# output messages from 10h20 today
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d from=10:20
# turn the output off
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d enabled=false

Use the to or toId parameters to have the output stop automatically on reaching the specified timestamp or id respectively:

# output messages from 10h20 to 10h30 (exclusive) today
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d from=10:20 -d to=10:30

Use the limit parameter to have the output stop (disable itself) after outputting that many messages:

# output 1000 messages from 10h20 today then stop
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d from=10:20 -d limit=1000
# output another 1000 messages from the current position
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d limit=1000 -d enabled=true

Output Status

This endpoint shows the current values of the configurable parameters of the output as well as its current status. You can query all outputs for a queue or individual outputs:

$ curl http://127.0.0.1:9554/q/mentions/out
[ {
  "id" : "rabbit",
  "version" : 316690,
  "type" : "rabbitmq",
  "url" : "amqps://user:password@example.com",
  "enabled" : true,
  "atId" : 5503589683,
  "at" : "2013-07-21T18:39:05.728+0100",
  "status": "OK",
  "behindBy" : "23 ms",
  "behindByBytes" : "1.2 kB",
  "behindByPercentage" : 0.0,
  "queues" : [ "pork-mentions", "goose-mentions" ],
  "exchange" : "mentions"
} ]
$ curl http://127.0.0.1:9554/q/mentions/out/rabbit
# .. same information just for the output with id "rabbit" ..

The behind* fields show how well the output is keeping up as new messages are appended to the queue. In this case there are 23 milliseconds or 1.2 kB of messages in the queue that have not been sent to RabbitMQ. The behindByPercentage field is behindByBytes expressed as a percentage of the queue maxSize. If it reaches 100% some messages in the queue are going to be deleted before being processed by the output. You can increase the size of the queue at any time to avoid data loss:

$ curl http://127.0.0.1:9554/q/mentions
{
  ...
  "maxSize" : "20 GB",
}
$ curl http://127.0.0.1:9554/q/mentions -d maxSize=40g

Output Alerts

QDB can monitor behindByPercentage and log warnings and errors when it exceeds configurable thresholds, warnAfter and/or errorAfter respectively:

# The current WARN or ERROR status is reflected in the output status
$ curl http://127.0.0.1:9554/q/mentions/out/rabbit -d warnAfter=10 -d errorAfter=20
{
  "id" : "rabbit",
  ...
  "status" : "WARN: 19.8% of queue used",
  ...
  "behindByPercentage" : 19.8,
  "warnAfter" : 10.0,
  "errorAfter" : 20.0,
  ...
}

Changes in status are logged. The WARN and ERROR messages are repeated every 5 minutes (this is configurable):

23:31:56.539 [output-status-monitor] WARN  ... - /q/mentions/out/rabbit: 19.8% of queue used
23:35:56.540 [output-status-monitor] ERROR ... - /q/mentions/out/rabbit: 22.5% of queue used
23:40:56.540 [output-status-monitor] ERROR ... - /q/mentions/out/rabbit: 26.3% of queue used
23:42:12.537 [output-status-monitor] INFO  ... - /q/mentions/out/rabbit has recovered

Output Filtering

Outputs can filter messages by routingKey and/or grep on the message body as described in queues.

# Output messages from 10h20 today containing a line starting with a number
# The %2b is the + character url encoded
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d from=10:20 -d grep=^[0-9]%2b
# Output messages with routingKey matching a RabbitMQ expression
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d "routingKey=foo.*"
# Fetch messages with routingKey matching a Java regular expression
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d "routingKey=/^[0-9]%2b/"

RabbitMQ

Use type=rabbitmq to output to RabbitMQ. The url and either an exchange and/or a queue are required. If only a queue is given then an exchange with the same name is created. The type and durability of the exchange and the durability of the queue(s) can be set:

# default exchange type is fanout and is durable, this will create non-durable exchange
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d exchange=foobar#fanout#false
# queue foo is non-durable, bar is durable
$ curl http://127.0.0.1:9554/q/foo/out/rabbit -d queues=foo#false,bar#true

Other parameters (with defaults):

url (required) amqp[s]://user:password@host:port[/vhost]
heartbeat 30 heartbeat in seconds, 0 to disable
persistentMessages false create durable messages (delivery mode 2)

Inputs

Inputs are attached to queues to fetch and append messages from somewhere else (e.g. a RabbitMQ queue). It is very easy to setup an input to backup data arriving on a RabbitMQ queue to be played back when needed (using an output, be careful not to create a loop!).

# create a qdb queue 'backup'
$ curl -XPOST http://127.0.0.1:9554/q/backup
# fetch messages from RabbitMQ queue 'backup' bound to exchange 'incoming'
$ curl http://127.0.0.1:9554/q/backup/in/rabbit -d type=rabbitmq \
    -d url=amqp://127.0.0.1 -d queue=backup -d exchange=incoming

The queue and (optional) exchange are created if they do not exist and bound together. Likewise if the input already exists it is updated. More input types will be added in future releases.

Input Status

This endpoint shows the current values of the configurable parameters of the input as well as its current status. You can query all inputs for a queue or individual inputs:

$ curl http://127.0.0.1:9554/q/backup/in
[ {
  "id" : "rabbit",
  "version" : 1298,
  "type" : "rabbitmq",
  "url" : "amqp://127.0.0.1",
  "enabled" : true,
  "updateIntervalMs" : 1000,
  "errorAfter" : "0:01:00",
  "status" : "OK",
  "lastMessageId" : 28122142,
  "lastMessageTimestamp" : "2013-08-04T09:55:10.214+0000",
  "lastMessageAppended" : "0:00:01 ago",
  "queue" : "backup",
  "exchange" : "incoming"
} ]
$ curl http://127.0.0.1:9554/q/backup/in/rabbit
# .. same information just for the input with id "rabbit" ..

The lastMessageAppended field shows how long ago the last message was fetched and appended to the QDB queue. In this case it has been 1 second since the last messaged from RabbitMQ was appended.

Input Alerts

QDB can monitor lastMessageAppended and log warnings and errors when it exceeds configurable thresholds, warnAfter and/or errorAfter respectively:

# The current WARN or ERROR status is reflected in the input status
$ curl http://127.0.0.1:9554/q/backup/in/rabbit -d warnAfter=10 -d errorAfter=20
{
  "id" : "rabbit",
  ...
  "status" : "WARN: Last message appended 0:2:32 ago",
  ...
}

Changes in status are logged. The WARN and ERROR messages are repeated every 5 minutes (this is configurable):

23:31:56.539 [input-status-monitor] WARN  ... - /q/backup/in/rabbit: Last message appended 0:00:10 ago
23:35:56.540 [input-status-monitor] ERROR ... - /q/backup/in/rabbit: Last message appended 0:00:20 ago
23:40:56.540 [input-status-monitor] ERROR ... - /q/backup/in/rabbit: Last message appended 0:05:20 ago
23:42:12.537 [input-status-monitor] INFO  ... - /q/backup/in/rabbit has recovered

RabbitMQ

Use type=rabbitmq to fetch messages from RabbitMQ. The url and a queue are required. If an exchange is also provided then an exchange is created and bound to the queue. The type and durability of the exchange and the durability of the queue(s) can be set:

# default exchange type is fanout and is durable, this will create non-durable exchange
$ curl http://127.0.0.1:9554/q/backup/in/rabbit -d exchange=incoming#fanout#false
# queue foo is non-durable, bar is durable
$ curl http://127.0.0.1:9554/q/backup/in/rabbit -d queue=backup#false

Other parameters (with defaults):

url (required) amqp[s]://user:password@host:port[/vhost]
heartbeat 30 heartbeat in seconds, 0 to disable
routingKey routingKey used to bind the queue to the exchange
autoAck false if true messages are not individually acked

Databases

Queues in QDB are grouped into databases. There is a database called "default" that is assumed when no database is specified:

# these two calls return information about the same queue (foo)
$ curl http://127.0.0.1:9554/q/foo
$ curl http://127.0.0.1:9554/db/default/q/foo
# list databases
$ curl http://127.0.0.1:9554/db

Security & Users

QDB uses HTTP basic authentication. The QDB server creates a user called "admin" on startup with full access to everything. The default password for the "admin" user is "admin" unless it is changed in the configuration file or after startup using the API. If no authentication information is supplied then admin:admin is assumed. Non-admin users can only access databases they own.

# create an admin user "david" with password "secret"
$ curl http://127.0.0.1:9554/users/david -d admin=true -d password=secret
# list all users
$ curl http://127.0.0.1:9554/users
# get details of the "david" user
$ curl http://127.0.0.1:9554/users/david
# authenticate as "david" and list all databases visible to "david"
$ curl --user david:secret http://127.0.0.1:9554/db
# change the password for the "admin" user
$ curl http://127.0.0.1:9554/users/admin -d password=oink
# list users
$ curl --user admin:oink http://127.0.0.1:9554/users

Appendix

Date & Time Formats

Time based parameters (e.g. from and to) can be specified in a number of formats:

yyyy-MM-dd'T'HH:mm:ss.SSSZ 2013-07-15T17:26:42.491+0200 Full date, time and milliseconds with timezone
yyyy-MM-dd'T'HH:mm:ss.SSS 2013-07-15T17:26:42.491 Timezone is that of the server
yyyy-MM-dd'T'HH:mm:ssZ 2013-07-15T17:26:42+0200 Full date and time with timezone
yyyy-MM-dd'T'HH:mm:ss 2013-07-15T17:26:42 Timezone is that of the server
yyyy-MM-dd'T'HH:mm 2013-07-15T17:26 No secs, timezone is that of the server
yyyy-MM-dd 2013-07-15 Date only, timezone is that of the server
HH:mm:ss 17:26:42 That time today in servers timezone
HH:mm 17:26 That time today in servers timezone
[0-9]+ 1373824041241 Milliseconds since 1970