tigerbeetle
The TigerBeetle client for Python.
Prerequisites
Linux >= 5.6 is the only production environment we support. But for ease of development we also support macOS and Windows.
- Python (or PyPy, etc) >=
3.7
Setup
First, create a directory for your project and cd
into
the directory.
Then, install the TigerBeetle client:
pip install tigerbeetle
Now, create main.py
and copy this into it:
import os
import tigerbeetle as tb
print("Import OK!")
# To enable debug logging, via Python's built in logging module:
# logging.basicConfig(level=logging.DEBUG)
# tb.configure_logging(debug=True)
Finally, build and run:
python3 main.py
Now that all prerequisites and dependencies are correctly set up, let’s dig into using TigerBeetle.
Sample projects
This document is primarily a reference guide to the client. Below are various sample projects demonstrating features of TigerBeetle.
- Basic: Create two accounts and transfer an amount between them.
- Two-Phase Transfer: Create two accounts and start a pending transfer between them, then post the transfer.
- Many Two-Phase Transfers: Create two accounts and start a number of pending transfer between them, posting and voiding alternating transfers.
Creating a Client
A client is created with a cluster ID and replica addresses for all replicas in the cluster. The cluster ID and replica addresses are both chosen by the system that starts the TigerBeetle cluster.
Clients are thread-safe and a single instance should be shared between multiple concurrent tasks.
Multiple clients are useful when connecting to more than one TigerBeetle cluster.
In this example the cluster ID is 0
and there is one
replica. The address is read from the TB_ADDRESS
environment variable and defaults to port 3000
.
with tb.ClientSync(cluster_id=0, replica_addresses=os.getenv("TB_ADDRESS", "3000")) as client:
# Use the client.
pass
The following are valid addresses:
3000
(interpreted as127.0.0.1:3000
)127.0.0.1:3000
(interpreted as127.0.0.1:3000
)127.0.0.1
(interpreted as127.0.0.1:3001
,3001
is the default port)
Creating Accounts
See details for account fields in the Accounts reference.
= tb.Account(
account id=tb.id(), # TigerBeetle time-based ID.
=0,
debits_pending=0,
debits_posted=0,
credits_pending=0,
credits_posted=0,
user_data_128=0,
user_data_64=0,
user_data_32=1,
ledger=718,
code=0,
flags=0,
timestamp
)
= client.create_accounts([account])
account_errors # Error handling omitted.
See details for the recommended ID scheme in time-based identifiers.
Account Flags
The account flags value is a bitfield. See details for these flags in the Accounts reference.
To toggle behavior for an account, combine enum values stored in the
AccountFlags
object (it’s an enum.IntFlag
)
with bitwise-or:
AccountFlags.linked
AccountFlags.debits_must_not_exceed_credits
AccountFlags.credits_must_not_exceed_credits
AccountFlags.history
For example, to link two accounts where the first account
additionally has the debits_must_not_exceed_credits
constraint:
= tb.Account(
account0 id=100,
=0,
debits_pending=0,
debits_posted=0,
credits_pending=0,
credits_posted=0,
user_data_128=0,
user_data_64=0,
user_data_32=1,
ledger=1,
code=0,
timestamp=tb.AccountFlags.LINKED | tb.AccountFlags.DEBITS_MUST_NOT_EXCEED_CREDITS,
flags
)= tb.Account(
account1 id=101,
=0,
debits_pending=0,
debits_posted=0,
credits_pending=0,
credits_posted=0,
user_data_128=0,
user_data_64=0,
user_data_32=1,
ledger=1,
code=0,
timestamp=tb.AccountFlags.HISTORY,
flags
)
= client.create_accounts([account0, account1])
account_errors # Error handling omitted.
Response and Errors
The response is an empty array if all accounts were created successfully. If the response is non-empty, each object in the response array contains error information for an account that failed. The error object contains an error code and the index of the account in the request batch.
See all error conditions in the create_accounts reference.
= tb.Account(
account0 id=102,
=0,
debits_pending=0,
debits_posted=0,
credits_pending=0,
credits_posted=0,
user_data_128=0,
user_data_64=0,
user_data_32=1,
ledger=1,
code=0,
timestamp=0,
flags
)= tb.Account(
account1 id=103,
=0,
debits_pending=0,
debits_posted=0,
credits_pending=0,
credits_posted=0,
user_data_128=0,
user_data_64=0,
user_data_32=1,
ledger=1,
code=0,
timestamp=0,
flags
)= tb.Account(
account2 id=104,
=0,
debits_pending=0,
debits_posted=0,
credits_pending=0,
credits_posted=0,
user_data_128=0,
user_data_64=0,
user_data_32=1,
ledger=1,
code=0,
timestamp=0,
flags
)
= client.create_accounts([account0, account1, account2])
account_errors for error in account_errors:
if error.result == tb.CreateAccountResult.EXISTS:
print(f"Batch account at {error.index} already exists.")
else:
print(f"Batch account at ${error.index} failed to create: {error.result}.")
To handle errors you can compare the result code returned from
client.create_accounts
with enum values in the
CreateAccountResult
object.
Account Lookup
Account lookup is batched, like account creation. Pass in all IDs to fetch. The account for each matched ID is returned.
If no account matches an ID, no object is returned for that account. So the order of accounts in the response is not necessarily the same as the order of IDs in the request. You can refer to the ID field in the response to distinguish accounts.
= client.lookup_accounts([100, 101]) accounts
Create Transfers
This creates a journal entry between two accounts.
See details for transfer fields in the Transfers reference.
= [tb.Transfer(
transfers id=tb.id(), # TigerBeetle time-based ID.
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=0,
flags=0,
timestamp
)]
= client.create_transfers(transfers)
transfer_errors # Error handling omitted.
See details for the recommended ID scheme in time-based identifiers.
Response and Errors
The response is an empty array if all transfers were created successfully. If the response is non-empty, each object in the response array contains error information for a transfer that failed. The error object contains an error code and the index of the transfer in the request batch.
See all error conditions in the create_transfers reference.
= [tb.Transfer(
batch id=1,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=0,
flags=0,
timestamp
),
tb.Transfer(id=2,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=0,
flags=0,
timestamp
),
tb.Transfer(id=3,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=0,
flags=0,
timestamp
)]
= client.create_transfers(batch)
transfer_errors for error in transfer_errors:
if error.result == tb.CreateTransferResult.EXISTS:
print(f"Batch transfer at {error.index} already exists.")
else:
print(f"Batch transfer at {error.index} failed to create: {error.result}.")
To handle errors you can compare the result code returned from
client.create_transfers
with enum values in the
CreateTransferResult
object.
Batching
TigerBeetle performance is maximized when you batch API requests. The client does not do this automatically for you. So, for example, you can insert 1 million transfers one at a time like so:
= [] # Array of transfer to create.
batch for transfer in batch:
= client.create_transfers([transfer])
transfer_errors # Error handling omitted.
But the insert rate will be a fraction of potential. Instead, always batch what you can.
The maximum batch size is set in the TigerBeetle server. The default is 8190.
= [] # Array of transfer to create.
batch = 8190 #FIXME
BATCH_SIZE for i in range(0, len(batch), BATCH_SIZE):
= client.create_transfers(
transfer_errors min(len(batch), BATCH_SIZE)],
batch[i:
)# Error handling omitted.
Queues and Workers
If you are making requests to TigerBeetle from workers pulling jobs from a queue, you can batch requests to TigerBeetle by having the worker act on multiple jobs from the queue at once rather than one at a time. i.e. pulling multiple jobs from the queue rather than just one.
Transfer Flags
The transfer flags
value is a bitfield. See details for
these flags in the Transfers
reference.
To toggle behavior for a transfer, combine enum values stored in the
TransferFlags
object (it’s an enum.IntFlag
)
with bitwise-or:
TransferFlags.linked
TransferFlags.pending
TransferFlags.post_pending_transfer
TransferFlags.void_pending_transfer
For example, to link transfer0
and
transfer1
:
= tb.Transfer(
transfer0 id=4,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=tb.TransferFlags.LINKED,
flags=0,
timestamp
)= tb.Transfer(
transfer1 id=5,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=0,
flags=0,
timestamp
)
# Create the transfer
= client.create_transfers([transfer0, transfer1])
transfer_errors # Error handling omitted.
Two-Phase Transfers
Two-phase transfers are supported natively by toggling the
appropriate flag. TigerBeetle will then adjust the
credits_pending
and debits_pending
fields of
the appropriate accounts. A corresponding post pending transfer then
needs to be sent to post or void the transfer.
Post a Pending Transfer
With flags
set to post_pending_transfer
,
TigerBeetle will post the transfer. TigerBeetle will atomically roll
back the changes to debits_pending
and
credits_pending
of the appropriate accounts and apply them
to the debits_posted
and credits_posted
balances.
= tb.Transfer(
transfer0 id=6,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=tb.TransferFlags.PENDING,
flags=0,
timestamp
)
= client.create_transfers([transfer0])
transfer_errors # Error handling omitted.
= tb.Transfer(
transfer1 id=7,
=102,
debit_account_id=103,
credit_account_id# Post the entire pending amount.
=tb.amount_max,
amount=6,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=tb.TransferFlags.POST_PENDING_TRANSFER,
flags=0,
timestamp
)
= client.create_transfers([transfer1])
transfer_errors # Error handling omitted.
Void a Pending Transfer
In contrast, with flags
set to
void_pending_transfer
, TigerBeetle will void the transfer.
TigerBeetle will roll back the changes to debits_pending
and credits_pending
of the appropriate accounts and
not apply them to the debits_posted
and
credits_posted
balances.
= tb.Transfer(
transfer0 id=8,
=102,
debit_account_id=103,
credit_account_id=10,
amount=0,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=tb.TransferFlags.PENDING,
flags=0,
timestamp
)
= client.create_transfers([transfer0])
transfer_errors # Error handling omitted.
= tb.Transfer(
transfer1 id=9,
=102,
debit_account_id=103,
credit_account_id=10,
amount=8,
pending_id=0,
user_data_128=0,
user_data_64=0,
user_data_32=0,
timeout=1,
ledger=720,
code=tb.TransferFlags.VOID_PENDING_TRANSFER,
flags=0,
timestamp
)
= client.create_transfers([transfer1])
transfer_errors # Error handling omitted.
Transfer Lookup
NOTE: While transfer lookup exists, it is not a flexible query API. We are developing query APIs and there will be new methods for querying transfers in the future.
Transfer lookup is batched, like transfer creation. Pass in all
id
s to fetch, and matched transfers are returned.
If no transfer matches an id
, no object is returned for
that transfer. So the order of transfers in the response is not
necessarily the same as the order of id
s in the request.
You can refer to the id
field in the response to
distinguish transfers.
= client.lookup_transfers([1, 2]) transfers
Get Account Transfers
NOTE: This is a preview API that is subject to breaking changes once we have a stable querying API.
Fetches the transfers involving a given account, allowing basic filter and pagination capabilities.
The transfers in the response are sorted by timestamp
in
chronological or reverse-chronological order.
filter = tb.AccountFilter(
=2,
account_id=0, # No filter by UserData.
user_data_128=0,
user_data_64=0,
user_data_32=0, # No filter by Code.
code=0, # No filter by Timestamp.
timestamp_min=0, # No filter by Timestamp.
timestamp_max=10, # Limit to ten balances at most.
limit=tb.AccountFilterFlags.DEBITS | # Include transfer from the debit side.
flags| # Include transfer from the credit side.
tb.AccountFilterFlags.CREDITS # Sort by timestamp in reverse-chronological order.
tb.AccountFilterFlags.REVERSED,
)
= client.get_account_transfers(filter) account_transfers
Get Account Balances
NOTE: This is a preview API that is subject to breaking changes once we have a stable querying API.
Fetches the point-in-time balances of a given account, allowing basic filter and pagination capabilities.
Only accounts created with the flag history
set retain historical
balances.
The balances in the response are sorted by timestamp
in
chronological or reverse-chronological order.
filter = tb.AccountFilter(
=2,
account_id=0, # No filter by UserData.
user_data_128=0,
user_data_64=0,
user_data_32=0, # No filter by Code.
code=0, # No filter by Timestamp.
timestamp_min=0, # No filter by Timestamp.
timestamp_max=10, # Limit to ten balances at most.
limit=tb.AccountFilterFlags.DEBITS | # Include transfer from the debit side.
flags| # Include transfer from the credit side.
tb.AccountFilterFlags.CREDITS # Sort by timestamp in reverse-chronological order.
tb.AccountFilterFlags.REVERSED,
)
= client.get_account_balances(filter) account_balances
Query Accounts
NOTE: This is a preview API that is subject to breaking changes once we have a stable querying API.
Query accounts by the intersection of some fields and by timestamp range.
The accounts in the response are sorted by timestamp
in
chronological or reverse-chronological order.
= tb.QueryFilter(
query_filter =1000, # Filter by UserData.
user_data_128=100,
user_data_64=10,
user_data_32=1, # Filter by Code.
code=0, # No filter by Ledger.
ledger=0, # No filter by Timestamp.
timestamp_min=0, # No filter by Timestamp.
timestamp_max=10, # Limit to ten balances at most.
limit=tb.QueryFilterFlags.REVERSED, # Sort by timestamp in reverse-chronological order.
flags
)
= client.query_accounts(query_filter) query_accounts
Query Transfers
NOTE: This is a preview API that is subject to breaking changes once we have a stable querying API.
Query transfers by the intersection of some fields and by timestamp range.
The transfers in the response are sorted by timestamp
in
chronological or reverse-chronological order.
= tb.QueryFilter(
query_filter =1000, # Filter by UserData.
user_data_128=100,
user_data_64=10,
user_data_32=1, # Filter by Code.
code=0, # No filter by Ledger.
ledger=0, # No filter by Timestamp.
timestamp_min=0, # No filter by Timestamp.
timestamp_max=10, # Limit to ten balances at most.
limit=tb.QueryFilterFlags.REVERSED, # Sort by timestamp in reverse-chronological order.
flags
)
= client.query_transfers(query_filter) query_transfers
Linked Events
When the linked
flag is specified for an account when
creating accounts or a transfer when creating transfers, it links that
event with the next event in the batch, to create a chain of events, of
arbitrary length, which all succeed or fail together. The tail of a
chain is denoted by the first event without this flag. The last event in
a batch may therefore never have the linked
flag set as
this would leave a chain open-ended. Multiple chains or individual
events may coexist within a batch to succeed or fail independently.
Events within a chain are executed within order, or are rolled back
on error, so that the effect of each event in the chain is visible to
the next, and so that the chain is either visible or invisible as a unit
to subsequent events after the chain. The event that was the first to
break the chain will have a unique error result. Other events in the
chain will have their error result set to
linked_event_failed
.
= [] # List of tb.Transfers to create.
batch = 0
linkedFlag |= tb.TransferFlags.LINKED
linkedFlag
# An individual transfer (successful):
id=1))
batch.append(tb.Transfer(
# A chain of 4 transfers (the last transfer in the chain closes the chain with linked=false):
id=2, flags=linkedFlag)) # Commit/rollback.
batch.append(tb.Transfer(id=3, flags=linkedFlag)) # Commit/rollback.
batch.append(tb.Transfer(id=2, flags=linkedFlag)) # Fail with exists
batch.append(tb.Transfer(id=4, flags=0)) # Fail without committing.
batch.append(tb.Transfer(
# An individual transfer (successful):
# This should not see any effect from the failed chain above.
id=2, flags=0 ))
batch.append(tb.Transfer(
# A chain of 2 transfers (the first transfer fails the chain):
id=2, flags=linkedFlag))
batch.append(tb.Transfer(id=3, flags=0))
batch.append(tb.Transfer(
# A chain of 2 transfers (successful):
id=3, flags=linkedFlag))
batch.append(tb.Transfer(id=4, flags=0))
batch.append(tb.Transfer(
= client.create_transfers(batch)
transfer_errors # Error handling omitted.
Imported Events
When the imported
flag is specified for an account when
creating accounts or a transfer when creating transfers, it allows
importing historical events with a user-defined timestamp.
The entire batch of events must be set with the flag
imported
.
It’s recommended to submit the whole batch as a linked
chain of events, ensuring that if any event fails, none of them are
committed, preserving the last timestamp unchanged. This approach gives
the application a chance to correct failed imported events,
re-submitting the batch again with the same user-defined timestamps.
# External source of time.
= 0
historical_timestamp # Events loaded from an external source.
= [] # Loaded from an external source.
historical_accounts = [] # Loaded from an external source.
historical_transfers
# First, load and import all accounts with their timestamps from the historical source.
= []
accounts for index, account in enumerate(historical_accounts):
# Set a unique and strictly increasing timestamp.
+= 1
historical_timestamp = historical_timestamp
account.timestamp # Set the account as `imported`.
= tb.AccountFlags.IMPORTED
account.flags # To ensure atomicity, the entire batch (except the last event in the chain)
# must be `linked`.
if index < len(historical_accounts) - 1:
|= tb.AccountFlags.LINKED
account.flags
accounts.append(account)
= client.create_accounts(accounts)
account_errors # Error handling omitted.
# The, load and import all transfers with their timestamps from the historical source.
= []
transfers for index, transfer in enumerate(historical_transfers):
# Set a unique and strictly increasing timestamp.
+= 1
historical_timestamp = historical_timestamp
transfer.timestamp # Set the account as `imported`.
= tb.TransferFlags.IMPORTED
transfer.flags # To ensure atomicity, the entire batch (except the last event in the chain)
# must be `linked`.
if index < len(historical_transfers) - 1:
|= tb.AccountFlags.LINKED
transfer.flags
transfers.append(transfer)
= client.create_transfers(transfers)
transfer_errors # Error handling omitted.
# Since it is a linked chain, in case of any error the entire batch is rolled back and can be retried
# with the same historical timestamps without regressing the cluster timestamp.