​
Kafka Advantages

Lower Latency

  • Shorter data pipeline
  • Less processing overhead compared to GraphQL subscriptions
  • No custom database or additional formatting logic needed

Better Reliability

  • More stable connection protocol than WebSocket
  • Optimized for persistent connections
  • Messages can be read from the latest offset

Superior Scalability

  • Multiple consumers can split the load
  • Automatic load redistribution
  • Better handling of high-volume data

​
Kafka Limitations

Browser Incompatibility

  • Server-side only implementation
  • No direct browser consumption

Limited Filtering

  • No pre-filtering capabilities
  • Pre-defined schema
  • Client-side post-processing required

Development Complexity

  • No IDE support for streams
  • Debugging limited to the consumer side

​
Technical Considerations

When using Kafka Streams, consider message retention policies, latency implications, and partitioning strategies. Messages have a default retention of 24 hours for critical data streams.

​
Implementation Guide

​
Message Retention

Retention Policies

  • Protobuf Streams: Retained for 24 hours
  • DEX Trades (JSON): Retained for 24 hours
  • Other JSON Streams: Retained for 4 hours

​
Connection Setup

const sasl_conf = {
  'bootstrap.servers': 'kfk0.chainstream.io:9093,kfk1.chainstream.io:9093,kfk2.chainstream.io:9093',
  'security.protocol': 'SASL_SSL',
  'sasl.mechanism': 'SCRAM-SHA-512',
  'sasl.username': '<YOUR USERNAME>',
  'sasl.password': '<YOUR PASSWORD>',
  'ssl.key.location': 'client.key.pem',
  'ssl.ca.location': 'server.cer.pem',
  'ssl.certificate.location': 'client.cer.pem',
  'ssl.endpoint.identification.algorithm': 'none'
}

​
Topic Structure

<BLOCKCHAIN_NAME>.<MESSAGE_TYPE>
<BLOCKCHAIN_NAME>.broadcasted.<MESSAGE_TYPE>
Message Types:
  • dextrades - DEX trades
  • dexorders - DEX orders
  • dexpools - DEX pools
  • transactions - Transactions
  • transfers - Transfers
  • instructions - Instructions
  • raw - Raw data

​
Best Practices for Kafka Streaming

Parallel Processing for High Throughput

  • Distribute consumers across multiple partitions to process data in parallel
  • Allocate one consumer thread per partition for efficient load balancing

Continuous Streaming for Stability

  • Keep Kafka consumers running to ensure seamless data flow
  • Process messages asynchronously to prevent bottlenecks
  • Avoid blocking the main consumption loop to maintain optimal throughput

Optimized Message Handling

  • Implement batch processing where applicable to reduce overhead
  • Use worker groups to improve concurrency
  • Continuously monitor processing latency to ensure real-time performance

​
Additional Code Examples

​
Prerequisites

ChainStream Kafka Server Access

Access to ChainStream Kafka brokers

Authentication Credentials

Username and Password for Kafka brokers

Topic Subscription

Topic name(s) to subscribe to

Go Installation

Go Version >= 1.16

Kafka Client

Confluent Kafka Go Client library

​
Dependencies Setup

go mod init kafka-consumer

​
Implementation

Kafka Client Initialization

Kafka Consumer Setup

Message Processing Loop

Message Processing Function

​
Running the Application

go mod init kafka-consumer

​
Execution Workflow

Kafka Client Initialization

  • Initialize with SSL and SASL configurations
  • Set unique group.id for independent message consumption

Consumer Setup

  • Create Kafka consumer
  • Subscribe to specified topic

Message Processing

  • Poll for new messages every 100ms
  • Parse JSON messages
  • Log processed data

Graceful Shutdown

  • Handle termination signals
  • Close consumer cleanly