Awesome Open Source
Awesome Open Source

Build Master Codacy Badge

KafkaFlow

KafkaFlow is a .NET framework to create Kafka based applications, simple to use and extend.

KafkaFlow uses Confluent Kafka Client.

Features

  • Multi-threaded consumer with message order guarantee
  • Middlewares support for producing and consuming messages
  • Support topics with different message types
  • Consumers with many topics
  • Serializer middleware with ApacheAvro (with Schema Registry), ProtoBuf and Json support
  • Compressor middleware (Gzip or implementing IMessageCompressor interface)
  • Graceful shutdown (wait to finish processing to shutdown)
  • Store offset when processing ends, avoiding message loss
  • Supports .NET Core and .NET Framework
  • Can be used with any dependency injection framework (see here)
  • Fluent configuration
  • Admin Web API that allows pause, resume and restart consumers, change workers count and rewind offsets, all at runtime

Packages

Packages Page

Basic Usage

.NET Core 2.1 and later using Hosted Service

public static void Main(string[] args)
{
    Host
        .CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            services.AddKafkaFlowHostedService(kafka => kafka
                .UseConsoleLog()
                .AddCluster(cluster => cluster
                    .WithBrokers(new[] { "localhost:9092" })
                    .AddConsumer(consumer => consumer
                        .Topic("sample-topic")
                        .WithGroupId("sample-group")
                        .WithBufferSize(100)
                        .WithWorkersCount(10)
                        .AddMiddlewares(middlewares => middlewares
                            .AddSerializer<NewtonsoftJsonMessageSerializer>()
                            .AddTypedHandlers(handlers => handlers
                                .AddHandler<SampleMessageHandler>())
                        )
                    )
                    .AddProducer("producer-name", producer => producer
                        .DefaultTopic("sample-topic")
                        .AddMiddlewares(middlewares => middlewares
                            .AddSerializer<NewtonsoftJsonMessageSerializer>()
                        )
                    )
                )
            );
        })
        .Build()
        .Run();
}

See the setup page and samples for more details

Documentation

Wiki Page

Contributing

  1. Fork this repository
  2. Follow project guidelines
  3. Do your stuff
  4. Open a pull request following conventional commits

Maintainers

License

MIT


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
c-sharp (11,947
csharp (1,044
dotnet (868
kafka (405
dotnet-core (396
nuget (150
workers (50
consumer (25

Find Open Source By Browsing 7,000 Topics Across 59 Categories