Построение микросервисных систем с использованием Kafka

iTechMeetup Mogilev #5

Mikalai Saskavets

Full Stack Web Developer

DevOps Enthusiast

• more than 12 years in web development

some experience with: php javascript python java perl golang lua nginx apache
postgres mysql redis rabbitmq cassandra dynamodb kafka docker kubernetes ansible
solr elasticsearch jquery react webpack django celery wordpress linux git latex

• last 6 years is falling into Python
• member of PyCon Belarus Program Committee
• contributing member of Minsk Python Meetup

Monolithic MVP

Scalability issues
Readability issues
Maintainability issues
Security issues

Microservices: Coupling and Cohesion?

Distributed Monolith?

Microservices: Synchronous Communication

( REST API )

Microservices: Synchronous Communication

( REST API )

Microservices: Async Communication (Celery)

shall we use
Kafka here???

What is Kafka?

Distributed Streaming Platform

Initially conceived as a messaging queue... Now Apache Kafka is a full-fledged distributed event streaming platform capable of handling trillions of events a day. ^[↪]

Kafka for what??

Messaging Metrics Website Activity Tracking Log Aggregation Stream Processing Event Sourcing Commit Log

Queue

Why Kafka???

up to 2,000,000 messages per sec
or even more... ^[↪]

Alternatives for Kafka

RabbitMQ ActiveMQ
NATS Apache Pulsar NSQ
AWS Kinesis

Kafka: Overview

Kafka: Partitions for Topics

Kafka: Partitions: few consumer groups

Kafka: Consumers Offsets Example for 1 Partition




          ┌───[Consumer Group 1]
          │
1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18
    │
    └───[Consumer Group 2]

Kafka: Consumers Offsets Example for 1 Partition




                  ┌───[Consumer Group 1]
                  │
1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20
      │
      └───[Consumer Group 2]

Kafka: Consumers Offsets Example for 1 Partition




                           ┌───[Consumer Group 1]
                           │
1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23
│           │
│           └───[Consumer Group 2]
│
└───[Consumer Group 3]

Kafka: Consumers Offsets Example for 1 Partition


                                    [Consumer Group 4]───┐
                                                         │
                           ┌───[Consumer Group 1]        │ 
                           │                             │
1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23
│           │
│           └───[Consumer Group 2]
│
└───[Consumer Group 3]

Kafka: Reset Consumer Group Offsets tooling


  Reset to Datetime 
  Reset from Duration
  Reset to Earliest
  Reset to Latest
  Reset to Current Time
  Reset to Offset
  Shift Offset by 'n' 
  Reset from file

Kafka: Consumers Offsets Example for 3 Partitions


                        ┌───[CG 1 C 1] 
                        │ 
Partition 1 >> 1¹|4²|7³|10⁴|13⁵|16⁶|19⁷|22⁸|25⁹
                     │
                     └───[CG 2 C 1]

                  ┌───[CG 1 C 2] 
                  │ 
Partition 2 >> 2¹|5²|8³|11⁴|14⁵|17⁶|20⁷|23⁸
                  │
                  └───[CG 2 C 2]

                            ┌───[CG 1 C 3] 
                            │ 
Partition 3 >> 3¹|6²|9³|12⁴|15⁵|18⁶|21⁷|24⁸
               │
               └───[CG 2 C 3]