Skip to content

Commit c2b22c9

Browse files
author
Sudev Ambadi
committed
updated readme
1 parent 908a532 commit c2b22c9

File tree

1 file changed

+7
-5
lines changed

1 file changed

+7
-5
lines changed

README.md

+7-5
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@
22

33
A simple handler for python logger to push logs into Kafka instead of writting them to disk.
44

5-
The logging handler will write to Kafka in async batch using kafka-python.
5+
The logging handler will write to Kafka in async batch using [pykafka](https://github.com/Parsely/pykafka). Current implementation works well, but is not fault tolerant if kafka fails (Server not available, Leader not available ..) ==> no logs.
66

7-
### Issues:
7+
### Todo:
88

9-
* Current version writes to Kafka using a different thread, so if the parent thread who was creating the logs exits then the messages won't be send to Kafka.
10-
- To overcome this issue we might have to use [pykafka](https://github.com/Parsely/pykafka), and block termination of thread until messages are sent
9+
* Logger fails when Kafka is down, need to tweak code such that Handler can be configured in runtime (If runtime change of handler is available!) else tweak to emit logs into file instead of pushing into Kafka.
10+
* Write test case where Kafka client connection is already established and then fails(Leader not available for partition and other Kafka errors)
11+
* Introduce snappy compression of messages.
12+
* Configure pykafka to use [librdkafka](https://github.com/edenhill/librdkafka), this will effectively better performance a lot, especially while using compression.
1113

12-
Basically a async version of https://github.com/taykey/python-kafka-logging
14+
Inspired from https://github.com/taykey/python-kafka-logging

0 commit comments

Comments
 (0)