Throughout the week, I read a lot of blog-posts, articles, and so forth that has to do with things that interest me:
- AI/data science
- data in general
- data architecture
- streaming
- distributed computing
- SQL Server
- transactions (both db as well as non db)
- and other “stuff”
This blog post is the “roundup” of the things that have been most interesting to me for the week just ending.
Machine Learning / AI
- A game-theoretic approach to provably correct and scalable offline RL. This post looks at reinforcement learning (RL) and a way to design a systematic and provably correct way to create offline RL algorithms that can learn good policies with state-of-the-art empirical performance.
Azure Data Explorer
- Generally available: Kusto Emulator. This is awesome! Microsoft has released a Docker image exposing the Kusto Query Engine endpoint in Azure Data Explorer. This means we can now create databases, and ingest and query data in a local environment!
Streaming
- IoT Real-Time Analytics — WAGO PLC with Databricks Auto Loader. Databricks Auto Loader allows you to stream or incrementally ingest and transform files from a data lake. This post looks at a solution whereby IoT messages are sent to an Azure IoT hub and ingested in Azure Data Lake Storage (ADLS). From ADLS, the data is streamed into Azure Databricks Delta Tables using Databricks Auto Loader. Very interesting, this may be something for us at Derivco!
- Welcome to Streamiz.Kafka.Net. If you have used Kafka, you are probably aware of KStreams, the library to create stream processing applications. You are also most likely aware that KStreams is Java/Scala only. That is about to change now with Streamiz, a .NET stream processing library for Kafka, i.e. similar to KStreams. It is in active development, and if you are into Kafka, stream processing and .NET, I urge you to look at it.
WIND (What Is Niels Doing)
I am prepping for this:
Yes, on September 22 at 12:30 UTC, I deliver the session ksqlDB - The Real Time Streaming Database at Data Platform Summit 2022.
The conference is free; you register here.
~ Finally
That’s all for this week. I hope you enjoy what I did put together. Please comment on this post or ping me if you have ideas for what to cover.
comments powered by Disqus