This blog is devoted to the community Nerd or Geek, for those who like IT and coffee, and containing random thoughts and opinions on things that interest me.
This is not an exhaustive list, so if you know someone that you think should be here, please post a comment with. And of course, some here are Kafka related.
Starting with the basics, I add the official ones;
Feel free to send anyone that you think should be here.
I couldn’t find anyone for IBM and Google clouds.
I couldn’t find anyone for Python Faust as well.
This blog post is part of my series of posts regarding Kafka Connect.
If you’re not familiar with Kafka, I suggest you have a look at some of my previous post;
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
TL;DR
This post is about code, I’ll show all steps to develop your Sink Kafka Connector.
Table of contents
1. Use Case
2. Code
3. Links
1. Use Case
One scenario that has become popular and I came across with some questions around in the last months is to use Kafka to trigger functions as service.
In simple words: Send a message in your kafka topic and your function(s) gets invoked with the payload which you sent to Kafka, you can now have a function trigger in response to messages in Kafka Topics.
My example is using OCI functions.
2. Code
The connector needs extends SinkConnector. Is almost the same from source and you need to implement six methods here.
@Override publicvoid put(Collection<SinkRecord> records){ // here is where the magic happenings. // just add a boolean triggerFn method that trigger the function. }
@Override publicvoid stop(){ // TODO Auto-generated method stub
This blog post is part of my series of posts regarding Kafka Connect.
If you’re not familiar with Kafka, I suggest you have a look at some of my previous post;
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
TL;DR
This post is about code, I’ll show all steps to develop your own Source Kafka Connector, and in the end, I’ll show an easy way to deploy your code. Openweathermap source kafka Connector.
Table of contents
1. Get Started
2. Use Case
3. Shema
4. AbstractConfig
5. Connector
6. Task
7. Code
8. SMTs
9. Converter
10. Transformation
11. Deploy
12. Links
1. Get Started
Kafka Connect aims to reduce the burden of connecting Kafka with external systems such as databases, key-value stores, search indexes, and even file systems. Connectors are the high-level abstractions that manage the streaming of data from and to Kafka. Each connector speaks the language of the external system it connects to, hiding this complexity from the application developer who can focus on more valuable business logic.
The best place to start when implementing your Source Connector is the Confluent Connector Development Guide.
OpenWeatherMap real-time weather data website. This API is already used in a lot of examples around Development. I already used some time for IoT and Data analyse demos.
They have a free API tier, which is limited to 1 request per second, which is quite enough for tracking changes in weather. Their different API schemas return plenty of numeric and textual data, all interesting for analysis. This is a perfect example for us. I tried to choose an API that contains a more complex Json, as a result, to show more in a didactic way the Shema part.
A partition is a division of source records that usually depends on the source medium.
In our case, there is only one API URL and we are only ever requesting current data, so a very good partition would be query weather data for different cities and each partition would be processed by a separate task.
3. Schema
The Schema interface is one of the most important components of Kafka Connect. This allows you to define what the data looks like without having to worry about the way the data is stored.
Here if you have a look in the json you are going to see that the json has some structures inside, for example, you can see weather that contains, id, main, descriptions and so on.
AbstractConfig is an abstract class that defines an interface that our WeatherAPIConfig
needs to adhere to, and here is where we get the configuration for our connector.
We need to pass the openWeatherApiKey, a list of cities, that we want to get the weather data and the kafka topic. I add pollFrequency as a hardcoded value because the API has a limitation of the number of requests in the free tier.
@Override publicString version(){ // you can add your own version strategy returnnull; }
@Override publicvoid start(Map<String, String> props){ // here is were we implement the configuration class }
@Override public Class<?extends Task> taskClass(){ // just need pass the name of the class returnnull; }
@Override public List<Map<String, String>> taskConfigs(int maxTasks){ // here is a function partition strategy to split the list o cities to a different task returnnull; }
@Override publicvoid stop(){ // since we don't start anything on Star method we don't need do anything here }
@Override public ConfigDef config(){ // here we already defined in the weatherAPI class returnnull; }
6. Task
In a similar manner to the Source Connector class, we implement the Source Task abstract class. Here we have only 4 methods.
@Override publicString version(){ // you can add your own version strategy returnnull; }
@Override publicvoid start(Map<String, String> props){ // initialize the configuration and the weather API client }
@Override public List<SourceRecord> poll()throwsInterruptedException{ // this method is going to be called continuous and because of the API limitation, we need to add a sleep to decrease the poll frequency. returnnull; }
@Override publicvoid stop(){ // TODO Auto-generated method stub }
7. Code
I create one package model where I add my DTO to the fields that I used in my schema and package shema.
WeatherAPIClient this class just calls the API and passes the list of cities.
I use the WeatherAPIConfig config to get all configurations;
From an architectural point of view, SMT resides exactly between the connectors and the converters and they apply their data transformation just before storing the data into Kafka and right before extracting the data from it.
The Kafka Connect frameworks come with many built-in SMT to address the most common transformations use cases like for example:
mask sensitive message fields,
add identifiers,
tag events,
cast data types to comply to the destination,
convert data (e.g. timestamp conversion like time-based data conversion standardization),
As you can see SMT can be combined and chained pretty easily by simply configuring them next to the Kafka Connect sinks and sources.
But the most important thing that should always be kept in mind is the “one message at a time”.
This is particularly crucial because it strongly limits SMT’s usage. In particular, it doesn’t permit any kind of transformation that is characterized by any kind of aggregation logic. Processing one message at a time is mandatory due to how partitions work and how the offset semantic is implemented in Kafka.
9. Converter
Here I create a package converter where I add an example of a converter, Serialize, and Deserializer. Here we have only three methods to implement.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
@Override publicvoid configure(Map<String, ?> configs, boolean isKey){ // get the properties }
@Override publicbyte[] fromConnectData(String topic, Schema schema, Object value){ // convert to array of bytes returnnull; }
@Override public SchemaAndValue toConnectData(String topic, byte[] value){ // convert to schema value returnnull; }
GitHub
You can check the full code in my Github, and I did a Scala demo as well, where I just follow the same structure of folders, packages and classes.
This blog post is part of my series of posts regarding Kafka Connect.
If you’re not familiar with Kafka, I suggest you have a look at some of my previous post;
Standalone mode is the best way to get started with minimum config and infrastructure setup. In this section, we will see how to configure a connector in Standalone mode and transfer file content to Kafka topic.
In the same directory where we have created the yaml file execute the following command to start Kafka cluster. When the below command runs for the very first time it downloads the image. Once the image is downloaded it creates a Kafka cluster.
1
docker-compose up
For starting any Kafka connect cluster we require – workers config and connector (file-stream) config.
Create two files: workers-config.properties and file-stream-connector-properties.
Workers-config.properties:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
bootstrap.servers=127.0.0.1:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false # we always leave the internal key to JsonConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter.schemas.enable=false # Rest API
rest.port=8086
rest.host.name=127.0.0.1 # this config is only for standalone workers
offset.storage.file.filename=standalone.offsets
offset.flush.interval.ms=10000
File-stream-connector-properties:
1 2 3 4 5 6 7
# These are standard kafka connect parameters, need for ALL connectors
name=file-stream-kafka-connect-standalone
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1 file/FileStreamSourceConnector.java file=source-input.txt
topic=kafka-connect-standalone
You should see three files in the folder; docker-compose.yml, file-stream-connector-properties and workers-config.properties
Create an input file source-input.txt, the content of the file is transferred to Kafka topic.
1
touch source-input.txt
Mount a host directory in a docker container: Make sure we are in the directory where we have created the files. After mount, we automatically switch to cp-kafka-connect.
1
docker run --rm -it -v "$(pwd)":/kafka-connect/ --net=host confluentinc/cp-kafka-connect
This docker image expect the connector in this folder;
/etc/kafka-connect/jars
Create Kafka topic “kafka-connect-standalone” with 3 partitions and replication factor 1.
Now you can test: Open file source-input.txt and type some message to it & save it. The message should have been transferred to Kafka topic.
What happened in the background: We wrote data to the source file and Kafka connect standalone pushed the data to topic. No programming, just configs.
Now stop Kafka Connect (press Ctrl/Command + C). Once Kafka connect gracefully stopped, list all files in the given directory. We have a new guest in the form of a file “standalone.offsets”.
This file is created by Kafka connect to keep track of from where it should resume reading messages from the source file on re-starts.
When Kafka connect starts again it should resume reading without publishing duplicate messages to the topic.
Try to execute the above command (connect-standalone workers-config.properties file-stream-connector-properties) again and validate if you do not have a duplicate message.
2. Source Distributed mode
Now let’s configure a connector in distributed mode.
Create Kafka topic “kafka-connect-distibuted” with 3 partitions and replication factor 1.
curl -d @<path-to-config-file>/connect-source.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors
Now write some message in the source file and once the file is saved, all messages are posted in topic “kafka-connect-distributed”.
check for the message copied from file to topic by FileSourceConnector. Note that messages are stored in JSON format as the connector topic created earlier with config value.converter.schemas.enable=true.
Strimzi simplifies the process of running Apache Kafka in a Kubernetes cluster. It provides container images and Operators for running Kafka on Kubernetes.
This blog post is part of my series of posts regarding “Kafka Connect Overview“.
If you’re not familiar with Kafka, I suggest you have a look at my previous post “What is Kafka?” before.
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
Kafka Connects goal of copying data between systems has been tackled by a variety of frameworks, many of them still actively developed and maintained. This section explains the motivation behind Kafka Connect, where it fits in the design space, and its unique features and design decisions
Kafka Connect has three major models in its design:
Connector model
Worker model
Data model
The connector model addresses three key user requirements. First, Kafka Connect performs broad copying by default by having users define jobs at the level of Connectors which then break the job into smaller Tasks. This two level scheme strongly encourages connectors to use configurations that encourage copying broad swaths of data since they should have enough inputs to break the job into smaller tasks. It also provides one point of parallelism by requiring Connectors to immediately consider how their job can be broken down into subtasks, and select an appropriate granularity to do so. Finally, by specializing source and sink interfaces, Kafka Connect provides an accessible connector API that makes it very easy to implement connectors for a variety of systems.
The worker model allows Kafka Connect to scale to the application. It can run scaled down to a single worker process that also acts as its own coordinator, or in clustered mode where connectors and tasks are dynamically scheduled on workers. However, it assumes very little about the process management of the workers, so it can easily run on a variety of cluster managers or using traditional service supervision. This architecture allows scaling up and down, but Kafka Connect’s implementation also adds utilities to support both modes well. The REST interface for managing and monitoring jobs makes it easy to run Kafka Connect as an organization-wide service that runs jobs for many users. Command line utilities specialized for ad hoc jobs make it easy to get up and running in a development environment, for testing, or in production environments where an agent-based approach is required.
The data model addresses the remaining requirements. Many of the benefits come from coupling tightly with Kafka. Kafka serves as a natural buffer for both streaming and batch systems, removing much of the burden of managing data and ensuring delivery from connector developers. Additionally, by always requiring Kafka as one of the endpoints, the larger data pipeline can leverage the many tools that integrate well with Kafka. This allows Kafka Connect to focus only on copying data because a variety of stream processing tools are available to further process the data, which keeps Kafka Connect simple, both conceptually and in its implementation. This differs greatly from other systems where ETL must occur before hitting a sink. In contrast, Kafka Connect can bookend an ETL process, leaving any transformation to tools specifically designed for that purpose. Finally, Kafka includes partitions in its core abstraction, providing another point of parallelism
In simples words
Kafka Connect is a distributed, scale, fault-tolerant service designed to reliably stream data between Kafka and other data systems. Data is produced from a source and consumed to a sink.
Connect tracks the offset that was last consumed for a source, to restart task at the correct starting point after a failure. These offsets are different from Kafka offsets, they are based on the sourse system like database, file, etc.
In standalone mode, the source offset is tracked in a local file and in a distributed mode, the source offset is tracked in a Kafka topic.
2. Source & Sink connectors
Producers and Consumers provide complete flexibility to send any data to Kafka or process in any way. This flexibility means you do everything yourself.
Kafka Connect’s simple frameworks allows;
developers to create connectors that copy data to or from others systems.
Operators to use said connectors just by writing configuration files and submitting them to Connect. (No code)
Community and 3rd-party engineers to build reliable plugins for common data sources and sinks.
Deployments to deliver fault-tolerant and automated load balance out-of-the-box.
And the frameworks do the hard work;
Serialization and deserialization.
Schema registry integration.
Fault-tolerant and failover.
Partitioning and scale-out.
And let the developers focus on domain specific details.
3. Standalone & Distributed
In standalone mode we have a source or a Sink and a Kafka broker, when we deploy Kafka connect in the standalone we actually need to pass a configuration file containing all the connection properties that we need to run. So standalone’s main way of providing configuration to our connector is by using properties files and not the Rest API.
In the distributed mode we usually have more than one worker, since these workers can be in different machines or containers they can not share the same storage space, so a properties file is out of the question. Instead kafka connect in distributed mode leverage kafka topics in order to sink between themselves.
4. Converters & Transforms
Pluggable API to convert data between native formats and Kafka. Just like the name says. Converters are used to come for data from a format to another.
In the Source connectors converters are invoked after the data has been fetched from the source and before it is published to kafka.
In the Sink connectors converters are invoked after the data has been consumed from Kafka and before it is stored in the sink.
Apache Kafka ships with Json Converter.
Transform is a simple operation that can be applied in the message level.
There’s a nice blog post about Single message transforms here.
Single message transforms SMT modify events before storing in Kafka, mask sensitive information, add identifiers, tag events, remove unnecessary columns and more, modify events going out of Kafka, route high priority events to faster datastore, cast data types to match destination and more.
5. Life cycle
I build this animation using the slides from (Randall Hauch, Confluent) Kafka Summit SF 2018 here.
Sequence diagram
6. Code
Confluent makes available a code example that you can found here.
If you’re not familiar with Kafka, I suggest you have a look at my previous post “What is Kafka?” before.
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion. The idea is that I’ll create a series of posts regarding Kafka Connect. Stay tuned!
Table of contents
1. What is Kafka Connect?
2. Concepts
3. Features
4. Why Kafka connect?
5. Courses
6. Books
7. Influencers List
8. Link
1. What is Kafka Connect?
Kafka Connect (or Connect API) is a framework to import/export data from/to other systems. It was added in the Kafka 0.9.0.0 release and uses the Producer and Consumer API internally. The Connect framework itself executes so-called “connectors” that implement the actual logic to read/write data from other systems. The Connect API defines the programming interface that must be implemented to build a custom connector. Many open source and commercial connectors for popular data systems are available already. However, Apache Kafka itself does not include production ready connectors.
Connectors are meant to provide a simple way of connecting to external systems, only requiring a configuration file, while the scaling, distribution and persistence of state is handled by the framework for you. Connectors for common things like JDBC exist already at the Confluent Hub. Official blog announcement and overview
For example;
Sometimes you need to process streams of data that are not in your Kafka cluster. These data may be located in a SQL database like SQL Server, MySQL or a simple CSV file. In order to process those data, you have to move them from your database to the Kafka cluster. To this end, you have some options and two of them are:
Create an application that reads data from your source storage system and produces them to Kafka cluster.
Or use Kafka Connect to move your data easily from source storage system to your Kafka cluster.
If you choose the first option you need to write codes that move your data to the Kafka cluster. Your code must deal with the failure of your application (for example it must store the offset of the last record of tables that are moved to Kafka, so it can continue to copy the records that were not inserted into Kafka), scalability, polling and much more.
But if you choose the second option you can move data without writing any code. The Kafka Connect does the same job as the first option but in a scalable and fault-tolerant way. The process of copying data from a storage system and move it to Kafka Cluster is so common that Kafka Connect tool is created to address this problem.
Kafka connectors provide some powerful features. They can be easily configured to route unprocessable or invalid messages to a dead letter queue, apply Single Message Transforms before a message is written to Kafka by a source connector or before it is consumed from Kafka by a sink connector, integrate with Confluent Schema Registry for automatic schema registration and management, and convert data into types such as Avro or JSON. By leveraging existing connectors, developers can quickly create fault-tolerant data pipelines that reliably stream data from an external source into records in Kafka topics or from Kafka topics into an external sink, all with mere configuration and no code!
2. Concept
To efficiently discuss the inner workings of Kafka Connect, it is helpful to establish a few major concepts, and of course, I suggest a look in the official docs, here.
Connectors – the high level abstraction that coordinates data streaming by managing tasks
Tasks – the implementation of how data is copied to or from Kafka
Workers – the running processes that execute connectors and tasks
Converters – the code used to translate data between Connect and the system sending or receiving data
Transforms – simple logic to alter each message produced by or sent to a connector
Dead Letter Queue – how Connect handles connector errors
Each connector instance can break down its job into multiple tasks, thereby parallelizing the work of copying data and providing scalability. When a connector instance starts up a task, it passes along the configuration properties that each task will need. The task stores this configuration as well as the status and the latest offsets for the records it has produced or consumed externally in Kafka topics. Since the task does not store any state, tasks can be stopped, started, or restarted at any time. Newly started tasks will simply pick up the latest offsets from Kafka and continue on their merry way.
Workers are a physical concept. They are processes that run inside JVM. Your job in Kafka Connect concepts is called a connector. It is something like this:
Copy records from table ‘accounts’ of my MySQL to Kafka topic ‘accounts’. It is called a source connector because you move data from external storage to Kafka.
Copy each message from Kafka topic ‘product-events’ to a CSV file ‘myfile.csv’. It is called a sink connector because you move data from Kafka to external storage.
Kafka Connect uses workers for moving data. Workers are just simple Linux (or any other OS) processes. Kafka Connect can create a cluster of workers to make the copying data process scalable and fault tolerant. Workers need to store some information about their status, their progress in reading data from external storage and so on. To store that data, they use Kafka as their storage. Note that Kafka Connect cluster (which is a cluster of workers) is completely different from the Kafka cluster (which is a cluster of Kafka brokers). More workers mean that your copying process is more fault tolerant.
Standalone vs. Distributed Mode
There are two modes for running workers: standalone mode and distributed mode. You should identify which mode works best for your environment before getting started.
Standalone mode is useful for development and testing Kafka Connect on a local machine. It can also be used for environments that typically use single agents (for example, sending web server logs to Kafka).
Distributed mode runs Connect workers on multiple machines (nodes). These form a Connect cluster. Kafka Connect distributes running connectors across the cluster. You can add more nodes or remove nodes as your needs evolve.
Distributed mode is also more fault tolerant. If a node unexpectedly leaves the cluster, Kafka Connect automatically distributes the work of that node to other nodes in the cluster. And, because of Kafka Connect stores connector configurations, status, and offset information inside the Kafka cluster where it is safely replicated, losing the node where a Connect worker runs do not result in any lost data.
Converters are necessary to have a Kafka Connect deployment support a particular data format when writing to or reading from Kafka. Tasks use converters to change the format of data from bytes to a Connect internal data format and vice versa.
3. Features
A common framework for Kafka connectors
It standardizes the integration of other data systems with Kafka. Also, it simplifies connector development, deployment, and management.
Distributed and standalone modes
Scale up to a large, centrally managed service supporting an entire organization or scale down to development, testing, and small production deployments.
REST interface
By an easy to use REST API, we can submit and manage connectors to our Kafka Connect cluster.
Automatic offset management
However, Kafka Connect can manage the offset commit process automatically even with just a little information from connectors. Hence, connector developers do not need to worry about this error-prone part of connector development.
Distributed and scalable by default
It builds upon the existing group management protocol. And to scale up a Kafka Connect cluster we can add more workers.
Streaming/batch integration
We can say for bridging streaming and batch data systems, Kafka Connect is an ideal solution.
Transformations
Enable to make simple and lightweight modifications to individual messages
Auto-recovery After Failure
To each record, a “source” connector can attach arbitrary “source location” information which it passes to Kafka Connect. Hence, at the time of failure Kafka Connect will automatically provide this information back to the connector. In this way, it can resume where it failed. Additionally, auto recovery for “sink” connectors is even easier.
Auto-failover
Auto-failover is possible because the Kafka Connect nodes build a Kafka cluster. That means if suppose one node fails the work that it is doing is redistributed to other nodes.
Simple Parallelism
A connector can define data import or export tasks, especially which execute in parallel.
It is a good chance that you are reading this while self quarantined at your home, social distancing from public and like many, have very much extra time to deal with.
TL;DR
I came with the idea to create this blog post to maybe help or give some ideas for you to start something. if you are familiar with programming, Developer, Coder, Data something, you should give a go and try or start something.
This post is a collection of links, videos, tutorials and blogs that I found mixed with my opinion.
Table of contents
1. Some articles to start
2. Datasets
3. APIs
4. Dashboards
5. Ideas
6. Code
7. Link
In the wake of the Coronavirus outbreak, many data sources have been made available to the public in an effort to encourage research in the field. Recently, the White House and a group of leading researchers published the COVID-19 Open Research Dataset (CORD-19), which is available on Kaggle. Kaggle calls data scientists to action on COVID-19
There are several ideas around, and because the ideas are based on data analyses and similar topics Oracle OCI data Science is a good place to give a go. Start free Oracle DataScience Docs video
You can use an open-source OCI Marketplace image as well, (GPU as optional), and talking about GPU, NVidia provides a free 90-day license to Parabricks
I created a list of links with some ideas that I found;
def plt1color(df):
cols=[] for l in df.index: if df.loc[l,'Covid-19']==True: #Infected
cols.append('red') elif df.loc[l,'Covid-19']==666: #Dead
cols.append('black') elif df.loc[l,'Covid-19']==115: #Hospitalized
cols.append('yellow') elif df.loc[l,'Covid-19']==7: #Cured
cols.append('green') else:
cols.append('blue')#Healthy return cols
def plt2color(Stat):
cols=[] for i in Stat.columns: if i=='Covid-19(+)': #Infected
cols.append('red') elif i=='Dead': #Dead
cols.append('black') elif i=='Hospitalized': #Hospitalized
cols.append('yellow') elif i=='Cured': #Cured
cols.append('green') else:
cols.append('blue')#Healthy return cols
def Plot(): global df, fig, Stat, Day, Moverslist
cols=plt1color(df)
ld =['Healthy','Covid-19(+)','Hospitalized','Cured','Death Toll']
axs[0].cla()
axs[0].scatter(df['X'],df['Y'],s=1,c=cols) for i in MoversList:
axs[0].scatter(df.loc[i,'X'],df.loc[i,'Y'],s=6,facecolors='none', edgecolors='black')
axs[0].text(df.loc[i,'X']+0.02, df.loc[i,'Y']+0.02,str(i), fontsize=5)
cols=plt2color(Stat)
sDay =str(Day)
title ='Day' + sDay
axs[0].set_title(title,loc='left')
axs[0].set_yticklabels([])
axs[0].set_xticklabels([])
axs[0].tick_params( # axis='both', # changes apply to the x-axis
which='both',# both major and minor ticks are affected
bottom=False,# ticks along the bottom edge are off
top=False,# ticks along the top edge are off
right=False,# ticks along the right edge are off
left=False,# ticks along the left edge are off
labelbottom=False)# labels along the bottom edge are off
axs[1].cla()
axs[1].plot(Stat.Healthy,label=ld[0],color=cols[0])
axs[1].plot(Stat['Covid-19(+)'],label=ld[1],color=cols[1])
axs[1].plot(Stat.Hospitalized,label=ld[2],color=cols[2])
axs[1].plot(Stat.Cured,label=ld[3],color=cols[3])
axs[1].plot(Stat.Dead,label=ld[4],color=cols[4]) # axs[1].set_prop_cycle(color=cols)
axs[1].legend(bbox_to_anchor=(0,1), loc='upper left', borderaxespad=0.)
plt.xlabel('Days') # plt.show() if Day<10 : sDay ='0' + sDay
title ='Day' + sDay + '.png'
plt.savefig(title)
return
def Png_to_gif(): # Create frames
frames =[]
imgs =sorted(glob.glob("*.png")) for i in imgs:
new_frame = Image.open(i)
frames.append(new_frame)
# Save into GIF
frames[0].save('png_to_gif.gif', format='GIF',
append_images=frames[1:],
save_all=True,
duration=500, loop=0)
def infect(Person): global df,Day ifrandom.random()>0.25and Day>3 : return if df.loc[Person,'Covid-19']==False:
df.loc[Person,'Covid-19'], df.loc[Person,'Day']=True, Day
def Move(xlimit,ylimit): """
Move Movers Randomly
""" global df, MoversList for i in MoversList: if(df.loc[i,'Covid-19']==115)or(df.loc[i,'Covid-19']==666) :
MoversList.remove(i)
df.loc[i,'X'], df.loc[i,'Y']=(df.loc[i,'X']+random.uniform(1,xlimit/3))%xlimit,(df.loc[i,'Y']+random.uniform(1,ylimit/3))%ylimit
def check(i,j): global df, YesterdayPatients, Distlimit
Dist =math.sqrt((df.loc[i,'X']-df.loc[j,'X'])**2+(df.loc[i,'Y']-df.loc[j,'Y'])**2)
flag =((YesterdayPatients[i]==True) ^ (YesterdayPatients[j]==True))and Dist<Distlimit return flag
def interact(): global Day, df for i inrange(len(df)): for j inrange(i): if check(i,j): if(df.loc[i,'Covid-19']==False) :
infect(i) else:
infect(j)
def kill(): global df
samplesize =math.floor(len(df[df['Covid-19']==True])*.005+len(df[df['Covid-19']==115])*.005) if samplesize>len(df[df['Covid-19']==True]): return
df.loc[df[df['Covid-19']==True].sample(n = samplesize).index.values.tolist(),'Covid-19']=666 return
def hospitilize(): global df
samplesize =math.floor(len(df[df['Covid-19']==True])*0.03) if samplesize>len(df[df['Covid-19']==True]): return
df.loc[df[df['Covid-19']==True].sample(n = samplesize).index.values.tolist(),'Covid-19']=115 return
def cure(): global df, Day
df.loc[(df['Day']<Day-10) & (df['Covid-19']==True),'Covid-19']=7
df.loc[(df['Day']<Day-21) & (df['Covid-19']==115),'Covid-19']=7 return
def Tomorrow(): # To Be checked and Resolved!!! global df, Day
Day +=1
kill()
hospitilize()
cure()
Move(xlimit,ylimit)
interact()
def Count(Day): global df, Stat
List =list(df['Covid-19'])
Stat.loc[Day,'Healthy']= List.count(False)
Stat.loc[Day,'Covid-19(+)']= List.count(True)
Stat.loc[Day,'Hospitalized']= List.count(115)
Stat.loc[Day,'Cured']= List.count(7)
Stat.loc[Day,'Dead']= List.count(666)
return
def write_log(*args): global log_file
line =' '.join([str(a)for a in args])
log_file.write(line+'\n') print(line)
# Main ---
log_file =open("Log.txt","w+")
n =1000
xlimit,ylimit=30,30
Distlimit =1.5
write_log(31*'-')
write_log("Here's the Input Data:")
write_log(8*'- - ')
write_log('Numper of Sample:',n)
write_log('X & Y limites: ',xlimit,', ',ylimit)
write_log('Distance required for Contamination:', Distlimit)
import requests importtime importsignal from gfxhat import touch, lcd, backlight, fonts from PIL import Image, ImageFont, ImageDraw
resp = requests.get('https://thevirustracker.com/free-api?global=stats') if resp.status_code!=200: # This means something went wrong. raise ApiError('GET /tasks/ {}'.format(resp.status_code))
Don’t forget to search for tweets with #ThanksOGB.
Oracle blog post my article about Face Recognition in 4 lines of code.
Today I want to show how you can create an Oracle OCI instance using the Marketplace and configure the environment to run this example easily with the Nvidia image.
class StreamingOutput(object): def__init__(self): self.frame=None self.buffer= io.BytesIO() self.condition= Condition()
def write(self, buf): if buf.startswith(b'\xff\xd8'): # New frame, copy the existing buffer's content and notify all # clients it's available self.buffer.truncate() withself.condition: self.frame=self.buffer.getvalue() self.condition.notify_all() self.buffer.seek(0) returnself.buffer.write(buf)
class StreamingServer(socketserver.ThreadingMixIn, server.HTTPServer):
allow_reuse_address =True
daemon_threads =True
with picamera.PiCamera(resolution='640x480', framerate=24)as camera:
output = StreamingOutput() #Uncomment the next line to change your Pi's Camera rotation (in degrees) #camera.rotation = 90
camera.start_recording(output, format='mjpeg') try:
address =('',8000)
server = StreamingServer(address, StreamingHandler)
server.serve_forever() finally:
camera.stop_recording()
Just change the code for instead of 0 use the streaming url, in the example here: http://ipaddress:8000/stream
if you’re not familiar with Big Data or Data lake, I suggest you have a look at my previous post “What is Big Data?” and “What is data lake?” before.
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
Table of contents
01. What is Stream processing?
02. Martin Kleppmann
03. Typical use cases
04. Pattern
05. Evaluation: Choose a Stream Processing Framework or a Product or Both?
06. Vertical vs. Horizontal Scaling
07. Streaming is better with SQL
08. Streaming Windows
09. Why Stream Processing
10. Final considerations
11. Book
12. Influence’s List
13. Links
Stream processing is becoming something like a “grand unifying paradigm” for data processing. Outgrowing its original space of real-time data processing, stream processing is becoming a technology that offers new approaches to data processing (including batch processing), real-time applications, and even distributed transactions.
1. What is Stream Processing?
Stream processing is the act of continuous incorporate new data to compute a result. In stream processing, the input data is unbounded and has no predetermined beginning or end. It simply forms a series of events that arrives at the stream processing system e.g. credit card transactions, clicks on a website, or sensor readings from internet of things devices.
Streaming is a data distribution technique where data producers write data records into an ordered data stream from which data consumers can read that data in the same order. Here is a simple data streaming diagram illustrating a data producer, a data stream and a data consumer
Each data streaming product makes a certain set of assumptions about the use cases and processing techniques to support. These assumptions leads to certain design choices, which affect what types of stream processing behaviour you can implement with them.
From wikipedia;
Stream processing is a computer programming paradigm, equivalent to dataflow programming, event stream processing, and reactive programming, that allows some applications to more easily exploit a limited form of parallel processing.
Stream Processing is a powerful technology that can scan huge volumes of data coming from sensors, credit card swipes, clickstreams and other inputs, and find actionable insights nearly instantaneously. For example, Stream Processing can detect a single fraudulent transaction in a stream containing millions of legitimate purchases, act as a recommendation engine to determine what ad or promotion to display for a particular customer while he or she is actually shopping or compute the optimal price for a car service ride in only a few seconds.
The term “Stream Processing” means that the data is coming into the processing engine as a continuous “stream” of events produced by some outside system or systems, and the processing engine works so fast that all decisions are made without stopping the data stream and storing the information first.
Streaming data and event-driven architectures are rising in popularity. The ideas have been around for a while, but technological and architectural advances have made into reality capabilities like stream processing and even function-based (aka “serverless”) computing. In many cases, the ability to act on data quickly is more valuable than a new method for batch-processing or historical data analysis.
I Googled about and I found;
Streaming is processing of data in motion.
Streaming is data that is continuously generated by different sources.
Streaming is the continuous high-speed transfer of large amounts of data from a source system to a target.
Programming paradigm that allows some applications to more easily exploit a limited form of parallel processing.
Streaming decouple data producers and data consumers from each other. When a data producer simply writes its data to a data stream, the producer does not need to know the consumers that read the data. Consumers can be added and removed independently of the producer. Consumers can also start and stop or pause and resume their consumption without the data producer needing to know about it. This decoupling simplifies the implementation of both data producers and consumers.
A data stream can be persistent, in which case it is sometimes referred to as a log or a journal. A persistent data stream has the advantage that the data in the stream can survive a shutdown of the data streaming service, so no data records are lost.
Persistent data streaming services can typically hold larger amounts of historic data than a data streaming service that only holds records in memory. Some data streaming services can even hold historic data all the way back to the first record written to the data stream. Others only hold e.g. a number of days of historic data.
In the cases where a persistent data stream holds the full history of records, consumers can replay all these records and recreate their internal state based on these records. In case a consumer discovers a bug in its own code, it can correct that code and replay the data stream to recreate its internal database.
2. Martin Kleppmann
Martin Kleppmann is the author of the book “Designing Data Intensive Applications”, and he has some nice papers/presentations;
The concept of Streaming came from Database “Replication”;
1. BuzzWords
Some people call it stream processing. Others call it Event Sourcing or CQRS. Some even call it Complex Event Processing. Sometimes, such self-important buzzwords are just smoke and mirrors, invented by companies who want to sell you stuff. But sometimes, they contain a kernel of wisdom which can really help us design better systems.
The idea of structuring data as a stream of events is nothing new, and it is used in many different fields. Even though the underlying principles are often similar, the terminology is frequently inconsistent across different fields, which can be quite confusing. Although the jargon can be intimidating when you first encounter it, don’t let that put you off; many of the ideas are quite simple when you get down to the core.
if we took that replication stream, and made it a first-class citizen in our data architecture? What if we changed our infrastructure so that the replication stream was not an implementation detail, but a key part of the public interface of the database? What if we turn the database inside out, take the implementation detail that was previously hidden, and make it a top-level concern? What would that look like?
3. Typical use cases
Stream Processing is rapidly gaining popularity and finding applications in various business domains. Found its first uses in the finance industry, as stock exchanges moved from floor-based trading to electronic trading. Today, it makes sense in almost every industry – anywhere where you generate stream data through human activities, machine data or sensors data. Assuming it takes off, the Internet of Things will increase volume, variety and velocity of data, leading to a dramatic increase in the applications for stream processing technologies.
Some use cases where stream processing can solve business problems include:
Network monitoring
Intelligence and surveillance
Risk management
E-commerce
Fraud detection
Smart order routing
Transaction cost analysis
Pricing and analytics
Market data management
Algorithmic trading
Data warehouse augmentation
Here is a short list of well-known, proven applications of Stream Processing:
Clickstream analytics can act as a recommendation engine providing actionable insights used to personalize offers, coupons and discounts, customize search results, and guide targeted advertisements — all of which help retailers enhance the online shopping experience, increase sales, and improve conversion rates.
Preventive maintenance allows equipment manufacturers and service providers to monitor quality of service, detect problems early, notify support teams, and prevent outages.
Fraud detection alerts banks and service providers of suspected frauds in time to stop bogus transactions and quickly notify affected accounts.
Emotions analytics can detect an unhappy customer and help customer service augment the response to prevent escalations before the customer’s unhappiness boils over into anger.
A dynamic pricing engine determines the price of a product on the fly based on factors such as current customer demand, product availability, and competitive prices in the area.
Common Usage Pattern for In-Stream Analytics
4. Pattern
Writing Streaming Applications requires very different thinking patterns from writing code with a language like Java. A better understanding of common patterns in Stream Processing will let us understand the domain better and build tools that handle those scenarios.
Pattern 1: Preprocessing
Preprocessing is often done as a projection from one data stream to the other or through filtering. Potential operations include
Filtering and removing some events
Reshaping a stream by removing, renaming, or adding new attributes to a stream
Splitting and combining attributes in a stream
Transforming attributes
For example, from a twitter data stream, we might choose to extract the fields: author, timestamp, location, and then filter them based on the location of the author.
Pattern 2: Alerts and Thresholds
This pattern detects a condition and generates alerts based on a condition. (e.g. Alarm on high temperature). These alerts can be based on a simple value or more complex conditions such as rate of increase etc.
For an example, in TFL (Transport for London) Demo video based on transit data from London, we trigger a speed alert when the bus has exceeded a given speed limit.
We can generate alerts for scenarios such as the server room temperature is continually increasing for the last 5 mins.
Pattern 3: Simple Counting and Counting with Windows
This pattern includes aggregate functions like Min, Max, Percentiles etc, and they can be counted without storing any data. (e.g. counting the number of failed transactions).
However, counts are often used with a time window attached to it. ( e.g. failure count last hour). There are many types of windows: sliding windows vs. batch (tumbling) windows and time vs. length windows. There are four main variations.
Time, Sliding window: keeps each event for the given time window, produce an output whenever a new event has added or removed.
Time, Batch window: also called tumbling windows, they only produce output at the end of the time window
Length, Sliding: same as the time, sliding window, but keeps a window of n events instead of selecting them by time.
Length, Batch window: same as the time, batch window, but keeps a window of n events instead of selecting them by time
There are special windows like decaying windows and unique windows.
Pattern 4: Joining Event Streams
The main idea behind this pattern is to match up multiple data streams and create a new event steam. For an example, let’s assume we play a football game with both the players and the ball having sensors that emit events with current location and acceleration. We can use “joins” to detect when a player has kicked the ball. To that end, we can join the ball location stream and the player stream on the condition that they are close to each other by one meter and the ball’s acceleration has increased by more than 55m/s^2.
Among other use cases are combining data from two sensors, and detecting the proximity of two vehicles. Please refer to Stream Processing 101: From SQL to Streaming SQL in 10 Minutes for more details.
Pattern 5: Data Correlation, Missing Events, and Erroneous Data
This pattern and the pattern four a has lot in common where here too we match up multiple streams. In addition, we also correlate the data within the same stream. This is because different data sensors can send events at different rates, and many use cases require this fundamental operator.
Following are some possible scenarios.
Matching up two data streams that send events at different speeds
Detecting a missing event in a data stream ( e.g. detect a customer request that has not been responded within 1 hour of its reception. )
Detecting erroneous data (e.g. Detect failed sensors using a set of sensors that monitor overlapping regions and using those redundant data to find erroneous sensors and removing their data from further processing)
Pattern 6: Interacting with Databases
Often we need to combine the real time data against the historical data stored in a disk. Following are a few examples.
When a transaction happened, look up the age using the customer ID from customer database to be used for Fraud detection (enrichment)
Checking a transaction against blacklists and whitelists in the database
Receive an input from the user (e.g. Daily discount amount may be updated in the database, and then the query will pick it automatically without human intervention.)
Using regular expressions with strings, we detect a pattern of characters from a sequence of characters. Similarly, given a sequence of events, we can write a regular expression to detect a temporal sequence of events arranged on time where each event or condition about the event is parallel to a character in a string in the above example.
A frequently cited example, although bit simplistic, is that a thief, having stolen a credit card, would try a smaller transaction to make sure it works and then do a large transaction. Here the small transaction followed by a large transaction is a temporal sequence of events arranged on time and can be detected using a regular expression written on top of an event sequence.
Such temporal sequence patterns are very powerful. For example, the following video shows a real time analytics done using the data collected from a real football game. This was the dataset taken from DEBS 2013 Grand Challenge.
In the video, we used patterns on event sequence to detect the ball possession, the time period a specific player controlled the ball. A player possessed the ball from the time he hits the ball until someone else hits the ball. This condition can be written as a regular expression: a hit by me, followed by any number of hits by me, followed by a hit by someone else. (We already discussed how to detect the hits on the ball in Pattern 4: Joins).
Pattern 8: Tracking
The eighth pattern tracks something over space and time and detects given conditions.
Following are few examples
Tracking a fleet of vehicles, making sure that they adhere to speed limits, routes, and geo-fences.
Tracking wildlife, making sure they are alive (they will not move if they are dead) and making sure they will not go out of the reservation.
Tracking airline luggage and making sure they are not been sent to wrong destinations
Tracking a logistic network and figure out bottlenecks and unexpected conditions.
For example, TFL Demo we discussed under pattern 2 shows an application that tracks and monitors London buses using the open data feeds exposed by TFL(Transport for London).
Pattern 9: Detecting Trends
We often encounter time series data. Detecting patterns from time series data and bringing them into operator attention are common use cases.
Following are some of the examples of tends.
Rise, Fall
Turn (switch from a rise to a fall)
Outliers
Complex trends like triple bottom etc.
These trends are useful in a wide variety of use cases such as
Stock markets and Algorithmic trading
Enforcing SLA (Service Level Agreement), Auto Scaling, and Load Balancing
Predictive maintenance ( e.g. guessing the Hard Disk will fill within next week)
Pattern 10: Running the same Query in Batch and Realtime Pipelines
This pattern runs the same query in both Relatime and batch pipeline. It is often used to fill the gap left in the data due to batch processing. For example, if batch processing takes 15 minutes, results would lack the data for the last 15 minutes.
The idea of this pattern, which is sometimes called “Lambda Architecture” is to use real time analytics to fill the gap. Jay Kreps’s article “Questioning the Lambda Architecture” discusses this pattern in detail.
Pattern 11: Detecting and switching to Detailed Analysis
The main idea of the pattern is to detect a condition that suggests some anomaly, and further analyze it using historical data. This pattern is used with the use cases where we cannot analyze all the data with full detail. Instead, we analyze anomalous cases in full detail. Following are a few examples.
Use basic rules to detect Fraud (e.g. large transaction), then pull out all transactions done against that credit card for a larger time period (e.g. 3 months data) from a batch pipeline and run a detailed analysis
While monitoring weather, detect conditions like high temperature or low pressure in a given region and then start a high resolution localized forecast on that region.
Detect good customers, for example through the expenditure of more than $1000 within a month, and then run a detailed model to decide the potential of offering a deal.
Pattern 12: Using a Model
The idea is to train a model (often a Machine Learning model), and then use it with the Realtime pipeline to make decisions. For example, you can build a model using R, export it as PMML (Predictive Model Markup Language) and use it within your realtime pipeline.
Among examples is Fraud Detections, Segmentation, Predict next value, Predict Churn. Also see InfoQ article, Machine Learning Techniques for Predictive Maintenance, for a detailed example of this pattern.
Pattern 13: Online Control
There are many use cases where we need to control something online. The classical use cases are the autopilot, self-driving, and robotics. These would involve problems like current situation awareness, predicting the next value(s), and deciding on corrective actions.
You can implement most of these use cases with a Stream Processor that supports a Streaming SQL language.
You can find details about pattern implementations and source code from here.
Monal Daxini presents a blueprint for streaming data architectures and a review of desirable features of a streaming engine. He also talks about streaming application patterns and anti-patterns, and use cases and concrete examples using Apache Flink. Patterns of Streaming Applications
5. Evaluation: Choose a Stream Processing Framework or a Product or Both?
There are many different data streaming products, and it can be hard to know where to start studying them, and which products do what etc.
The typical evaluation process (long list, short list, proof of concept) is obligatory before making a decision.
A stream processing programming language for streaming analytics
Visual development and debugging instead of coding
Real-time analytics
Monitoring and alerts
Support for fault tolerance, and highly optimized performance
Product maturity
In the case of TIBCO, a live data mart and operational command and control center for business users
Out-of-the-box connectivity to plenty of streaming data sources
Commercial support
Professional services and training.
Think about which of the above features you need for your project. In addition, you have to evaluate the costs of using a framework against productivity, reduced effort and time-to-market using a product before making your choice.
Besides evaluating the core features of stream processing products, you also have to check integration with other products. Can a product work together with messaging, Enterprise Service Bus (ESB), Master Data Management (MDM), in-memory stores, etc. in a loosely coupled, but highly integrated way? If not, there will be a lot of integration time and high costs.
6. Vertical vs. Horizontal Scaling
Vertical scaling means running your data streaming storage and processors on a more powerful computer. Vertical scaling is also sometimes referred to as scaling up. You scale up the size and speed of its disk, memory, speed of CPUs, possibly CPU cores too, graphics cards etc.
Horizontal scaling means distributing the workload among multiple computers. Thus, the data in the data stream is distributed among multiple computers, and the applications processing the data streams are too (or at least they can be). Horizontal scaling is also sometimes referred to as scaling out. You scale out from a single computer to multiple computers.
Distributing the messages of a data stream onto multiple computers is also referred to as partitioning the data stream.
1. Round Robin Partitioning
Round robin data stream partitioning is the simplest way to partition the messages of a data stream across multiple computers. The round robin partitioning method simply distributes the messages evenly and sequentially among the computers. In other words, the first message is stored on the first computer, the second message on the second computer etc. When all computers have received a message from the stream, the round robin method starts from the first computer again.
2. Key Based Partitioning
Key based partitioning distributes the message across different computers based on a certain key value read from each message. Commonly the identifying id (e.g. primary key) is used as key to distribute the messages. Typically, a hash value is calculated from each key value, and that hash value is then used to map the message to one of the computers in the cluster.
Stream Processing and DWH
A DWH is a great tool to store and analyze structured data. You can store terabytes of data and get answers to your queries about historical data within seconds. DWH products such as Teradata or HP Vertica were built for this use case. However the ETL processes often take too long. Business wants to query up-to-date information instead of using an approach where you may only get information about what happened yesterday. This is where stream processing comes in and feeds all new data into the DWH immediately.
Stream Processing and Hadoop
A big data architecture contains stream processing for real-time analytics and Hadoop for storing all kinds of data and long-running computations.
Hadoop initially started with MapReduce, which offers batch processing where queries take hours, minutes or at best seconds. This is and will be great for complex transformations and computations of big data volumes. However, it is not so good for ad hoc data exploration and real-time analytics. Multiple vendors have though made improvements and added capabilities to Hadoop that make it capable of being more than just a batch framework.
DWH, Hadoop and stream processing complement each other very well. Therefore, the integration layer is even more important in the big data era, because you have to combine more and more different sinks and sources.
Since 2016, a new idea called Streaming SQL has emerged. We call a language that enables users to write SQL like queries to query streaming data as a “Streaming SQL” language. Almost all Stream Processors now support Streaming SQL.
7. Streaming is better with SQL
Let’s assume that you picked a stream processor, implemented some use cases, and it’s working. Now you sit down to savor the win. However, given that you can simply write SQL or something like SQL when doing batch processing, why should you have to write all this code? Shouldn’t you be able to do streaming with SQL? The answer is yes, you should. Such streaming SQL exists. Again there are many offerings. Unfortunately, unlike SQL, there is no standard streaming SQL syntax. There are many favors, which follow SQL but have variations.
SQL is a powerful language for querying structured data. It is designed as a set of independent operators: projection, filter, joins, and grouping, which can be recombined to create very powerful queries.
Following are some advantages of streaming SQL languages:
It’s easy to follow and learn for the many people who know SQL.
It’s expressive, short, sweet and fast!!
It defines core operations that cover 90% of problems.
Streaming SQL language experts can dig in when they like by writing extensions!
A query engine can better optimize the executions with a streaming SQL model. Most optimizations are already studied under SQL, and there is much we can simply borrow from database optimizations.
Let us walk through a few of the key operators. Just as SQL can cover most data queries on data stored in a disk, streaming SQL can cover most of the queries on streaming data. Without streaming SQL, programmers would have to hand code each operator, which is very complicated and hard work.
Concepts in SQL, such as “group by” and “having” clauses, usually work similarly with streaming SQL languages.
Streaming SQL has two additional concepts not covered by SQL: windows and joins, which handle the complexities of streaming. Let’s understand each of them.
8. Streaming Windows
Although batch can be handled as a special case of stream processing, analyzing never-ending streaming data often requires a shift in the mindset and comes with its own terminology (for example, “windowing” and “at-least-once”/”exactly-once” processing). This shift and the new terminology can be quite confusing for people being new to the space of stream processing.
Consider the example of a traffic sensor that counts every 15 seconds the number of vehicles passing a certain location. The resulting stream could look like:
If you would like to know, how many vehicles passed that location, you would simply sum the individual counts. However, the nature of a sensor stream is that it continuously produces data. Such a stream never ends and it is not possible to compute a final sum that can be returned. Instead, it is possible to compute rolling sums, i.e., return for each input event an updated sum record. This would yield a new stream of partial sums.
However, a stream of partial sums might not be what we are looking for, because it constantly updates the count and even more important, some information such as variation over time is lost. Hence, we might want to rephrase our question and ask for the number of cars that pass the location every minute. This requires us to group the elements of the stream into finite sets, each set corresponding to sixty seconds. This operation is called a tumbling windows operation.
Tumbling windows discretize a stream into non-overlapping windows. For certain applications it is important that windows are not disjunct because an application might require smoothed aggregates. For example, we can compute every thirty seconds the number of cars passed in the last minute. Such windows are called sliding windows.
This is because each element of a stream must be processed by the same window operator that decides which windows the element should be added to. For many applications, a data stream needs to be grouped into multiple logical streams on each of which a window operator can be applied. Think for example about a stream of vehicle counts from multiple traffic sensors (instead of only one sensor as in our previous example), where each sensor monitors a different location. By grouping the stream by sensor id, we can compute windowed traffic statistics for each location in parallel.
The following figure shows tumbling windows that collect two elements over a stream of (sensorId, count) pair elements.
Generally speaking, a window defines a finite set of elements on an unbounded stream. This set can be based on time (as in our previous examples), element counts, a combination of counts and time, or some custom logic to assign elements to windows.
Streaming Joins
If we want to handle data from multiple tables, we use the JOIN operator in SQL. Similarly, if you want to handle data from multiple streams, there are two options. First is to join the two and create one stream while the second is to write patterns across multiple streams.
We have entered an era where competitive advantage comes from analyzing, understanding, and responding to an organization’s data. When doing this, time is of the essence, and speed will decide the winners and losers.
Stream processing is required when data has to be processed fast and / or continuously, i.e. reactions have to be computed and initiated in real time. This requirement is coming more and more into every vertical. Many different frameworks and products are available on the market already.
Many use cases need fast, real-time decisions. Although it is possible to implement them using databases or batch processing, these technologies quickly introduce complexities because there is a fundamental impedance mismatch between the use cases and the tools employed. In contrast, streaming provides a much more natural model to think about, capture, and implement those real-time streaming use cases. Streaming SQL provides a simple yet powerful language to program streaming use cases.
The reality is that the value of most data degrades with time. It’s interesting to know that yesterday there was a traffic jam, or 10 fraud incidents, or 10 people who had heart attacks. From that knowledge, we can learn how to mitigate or prevent those incidents in the future. However, it is much better if we can gain those insights at the time they are occurring so that we can intervene and manage the situation.
The most popular Stream processing framework is Kafka. You can check my previous post here
Stream processing myths debunked – Six Common Streaming Misconceptions
Myth 1: There’s no streaming without batch (the Lambda Architecture)
Myth 2: Latency and Throughput: Choose One
Myth 3: Micro-batching means better throughput
Myth 4: Exactly once? Completely impossible.
Myth 5: Streaming only applies to “real-time”
Myth 6: So what? Streaming is too hard anyway.
Yesterday I went to Predict Conference, and I need to say thanks to Oracle for providing this experience for me.
Takeaways
My first time at Predict Conference, my first impression: it is a nice conference to get what are the trending topics in the area of Data in Ireland.
With nearly 1000 attendees, more than 30 speakers, over 2 stages.
Women are more and more present in tech.
But what is Predict Conference?
The Predict Conference is unique in its approach to combining business, data science and technology under one roof.
Predict is a holistic and continuous experience. It combines great talks, inspiration, face-to-face serendipity, experience zone, and hands-on data modeling innovation to equip you for decision making in the data age.
Predict is Europe’s leading data conference. It is designed to bring together thought leaders and innovators in the fields of data science, predictive analytics, artificial intelligence and technology.
Tickets include:
– Full access to the main event on the 1st of October 2019: all talks, access to the Predict experience area, all-day coffee, lunch and evening drinks reception.
Creme Global started Predict in 2015; it is now in its 5th year and has gone from strength to strength.
What exciting things can people look forward to?
The major themes at Predict this year where Health & Life Sciences, AI and Machine Learning along with Technology and Society (Sustainability/ Government/ Cities/ Privacy);
It’s not a very deep dive tech conference
The Predict Conference is just about talks around data topics, but it isn’t the most techie one for sure. The idea of this conference was to bring the tech people and industries together — and this objective is being realized year after year with great success.
From video analysis to APIs and IOT
The range of topics was very broad: but for me, everything was around 3 main topics; Video analizes, text analysis and IOT.
Talks
The talks were divided into some main categories;
Health;
Applied;
Sustainability & Cities;
Customer first;
Experiments;
I create my list of talks that I had the opportunity to see there. This is not an exhaustive list!
Talk:
From Cups to Consciousness: The Roadmap to AGI – Ben Duffy
He talked about AGI, Artificial general intelligence that is the intelligence of a machine that has the capacity to understand or learn any intellectual task that a human being can. It is a primary goal of some artificial intelligence research and a common topic in science fiction and future studies, and also open the floor for debate about what the development of AI and robotics will look like over the next few years.
Talk:
Onboard Artificial Intelligence: train, deploy and use Deep Learning on an edge device (Raspberry Pi) – Constant Bridon
He showed a live demo of a model designed to recognised car drawings via the camera of a Raspberry Pi. He also talked about an IOT car competition in the Us where the car run using the feeds of the camera.
There’s a nice one in UK called FormulaPI. https://www.formulapi.com/
Talk:
The history of Creative AI – Eric Risser
He talked about artificial intelligence and computer graphics to lead both the vision and the core technology.
There’s a nice colleague video about showing how do do it a demo using Oracle Cloud and GPU
Talks:
Top 3 things I’ve learned in 3 decades of Data Science – Dr. John F Elder
He talked about Ensembles, we can’t do it alone, we need alternative perspectives, Target Shuffling, some of our success is luck, simulation can find out how much and Leaks from the future, we can’t blindly trust results, if it looks good to good to be true, it is.
Talk:
Uncertainty and Interpretability – Kevin Kuo
Talk:
Low-Latency Model Prediction with Video
He talked about TensorFlow and buy the way there’s a 2.0 version now.
Talk:
AI Live: No Experience Required
He showed one Wolfram notebook.
But the main topic, in my opinion, was ML & DL buzzwords
A lot of presentation was trying to explain that Machine Learning and Deep Learning are not the same things, and that there’s a lot of other buzzwords around.
Partners
A stands area with some nice content. Experience Zone.
Wolfram – they are showing case the computing program Wolfram Mathematica.
Equinix – specializes in internet connection and data centers
Rstudio – they are showing case the IDE and language. I had a nice talk with the guy there about R is not dying and is Python that becoming more strong.
