Awesome Open Source
Awesome Open Source

CS-744-Course-Project

Documents

Paper: https://www.overleaf.com/read/qfvgwpjvvcbh

Poster: https://github.com/ShawnZhong/CS-744-Course-Project/blob/master/Poster.pdf

SSH commands

Parallel scp
pscp -h slaves -O StrictHostKeyChecking=no src /users/szhong
Parallel ssh
pssh -i -h slaves -O StrictHostKeyChecking=no cmd

Kubernetes

Reset Kubernetes
kubectl delete daemonsets,replicasets,services,deployments,pods,rc --all --grace-period=0 --force
Kill a topology
heron kill kubernetes WindowedWordCountTopology
Start proxy
kubectl proxy -p 8001

Helm

Init
helm init

kubectl create serviceaccount tiller \
--namespace kube-system \

kubectl create clusterrolebinding tiller-cluster-rule \
--clusterrole cluster-admin \
--serviceaccount kube-system:tiller

kubectl patch deploy tiller-deploy \
--namespace kube-system \
--patch '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

Flink

https://awesomeopensource.com/project/docker-flink/examples

helm package helm/flink/
helm install --name flink --set flink.num_taskmanagers=5 flink*.tgz

Spark

helm install --name spark stable/spark

Kafka

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm install --name my-kafka incubator/kafka

Heron

Links to docs

https://apache.github.io/incubator-heron/docs/operators/deployment/schedulers/kubernetes/

https://github.com/apache/incubator-heron/tree/master/deploy/kubernetes/general

Helm
helm repo add heron-charts https://storage.googleapis.com/heron-charts

helm install heron-charts/heron \
--set platform=gke \
--name kubernetes \
--values https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/gke/small.yaml
Link to Heron tracker

http://localhost:8001/api/v1/namespaces/default/services/kubernetes-tracker:8888/proxy/topologies

Link to Heron UI

http://localhost:8001/api/v1/namespaces/default/services/kubernetes-ui:8889/proxy

Link to Heron API Server

http://localhost:8001/api/v1/namespaces/default/services/kubernetes-apiserver:9000/proxy/

http://localhost:8001/api/v1/namespaces/default/services/kubernetes-apiserver:9000/proxy/api/v1/version

Set Service URL
heron config kubernetes \
set service_url http://localhost:8001/api/v1/namespaces/default/services/kubernetes-apiserver:9000/proxy
Submit Topology
heron submit kubernetes \
~/.heron/examples/heron-streamlet-examples.jar \
com.twitter.heron.examples.streamlet.WindowedWordCountTopology \
wwc
heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
com.twitter.heron.examples.api.AckingTopology acking
Related Awesome Lists
Top Programming Languages
Top Projects

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Kubernetes (24,540
Stream (21,145
Tex (19,400
Ssh (14,447
Parallel (7,126
Hadoop (5,354
Helm (4,692
Benchmarking (2,719
Topology (2,664
Apache Spark (1,190
Flink (1,100
Mesos (1,085
Apache Kafka (691
Stream Processing (577
Performance Analysis (523
Kafka Streams (413
Spark Streaming (363
Apache Storm (133
Cluster Manager (23
Heron (7