Ecclesiastical Latin IPA: /i.o/, [i.o], [i.io] Verb: I can, know, understand, have knowledge.
Scio 0.3.0 and future versions depend on Apache Beam (
org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (
com.google.cloud.dataflow). See this page for a list of breaking changes.
* provided by Google Cloud Dataflow
Download and install the Java Development Kit (JDK) version 8.
Use our giter8 template to quickly create a new Scio job repository:
sbt new spotify/scio.g8
Switch to the new repo (default
scio-job) and build it:
cd scio-job sbt stage
Run the included word count example:
List result files and inspect content:
ls -l wc cat wc/part-00000-of-00004.txt
Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.
Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.
Scio includes the following artifacts:
scio-core: core library
scio-test: test utilities, add to your project as a "test" dependency
scio-avro: add-on for Avro, can also be used standalone
scio-google-cloud-platform: add-on for Google Cloud IO's: BigQuery, Bigtable, Pub/Sub, Datastore, Spanner
scio-cassandra*: add-ons for Cassandra
scio-elasticsearch*: add-ons for Elasticsearch
scio-extra: extra utilities for working with collections, Breeze, etc., best effort support
scio-jdbc: add-on for JDBC IO
scio-parquet: add-on for Parquet
scio-tensorflow: add-on for TensorFlow TFRecords IO and prediction
scio-redis: add-on for Redis
scio-smb: add-on for Sort Merge Bucket operations
scio-repl: extension of the Scala REPL with Scio specific operations
Copyright 2021 Spotify AB.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0