Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Smart_open | 2,803 | 320 | 156 | 5 days ago | 54 | April 24, 2022 | 77 | mit | Python | |
Utils for streaming large files (S3, HDFS, gzip, bz2...) | ||||||||||
Storagetapper | 269 | a year ago | 4 | November 19, 2021 | 21 | mit | Go | |||
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service | ||||||||||
Afs | 226 | 14 | 2 days ago | 71 | October 27, 2021 | 2 | apache-2.0 | Go | ||
Abstract File Storage | ||||||||||
Tus Ruby Server | 188 | 2 years ago | 5 | mit | Ruby | |||||
Ruby server for tus resumable upload protocol | ||||||||||
S3 Streaming Upload | 119 | 32 | 10 | 2 years ago | 26 | January 21, 2021 | 2 | mit | JavaScript | |
s3-streaming-upload is node.js library that listens to your stream and upload its data to Amazon S3 using ManagedUpload API. | ||||||||||
Node.js Amazon S3 | 96 | 12 years ago | 7 | mit | JavaScript | |||||
An S3 library for node.js. Supports automatic md5 of content, streaming, 100-continues, and very large files | ||||||||||
Streamx | 95 | 4 years ago | 26 | apache-2.0 | Java | |||||
kafka-connect-s3 : Ingest data from Kafka to Object Stores(s3) | ||||||||||
S3 Sync | 79 | 30 | 14 | 7 years ago | 14 | October 29, 2015 | 14 | other | JavaScript | |
A streaming interface for uploading multiple files to S3. | ||||||||||
Megfile | 78 | 2 | a day ago | 23 | April 26, 2022 | 14 | apache-2.0 | Python | ||
Megvii FILE Library - Working with Files in Python | ||||||||||
Media Services Application Mapper | 76 | a month ago | n,ull | apache-2.0 | JavaScript | |||||
Media Services Application Mapper is a browser-based tool that allows operators to visualize the structure and logical connections among AWS Media Services and supporting services in the cloud. The tool can be used as a top-down resource monitoring tool when integrated with CloudWatch. |
This project has been merged into Alpakka. The version found here is not actively maintained.
Akka Streaming Client for S3 and Supporting Libraries
A library built around Akka-Http to stream an upload to S3. The initialization request will be sent, incoming stream will be chunked and uploaded, then the completion request is sent.
Add to your build.sbt
:
resolvers += Resolver.jcenterRepo
libraryDependencies += "com.bluelabs" %% "s3-stream" % "0.0.3"
Then in your application:
implicit val system = ActorSystem()
implicit val mat: Materializer = ActorMaterializer()
implicit val ec = mat.executionContext
val creds = AWSCredentials("KEYGOESHERE", "SECRETGOESHERE")
val stream: S3Stream = new S3Stream(creds)
val input: Source[ByteString, Future[IOResult]] = StreamConverters.fromInputStream(...whatever) // Or something else to generate a stream of ByteStrings
val sink: Sink[ByteString, Future[CompleteMultipartUploadResult]] = stream.multipartUpload(S3Location("bucketGoesHere", "keygoeshere"))
val result: Future[CompleteMultipartUploadResult] = input.runWith(sink)
Sign Akka-HTTP requests using AWS credentials, following the AWS Signature v4 process.
Add to your build.sbt
:
resolvers += Resolver.jcenterRepo
libraryDependencies += "com.bluelabs" %% "akka-http-aws" % "0.0.3"
Then in your application:
val credentials = AWSCredentials("KEYGOESHERE", "SECRETGOESHERE")
val signingKey = SigningKey(credentials, CredentialScope(LocalDate.now(), region, "s3"))
val req = HttpRequest(... with assorted params ...)
val signedRequest = Signer.signedRequest(req, signingKey)