Alchemist is an JavaScript ETL(Extract, Transform, Load) engine focused on managing data coming from multiple sources, in an efficient and streamlined way, by using pipelines.
If your data-related needs oscillate around any of the following:
then Alchemist will be the right tool for you!
Alchemist was designed with 2 goals in mind:
While the typical flow of data in an ETL engine is:
Alchemist offers more flexibility, by storing both valid (parseable) and invalid (malformed) data between steps in a pipeline. This distinction allows for deeper and more detailed processing of valid data, while keeping seemingly invalid data safely stored for further analysis and improvements.
The full data processing pipeline is demonstrated in the following diagram:
A more in depth look about how we use alchemist
Use npm
to add the dependency in your package.json
file:
$ npm install alchemist
Or install it globally with:
$ npm install --global alchemist
Each pipeline is an AWS Lambda function, e.g.:
async call() {
// Step 1: Define input from Kinesis:
let input = Input.instanceFor(this.adapterRegistry, 'KinesisInput', { events: this.event.events() })
// Step 2: Define transformations sequence:
let transformations = [
Transformation.instanceFor(this.adapterRegistry, 'FirstEventTransformation'),
Transformation.instanceFor(this.adapterRegistry, 'SecondEventTransformation'),
Transformation.instanceFor(this.adapterRegistry, 'ThirdEventTransformation')
]
// Step 3: Define output to Kinesis:
let output = Output.instanceFor(this.adapterRegistry, 'KinesisOutput', {stream_name: 'output-kinesis-stream'})
// Step 4: Use console for invalid output
// Note: ConsoleOutput is a preregistered Adapter!
let invalidOutput = Output.instanceFor(this.adapterRegistry, 'ConsoleOutput', { })
// Step 5: Execute the pipeline:
let pipeline = new Pipeline({
input: input,
transformations: transformations,
output: output,
invalidOutput: invalidOutput
})
await pipeline.call()
}
Review a complete example to learn more about the complete pipeline structure.
After checking out the repo, run yarn install
to install dependencies. Then, run docker-compose run --rm test
to run the tests.
To install this library onto your local machine, run npm install
. To release a new version, run npm version <update_type>
to update the version number in package.json
, and then run npm publish
, which will push the library to npmjs.com.
The npm package is available as open source under the terms of the MIT License.
Everyone interacting in the Alchemist project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.