Awesome Open Source
Awesome Open Source

kvstore

Kvstore for time series data, with indexing and data compression.

Features

  • Store data with a uint64 key, for example use date(20160101) as key
  • Support Get and Append
  • Support data compression, both gzip and snappy

Architectures

CompressCodec FileCapacity FileDataNums
Index(Key, Offset, Size) Index ... ...
Data Block Data Block ... ...

Each datafile contains three parts:

  • File header
    • File header contains CompressCodec, FileCapacity and FileDataNums.
  • Indexes
    • Indexes are sorted by key. Each index contains Key, Offset and Size for the corresponding data block.
  • Data blocks
    • Data blocks can be compressed, support gzip and snappy.

Usages

var (
	key1 uint64 = 20150110
	key2 uint64 = 20150120
	key3 uint64 = 20150130

	buf1 []byte = []byte("hello world 1.")
	buf2 []byte = []byte("hello world 2.")
	buf3 []byte = []byte("hello world 3.")
)

func main() {
	c := &Config{
		FileName:      "./data/gzip.file",
		FileCapacity:  3,
		CompressCodec: 1}

	kv, _ := New(c)

	kv.Append(key1, buf1)
	kv.Append(key2, buf2)
	kv.Append(key3, buf3)

	buf, _ := kv.Get(key1)

	kv.Close()
}

Discussion

Discussion at Go Forum.


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Go (194,369
Indexing (291
Binary Search (196
Related Projects