Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Yq | 9,419 | 70 | 2 days ago | 126 | July 12, 2023 | 101 | mit | Go | ||
yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor | ||||||||||
Structured Text Tools | 6,759 | 2 days ago | 3 | |||||||
A list of command line tools for manipulating structured text data | ||||||||||
Countries | 5,795 | 399 | 100 | 7 days ago | 21 | April 04, 2020 | 18 | odbl-1.0 | PHP | |
World countries in JSON, CSV, XML and Yaml. Any help is welcome! | ||||||||||
Countries States Cities Database | 5,483 | 5 days ago | 68 | odbl-1.0 | PHP | |||||
🌍 Discover our global repository of countries, states, and cities! 🏙️ Get comprehensive data in JSON, SQL, XML, YAML, and CSV formats. Access ISO2, ISO3 codes, country code, capital, native language, timezones (for countries), and more. #countries #states #cities | ||||||||||
Dasel | 4,544 | 12 | 7 days ago | 92 | June 01, 2023 | 31 | mit | Go | ||
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package. | ||||||||||
Django Tastypie | 3,866 | 2,508 | 25 | a month ago | 26 | January 24, 2023 | 409 | other | Python | |
Creating delicious APIs for Django apps since 2010. | ||||||||||
Luban | 2,466 | 5 days ago | mit | C# | ||||||
luban是一个强大、易用、优雅、稳定的游戏配置解决方案。luban is a powerful, easy-to-use, elegant and stable game configuration solution. | ||||||||||
Yq | 2,281 | 18 | 8 | 20 days ago | 41 | April 22, 2023 | 25 | apache-2.0 | Python | |
Command-line YAML, XML, TOML processor - jq wrapper for YAML/XML/TOML documents | ||||||||||
Jmsserializerbundle | 1,774 | 5,708 | 743 | a month ago | 54 | June 13, 2023 | 95 | mit | PHP | |
Easily serialize, and deserialize data of any complexity (supports XML, JSON, YAML) | ||||||||||
Python Benedict | 1,158 | 30 | a day ago | 61 | July 16, 2023 | 14 | mit | Python | ||
:blue_book: dict subclass with keylist/keypath support, built-in I/O operations (base64, csv, ini, json, pickle, plist, query-string, toml, xls, xml, yaml), s3 support and many utilities. |
python-benedict is a dict subclass with keylist/keypath/keyattr support, I/O shortcuts (base64
, cli
, csv
, ini
, json
, pickle
, plist
, query-string
, toml
, xls
, xml
, yaml
) and many utilities... for humans, obviously.
NEW
Keyattr support for get/set items using keys as attributes.[n]
suffix.base64
, cli
, csv
, ini
, json
, pickle
, plist
, query-string
, toml
, xls
, xml
, yaml
.file-system
(read/write), url
(read-only), s3
(read/write).If you want to install everything:
pip install "python-benedict[all]"
alternatively you can install the main package:
pip install python-benedict
, then install only the optional requirements you need.Here the hierarchy of possible installation targets available when running pip install "python-benedict[...]"
(each target installs all its sub-targets):
[all]
[io]
[toml]
[xls]
[xml]
[yaml]
[parse]
[s3]
benedict
is a dict
subclass, so it is possible to use it as a normal dictionary (you can just cast an existing dict).
from benedict import benedict
# create a new empty instance
d = benedict()
# or cast an existing dict
d = benedict(existing_dict)
# or create from data source (filepath, url or data-string) in a supported format:
# Base64, CSV, JSON, TOML, XML, YAML, query-string
d = benedict("https://localhost:8000/data.json", format="json")
# or in a Django view
params = benedict(request.GET.items())
page = params.get_int("page", 1)
It is possible to get/set items using keys as attributes (dotted notation).
d = benedict(keyattr_dynamic=True) # default False
d.profile.firstname = "Fabio"
d.profile.lastname = "Caccamo"
print(d) # -> { "profile":{ "firstname":"Fabio", "lastname":"Caccamo" } }
By default, if the keyattr_dynamic
is not explicitly set to True
, this functionality works for get/set only already existing items.
You can disable the keyattr functionality passing keyattr_enabled=False
option in the constructor.
d = benedict(existing_dict, keyattr_enabled=False) # default True
or using the getter/setter
property.
d.keyattr_enabled = False
You can enable the dynamic attributes access functionality passing keyattr_dynamic=True
in the constructor.
d = benedict(existing_dict, keyattr_dynamic=True) # default False
or using the getter/setter
property.
d.keyattr_dynamic = True
Warning - even if this feature is very useful, it has some obvious limitations: it works only for string keys that are unprotected (not starting with an
_
) and that don't clash with the currently supported methods names.
Wherever a key is used, it is possible to use also a list (or a tuple) of keys.
d = benedict()
# set values by keys list
d["profile", "firstname"] = "Fabio"
d["profile", "lastname"] = "Caccamo"
print(d) # -> { "profile":{ "firstname":"Fabio", "lastname":"Caccamo" } }
print(d["profile"]) # -> { "firstname":"Fabio", "lastname":"Caccamo" }
# check if keypath exists in dict
print(["profile", "lastname"] in d) # -> True
# delete value by keys list
del d["profile", "lastname"]
print(d["profile"]) # -> { "firstname":"Fabio" }
.
is the default keypath separator.
If you cast an existing dict and its keys contain the keypath separator a ValueError
will be raised.
In this case you should use a custom keypath separator or disable keypath functionality.
d = benedict()
# set values by keypath
d["profile.firstname"] = "Fabio"
d["profile.lastname"] = "Caccamo"
print(d) # -> { "profile":{ "firstname":"Fabio", "lastname":"Caccamo" } }
print(d["profile"]) # -> { "firstname":"Fabio", "lastname":"Caccamo" }
# check if keypath exists in dict
print("profile.lastname" in d) # -> True
# delete value by keypath
del d["profile.lastname"]
You can customize the keypath separator passing the keypath_separator
argument in the constructor.
If you pass an existing dict to the constructor and its keys contain the keypath separator an Exception
will be raised.
d = benedict(existing_dict, keypath_separator="/")
You can change the keypath_separator
at any time using the getter/setter
property.
If any existing key contains the new keypath_separator
an Exception
will be raised.
d.keypath_separator = "/"
You can disable the keypath functionality passing keypath_separator=None
option in the constructor.
d = benedict(existing_dict, keypath_separator=None)
or using the getter/setter
property.
d.keypath_separator = None
List index are supported, keypaths can include indexes (also negative) using [n]
, to perform any operation very fast:
# Eg. get last location cordinates of the first result:
loc = d["results[0].locations[-1].coordinates"]
lat = loc.get_decimal("latitude")
lng = loc.get_decimal("longitude")
For simplifying I/O operations, benedict
supports a variety of input/output methods with most common formats: base64
, cli
, csv
, ini
, json
, pickle
, plist
, query-string
, toml
, xls
, xml
, yaml
.
It is possible to create a benedict
instance directly from data-source (filepath
, url
, s3
or data-string
) by passing the data source and the data format (optional, default "json") in the constructor.
# filepath
d = benedict("/root/data.yml", format="yaml")
# url
d = benedict("https://localhost:8000/data.xml", format="xml")
# s3
d = benedict("s3://my-bucket/data.xml", s3_options={"aws_access_key_id": "...", "aws_secret_access_key": "..."})
# data-string
d = benedict('{"a": 1, "b": 2, "c": 3, "x": 7, "y": 8, "z": 9}')
from_*
followed by the format name.to_*
followed by the format name.filepath="..."
kwarg is specified, the output will be also saved at the specified filepath.Here are the details of the supported formats, operations and extra options docs.
format | input | output | extra options docs |
---|---|---|---|
base64 |
✅ | ✅ | - |
cli |
✅ | ❌ | argparse |
csv |
✅ | ✅ | csv |
ini |
✅ | ✅ | configparser |
json |
✅ | ✅ | json |
pickle |
✅ | ✅ | pickle |
plist |
✅ | ✅ | plistlib |
query-string |
✅ | ✅ | - |
toml |
✅ | ✅ | toml |
xls |
✅ | ❌ | openpyxl - xlrd |
xml |
✅ | ✅ | xmltodict |
yaml |
✅ | ✅ | PyYAML |
Utility methods
I/O methods
Parse methods
These methods are common utilities that will speed up your everyday work.
Utilities that accept key argument(s) also support keypath(s).
Utilities that return a dictionary always return a new benedict
instance.
clean
# Clean the current dict instance removing all empty values: None, "", {}, [], ().
# If strings or collections (dict, list, set, tuple) flags are False,
# related empty values will not be deleted.
d.clean(strings=True, collections=True)
clone
# Return a clone (deepcopy) of the dict.
c = d.clone()
dump
# Return a readable representation of any dict/list.
# This method can be used both as static method or instance method.
s = benedict.dump(d.keypaths())
print(s)
# or
d = benedict()
print(d.dump())
filter
# Return a filtered dict using the given predicate function.
# Predicate function receives key, value arguments and should return a bool value.
predicate = lambda k, v: v is not None
f = d.filter(predicate)
find
# Return the first match searching for the given keys/keypaths.
# If no result found, default value is returned.
keys = ["a.b.c", "m.n.o", "x.y.z"]
f = d.find(keys, default=0)
flatten
# Return a new flattened dict using the given separator to join nested dict keys to flatten keypaths.
f = d.flatten(separator="_")
groupby
# Group a list of dicts at key by the value of the given by_key and return a new dict.
g = d.groupby("cities", by_key="country_code")
invert
# Return an inverted dict where values become keys and keys become values.
# Since multiple keys could have the same value, each value will be a list of keys.
# If flat is True each value will be a single value (use this only if values are unique).
i = d.invert(flat=False)
items_sorted_by_keys
# Return items (key/value list) sorted by keys.
# If reverse is True, the list will be reversed.
items = d.items_sorted_by_keys(reverse=False)
items_sorted_by_values
# Return items (key/value list) sorted by values.
# If reverse is True, the list will be reversed.
items = d.items_sorted_by_values(reverse=False)
keypaths
# Return a list of all keypaths in the dict.
# If indexes is True, the output will include list values indexes.
k = d.keypaths(indexes=False)
match
# Return a list of all values whose keypath matches the given pattern (a regex or string).
# If pattern is string, wildcard can be used (eg. [*] can be used to match all list indexes).
# If indexes is True, the pattern will be matched also against list values.
m = d.match(pattern, indexes=True)
merge
# Merge one or more dictionary objects into current instance (deepupdate).
# Sub-dictionaries keys will be merged together.
# If overwrite is False, existing values will not be overwritten.
# If concat is True, list values will be concatenated together.
d.merge(a, b, c, overwrite=True, concat=False)
move
# Move an item from key_src to key_dst.
# It can be used to rename a key.
# If key_dst exists, its value will be overwritten.
d.move("a", "b", overwrite=True)
nest
# Nest a list of dicts at the given key and return a new nested list
# using the specified keys to establish the correct items hierarchy.
d.nest("values", id_key="id", parent_id_key="parent_id", children_key="children")
remove
# Remove multiple keys from the dict.
# It is possible to pass a single key or more keys (as list or *args).
d.remove(["firstname", "lastname", "email"])
rename
# Rename a dict item key from "key" to "key_new".
# If key_new exists, a KeyError will be raised.
d.rename("first_name", "firstname")
search
# Search and return a list of items (dict, key, value, ) matching the given query.
r = d.search("hello", in_keys=True, in_values=True, exact=False, case_sensitive=False)
standardize
# Standardize all dict keys, e.g. "Location Latitude" -> "location_latitude".
d.standardize()
subset
# Return a dict subset for the given keys.
# It is possible to pass a single key or more keys (as list or *args).
s = d.subset(["firstname", "lastname", "email"])
swap
# Swap items values at the given keys.
d.swap("firstname", "lastname")
traverse
# Traverse a dict passing each item (dict, key, value) to the given callback function.
def f(d, key, value):
print(f"dict: {d} - key: {key} - value: {value}")
d.traverse(f)
unflatten
# Return a new unflattened dict using the given separator to split dict keys to nested keypaths.
u = d.unflatten(separator="_")
unique
# Remove duplicated values from the dict.
d.unique()
These methods are available for input/output operations.
from_base64
# Try to load/decode a base64 encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to choose the subformat used under the hood:
# ('csv', 'json', 'query-string', 'toml', 'xml', 'yaml'), default: 'json'.
# It's possible to choose the encoding, default 'utf-8'.
# A ValueError is raised in case of failure.
d = benedict.from_base64(s, subformat="json", encoding="utf-8", **kwargs)
from_cli
# Load and decode data from a string of CLI arguments.
# ArgumentParser specific options can be passed using kwargs:
# https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser
# Return a new dict instance. A ValueError is raised in case of failure.
d = benedict.from_cli(s, **kwargs)
from_csv
# Try to load/decode a csv encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to specify the columns list, default: None (in this case the first row values will be used as keys).
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/csv.html
# A ValueError is raised in case of failure.
d = benedict.from_csv(s, columns=None, columns_row=True, **kwargs)
from_ini
# Try to load/decode a ini encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/configparser.html
# A ValueError is raised in case of failure.
d = benedict.from_ini(s, **kwargs)
from_json
# Try to load/decode a json encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/json.html
# A ValueError is raised in case of failure.
d = benedict.from_json(s, **kwargs)
from_pickle
# Try to load/decode a pickle encoded in Base64 format and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/pickle.html
# A ValueError is raised in case of failure.
d = benedict.from_pickle(s, **kwargs)
from_plist
# Try to load/decode a p-list encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://docs.python.org/3/library/plistlib.html
# A ValueError is raised in case of failure.
d = benedict.from_plist(s, **kwargs)
from_query_string
# Try to load/decode a query-string and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# A ValueError is raised in case of failure.
d = benedict.from_query_string(s, **kwargs)
from_toml
# Try to load/decode a toml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://pypi.org/project/toml/
# A ValueError is raised in case of failure.
d = benedict.from_toml(s, **kwargs)
from_xls
# Try to load/decode a xls file (".xls", ".xlsx", ".xlsm") from url, filepath or data-string.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# - https://openpyxl.readthedocs.io/ (for .xlsx and .xlsm files)
# - https://pypi.org/project/xlrd/ (for .xls files)
# A ValueError is raised in case of failure.
d = benedict.from_xls(s, sheet=0, columns=None, columns_row=True, **kwargs)
from_xml
# Try to load/decode a xml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://github.com/martinblech/xmltodict
# A ValueError is raised in case of failure.
d = benedict.from_xml(s, **kwargs)
from_yaml
# Try to load/decode a yaml encoded data and return it as benedict instance.
# Accept as first argument: url, filepath or data-string.
# It's possible to pass decoder specific options using kwargs:
# https://pyyaml.org/wiki/PyYAMLDocumentation
# A ValueError is raised in case of failure.
d = benedict.from_yaml(s, **kwargs)
to_base64
# Return the dict instance encoded in base64 format and optionally save it at the specified 'filepath'.
# It's possible to choose the subformat used under the hood:
# ('csv', json', 'query-string', 'toml', 'xml', 'yaml'), default: 'json'.
# It's possible to choose the encoding, default 'utf-8'.
# It's possible to pass decoder specific options using kwargs.
# A ValueError is raised in case of failure.
s = d.to_base64(subformat="json", encoding="utf-8", **kwargs)
to_csv
# Return a list of dicts in the current dict encoded in csv format and optionally save it at the specified filepath.
# It's possible to specify the key of the item (list of dicts) to encode, default: 'values'.
# It's possible to specify the columns list, default: None (in this case the keys of the first item will be used).
# A ValueError is raised in case of failure.
s = d.to_csv(key="values", columns=None, columns_row=True, **kwargs)
to_ini
# Return the dict instance encoded in ini format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/configparser.html
# A ValueError is raised in case of failure.
s = d.to_ini(**kwargs)
to_json
# Return the dict instance encoded in json format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/json.html
# A ValueError is raised in case of failure.
s = d.to_json(**kwargs)
to_pickle
# Return the dict instance as pickle encoded in Base64 format and optionally save it at the specified filepath.
# The pickle protocol used by default is 2.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/pickle.html
# A ValueError is raised in case of failure.
s = d.to_pickle(**kwargs)
to_plist
# Return the dict instance encoded in p-list format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://docs.python.org/3/library/plistlib.html
# A ValueError is raised in case of failure.
s = d.to_plist(**kwargs)
to_query_string
# Return the dict instance as query-string and optionally save it at the specified filepath.
# A ValueError is raised in case of failure.
s = d.to_query_string(**kwargs)
to_toml
# Return the dict instance encoded in toml format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://pypi.org/project/toml/
# A ValueError is raised in case of failure.
s = d.to_toml(**kwargs)
to_xml
# Return the dict instance encoded in xml format and optionally save it at the specified filepath.
# It's possible to pass encoder specific options using kwargs:
# https://github.com/martinblech/xmltodict
# A ValueError is raised in case of failure.
s = d.to_xml(**kwargs)
to_yaml
# Return the dict instance encoded in yaml format.
# If filepath option is passed the output will be saved ath
# It's possible to pass encoder specific options using kwargs:
# https://pyyaml.org/wiki/PyYAMLDocumentation
# A ValueError is raised in case of failure.
s = d.to_yaml(**kwargs)
These methods are wrappers of the get
method, they parse data trying to return it in the expected type.
get_bool
# Get value by key or keypath trying to return it as bool.
# Values like `1`, `true`, `yes`, `on`, `ok` will be returned as `True`.
d.get_bool(key, default=False)
get_bool_list
# Get value by key or keypath trying to return it as list of bool values.
# If separator is specified and value is a string it will be splitted.
d.get_bool_list(key, default=[], separator=",")
get_date
# Get value by key or keypath trying to return it as date.
# If format is not specified it will be autodetected.
# If choices and value is in choices return value otherwise default.
d.get_date(key, default=None, format=None, choices=[])
get_date_list
# Get value by key or keypath trying to return it as list of date values.
# If separator is specified and value is a string it will be splitted.
d.get_date_list(key, default=[], format=None, separator=",")
get_datetime
# Get value by key or keypath trying to return it as datetime.
# If format is not specified it will be autodetected.
# If choices and value is in choices return value otherwise default.
d.get_datetime(key, default=None, format=None, choices=[])
get_datetime_list
# Get value by key or keypath trying to return it as list of datetime values.
# If separator is specified and value is a string it will be splitted.
d.get_datetime_list(key, default=[], format=None, separator=",")
get_decimal
# Get value by key or keypath trying to return it as Decimal.
# If choices and value is in choices return value otherwise default.
d.get_decimal(key, default=Decimal("0.0"), choices=[])
get_decimal_list
# Get value by key or keypath trying to return it as list of Decimal values.
# If separator is specified and value is a string it will be splitted.
d.get_decimal_list(key, default=[], separator=",")
get_dict
# Get value by key or keypath trying to return it as dict.
# If value is a json string it will be automatically decoded.
d.get_dict(key, default={})
get_email
# Get email by key or keypath and return it.
# If value is blacklisted it will be automatically ignored.
# If check_blacklist is False, it will be not ignored even if blacklisted.
d.get_email(key, default="", choices=None, check_blacklist=True)
get_float
# Get value by key or keypath trying to return it as float.
# If choices and value is in choices return value otherwise default.
d.get_float(key, default=0.0, choices=[])
get_float_list
# Get value by key or keypath trying to return it as list of float values.
# If separator is specified and value is a string it will be splitted.
d.get_float_list(key, default=[], separator=",")
get_int
# Get value by key or keypath trying to return it as int.
# If choices and value is in choices return value otherwise default.
d.get_int(key, default=0, choices=[])
get_int_list
# Get value by key or keypath trying to return it as list of int values.
# If separator is specified and value is a string it will be splitted.
d.get_int_list(key, default=[], separator=",")
get_list
# Get value by key or keypath trying to return it as list.
# If separator is specified and value is a string it will be splitted.
d.get_list(key, default=[], separator=",")
get_list_item
# Get list by key or keypath and return value at the specified index.
# If separator is specified and list value is a string it will be splitted.
d.get_list_item(key, index=0, default=None, separator=",")
get_phonenumber
#Get phone number by key or keypath and return a dict with different formats (e164, international, national).
# If country code is specified (alpha 2 code), it will be used to parse phone number correctly.
d.get_phonenumber(key, country_code=None, default=None)
get_slug
# Get value by key or keypath trying to return it as slug.
# If choices and value is in choices return value otherwise default.
d.get_slug(key, default="", choices=[])
get_slug_list
# Get value by key or keypath trying to return it as list of slug values.
# If separator is specified and value is a string it will be splitted.
d.get_slug_list(key, default=[], separator=",")
get_str
# Get value by key or keypath trying to return it as string.
# Encoding issues will be automatically fixed.
# If choices and value is in choices return value otherwise default.
d.get_str(key, default="", choices=[])
get_str_list
# Get value by key or keypath trying to return it as list of str values.
# If separator is specified and value is a string it will be splitted.
d.get_str_list(key, default=[], separator=",")
get_uuid
# Get value by key or keypath trying to return it as valid uuid.
# If choices and value is in choices return value otherwise default.
d.get_uuid(key, default="", choices=[])
get_uuid_list
#Get value by key or keypath trying to return it as list of valid uuid values.
# If separator is specified and value is a string it will be splitted.
d.get_uuid_list(key, default=[], separator=",")
# clone repository
git clone https://github.com/fabiocaccamo/python-benedict.git && cd python-benedict
# create virtualenv and activate it
python -m venv venv && . venv/bin/activate
# upgrade pip
python -m pip install --upgrade pip
# install requirements
pip install -r requirements.txt -r requirements-test.txt
# install pre-commit to run formatters and linters
pre-commit install --install-hooks
# run tests using tox
tox
# or run tests using unittest
python -m unittest
Released under MIT License.
python-fontbro
- friendly font operations.
python-fsutil
- file-system utilities for lazy devs.