Configuration Management

We provide a mechanism for storing and retrieving configurations of different clients, servers and services securely, any type of a resource or a client that needs configuration, it can simply implemented using base classes.

Introduction

To define any type of resource, you can create a class, with different configuration fields, and a factory, which will manage this class instances.

Consider having remote machines, where every machine have an ip address and a region:

from enum import Enum

from jumpscale.core.base import Base, fields, StoredFactory
from jumpscale.clients.sshclient import sshclient

class Region(Enum):
    US = "us"
    EU = "eu"

class Machine(Base):
    ip = fields.IPAddress()
    region = fields.Enum(Region)

machines = StoredFactory(Machine)

This simply define a resource called Machine, which have two fields.

machines here is factory, and it's the entry point where we can create and manage instances of this machines:

m1 = machines.get("m1")
m1.ip = "127.0.0.1"
m1.region = "us"
m1.save()

save will store this configuration to current configured storage, and at any time, you can list current machines, delete any of them...etc:

print(machines.list_all())
machines.delete("m1")

configurations

"""Config module is the single entry of configurations across the framework.

It allows - resolving configurations paths for configuration directoryconfig_root, or configuration file path config_path - rebuildling default configurations or retrieving them (using get_default_config) - Getting configurations using get_default_config - Updating configuration using update_config

JS-NG> j.core.config

Getting where is the config.toml path

JS-NG> j.core.config.config_path
'/home/ahmed/.config/jumpscale/config.toml'

Getting configuration directory

JS-NG> j.core.config.config_root
'/home/ahmed/.config/jumpscale'

Getting default configurations of js-ng (used in case of no configuration found)

JS-NG> j.core.config.get_default_config()
{'debug': True, 'logging': {'redis': {'enabled': True, 'level': 15, 'max_size': 1000, 'dump': True, 'dump_dir': '/home/ahmed/.config/jumpscale/logs/redis'}, 'filesystem': {'enabled': True, 'level': 15, 'log_dir': '/home/ahmed/.config/jumpscale/logs/fs/log.txt', 'rotation': '5 MB'}}, 'alerts': {'enabled': True, 'level': 40}, 'ssh_key_path': '', 'private_key_path': '', 'stores': {'redis': {'hostname': 'localhost', 'port': 6379}, 'filesystem': {'path': '/home/ahmed/.config/jumpscale/secureconfig'}}, 'store': 'filesystem', 'threebot': {'default': ''}, 'explorer': {'default_url': 'https://explorer.testnet.grid.tf/explorer'}}

Getting the current configurations

JS-NG> j.core.config.get_config()
{'debug': True, 'ssh_key_path': '', 'log_to_redis': False, 'log_to_files': True, 'log_level': 15, 'private_key_path': '/home/ahmed/.config/jumpscale/mykey.priv', 'secure_config_path': '/home/ahmed/.config/jumpscale/secureconfig', 'store': 'filesystem', 'logging': {'handlers': [{'sink': 'sys.stdout', 'format': '{time} - {message}', 'colorize': True, 'enqueue': True}, {'sink': '/home/ahmed/.config/jumpscale/logs/file_jumpscale.log', 'serialize': True, 'enqueue': True}], 'redis': {'enabled': True, 'level': 15, 'max_size': 1000, 'dump': True, 'dump_dir': '/home/ahmed/.config/jumpscale/logs/redis'}, 'filesystem': {'enabled': True, 'level': 15, 'log_dir': '/home/ahmed/.config/jumpscale/logs/fs/log.txt', 'rotation': '5 MB'}}, 'stores': {'redis': {'hostname': 'localhost', 'port': 6379}, 'filesystem': {'path': '/home/ahmed/.config/jumpscale/secureconfig'}}, 'alerts': {'enabled': True, 'level': 40}, 'threebot': {'default': ''}, 'explorer': {'default_url': 'https://explorer.testnet.grid.tf/explorer'}}

Updating configurations

JS-NG> conf = j.core.config.get_default_config()
JS-NG> conf
{'debug': True, 'ssh_key_path': '', 'log_to_redis': False, 'log_to_files': True, 'log_level': 15, 'private_key_path': '/home/ahmed/.config/jumpscale/mykey.priv', 'secure_config_path': '/home/ahmed/.config/jumpscale/secureconfig', 'store': 'filesystem', 'logging': {'handlers': [{'sink': 'sys.stdout', 'format': '{time} - {message}', 'colorize': True, 'enqueue': True}, {'sink': '/home/ahmed/.config/jumpscale/logs/file_jumpscale.log', 'serialize': True, 'enqueue': True}], 'redis': {'enabled': True, 'level': 15, 'max_size': 1000, 'dump': True, 'dump_dir': '/home/ahmed/.config/jumpscale/logs/redis'}, 'filesystem': {'enabled': True, 'level': 15, 'log_dir': '/home/ahmed/.config/jumpscale/logs/fs/log.txt', 'rotation': '5 MB'}}, 'stores': {'redis': {'hostname': 'localhost', 'port': 6379}, 'filesystem': {'path': '/home/ahmed/.config/jumpscale/secureconfig'}}, 'alerts': {'enabled': True, 'level': 40}, 'threebot': {'default': ''}, 'explorer': {'default_url': 'https://explorer.testnet.grid.tf/explorer'}}

JS-NG> conf['favcolor'] = 'blue'
JS-NG> j.core.config.update_config(conf)
JS-NG> j.core.config.get_config()
{'debug': True, 'ssh_key_path': '', 'log_to_redis': False, 'log_to_files': True, 'log_level': 15, 'private_key_path': '/home/ahmed/.config/jumpscale/mykey.priv', 'secure_config_path': '/home/ahmed/.config/jumpscale/secureconfig', 'store': 'filesystem', 'favcolor': 'blue', 'logging': {'handlers': [{'sink': 'sys.stdout', 'format': '{time} - {message}', 'colorize': True, 'enqueue': True}, {'sink': '/home/ahmed/.config/jumpscale/logs/file_jumpscale.log', 'serialize': True, 'enqueue': True}], 'redis': {'enabled': True, 'level': 15, 'max_size': 1000, 'dump': True, 'dump_dir': '/home/ahmed/.config/jumpscale/logs/redis'}, 'filesystem': {'enabled': True, 'level': 15, 'log_dir': '/home/ahmed/.config/jumpscale/logs/fs/log.txt', 'rotation': '5 MB'}}, 'stores': {'redis': {'hostname': 'localhost', 'port': 6379}, 'filesystem': {'path': '/home/ahmed/.config/jumpscale/secureconfig'}}, 'alerts': {'enabled': True, 'level': 40}, 'threebot': {'default': ''}, 'explorer': {'default_url': 'https://explorer.testnet.grid.tf/explorer'}}

Get/Set

you can use j.core.config.get and j.core.config.set to retrive values of keys or set the value of a key respectively.

e.g

JS-NG> j.core.config.get("favcolor")
'blue'

JS-NG> j.core.config.set("favcolor", "grey")
JS-NG> j.core.config.get("favcolor")
'grey'

Storage

Any stored factory has a store, which is selected based configuration.

For now, we have two types of storage:

  • File system: by default, stores secure config under ~/.config/jumpscale/secureconfig
  • Redis: stores configuration at redis.

Implementation can be found at jumpscale.core.base.store, the main idea is that, every factory should store configuration on a unique location, this location is automatically generated, and instances are stored in a tree-like hierarchy.

For example:

➜  js-ng git:(development) ✗ tree ~/.config/jumpscale/secureconfig
/home/abom/.config/jumpscale/secureconfig
├── jumpscale
│   ├── clients
│   │   ├── docker
│   │   │   └── docker
│   │   │       └── DockerClient
│   │   │           └── test
│   │   │               └── data
│   │   ├── gdrive
│   │   │   └── gdrive
│   │   │       └── GdriveClient
│   │   │           └── test
│   │   │               └── data
│   │   ├── github
│   │   │   └── github
│   │   │       ├── Github
│   │   │       │   ├── g1
│   │   │       │   │   ├── data
│   │   │       │   │   └── users
│   │   │       │   │       └── jumpscale
│   │   │       │   │           └── clients
│   │   │       │   │               └── github
│   │   │       │   │                   └── github
│   │   │       │   │                       └── User
│   │   │       │   │                           └── u1
│   │   │       │   │                               └── data

Serialization:

Only json serialization is now supported, if you need to get a serializable dict for any base class instance, just use to_dict() method.

Encryption

We use public key encryption, implementation is at j.data.nacl. For keys, we use the key that's generated and saved at private_key_path. There is a key that's auto-generated for you, to generate a new key, use hush_keygen and jsctl to set the path:

hush_keygen --name test
jsctl config update private_key_path "`pwd`/test.priv"

Only secret fields are encrypted by default, so, if you have a password field for example, it's better to use fields.Secret for it:

class Region(enum):
    US = "us"
    EU = "eu"

class Machine(Base):
    ip = fields.IPAddress()
    region = fields.Enum(Region)
    username = fields.Secret()
    password = fields.Secret()

Now when the store writes any instance configuration of Machine, it will encrypt it. If you load this instance again, it will be decrypted.

More on base classes

You can check what base classes can provide, what are supported field types and more at Base classes.