Contributor Notes

Project General

This project follows Semantic Versioning. As it is currently in 1.x, you can be confident that console commands that work now will keep working in other 1.x versions. As the internal API is not currently intended for out-of-app use, there may be breaking changes in minor versions. However, the internal API should remain stable through patch versions, and future releases that expose this API should fully stabilise it.

Internal Design Policy

CFS_Manager follows an internal interaction model where commands from the CLI call upon the top-level manager, which calls upon file-system-object abstractions of each storage provider. These file system objects then interface with the provider’s API via a wrapper written to increase the interface similarity, from the internal perspective, between the different providers. Generally, the provider’s official Python SDK sits between the in-house wrapper and the provider’s API.

It is designed to be easily extended with new service providers, and ensures this by providing a very specific interface for the implementation of new file system classes. All file systems inherit from the carefully described Cloud File System class, and their methods are called interchangably by the top-level manager.

It anticipates lower variance as information cascades upward. Each wrapper is extremely ideosyncratic because it talks to an API with a unique design philosophy. However, they use similar functions to pass similar types of data to their file system object. The file system object manipulate this data with different internal logic, but passing the same results up to the manager. The manager accepts data in several formats, but collapses them all into a standardised format. And when the CLI displays data to the user or accepts their commands, that user should see no indication that the provider APIs are all wildly different.

Comment Policy

The general approach taken is to err on the side of too much commentary. The goal is to make it easy to know exactly what’s happening upon skimming a plain-English description. Hopefully, one should not have to read the entire codebase to know what they need to change to improve their workflow or make a pull request. If you benefit from the current level of descriptiveness, please pass it on by including comments in your commits!

Direct contributions to the documentation or the UI text are considered as valuable as code contribution. If there was anything you had to figure out on your own that you wish had been in the standard docs, please write it up! The world will thank you (or, at least, Alison will).

Adding Support For Storage Providers

While all current support is built on top of externally sourced wrappers, this is not a requirement for contribution. If you’d like to build integration for the bare REST API into this project, please go ahead! In fact, the use of a more minimal wrapper that only supports the operations CFS_Manager requires would make the whole project lighter-weight.

When adding an externally-sourced SDK to the dependency tree, check the license! It has to be compatible with imports from an Apache 2.0 project. Preferred licenses for linked SDKs include the Apache, MIT, BSD, and ISC. (The LGPL is also compatible, but full GPL is not.)

If you’d like to suggest another storage provider to integrate, but don’t want to write the code, go ahead. However, if Alison is the one writing the code, she’ll prioritise providers based on how simple the integration seems. Providers with up-to-date SDKs and readable documentation go to the top of the stack. If you’re a fan/user/developer of a provider and really want support for it, and you want to speed it up, the best way is to write a wrapper yourself. (The second best way is to improve their documentation and be on call for CFS_Manager developers to question you.)

Storage Provider Integration Process

If you add a new provider, there are a few places where updates will need to happen to fully integrate them:

  1. You’ll need to write the in-project wrapper. The current style of the project involves writing an internal wrapper to talk to external ones. Any new wrappers written should losely comply with the conventions of the other ones in terms of function names and what results they achieve.
  2. You’ll need to write a new class in This class must inherit from CloudFileSystem and include the same methods. What those methods do is stated in their doc strings in the CloudFileSystem class definition. The consistency is necessary, as it’s part of the internal API. Those methods will be called, by name, by, which is a high-level abstraction over all the cloud accounts and must treat them interchangeably.
  3. You’ll need to import this class to, and add it to the list ‘fs_classes’. This will be how the manager instantiates these filesystems.
  4. Create a function to convert the file metadata returned by the specific API to the general metadata format used by the manager.
  5. Depending on what the access credentials needed by the file system are, you should add a setup step to the Ideally, users should save as much reusable info as possible (passwords excepted). Configuration imports fs_classes from, so your addition will automatically show up in the list of file systems for users to approve. However, you’ll still need a step to write its name to the system_configuration.txt file.

Once you’ve completed those five steps, the newly-added filesystem should work just as well as all the others.