mandacode
M-Drive: POSIX Filesystem on S3
The design philosophy and technical decisions of M-Drive, combining the scalability of object storage with the convenience of POSIX.
Problem Awareness
Table-based file management systems offer excellent stability, but expressing folder structures is complex, permission management is difficult, and flexible operation is challenging.
M-Drive reinterprets Linux's Inode structure to compose a filesystem and implements a POSIX filesystem interface, making it friendly for Linux users and allowing flexible operation for general users.
Core Design 1: Index Node System
The major question during system design was “How to provide a flexible and efficient filesystem?”
We determined that as tables become more complex and relations increase, query efficiency can decrease, management can become complicated, and many changes in the system's core logic could break the structure.
Then while looking at Linux's inode structure, we gained inspiration. The Inode is a data structure containing crucial information about files in Linux and Unix-based filesystems. Even folders or symbolic links are represented as files, maintaining consistency in data storage.
By adopting this into the service, we could store data consistently while retrieving it in one go without join operations. This led to performance improvements and allowed for creating an immutable core logic to enhance stability.
Field Name | Type | Description | Constraints / Attributes |
|---|---|---|---|
id | uuid.UUID | Basic Identifier (UUID) | Unique, Immutable |
type | enum | Node Type | Default: file |
size | int64 | Size (external storage data size for objects) | Default: 0 |
nlink | uint32 | Number of Hard Links | Default: 1 |
content | []byte | Inline Content | Up to 4KiB |
atime | time.Time | Node Access Time | |
mtime | time.Time | Node Modification Time | |
ctime | time.Time | Node Change Time | |
crtime | time.Time | Node Creation Time | |
flags | uint32 | Flag Bitmask | Default: 0 |
revision | string | Generation Identifier (ULID) | Up to 26 characters |
Rather than completely following the node system, we designed the table to suit the M-Drive system as shown above.
We simplified the nodes by removing parts considered excessive for the current system or roles of external systems such as permission and block data storage.
i_block[EXT4_N_BLOCKS=15] Also, considering that data is stored in external storage, []byte form was used to content field to serialize and store json data defined by type.
Core Design 2: POSIX-Based File Management
One of the main features of M-Drive is its flexible and precise drive manipulation feature. It provides various interfaces for users to manipulate files flexibly across various environments, from the web to CLI tools to be added later, unlike typical storage services. Additionally, we believed combining this with the Inode system would create a user-friendly structure that operates with the operating system's filesystem.
Thus, we adopted the POSIX file interface into the M-Drive system. It allows file manipulation using commands and features familiar to users like rm, ln, cp, thereby making it suitable for use in drive systems.