An efficient and flexible metadata management layer for local file systems

Yubo Liu, Hongbo Li, Yutong Lu, Zhiguang Chen, Ming Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The efficiency of metadata processing affects the file system performance significantly. There are two bottlenecks in metadata management in existing local file systems: 1) Path lookup is costly because it causes a lot of disk I/Os, which makes metadata operations inefficient. 2) Existing file systems have deep I/O stack in metadata management, resulting in additional processing overhead. To solve these two bottlenecks, we decoupled data and metadata management and proposed a metadata management layer for local file systems. First, we separated the metadata based on their locations in the namespace tree and aggregated the metadata into fixed-size metadata buckets (MDBs). This design fully utilizes the metadata locality and improves the efficiency of disk I/O in the path lookup. Second, we customized an efficient MDB storage system on the raw storage device. This design simplifies the file system I/O stack in the metadata management and allows metadata lookup to be completed with constant time complexity. Finally, this metadata management layer gives users the flexibility to choose metadata storage devices. We implemented a prototype called Otter. Our evaluation demonstrated that Otter outperforms native EXT4, XFS, Btrfs, BetrFS and TableFS in many metadata operations. For instance, Otter has 1.2 times to 9.6 times performance improvement over other tested file systems in file opening.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE International Conference on Computer Design, ICCD 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages208-216
Number of pages9
ISBN (Electronic)9781538666487
DOIs
StatePublished - Nov 2019
Externally publishedYes
Event37th IEEE International Conference on Computer Design, ICCD 2019 - Abu Dhabi, United Arab Emirates
Duration: Nov 17 2019Nov 20 2019

Publication series

NameProceedings - 2019 IEEE International Conference on Computer Design, ICCD 2019

Conference

Conference37th IEEE International Conference on Computer Design, ICCD 2019
CountryUnited Arab Emirates
CityAbu Dhabi
Period11/17/1911/20/19

Keywords

  • File system
  • Metadata
  • Namespace
  • Path lookup

ASJC Scopus subject areas

  • Information Systems and Management
  • Computer Networks and Communications
  • Control and Optimization
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'An efficient and flexible metadata management layer for local file systems'. Together they form a unique fingerprint.

Cite this