File System Crawler: diskover

2018-05-31T18:56:35
ID N0WHERE:172763
Type n0where
Reporter N0where
Modified 2018-05-31T18:56:35

Description

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases.

As the amount of file data generated by business’ continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

Using diskover, users can identify old and unused files and give better insights into data change, file duplication and wasted space.

Requirements

  • Linux or OS X/macOS (tested on OS X 10.11.6, Ubuntu 16.04/18.04)
  • Python 2.7. or Python 3.5./3.6. (tested on Python 2.7.14, 3.5.3, 3.6.4) Python 3 recommended.
  • Python elasticsearch client module
  • Python requests module
  • Python scandir module
  • Python progressbar2 module
  • Python redis module
  • Python rq module
  • Elasticsearch 5 (local or AWS ES Service, tested on Elasticsearch 5.4.2, 5.6.4, 5.6.9) Elasticsearch 6 is not supported yet.
  • Redis (tested on 4.0.8)

Install the above Python modules using pip.

Optional Installs

  • diskover-web (diskover’s web file manager and analytics app)
  • Redis RQ Dashboard (for monitoring redis queue)
  • sharesniffer (for scanning your network for file shares and auto-mounting for crawls)
  • Kibana (for visualizing Elasticsearch data, tested on Kibana 5.4.2, 5.6.4)
  • X-Pack (Kibana plugin for graphs, reports, monitoring and http auth)
  • Gource (for Gource visualizations of diskover Elasticsearch data, see videos above)
  • Python qumulo-api module (for using Qumulo storage api, –qumulo cli option, install with pip, Python 2.7. only)

File System Crawler: diskover wiki

File System Crawler: diskover Download