Minerva is a storage plugin of Drill that connects IPFS's decentralized storage and Drill's flexible query engine. Any data file stored on IPFS can be easily accessed from Drill's query interface, just like a file stored on a local disk. Moreover, with Drill's capability of distributed execution, other instances who are also running Minerva can help accelerate the execution: the data stays where it was, and the queries go to the most suitable nodes which stores the data locally and from there the operations can be performed most efficiently.
The jars are in the storage-ipfs/target directory.
Install
The executables and configurations are in distribution/target/apache-drill-1.16.0. Copy the entire directory to somewhere outside the source tree, and name it drill-run e.g., for testing later.
Copy the drill-ipfs-storage-{version}.jar generated jar file to drill-run/jars.
Copy java-api-ipfs-v1.2.2.jar which is IPFS's Java API, along with its dependencies provided as jar files:
Optionally, copy the configuration override file storage-plugin-override.conf to drill-run/conf, if you want Drill to auto configure and enable IPFS storage plugin at every (re)start.
Configuration
Set Drill hostname to the IP address of the node to run Drill:
Edit file conf/drill-env.sh and change the environment variable DRILL_HOST_NAME to the IP address of the node. Use private or global addresses, depending on whether you plan to run it on a cluster or the open Internet.
Configure the IPFS storage plugin:
If you are not using the configuration override file, you will have to manually configure and enable the plugin.
Run Drill according to Section Run and go to the webui of Drill (can be found at http://localhost:8047). Under the Storage tab, create a new storage plugin named ipfs and click the Create button.
Copy and paste the default configuration of the IPFS storage plugin located at storage-ipfs/src/resources/bootstrap-storage-plugins.json:
host and port are the host and API port where your IPFS daemon will be listening. Change it so that it matches the configuration of your IPFS instance.
max-nodes-per-leaf controls how many provider nodes will be considered when the query is being planned. A larger value increases the parallelization width but typically takes longer to find enough providers from DHT resolution. A smaller value does the opposite.
ipfs-timeouts set the maximum amount of time in seconds for various time consuming operations: find-provider is the time allowed to do DHT queries to find providers, find-peer-info is the time allowed to resolve the network addresses of the providers and fetch-data is the time the actual transmission is allowed to take.
groupscan-worker-threads limits the number of worker threads when the planner communicate with IPFS daemon to resolve providers and peer info.
formats specifies the formats of the files. It is unimplemented for now and does nothing.
Click the Update button after finishing editing. You should see the IPFS storage plugin is registered with Drill and you can enable it with the Enable button.
This flag indicates that an IPFS node is also capable of handling Drill quries and the planner will consider it when scheduling a query to execute distributedly. A node without this flag will be ignored.
Run
Embedded mode
Start IPFS daemon:
ipfs daemon &>/dev/null &
start drill-embedded:
drill-run/bin/drill-embedded
You can now execute queries via the command line as well as the web interface.
As a background service
You can run drill-embedded as a background process without controlling a terminal. This is done with the help of tmux, which is available in many distributions of Linux.
Edit the systemd service file drill-embedded.service, so that the environment variable DRILL_HOME pointes to where Drill is installed:
请发表评论