This is an automated email from the ASF dual-hosted git repository.
mwalch pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/fluo-muchos.git
The following commit(s) were added to refs/heads/master by this push:
new eb1346f Fixes #239 - Several updates (#242)
eb1346f is described below
commit eb1346f3a022c630aee49ca9241ba7072285ce87
Author: Mike Walch <[email protected]>
AuthorDate: Wed Dec 5 14:53:23 2018 -0500
Fixes #239 - Several updates (#242)
* boto3 should be installed by users using pip and
not installed by muchos script
* updated header to main.py for python2
* improved requirements documentation
* added quickstart
---
README.md | 78 +++++++++++++++++++++++++++++++++++++-----------------
bin/muchos | 39 +--------------------------
lib/muchos/main.py | 2 +-
3 files changed, 55 insertions(+), 64 deletions(-)
diff --git a/README.md b/README.md
index cd7544a..75b9f5a 100644
--- a/README.md
+++ b/README.md
@@ -21,22 +21,50 @@ Muchos is structured into two high level components:
Checkout [Uno] for setting up Accumulo or Fluo on a single machine.
-## Installation
+## Requirements
-First clone the Muchos repo:
+Muchos requires the following:
- git clone https://github.com/apache/fluo-muchos.git
+* Python 2
+* [awscli] & [boto3] libraries - Install using `pip install awscli boto3
--upgrade --user`
+* `ssh-agent` installed and running
+* An AWS account with your SSH public key uploaded. When you configure
[muchos.props], set `key.name`
+ to name of your key pair in AWS.
+* `~/.aws` [configured][aws-config] on your machine. Can be created manually
or using [aws configure][awscli-config].
-Now, create and modify your [muchos.props] configuration file for Muchos:
+## Quickstart
- cd muchos/
- cp conf/muchos.props.example conf/muchos.props
+The following commands will install Muchos, launch an EC2 cluster, and
setup/run Accumulo:
-In order to run the `muchos` command, you will need to create [AWS
configuration and credential files][aws-config]
-in your home directory. These files can be created by hand or by running `aws
configure` using the [AWS CLI][aws-cli].
+```bash
+git clone https://github.com/apache/fluo-muchos.git
+cd fluo-muchos/
+cp conf/muchos.props.example conf/muchos.props
+vim conf/muchos.props # Edit to configure
Muchos cluster
+./bin/muchos launch -c mycluster # Launches Muchos
cluster in EC2
+./bin/muchos setup # Set up cluster and
start Accumulo
+```
+
+The `setup` command can be run repeatedly to fix any failures and will not
repeat successful operations.
+
+After your cluster is launched, SSH to it using the following command:
+
+```bash
+./bin/muchos ssh
+```
+
+Run the following command to terminate your cluster. WARNING: All cluster data
will be lost.
-You will need to upload your public key to the AWS management console and set
`key.name` in
-[muchos.props] to the name of your key pair. If you want to give others
access to your cluster, add
+```bash
+./bin/muchos terminate
+```
+
+Please continue reading for more detailed Muchos instructions.
+
+## Launching an EC2 cluster
+
+Before launching a cluster, you will need to complete the requirements above,
clone the Muchos repo, and
+create [muchos.props]. If you want to give others access to your cluster, add
their public keys to a file named `keys` in your `conf/` directory. During
the setup of your
cluster, this file will be appended on each node to the
`~/.ssh/authorized_keys` file for the user
set by the `cluster.username` property.
@@ -49,26 +77,24 @@ error when you try to launch your cluster. By default, the
`aws_ami` property is
You will need to changes this value if a newer image has been released or if
you are running in different region
than `us-east-1`.
-## Launching an EC2 cluster
-
-After following the installation steps above, run the following command to
launch an EC2 cluster called `mycluster`:
+After following the steps above, run the following command to launch an EC2
cluster called `mycluster`:
- muchos launch -c mycluster
+ ./bin/muchos launch -c mycluster
After your cluster has launched, you do not have to specify a cluster anymore
using `-c` (unless you
have multiple clusters running).
Run the following command to confirm that you can ssh to the leader node:
- muchos ssh
+ ./bin/muchos ssh
You can check the status of the nodes using the EC2 Dashboard or by running
the following command:
- muchos status
+ ./bin/muchos status
## Set up the cluster
-The `muchos setup` command will set up your cluster and start Hadoop,
Zookeeper, & Accumulo. It
+The `./bin/muchos setup` command will set up your cluster and start Hadoop,
Zookeeper, & Accumulo. It
will download release tarballs of Fluo, Accumulo, Hadoop, etc. The versions of
these tarballs are
specified in [muchos.props] and can be changed if desired.
@@ -78,7 +104,7 @@ version of Fluo or Accumulo. Before running the `muchos
setup` command, you shou
version and SHA-256 hash of your tarball matches what is set in
[muchos.props]. Run the command
`shasum -a 256 /path/to/tarball` on your tarball to determine its hash.
-The `muchos setup` command will install and start Accumulo, Hadoop, and
Zookeeper. The optional
+The `muchos setup` command will install and start Accumulo, Hadoop, and
Zookeeper. The optional
services below will only be set up if configured in the `[nodes]` section of
[muchos.props]:
1. `fluo` - Fluo only needs to be installed and configured on a single node in
your cluster as Fluo
@@ -114,10 +140,10 @@ take over a minute, use `ctrl-c` to stop setup if it
hangs for a long time. Just
## Manage the cluster
The `setup` command is idempotent. It can be run again on a working cluster.
It will not change the
-cluster if everything is configured and running correctly. If a process has
stopped, the `setup`
+cluster if everything is configured and running correctly. If a process has
stopped, the `setup`
command will restart the process.
-The `muchos wipe` command can be used to wipe all data from the cluster and
kill any running
+The `./bin/muchos wipe` command can be used to wipe all data from the cluster
and kill any running
processes. After running the `wipe` command, run the `setup` command to start
a fresh cluster.
If you set `proxy_socks_port` in your [muchos.props], a SOCKS proxy will be
created on that port
@@ -142,7 +168,7 @@ with Muchos as it configures your shell with common
environment variables. To ru
application, SSH to a node on cluster where Fluo is installed and clone the
example repo:
```bash
-muchos ssh # SSH to cluster proxy node
+./bin/muchos ssh # SSH to cluster proxy node
ssh <node where Fluo is installed> # Nodes with Fluo installed is
determined by Muchos config
hub clone apache/fluo-examples # Clone repo of Fluo example
applications. Press enter for user/password.
```
@@ -165,7 +191,7 @@ mimic the scripts of example Fluo applications above.
## Customize your cluster
-After `muchos setup` is run, users can install additional software on the
cluster using their own
+After `./bin/muchos setup` is run, users can install additional software on
the cluster using their own
Ansible playbooks. In their own playbooks, users can reference any
configuration in the Ansible
inventory file at `/etc/ansible/hosts` which is set up by Muchos on the proxy
node. The inventory
file lists the hosts for services on the cluster such as the Zookeeper nodes,
Namenode, Accumulo
@@ -178,7 +204,7 @@ managed in their own git repository (see
[mikewalch/muchos-custom][mc] for an ex
If you launched your cluster on EC2, run the following command terminate your
cluster. WARNING - All
data on your cluster will be lost:
- muchos terminate
+ ./bin/muchos terminate
## Automatic shutdown of EC2 clusters
@@ -198,7 +224,7 @@ If you decide later to cancel the shutdown, run `muchos
cancel_shutdown`.
The `config` command allows you to retrieve cluster configuration for your own
scripts:
```bash
-$ muchos config -p leader.public.ip
+$ ./bin/muchos config -p leader.public.ip
10.10.10.10
```
@@ -222,12 +248,14 @@ The following command runs the unit tests:
[centos7]: https://aws.amazon.com/marketplace/pp/B00O7WM7QW
[aws-config]:
http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
-[aws-cli]:
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration
+[awscli]: https://docs.aws.amazon.com/cli/latest/userguide/installing.html
+[awscli-config]:
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration
[fluo-app]: https://github.com/apache/fluo/blob/master/docs/applications.md
[WebIndex]: https://github.com/apache/fluo-examples/tree/master/webindex
[Phrasecount]: https://github.com/apache/fluo-examples/tree/master/phrasecount
[Stresso]: https://github.com/apache/fluo-examples/tree/master/stresso
[boto]: http://boto.cloudhackers.com/en/latest/
+[boto3]: https://github.com/boto/boto3
[Ansible]: https://www.ansible.com/
[ti]: https://travis-ci.org/apache/fluo-muchos.svg?branch=master
[tl]: https://travis-ci.org/apache/fluo-muchos
diff --git a/bin/muchos b/bin/muchos
index 65f6c78..26cd0f4 100755
--- a/bin/muchos
+++ b/bin/muchos
@@ -16,48 +16,11 @@
bin="$(dirname $0)"
basedir="$( cd -P ${bin}/.. && pwd )"
-lib="$basedir/lib"
export MUCHOS_HOME=$basedir
if [ ! -f "$basedir/conf/muchos.props" ]; then
echo "Please create a configuration file at $basedir/conf/muchos.props"
exit 1
fi
-if [[ "$OSTYPE" == "darwin"* ]]; then
- export MD5="md5 -q"
-else
- export MD5="md5sum"
-fi
-
-function fetch_lib() {
- version=$1
- md5=$2
- url=$3
- tarball="$version.tar.gz"
- if [ ! -d "$lib/$version" ]; then
-
- wget -cq -P "$lib" "$url"
-
- actual_md5=$($MD5 "$lib/$tarball" | awk '{print $1}')
- if [[ "$actual_md5" != "$md5" ]]; then
- echo "The md5 checksum ($actual_md5) of $tarball does not match the
expected checksum ($md5)"
- exit 1
- fi
- tar xzf "$lib/$tarball" -C "$lib"
- rm "$lib/$tarball"
- fi
- export PYTHONPATH="$lib/$version:$PYTHONPATH"
-}
-
-boto3_version="boto3-1.4.7"
-boto3_md5="eb95e700fa9e06db6e9ff4ee159adbd9"
-boto3_url="https://pypi.python.org/packages/b6/7c/b9caa157b514e0a456286172862428a83814e4eda114ea1e80267de85378/boto3-1.4.7.tar.gz#md5=eb95e700fa9e06db6e9ff4ee159adbd9";
-
-bc_version="botocore-1.7.8"
-bc_md5="d7e3179b619b936048d763b2f579a896"
-bc_url="https://pypi.python.org/packages/d5/a1/499da39780f26bca1d2e652bd01f115addf57c6509d92f0d35334cbcacea/botocore-1.7.8.tar.gz#md5=d7e3179b619b936048d763b2f579a896";
-
-fetch_lib $boto3_version $boto3_md5 $boto3_url
-fetch_lib $bc_version $bc_md5 $bc_url
-python -Wdefault "$lib/muchos/main.py" "$@"
+python "$basedir/lib/muchos/main.py" "$@"
diff --git a/lib/muchos/main.py b/lib/muchos/main.py
index 930030c..ba2061b 100644
--- a/lib/muchos/main.py
+++ b/lib/muchos/main.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python2
+#!/usr/bin/env python2
# Copyright 2014 Muchos authors (see AUTHORS)
#
