Hi, I'm attempting to run "spark-ec2 launch" on AWS. My AWS instances
would be in our EC2 VPC (which seems to be causing a problem).
The two security groups MyClusterName-master and MyClusterName-slaves have
already been setup with the same ports open as the security group that
spark-ec2 tries to create. (My company has security rules where I don't
have permissions to create security groups, so they have to be created by
someone else ahead of time.)
I'm getting the error VPC security groups may not be used for a non-VPC
launch" when I try to run "spark-ec2 launch".
Is there something I need to do to make spark-ec2 launch the master and
slave instances within the VPC?
Here's the command-line and the error that I get...
command-line (I've changed the clustername to something generic):
$SPARK_HOME/ec2/spark-ec2 --key-pair=MyKeyPair
'--identity-file=~/.ssh/id_mysshkey' --slaves=2 --instance-type=m3.large
--region=us-east-1 --zone=us-east-1a --ami=myami --spark-version
=0.9.1 launch MyClusterName
error:
ERROR:boto:400 Bad Request
ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>VPC
security groups may not be used for a non-VPC
launch</Message></Error></Errors><RequestID>8374cac5-5869-4f38-a141-2fdaf3b18326</Requ
estID></Response>
Setting up security groups...
Searching for existing cluster MyClusterName...
Launching instances...
Traceback (most recent call last):
File "./spark_ec2.py", line 806, in <module>
main()
File "./spark_ec2.py", line 799, in main
real_main()
File "./spark_ec2.py", line 682, in real_main
conn, opts, cluster_name)
File "./spark_ec2.py", line 344, in launch_cluster
block_device_map = block_map)
File
"/opt/spark-0.9.1-bin-hadoop1/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/ec2/image.py",
line 255, in run
File
"/opt/spark-0.9.1-bin-hadoop1/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/ec2/connection.py",
line 678, in run_instances
File
"/opt/spark-0.9.1-bin-hadoop1/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/connection.py",
line 925, in get_object
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>VPC
security groups may not be used for a non-VPC
launch</Message></Error></Errors><RequestID>8374cac5-5869-4f38-a141-2fdaf3b18326</RequestID></Response>
Thanks for your help!!
Matt
