mmiklavc commented on issue #1537: METRON-2280 PCAP queries no longer work URL: https://github.com/apache/metron/pull/1537#issuecomment-544985811 ### Test Plan Pulled from https://github.com/apache/metron/pull/1523#issuecomment-540208150 ### PCAP Get PCAP data into Metron: 1. Install and setup pycapa (this has been updated in master recently) - https://github.com/apache/metron/blob/master/metron-sensors/pycapa/README.md#centos-6 2. (if using singlenode vagrant) Kill the enrichment, profiler, indexing, and sensor topologies via `for i in bro enrichment random_access_indexing batch_indexing yaf snort;do storm kill $i;done` 3. Start the pcap topology via $METRON_HOME/bin/start_pcap_topology.sh 4. Start the pycapa packet capture producer on eth1 ``` cd /opt/pycapa/pycapa-venv/bin pycapa --producer --kafka-topic pcap --interface eth1 --kafka-broker $BROKERLIST ``` 5. Watch the topology in the Storm UI and kill the packet capture utility started earlier when the number of packets ingested is over 3k. 6. You can leave your virtualenv session now via `deactivate` 7. Ensure that at at least 3 files exist on HDFS by running `hdfs dfs -ls /apps/metron/pcap/input` 8. Choose a file (denoted by $FILE) and dump a few of the contents using the pcap_inspector utility ``` FILE=<file path in hdfs> $METRON_HOME/bin/pcap_inspector.sh -i $FILE -n 5 ``` 9. Choose one of the lines in your output and note the protocol. e.g. ``` TS: October 9, 2019 8:43:39 PM UTC,ip_src_addr: 192.168.66.1,ip_src_port: 60911,ip_dst_addr: 192.168.66.121,ip_dst_port: 8080,protocol: 6 TS: October 9, 2019 8:43:39 PM UTC,ip_src_addr: 192.168.66.121,ip_src_port: 8080,ip_dst_addr: 192.168.66.1,ip_dst_port: 60911,protocol: 6 TS: October 9, 2019 8:43:39 PM UTC,ip_src_addr: 192.168.66.121,ip_src_port: 8080,ip_dst_addr: 192.168.66.1,ip_dst_port: 60911,protocol: 6 TS: October 9, 2019 8:43:39 PM UTC,ip_src_addr: 192.168.66.121,ip_src_port: 8080,ip_dst_addr: 192.168.66.1,ip_dst_port: 60911,protocol: 6 TS: October 9, 2019 8:43:39 PM UTC,ip_src_addr: 192.168.66.1,ip_src_port: 60911,ip_dst_addr: 192.168.66.121,ip_dst_port: 8080,protocol: 6 ``` **Note** when you run the fixed and query filter commands below, the resulting file will be placed in the execution directory where you kicked off the job from. #### Fixed filter 1. Run a fixed filter query by executing the following command with the values noted above (match your start_time format to the date format provided - default is to use millis since epoch) 2. `cd ~/; $METRON_HOME/bin/pcap_query.sh fixed -st <start_time> -df "yyyyMMdd" -p <protocol_num> -rpf 500` 3. Verify the MR job finishes successfully. Upon completion, you should see multiple files named with relatively current datestamps in your current directory, e.g. pcap-data-20160617160549737+0000.pcap 4. (Optional) Copy the files to your local machine and verify you can them it in Wireshark. I chose a middle file and the last file. The middle file should have 500 records (per the records_per_file option), and the last one will likely have a number of records <= 500. #### PCAP UI 1. Open the Metron Alerts UI http://node1:4201/login 2. Click on PCAP in the left menu bar 3. Use the same from/to date/time range as you used for the query filter test case and click the search button. A mapreduce job should kick off in the background. You can wait for the Metron UI to show complete (it will fail at the end due to tshark not being installed - this isn't important for this test), or watch the progress in the Resource Manager UI - http://node1:8088/cluster. 4. When the job completes, verify it shows as "SUCCEEDED" in the job history http://node1:19888/jobhistory.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
