Changeset: 2a0a5ab57c22 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2a0a5ab57c22
Added Files:
clients/iotclient/src/Utilities/customthreading.py
Modified Files:
clients/iotclient/documentation/index.rst
clients/iotclient/documentation/introduction.rst
clients/iotclient/documentation/iot_server_arguments.rst
clients/iotclient/documentation/restful_resources.rst
clients/iotclient/documentation/streams_data_types.rst
clients/iotclient/requirements.txt
clients/iotclient/src/Flask/restresources.py
clients/iotclient/src/Settings/filesystem.py
clients/iotclient/src/Settings/iotlogger.py
clients/iotclient/src/Settings/mapiconnection.py
clients/iotclient/src/Streams/datatypes.py
clients/iotclient/src/Streams/flushing.py
clients/iotclient/src/Streams/streams.py
clients/iotclient/src/main.py
Branch: iot
Log Message:
Small corrections on code and documentation
diffs (truncated from 583 to 300 lines):
diff --git a/clients/iotclient/documentation/index.rst
b/clients/iotclient/documentation/index.rst
--- a/clients/iotclient/documentation/index.rst
+++ b/clients/iotclient/documentation/index.rst
@@ -18,4 +18,3 @@ Contents:
restful_resources.rst
streams_data_types.rst
conclusion.rst
-
diff --git a/clients/iotclient/documentation/introduction.rst
b/clients/iotclient/documentation/introduction.rst
--- a/clients/iotclient/documentation/introduction.rst
+++ b/clients/iotclient/documentation/introduction.rst
@@ -1,14 +1,13 @@
.. _instroduction:
-
************
Introduction
************
-This appplication is a Python web server destinated to extend MonetDB RDBMS to
the Internet-of-Things while using a streaming engine. This project is derived
from Pedro Ferreira master thesis, a junior researcher at CWI. The objective of
the master thesis to re-evaluate the DataCell extension of MonetDB to the IOT
world, which was postponed for several years since its release. To accomplish
this, an IOT topology around MonetDB is being built, where this server aims to
analyse the input of streams. Later it will also be included another web server
to analyze the output of streams.
+This application is a Python web server destined to extend MonetDB RDBMS to
the Internet-of-Things while using a streaming engine. This project is derived
from Pedro Ferreira master thesis, a junior researcher at CWI. The objective of
the master thesis to re-evaluate the DataCell extension of MonetDB to the IOT
world, which was postponed for several years since its release. To accomplish
this, an IOT topology around MonetDB is being built, where this server aims to
analyse the input of streams. Later it will also be included another web server
to analyze the output of streams.
This web server is built using Python programming language, version 2.7, using
`Flask-RESTful <https://pypi.python.org/pypi/Flask-RESTful>`_ framework. The
required packages on requirements.txt should be installed before running the
server.
-The web server is capable of creating and deleting streams for the renewed
DataCell extension. After a stream is created, it's possible to make batch
inserts on it. The requests are made using RESTfull requests with JSON content.
Both stream creation and stream insertion are validated using `JSON Schema
Draft 4 <http://json-schema.org/documentation.html>`_. On a batch insert if a
tuple is invalid, then no tuples are inserted. The stream engine adds an
implicit timestamp column to recognize when the tuple was inserted. The
inserted tuples are later evaluated using MonetDB's relational engine with
continuous queries.
+The web server is capable of creating and deleting streams for the renewed
DataCell extension. After a stream is created, it's possible to make batch
inserts on it. The requests are made using RESTful requests with JSON content.
Both stream creation and stream insertion are validated using `JSON Schema
Draft 4 <http://json-schema.org/documentation.html>`_. On a batch insert if a
tuple is invalid, then no tuples are inserted. The stream engine adds an
implicit timestamp column to recognize when the tuple was inserted. The
inserted tuples are later evaluated using MonetDB's relational engine with
continuous queries.
The following chapters explain the server parameters, RESTful resources and
how to manage the streams.
diff --git a/clients/iotclient/documentation/iot_server_arguments.rst
b/clients/iotclient/documentation/iot_server_arguments.rst
--- a/clients/iotclient/documentation/iot_server_arguments.rst
+++ b/clients/iotclient/documentation/iot_server_arguments.rst
@@ -1,11 +1,12 @@
.. _starting_webserver:
-
***********************
Starting the web server
***********************
-The server is started as a regular Python program invoking the :code:`main.py`
file, however ther are several arguments that can be passed to change its
behaviour.
+The packages listened on requirements.txt must be installed before running the
server. To avoid conflicts with other versions of the packages existing on the
host machine, is recommended to create a Python virtual environment for the
server. `About Python virtual environments
<http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_.
+
+The server starts as a regular Python program invoking the :code:`main.py`
file, however there are several arguments that can be passed to change its
behaviour.
::
@@ -27,29 +28,27 @@ Paths
Set the filesystem directory where the baskets will be created. By default in
UNIX systems is on :code:`/etc/iotcollector` directory, while on Windows is on
the directory where the :code:`main.py` script was invoked. While running the
web server creates a :code:`baskets` directory.
-**-l - -logfile=**
+**-l - -log=**
Location of logfile. On the logfile is reported when streams are created or
removed, when tuples are inserted and when the baskets are flushed. By default
in UNIX systems is :code:`/var/log/iot/iot.log`, while on Windows is the
:code:`iot.log` on the directory where the :code:`main.py` script was invoked.
-**-c - -configfile=**
+**-c - -config=**
-Path of JSON file where the information of existing streams on the server will
be stored. By default is the :code:`config.json` file on filesystem directory.
-
+Location of the JSON file where the information of existing streams on the
server will be stored. By default is the :code:`config.json` file on filesystem
directory.
Host Identifier
---------------
If the *useidentifier* parameter is provided, an extra column on streams will
be added with a custom name of the host for later identification.
-**-u - -useidentifier=**
+**-ui - -useidentifier=**
Boolean to indicate if the extra column will be added.
-**-n - -name=**
+**-in - -name=**
Custom name of the host. By default is the host's MAC address.
-
Web Server Listening
--------------------
@@ -71,7 +70,6 @@ Listening host of the administration (IO
Listening port of the administration (IOT) server. By default is on port
:code:`8001`.
-
Database Connection
-------------------
@@ -79,18 +77,18 @@ Credentials for the MAPI connection to M
.. note:: The user password will be prompted during the initialization of the
server.
-**-dh - -dhostname=**
+**-d - -host=**
Listening host of MonetDB database. By default is on :code:`127.0.0.1`.
-**-dp - -dport=**
+**-d - -port=**
Listening port of MonetDB database. By default is on port :code:`50000`.
-**-du - -duser=**
+**-u - -user=**
Name of the user to authenticate. By default is user :code:`monetdb`.
-**-dd - -ddatabase=**
+**-d - -database=**
Name of database to use. By default is :code:`iotdb` database.
diff --git a/clients/iotclient/documentation/restful_resources.rst
b/clients/iotclient/documentation/restful_resources.rst
--- a/clients/iotclient/documentation/restful_resources.rst
+++ b/clients/iotclient/documentation/restful_resources.rst
@@ -1,11 +1,10 @@
.. _restful_resources:
-
*****************
RESTful Resources
*****************
-Bellow is lested the available RESTful resource for both servers on IOT
application. In case of error, the server will return the error message with
the respective HTTP response code.
+Bellow is listed the available RESTful resources for both servers on the
server. In case of error, the server will return the error message with the
respective HTTP response code.
.. important:: All the keywords provided on JSON must be lower case, as well
the column data types.
@@ -109,7 +108,7 @@ Same resource as streams_ .
**POST**
-Insert a batch of tuples on the provided stream in the URL. The insert must be
an array of JSON object with pairs of column-value. All tuples are validaded
according to the defined data types on each column. If there is an invalid
tuple, none of the tuples is inserted. The implicit timestamp and the host
identifier are automatically added. Bellow is shown an example to insert
several tuples on the stream in streams_:
+Insert a batch of tuples on the provided stream in the URL. The insert must be
an array of JSON object with pairs of column-value. All tuples are validated
according to the defined data types on each column. If there is an invalid
tuple, none of the tuples is inserted. The implicit timestamp and the host
identifier are automatically added. Bellow is shown an example to insert
several tuples on the stream in streams_:
.. code-block:: json
diff --git a/clients/iotclient/documentation/streams_data_types.rst
b/clients/iotclient/documentation/streams_data_types.rst
--- a/clients/iotclient/documentation/streams_data_types.rst
+++ b/clients/iotclient/documentation/streams_data_types.rst
@@ -4,7 +4,7 @@
Available Data Types
********************
-The following list contains the available data types for columns creation on
the stream engine. The list was made as much compatible with `MonetDB's data
types <https://www.monetdb.org/Documentation/Manuals/SQLreference/Datatypes>`_.
For each column definiton, one of the types from the list must be provided
along with its name.
+The following list contains the available data types for columns creation on
the stream engine. The list was made as much compatible with `MonetDB's data
types <https://www.monetdb.org/Documentation/Manuals/SQLreference/Datatypes>`_.
For each column definition, one of the types from the list must be provided
along with its name.
By default a column is not nullable, but the pair :code:`"nullable": true` can
be provided to make it nullable. There is also the possibility to add a default
value if the column value is not present on a JSON insert tuple (ex:
:code:`"default": "text"`). The default value is validated against the column's
validation during the stream creation.
@@ -23,7 +23,7 @@ MonetDB's string types with unbounded le
Char, Character, Varchar, Character Varying
-------------------------------------------
-MonetDB's string types with a bounded length. The :code:`limit` parameter must
be provided as an interger. The insertion must be provided as a JSON string
within the limit.
+MonetDB's string types with a bounded length. The :code:`limit` parameter must
be provided as an integer. The insertion must be provided as a JSON string
within the limit.
UUID
----
@@ -55,7 +55,7 @@ An *IPv6* address. The value must be a J
Regex
-----
-A string allways validated with a provided regular expression. The JSON must
contain a :code:`regex` key with the regular expression. As MonetDB doesn't
have a Regex equivalent data type yet, the data is stored as :code:`string`.
+A string always validated with a provided regular expression. The JSON must
contain a :code:`regex` key with the regular expression. As MonetDB doesn't
have a Regex equivalent data type yet, the data is stored as :code:`string`.
Enum
----
diff --git a/clients/iotclient/requirements.txt
b/clients/iotclient/requirements.txt
--- a/clients/iotclient/requirements.txt
+++ b/clients/iotclient/requirements.txt
@@ -2,8 +2,8 @@ Flask-RESTful==0.3.5
jsonschema==2.5.1
rfc3987==1.3.5
strict-rfc3339==0.6
-python-dateutil==2.5.2
-pytz==2016.3
+python-dateutil==2.5.3
+pytz==2016.4
pymonetdb==0.1.1
tzlocal==1.2.2
Sphinx==1.4.1
diff --git a/clients/iotclient/src/Flask/restresources.py
b/clients/iotclient/src/Flask/restresources.py
--- a/clients/iotclient/src/Flask/restresources.py
+++ b/clients/iotclient/src/Flask/restresources.py
@@ -26,7 +26,7 @@ def init_rest_resources():
try:
Streams_Context = IOTStreams()
except BaseException as ex:
- print >> sys.stderr, ex
+ print >> sys.stdout, ex
add_log(50, ex.message)
sys.exit(1)
@@ -56,7 +56,7 @@ class StreamInput(Resource): # TODO the
except BaseException as ex:
add_log(50, ex.message)
return ex.message, 400
- return '', 201
+ return 'The insertions were made with success!', 201
class StreamsInfo(Resource):
@@ -81,7 +81,7 @@ class StreamsHandling(Resource):
add_log(50, ex.message)
return ex.message, 400
else:
- return '', 201
+ return 'The stream was created with success!', 201
def delete(self):
try:
@@ -96,4 +96,4 @@ class StreamsHandling(Resource):
except BaseException as ex:
add_log(50, ex.message)
return ex.message, 404
- return '', 204
+ return 'The stream was deleted with success!', 204
diff --git a/clients/iotclient/src/Settings/filesystem.py
b/clients/iotclient/src/Settings/filesystem.py
--- a/clients/iotclient/src/Settings/filesystem.py
+++ b/clients/iotclient/src/Settings/filesystem.py
@@ -38,7 +38,7 @@ def init_file_system(host_identifier=Non
Host_Identifier = host_identifier
except (Exception, OSError) as ex:
- print >> sys.stderr, ex
+ print >> sys.stdout, ex
add_log(50, ex.message)
sys.exit(1)
diff --git a/clients/iotclient/src/Settings/iotlogger.py
b/clients/iotclient/src/Settings/iotlogger.py
--- a/clients/iotclient/src/Settings/iotlogger.py
+++ b/clients/iotclient/src/Settings/iotlogger.py
@@ -7,7 +7,6 @@ if sys.platform in ("linux", "linux2", "
elif sys.platform == "win32":
logging_location = os.path.join(os.path.dirname(__file__), os.pardir,
'iot.log')
-logging_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
logger = logging.getLogger("IOTLog")
@@ -21,7 +20,7 @@ def init_logging():
try:
logger = logging.getLogger("IOTLog")
logger.setLevel(logging.DEBUG)
- formatter = logging.Formatter(logging_format)
+ formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s
- %(message)s')
logging_path = os.path.dirname(logging_location)
if not os.path.exists(logging_path):
@@ -30,7 +29,7 @@ def init_logging():
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)
except (Exception, OSError) as ex:
- print >> sys.stderr, ex
+ print >> sys.stdout, ex
sys.exit(1)
diff --git a/clients/iotclient/src/Settings/mapiconnection.py
b/clients/iotclient/src/Settings/mapiconnection.py
--- a/clients/iotclient/src/Settings/mapiconnection.py
+++ b/clients/iotclient/src/Settings/mapiconnection.py
@@ -10,7 +10,10 @@ Connection = None
def init_monetdb_connection(hostname, port, user_name, database):
global Connection
- user_password = getpass.getpass(prompt='User password:')
+ user_password = getpass.getpass(prompt='Insert password for user ' +
user_name + ':')
+
+ if user_password == '':
+ user_password = 'monetdb'
try: # the autocommit is set to true so each statement will be independent
Connection = pymonetdb.connect(hostname=hostname, port=port,
username=user_name, password=user_password,
@@ -19,7 +22,7 @@ def init_monetdb_connection(hostname, po
print >> sys.stdout, log_message
add_log(20, log_message)
except BaseException as ex:
- print >> sys.stderr, ex.message
+ print >> sys.stdout, ex.message
add_log(50, ex.message)
sys.exit(1)
diff --git a/clients/iotclient/src/Streams/datatypes.py
b/clients/iotclient/src/Streams/datatypes.py
--- a/clients/iotclient/src/Streams/datatypes.py
+++ b/clients/iotclient/src/Streams/datatypes.py
@@ -571,11 +571,11 @@ class BaseDateTimeType(StreamDataType):
return self.pack_next_value(None, counter, parameters, errors)
parsed = self.parse_entry(entry)
if hasattr(self, '_minimum') and not hasattr(self, '_maximum') and
parsed < self._minimum:
- errors[counter] = 'The value is higher than the minimum: %s < %s!'
% (self._minimum_text, parsed)
+ errors[counter] = 'The value is higher than the minimum: %s < %s!'
% (parsed, self._minimum_text)
elif hasattr(self, '_maximum') and not hasattr(self, '_minimum') and
parsed > self._maximum:
errors[counter] = 'The value is higher than the maximum: %s > %s!'
% (parsed, self._maximum_text)
elif hasattr(self, '_maximum') and hasattr(self, '_minimum') and
parsed < self._minimum:
- errors[counter] = 'The value is out of range: %s < %s!' %
(self._minimum_text, parsed)
+ errors[counter] = 'The value is out of range: %s < %s!' % (parsed,
self._minimum_text)
elif hasattr(self, '_maximum') and hasattr(self, '_minimum') and
parsed > self._maximum:
errors[counter] = 'The value is out of range: %s > %s!' % (parsed,
self._maximum_text)
return self.pack_next_value(parsed, counter, parameters, errors)
diff --git a/clients/iotclient/src/Streams/flushing.py
b/clients/iotclient/src/Streams/flushing.py
_______________________________________________
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list
