Python cgi

[jbrewer]

I'm currently writing my first CGI script (in Python), and I keep
getting an error I don't know how to address.  I'm not sure if this is
a Python or Apache error, but I suspect it's an Apache config thing.
Anyway, in my code I need to upload a file, so in my HTML there's a
line like

File to upload 

and in my Python code I try to read the file following the Python docs
and the Python Cookbook like

form = cgi.FieldStorage()
fileitem = form["myfile"]
if fileitem.file:
# file upload details...
else:
# print error stuff to page

The problem is that the "if fileitem.file" test is never true.  After
some debugging I discovered that this is because fileitem is returned
as type MiniFieldStorage instead of FieldStorage, which is described as
"Like FieldStorage, for use when no file uploads are possible."  There
are other fields in the form that are read just fine.  Does anyone know
why no file uploads would be possible?  I know very little about
configuring Apache, unfortunately.

Also, I need to run an external program with my CGI script using
something like os.system with flags from input forms, which is a major
security risk.  Is it simply enough to test for flag.isalnum() or
should I do more to prevent random programs from being run?  I should
also do some minimal DOS protection as well, so information on how to
do that simply would be appreciated as well.

Some system info:
Fedora Core 3
Apache 2.0.53
Python 2.3.4

Thanks,
Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python cgi

[jbrewer]

I added enctype="multipart/form-data"  to the  tag, and that
seemed to solve it.  Thanks.

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

PIL FITs image decoder

[jbrewer]

I'm trying to read in a FITs image file for my research, and I decided
that writing a file decoder for the Python imaging library would be the
easiest way to accomplish this for my needs.  FITs is a raw data format
used in astronomy.

Anyway, I followed the example in the PIL documentation online, and I
also had a look at the FITs image stub file included with PIL 1.1.5.  I
cooked up something that should work (or nearly work), but I keep
running into either one of two errors (below).  The crux of the problem
appears to be that my plugin isn't being registered properly and isn't
being read at runtime when I call Image.open().

1.)  The library loads the FitsStubImagePlugin.py file from the
site-packages directory (instead of my plugin) and then gives the
following error:

  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/PIL/ImageFile.py",
line 255, in load
raise IOError("cannot find loader for this %s file" % self.format)
IOError: cannot find loader for this FITS file

2.)  I remove the FitsStubImagePlugin.py, pyc files and stick my plugin
in the directory instead (my plugin was already in the PYTHONPATH
before).  Then I get the following error:

Traceback (most recent call last):
  File "FitsImagePlugin.py", line 111, in ?
image = Image.open(fits_name).save(jpg_name)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/PIL/Image.py",
line 1745, in open
raise IOError("cannot identify image file")
IOError: cannot identify image file

This seems like I'm either doing (or not doing) something really stupid
or the docs are outdated.  Any help would be greatly appreciated!

Jeremy

 -- Code below
---

import Image, ImageFile

_handler = None

##
# Install application-specific FITS image handler.
#
# @param handler Handler object.

def register_handler(handler):
global _handler
_handler = handler

# 
# Image adapter

def _accept(prefix):
return prefix[:6] == "SIMPLE"

class FitsImageFile(ImageFile.StubImageFile):
#class FitsImageFile(ImageFile.ImageFile):

format = "FITS"
format_description = "FITs raw image"

def _open(self):

# byte offset for FITs images
byte_offset = 2880

# check header for valid FITs image
header_data = self.fp.read(byte_offset)

# headers are stored in 80 character strings, so sort them out
into a
# nice list
i = 0
headers = []
while header_data[i] != "\n" and i < len(header_data):
headers.append(header_data[i:i + 80])
i += 81

# parse individual headers
ok = False
for header in headers:
words = header.split()

try:
keyword = words[0]
value = words[2]
except IndexError:
ok = False
break

if keyword == "NAXIS" and value == 2:
ok = True
elif keyword == "NAXIS1":
xsize = value
elif keyword == "NAXIS2":
ysize = value

if not ok:
raise ValueError("file is not a valid FITs image")

# size in pixels (width, height)
self.size = (xsize, ysize)

# mode setting is always greyscale
self.mode = "F"

# data descriptor
self.tile = [("raw", (0, 0) + self.size, byte_offset, ("F", 0,
-1))]

# is this needed?
loader = self._load()
if loader:
loader.open(self)

def _load(self):
return _handler

def _save(im, fp, filename):
if _handler is None or not hasattr("_handler", "save"):
raise IOError("FITS save handler not installed")
_handler.save(im, fp, filename)

# register the plugin
Image.register_open(FitsImageFile.format, FitsImageFile, _accept)
Image.register_save(FitsImageFile.format, _save)

# method given in docs
#Image.register_open(FitsImageFile.format, FitsImageFile)

Image.register_extension(FitsImageFile.format, ".fits")
Image.register_extension(FitsImageFile.format, ".fit")

# test code
if __name__ == "__main__":
import os

fits_name = "fpC-001739-r1-0304.fit"
jpg_name = os.path.splitext(fits_name)[0] + ".jpg"

image = Image.open(fits_name).save(jpg_name)
print "Converted FITs image '%s' to jpeg image '%s'" % (fits_name,
jpg_name)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PIL FITs image decoder

[jbrewer]

>http://www.stsci.edu/resources/software_hardware/pyfits

I know and love PyFits, but I need to be able to do draw shapes on a
FITs image (and perhaps some other operations), and I don't believe
that PyFits allows these kinds of operations.  It might be possible to
import the data into a numarray object and turn that into something PIL
can read, though.

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PIL FITs image decoder

[jbrewer]

>If you can bear having two copies in memory, Image.frombuffer()
>generally does the trick.

What arguments do you pass to this function, and do you flatten the
array from the FITs image?   I this but got garbage out for the image.

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PIL FITs image decoder

[jbrewer]

>If you can bear having two copies in memory, Image.frombuffer()
>generally does the trick.

Also, does PIL have a contrast / scale option that is similar to zscale
in ds9 or equalize in Image Magick?

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PIL FITs image decoder

[jbrewer]

I tried following your simple example (I already had something similar)
but with no luck.  I'm completely stumped as to why this doesn't work.
I even tried manually scaling the data to be in the range 0-255 out of
desperation.  The data is definitely contiguous and 32 bit floating
point.  At this point I've tried everything I can think of.  Any ideas?

Jeremy

# open the fits file using pyfits
fits = pyfits.open(fitsfile)
xsize, ysize = fits[0].data.shape
data = fits[0].data.astype("Float32")
fits.close()

# now create an image object
image = Image.frombuffer("F", (xsize, ysize), data)

# scale the data be 256 bit unsigned integers
#int_data = numarray.zeros((xsize, ysize)).astype("UInt8")
#min = data.min()
#max = data.max()
#for i in arange(xsize):
#for j in arange(ysize):
#scaled_value = (data[i, j] - min) * (255.0 / (max - min))
#int_data[i, j] = int(scaled_value + 0.5)

#image = Image.frombuffer("L", (xsize, ysize), int_data)

-- 
http://mail.python.org/mailman/listinfo/python-list

Python CGI

[jbrewer]

I need to update a CGI script I have been working on to perform
validation of input files.  The basic idea is this:

1.) HTML page is served
2.) User posts file and some other info
3.) Check file for necessary data
* If data is missing, then post a 2nd page requesting needed data
* If data is present, continue to 4
4.) Process input file and display results (processing path depends on
whether data from 3 was present in file or had to be input by user)

I'm unsure of the best way to implement step 3.  Is it even possible to
put all of this into one script?  Or should I have one script check the
file for the data, then pass the info on to a 2nd script?  The
cgi.FieldStorage() function should only be called once, so it seems
like it's not possible to read a form and then repost another one and
read it.  If this is true and I need 2 scripts, how do I pass on the
information in step 3 automatically if the file has the proper data and
no user input is needed?

Sorry if this is a dumb question (or unclear), but this is my
(ever-growing) first CGI script.  Thanks.
Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python CGI

[jbrewer]

Thanks guys.  I didn't realize that hidden form fields were so easy to
use.

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python C module questions

[jbrewer]

>You probably shouldn't post such large pieces of code to the list.

OK.

>You mean a docstring on the module object itself?

Actually, I meant docstrings to the module and the functions, objects,
methods, whatever else in the module.  My code was derived from the
Python Cookbook, which left that part out (rather important for
building real C modules).

>You should give up C with a dumb algorithm running at fast speed.
>Implement a better algorithm in Python, maybe you can even outperform
>the dumb code.

That would be the next step if I needed even more speed, but the better
algorithm here would be to use KDTrees, which would be overkill (and
would require lots of development time).  The C brute force
implementation runs plenty fast for me.  It took only took a few hours
to implement and yielded a factor of 7 performance increase.  I was
mostly interested in feedback on whether I had done things in a
properly efficient way (there are so many Python list / tuple /
sequence functions).  I also find it hard to believe that there's no
standard Python function for converting sequences of one object to
arrays in C (a friend mentioned that Ruby's C API has this).

Another question: how does the distutils package handle version
upgrades?  Say for example I find some bugs in my C code and need to
recompile it, will it just overwrite what's present in the
site-packages directory?

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

XML-RPC + SimpleHTTPServer question

[jbrewer]

I'm currently implementing an XML-RPC service in Python where binary
data is sent to the server via URLs.  However, some clients that need
to access the server may not have access to a web server, and I need to
find a solution.  I came up with the idea of embedding a simple HTTP
server in the XML-RPC clients so that files can be sent in the
following way:

1.  Start an HTTP server on the client using e.g SImpleHTTPServer on a
user allowed port
2.  Call XML-RPC server with the URL to the file on the client to
upload
3.  Get XML-RPC server response, then shut down HTTP server on client

Does this sound reasonable?  I know that XML-RPC can send binary data,
but I presume that the URL method would be faster because the XML
parser could skip reading the binary file (is it base64 encoded as a
string like in SOAP?).

Anyway, I'm unsure of how to implement this in Python.  In particular,
how do I shut down a web server once I've started it with
server_forever()?  Should I run the server in a separate thread or run
it asynchronously?  Since I'm only uploading one file I don't really
need to handle multiple clients; I just need to be able to shut the
server down once remote call has finished.

Any help would be appreciated since I'm new to network programming.

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: XML-RPC + SimpleHTTPServer question

[jbrewer]

Fredrik Lundh wrote:

>why not just use an ordinary HTTP POST request ?

Sorry for such a simple question, but how would I do this?  XML-RPC
runs on top of HTTP, so can I do a POST without running a separate HTTP
server?

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: XML-RPC + SimpleHTTPServer question

[jbrewer]

>What server are you using?

Just SimpleXMLRPCServer from the standard library.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: XML-RPC + SimpleHTTPServer question

[jbrewer]

Fredrik Lundh wrote:

> the XML-RPC protocol uses HTTP POST, so if you can handle XML-RPC, you
> should be able to handle any POST request.  what server are you using ?

I need some clarification of your suggestion.  Instead of sending URLs,
I could read the file as a string, create a Binary object, and send
that via XML-RPC.  The parameters will be sent to the server via HTTP
POST.  However, the file will be encoded as a base64 string and
included in the body of the XML-RPC message, so it will have to be
parsed by the server.  In my experience with SOAP, I have found this to
be extremely inefficient.

Are you suggesting sending the file separately thought a 2nd HTTP POST
with no XML-RPC message body?  I guess the POST request would look
something like:

POST /path/file HTTP/1.0
From: ...
User-Agent: ...
Content-Type: /application/binary
Content-Length: 



I'm not sure how to add a 2nd request like this.  How would I alter a
simple call like that below to inlcude the 2nd post?  Do I need to use
httplib and the request() method of HTTPConnection?  Or can I make
POSTs through a ServerProxy object?

import xmlrpclib
server = xmlrpclib.ServerProxy("http://myserver";)
result = server.my_function(file, params)

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: XML-RPC + SimpleHTTPServer question

[jbrewer]

OK, I posted my previous message before I saw your reply on how to
handle the server side.  On the client side, should I use
httplib.HTTPConnection.request() to upload the data or can I do this
through xmlrpc.ServerProxy objects?

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: XML-RPC + SimpleHTTPServer question

[jbrewer]

Thank you very much, Fredrik.  Your code and suggestion worked
perfectly.  I haven't benchmarked the plain HTTP post vs Binary
wrapper, but strangely even using the naive Binary wrapper in Python
sends files much faster than how Java + Axis wraps byte arrays in SOAP
messages.

Jeremy

-- 
http://mail.python.org/mailman/listinfo/python-list