Struct.Pack and Binary files
Having trouble working out an appropriate format string for packing a
binary file.
The below is something I use for ASCII files but now I need something
equivalent for working with binary files i.e jpg, zips etc.
fileHandle = open("test.txt")
while loop:
fileBuffer = fileHandle.read(512)
format = "!hh%dc" % len(fileBuffer)
outdata = struct.pack(format, *fileBuffer)
clientSocket.sendto(outdata, DestAddress)
I've reused the basic structure below for a binary file, the issue I'm
having is working out the correct format string. At first I thought a
float or double would have been the one to use but the interpreter
complains about invalid types being passed.
fileHandle = open("test.zip", "rb")
while loop:
fileBuffer = fileHandle.read(512)
format = "!hh%dd" % len(fileBuffer)
outdata = struct.pack(format, *fileBuffer)
clientSocket.sendto(outdata, DestAddress)
If someone could shed some light on the problem it would be
appreciated, I'm clearly missing something fairly obvious.
Thanks in advance.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Struct.Pack and Binary files
On Jan 28, 1:48 pm, Grant Edwards <[EMAIL PROTECTED]> wrote:
> On 2008-01-28, PurpleServerMonkey <[EMAIL PROTECTED]> wrote:
>
> > Having trouble working out an appropriate format string for packing a
> > binary file.
>
> > The below is something I use for ASCII files but now I need something
> > equivalent for working with binary files i.e jpg, zips etc.
>
> > fileHandle = open("test.txt")
>
> > while loop:
> > fileBuffer = fileHandle.read(512)
> > format = "!hh%dc" % len(fileBuffer)
> > outdata = struct.pack(format, *fileBuffer)
>
> Assuming fileBuffer has 512 bytes in it, your format string is
> going to be "!hh512c". When struct.pack sees that, it expects
> 514 values to pack. You're only passing it 512. What data do
> you want to be packed into the two "h" fields?
>
> You're also wasting a lot of CPU time unpacking and then
> repacking the 512 bytes in filebufffer. I suspect what you
> want is:
>
>val1 = ???
>val2 = ???
>outdata = format.struct("!hh", val1, val2) + fileBuffer
>
> I don't know what val1 and val2 are supposed to be, since the
> values that are to be packed as the two "h" fields seems to be
> missing from your example code.
>
> --
> Grant
Thanks Grant,
Yeah I left out the two 'h' values by mistake but your suggestion
worked out really well.
--
http://mail.python.org/mailman/listinfo/python-list
Web Interface Recommendations
Looking for suggestions on the best framework to use for an applications web interface. The user interface doesn't need immediate feedback and will be cross platform so a web interface is a good solution especially since it operates as a client\server system and not standalone. However there's just so many frameworks to choose from it's very confusing. After a lot of reading it sounds like OSE or Cherrypy might be good options but I don't have any real world experience with these frameworks to go by. Basically the interface won't be the application, it's used to input data and have the application act on it. It's going to have a Postgresql database and a number of service\daemon processes that do the actual work. It will also contain some form based information for keeping track of requests etc. and grow to a fair number of screens over time. Given the above what framework would you recommend? -- http://mail.python.org/mailman/listinfo/python-list
Re: Web Interface Recommendations
On Jan 30, 12:55 pm, Graham Dumpleton <[EMAIL PROTECTED]> wrote: > On Jan 30, 12:00 pm, PurpleServerMonkey <[EMAIL PROTECTED]> > wrote: > > > > > Looking for suggestions on the best framework to use for an > > applications web interface. > > > The user interface doesn't need immediate feedback and will be cross > > platform so a web interface is a good solution especially since it > > operates as a client\server system and not standalone. > > > However there's just so many frameworks to choose from it's very > > confusing. After a lot of reading it sounds like OSE or Cherrypy might > > be good options but I don't have any real world experience with these > > frameworks to go by. > > > Basically the interface won't be the application, it's used to input > > data and have the application act on it. It's going to have a > > Postgresql database and a number of service\daemon processes that do > > the actual work. It will also contain some form based information for > > keeping track of requests etc. and grow to a fair number of screens > > over time. > > > Given the above what framework would you recommend? > > Surprised you even looked at OSE. Although OSE has some features for > building HTML based web interfaces, they are very basic and not really > intended for building major stuff. OSE can still be useful for writing > backend applications, but would very much suggest you use just the XML- > RPC interfaces it provides to talk into its distributed messaging > system and service agent framework. > > If you use the XML-RPC interfaces then you can use a proper web > application framework for doing the actual HTML based user interface > front end. At that point you can choose any of the major frameworks, > such as Django, Pylons, TurboGears, CherryPy or web.py. > > Splitting the front end from the backend like this also means that > backend itself could be swapped out. Thus, instead of using OSE in the > backend, you could use simpler XML-RPC enabled Python applications, or > even use Pyro. In other words, you avoid intertwining code for front > end and backend too much, thus perhaps making it easier to change and > adapt as it grows. > > Graham Thanks Graham, decoupling the user interface and backend logic makes sense and definitely the way I want to go. Out of the major frameworks is there one that stands out as being particularly well suited for what I'm trying to do? Django and CherryPy are on the short list so I'll give them a detailed look although Pylons does sound rather interesting as well. -- http://mail.python.org/mailman/listinfo/python-list
Re: Web Interface Recommendations
On Jan 30, 8:08 pm, Bruno Desthuilliers wrote: > PurpleServerMonkey a écrit : > (snip) > > > Out of the major frameworks is there one that stands out as being > > particularly well suited for what I'm trying to do? > > > Django and CherryPy are on the short list so I'll give them a detailed > > look although Pylons does sound rather interesting as well. > > I guess you'll have to try them out to find the one that best match your > needs and personal preferences. Mostly: > > - CherryPy is more of a web application server than a framework per-se: > it's it's own HTTP server - which might or not be a good thing -, and > only deals with the "controler" part of the MVC triad. > > - Django is by now a mostly mature MVC framework, with more than a > couple years of production use on quite a lot of sites and applications, > good documentation and a somewhat large and active community. OTHO, it's > a very opiniated (and somewhat monolithic) framework, with most parts of > the stack (ORM, forms validation, template system etc) built > specifically for this framework (which was probably the sensible thing > to do by the time), so it's perhaps the less flexible of the three. > > - Pylons is IMHO very promising: wsgi from bottom to top, very flexible, > good default components choice (paste / Routes / SQLAlchemy / Mako / > FormEncode) but let you swap what you want in and out, and FWIW I've > seen so far a very sound architecture. FWIW, next Turbogears major > version will switch from CherryPy to Pylons. OTHO, it's still far from > being as mature and documented as Django. > > My 2 cents... Thanks Bruno, that's just the sort of info I was after. After reading more on the subject recently I'll be installing and testing Pylons and Turbogears first, think it will be a good fit for the project and if not there's no shortage of other offerings. -- http://mail.python.org/mailman/listinfo/python-list
Distributing Python Apps on Linux\BSD
Working on a rather large open source python application that I want to release for Linux and BSD and was wondering what methods others are using to distribute large and complex applications. Setuptools and friends seem to be focused on distributing modules, I'm at the other end of the scale where I want to distribute an entire application so that an Administrator can run a single install and have a fully operational product. A key requirement is that I want the application to fit in with what and admin would expect an application to look like at the system level i.e site-packages like structures aren't suitable. So far I've thought of using a configure script and make which would call some custom python installer script to do the actual install. It fits in nicely with what I want to achieve but are there any better options out there, how are others doing the same thing? -- http://mail.python.org/mailman/listinfo/python-list
Re: Distributing Python Apps on Linux\BSD
On Mar 22, 2:26 am, Miki <[EMAIL PROTECTED]> wrote: > Hello, > > Disclaimer: I'm not an expert on the subject. > > > Setuptools and friends seem to be focused on distributing modules, I'm > > at the other end of the scale where I want to distribute an entire > > application so that an Administrator can run a single install and have > > a fully operational product. A key requirement is that I want the > > application to fit in with what and admin would expect an application > > to look like at the system level i.e site-packages like structures > > aren't suitable. > > You do that with distutils as well. > > > So far I've thought of using a configure script and make which would > > call some custom python installer script to do the actual install. It > > fits in nicely with what I want to achieve but are there any better > > options out there, how are others doing the same thing? > > Every distro flavor has it's own installer: apt/deb, rpm, port, ... > On Windows you can use one of the free installer (InnoSetup and > friends). > > HTH, > -- > Miki <[EMAIL PROTECTED]>http://pythonwise.blogspot.com Well I'm not really interested in rpms or deb packages right now, I want to get it to the point where it will run on BSD and Linux without using distribution specific tools. Using a tarball or the standard python tools would be best. The problem is all the documentation surrounding distutils and setuptools refers to modules, now I'm not sure why but it seems most Python developers think an application is the same thing as a module. Unless you are writing very small applications that's definitely not the case. So I guess the question is, using distutils or setuptools is it possible for a user to select where to install the application i.e / usr/local? If not then I think it's going to be tarball deployment with a custom setup script, was hoping there was a better way. -- http://mail.python.org/mailman/listinfo/python-list
Python, Daemons and D-Bus
Seeking feedback from group members on a design I'm looking at using in a project. I've created an XML-RPC server and a number of Daemons, the idea is that the XML-RPC server gets a request from a user interface of some sort and then sends data to the Daemon processes to do work. Daemons will also need to feed back task information to the XML-RPC server. For the communications between the Daemons and XML-RPC server I was thinking of using D-Bus (The project is for Linux and *BSD) but wanted to hear what others thought of the idea? Would you use D-Bus or a more traditional IPC method such as sockets? Although D-Bus is relatively new it looks interesting, just not sure it would work well in this kind of project. Thanks in advance for your thoughts and opinions. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python, Daemons and D-Bus
On May 25, 5:46 am, Sebastian 'lunar' Wiesner <[EMAIL PROTECTED]> wrote: > [ PurpleServerMonkey <[EMAIL PROTECTED]> ] > > > Would you use D-Bus or a more traditional IPC method such as sockets? > > Although D-Bus is relatively new it looks interesting, just not sure > > it would work well in this kind of project. > > DBus is not really intended for private communication between processes of > the same application, but more for intercommunication between different > applications. If the IPC interface of your backend daemons is intended to > be used by other applications, DBus is the right choice, otherwise I would > choose something different. > > The reason is, that DBus doesn't know about applications. It exposes all > objects registered on the bus to every DBus client on the system and so > makes you application-private API available to the public (and spams the > bus with lots of generally useless objects ;) ). > > In case your IPC interface is application private, a custom IPC protocol > (probably using XML RPC over unix sockets) is better suited. > > Moreover you should make your choice dependent on the type of data you > transmit. Both DBus and XML-RPC wrap calls into XML messages, which is > terribly inefficient for large binary data. > > -- > Freedom is always the freedom of dissenters. > (Rosa Luxemburg) Thanks Sebastian, Your comments make a lot of sense. I was thinking of creating a custom session channel and using that for my purposes but as you mentioned it's not very secure and I do want to keep the server to daemon traffic private, the server has an XML-RPC interface with a public API. Will definitely look at using a different IPC mechanism for this part of the project. -- http://mail.python.org/mailman/listinfo/python-list
distutils as an application installer?
Was hoping someone could take a quick look at my distutils problem and give it a quick sanity check. Basically I have a server application (for Linux) that I want to install to /usr/local/, this location would have subdirectories for the daemon scripts, log files, plugins etc. After reading a bunch of distutils documentation it appears that the only good way out of this is to have my packages install to the system wide site-packages directory. Then using the data_files command in setup.py have it copy across an initial skeleton copy of the server directory and finally install the startup scripts. Does this sound correct, any other suggestions? The distutils documentation doesn't cover this scenario and I've spent days searching for information so any assistance is greatly appreciated. -- http://mail.python.org/mailman/listinfo/python-list
