determining bytes read from a file.
Hello all,
I have come across a weird problem, I need to determine the amount
of bytes read from a file, but couldn't figure it out ,
My program does this :
__
file = open("somefile")
data = file.read()
print "bytes read ", len(data)
---
But the bytes read is not being printed correctly, I think bytes are
being counted only till the first occurance of '\0' is encountered.
Even though the file is of a very large size, the bytes till the first
'\0' are counted.
Can someone pls advise me regarding this.
Thanks.
Best Regards,
Vineeth.
--
http://mail.python.org/mailman/listinfo/python-list
Re: determining bytes read from a file.
On Dec 13, 5:13 pm, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote:
> vineeth wrote:
> > Hello all,
> >I have come across a weird problem, I need to determine the amount
> > of bytes read from a file, but couldn't figure it out ,
> >My program does this :
> > __
> >file = open("somefile")
> >data = file.read()
> >print "bytes read ", len(data)
> > ---
>
> > But the bytes read is not being printed correctly, I think bytes are
> > being counted only till the first occurance of '\0' is encountered.
> > Even though the file is of a very large size, the bytes till the first
> > '\0' are counted.
>
> I doubt that. Python doesn't interpret data when reading, and byte-strings
> don't have a implicit 0-based length.
>
> So I think you must be doing something different - clearly the above is not
> actual code, but something made up for this post. Show us your actual code,
> please.
>
> diez
Hi,
The program tries to create a C Byte array of HEX data from a binary
input file (for ex : to embed a .png image with the application as an
array),
Here is the program :
"""
python script to create a bit stream of a input binary file.
Usage : bit_stream_creator.py -i input_file -b bytes_to_dump
"""
import sys
from binascii import hexlify
from optparse import OptionParser
if len(sys.argv) != 5:
print "incorrect args, usage : %s -i input_file -b bytes_to_dump" %
(sys.argv[0])
sys.exit(0)
parser = OptionParser()
parser.add_option("-i", "--input", dest="inputfilename")
parser.add_option("-b", "--bytes", dest="bytes")
(options, args) = parser.parse_args()
print "-i",options.inputfilename
print "-b",options.bytes
# open input file
infile = open(options.inputfilename)
# create the member variable name.
mem_var_name = options.inputfilename
mem_var_name = mem_var_name.replace(' ','_')
mem_var_name = mem_var_name.replace('.','_')
outfile_c = open(mem_var_name + ".c","w")
outfile_h = open(mem_var_name + ".h","w")
# read the data.
print " Reading %d bytes. " % (int(options.bytes))
bytes_reqd = int(options.bytes)
data = infile.read(bytes_reqd)
print "Bytes Read ", len(data)
# convert to hex decimal representation
hex_data = hexlify(data)
i = 0
# Write the c file with the memory dump.
outfile_c.write ( "unsigned char %s[%d] = {\n" %
(mem_var_name,bytes_reqd) )
while i < len(hex_data):
outfile_c.write( "0x%c%c" % ( hex_data[i],hex_data[i+1] ) )
i += 2
if i != len(hex_data):
outfile_c.write(",")
if i % 32 == 0:
outfile_c.write("\n")
outfile_c.write ( "\n};\n" )
# Write the .h file with forward declaration.
cpp_macro = "__"+mem_var_name.upper()+"_H__"
outfile_h.write("#ifndef "+cpp_macro + "\n")
outfile_h.write("#define "+cpp_macro + "\n")
outfile_h.write( "//%s, size %d \n" % (mem_var_name,len(data)) )
outfile_h.write( "extern unsigned char %s[%d];\n" %
(mem_var_name,bytes_reqd) )
outfile_h.write("#endif //"+cpp_macro + "\n")
#close the files.
outfile_c.close()
outfile_h.close()
infile.close()
But len(data) never proceeds beyond the NULL character.
Any help and tips is appreciated.
Thanks and Regards,
Vineeth.
--
http://mail.python.org/mailman/listinfo/python-list
Re: determining bytes read from a file.
On Dec 13, 5:27 pm, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> On Thu, 13 Dec 2007 04:04:59 -0800, vineeth wrote:
> >I have come across a weird problem, I need to determine the amount
> > of bytes read from a file, but couldn't figure it out ,
> >My program does this :
> > __
> >file = open("somefile")
> >data = file.read()
> >print "bytes read ", len(data)
> > ---
>
> > But the bytes read is not being printed correctly, I think bytes are
> > being counted only till the first occurance of '\0' is encountered.
> > Even though the file is of a very large size, the bytes till the first
> > '\0' are counted.
>
> If you want to deal with bytes better open the file in binary mode.
> Windows alters line endings and stops at a specific byte (forgot the
> value) otherwise.
>
> Ciao,
> Marc 'BlackJack' Rintsch
Thanks, opening the file in binary mode helped, thanks a lot.
--
http://mail.python.org/mailman/listinfo/python-list
Re: determining bytes read from a file.
On Dec 13, 7:50 pm, Chris <[EMAIL PROTECTED]> wrote:
> A couple potential optimizations:
>
>
>
> > # create the member variable name.
> > mem_var_name = options.inputfilename
> > mem_var_name = mem_var_name.replace(' ','_')
> > mem_var_name = mem_var_name.replace('.','_')
>
> mem_var_name = options.inputfilename.replace(' ','_').replace('.','_')
> No need to assign it 3 times.
>
> > while i < len(hex_data):
> > outfile_c.write( "0x%c%c" % ( hex_data[i],hex_data[i+1] ) )
> > i += 2
> > if i != len(hex_data):
> > outfile_c.write(",")
> > if i % 32 == 0:
> > outfile_c.write("\n")
>
> This could be re-written as such:
>
> for x in xrange(0, len(hex_data), 32):
> output_c.write('%s\n' % ','.join(['0x%s'%''.join(['%c'%a for a in
> hex_data[x:x+32][i:i+2]]) for i in xrange(0, 32, 2)]
Thanks everyone for all the suggestions, and the optimizations, I am
still a newbie and not aware of these ,
thanks!
--
http://mail.python.org/mailman/listinfo/python-list
problem writing excel sheet using python
Hi I am finding it difficult to create a excel sheet using openpyxl or xlsxwriter. The problem is that i am loading a table data from MYSQL db which has 600k rows and 15 columns (approximately 100mb data). The error that the terminal shows is that "MemoryError". I just wanted to know if it is possible to create a excel sheet with the above said data in less than one minute using any packages in python. Thanks, Vineeth -- https://mail.python.org/mailman/listinfo/python-list
Re: problem writing excel sheet using python
The belwo is the code, I have also tried writing it row by row with xlsxwrites
passing argument constant_memory:True. I donot know if my looping code is
ineffective.
for req_param in request.GET.get("Req-Tables").split(","):
sheet_names.append(req_param)
title.append(req_param)
report_data=db_exec_query( param_table_map[req_param])
sheet_names[key]=wb.active
if key!=0:
sheet_names[key]=wb.create_sheet(title= (title[key][:15] ) if
title[key] > 15 else title[key])
req_param, sys.getsizeof(report_data))
count= 0
for r in report_data:
try:
sheet_names[key].append(r)
except:
sheet_names[key].append(str(c).decode('cp1252') for c in r)
key=key+1
wb.save(response)
On Wednesday, July 13, 2016 at 2:41:43 PM UTC-7, Chris Angelico wrote:
> On Thu, Jul 14, 2016 at 7:29 AM, vineeth menneni
> wrote:
> > Hi I am finding it difficult to create a excel sheet using openpyxl or
> > xlsxwriter. The problem is that i am loading a table data from MYSQL db
> > which has 600k rows and 15 columns (approximately 100mb data). The error
> > that the terminal shows is that "MemoryError". I just wanted to know if it
> > is possible to create a excel sheet with the above said data in less than
> > one minute using any packages in python.
> >
>
> You can probably build it progressively, row by row. That way, you
> shouldn't need to keep everything in memory at once. But beyond that,
> I can't say without seeing your code.
>
> ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
apache not ale to execute a vbscript with python
Hi, I am trying to call a external vbscript through subprocess.call() it works fine with my local, but when i try to test apache says that no such file or directory. The problem here is that the vbscript should generate a .xlsx file and it is not being created. So apache gives a warning no such file or directory. Can you suggest any solutions. Thanks -- https://mail.python.org/mailman/listinfo/python-list
Re: I wrote a C++ code generator in Python, would anyone please help me to review the code? :)
Hi, I am interested in your proposal. I am basically a C/C++ programmer, but recently fell in love with python. Please send more details on fgen. We could probably start working. :) Kevin Wan wrote: > fgen is a free command line tool that facilitates cross platform c++ > development, including header generation, cpp file generation, makefile > generation, unit test framework generation, etc. > > http://sf.net/projects/fgen > > I'm not very familiar with Python. Any feedback are appreciated! Or > anyone like to develop it with me? > > Thanks. -- http://mail.python.org/mailman/listinfo/python-list
