script to upload from URL

[Hetz Ben Hamo]

Hi,

I've written a simple bash script to upload a file from a remote server as
a CGI script (yes, I know, I should use another language, but it's just a
proof-of-concept).

It goes like this: A simple HTML page gives the user a text line to enter a
URL and "upload" button, which submits the data using POST to a bash script
(I use the proccgi for transferring the values).

The scripts fetches the URL and launches wget to grab the file, rename it
and move it to a specific directory.

So far, so good. The script works well.

But I have one issue with it: those files are pretty big (1-3 GB) and wget
doesn't show anything while it uploads - in the web browser. I tried using
some redirect tricks, but it still doesn't show anything on the screen. I
can redirect the output to a text file and show it after the upload, but it
defeats the purpose of showing some activity.

So my question: how can I make WGET (or CURL) show anything on my browser
while it downloads the file (uploading it to the server)?

Thanks,
Hetz
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: script to upload from URL

[shimi]

2011/12/13 Hetz Ben Hamo 

> Hi,
>
> I've written a simple bash script to upload a file from a remote server as
> a CGI script (yes, I know, I should use another language, but it's just a
> proof-of-concept).
>
> It goes like this: A simple HTML page gives the user a text line to enter
> a URL and "upload" button, which submits the data using POST to a bash
> script (I use the proccgi for transferring the values).
>
> The scripts fetches the URL and launches wget to grab the file, rename it
> and move it to a specific directory.
>
> So far, so good. The script works well.
>
> But I have one issue with it: those files are pretty big (1-3 GB) and wget
> doesn't show anything while it uploads - in the web browser. I tried using
> some redirect tricks, but it still doesn't show anything on the screen. I
> can redirect the output to a text file and show it after the upload, but it
> defeats the purpose of showing some activity.
>
> So my question: how can I make WGET (or CURL) show anything on my browser
> while it downloads the file (uploading it to the server)?
>
> Thanks,
> Hetz
>
>
>
I am assuming I understood you correctly (and will answer upon that
assumption) :

You have a server that gets a request to upload a file from the client (the
file is NOT uploaded by the client, the client just asks for the file to be
uploaded)  - then the server uses cURL via system() or similar to do the
actual upload - and you want the CLIENT to see the progess of the cURL
process in real-time or close to real-time.

If so, how about this:

1. Redirect cURL's output to a temp file via -o - and background it
2. As a response to the client's request containing the form filled with
the filename to upload [which you started the cURL for in #1], you'll
return a simple HTML page with a DIV placeholder to show the progress. In
the end of the page, you'll call a JS function to run call an XHR that will
call a SECOND URL with the temp file in a parameter (you'll think on how to
secure it, if needed) - a URL that will run a script on the same said
server to read the temp file from #1 and display it (or a tail -n 5 of it,
or whatever).
3. After a successful return of the XHR, use
document.getElementyById('name-of-div-of-status').innerHTML=XHRObject.responseText
4. Repeat 2-3 until XHRObject.responseText contains some magic string from
SECOND URL that says "OK, I finished"

If I didn't get you right... please tell me where I was wrong :)

Hope this helps,

-- Shimi
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: script to upload from URL

[David Ronkin]

Call it asynchronously with ajax or jquery get/post - while waiting for
response you can print whatever you what.

David

-- 
בברכה,
דוד רונקין
נא בקרו בבלוג שלי: http://dronkin.blogspot.com


2011/12/13 Hetz Ben Hamo 

> Hi,
>
> I've written a simple bash script to upload a file from a remote server as
> a CGI script (yes, I know, I should use another language, but it's just a
> proof-of-concept).
>
> It goes like this: A simple HTML page gives the user a text line to enter
> a URL and "upload" button, which submits the data using POST to a bash
> script (I use the proccgi for transferring the values).
>
> The scripts fetches the URL and launches wget to grab the file, rename it
> and move it to a specific directory.
>
> So far, so good. The script works well.
>
> But I have one issue with it: those files are pretty big (1-3 GB) and wget
> doesn't show anything while it uploads - in the web browser. I tried using
> some redirect tricks, but it still doesn't show anything on the screen. I
> can redirect the output to a text file and show it after the upload, but it
> defeats the purpose of showing some activity.
>
> So my question: how can I make WGET (or CURL) show anything on my browser
> while it downloads the file (uploading it to the server)?
>
> Thanks,
> Hetz
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: script to upload from URL

[Evgeniy Ginzburg]

Hi.
Do second prototype using another language use libcurl it have
CURLOPT_NOPROGRESS, use AJAX as envelope.
For really big files it may be better to create some progress log or
read file size from filesysem directly with AJAX callback function to
check it status periodically.


2011/12/13 David Ronkin :
> Call it asynchronously with ajax or jquery get/post - while waiting for
> response you can print whatever you what.
>
> David
>
> --
> בברכה,
> דוד רונקין
> נא בקרו בבלוג שלי: http://dronkin.blogspot.com
>
>
> 2011/12/13 Hetz Ben Hamo 
>>
>> Hi,
>>
>> I've written a simple bash script to upload a file from a remote server as
>> a CGI script (yes, I know, I should use another language, but it's just a
>> proof-of-concept).
>>
>> It goes like this: A simple HTML page gives the user a text line to enter
>> a URL and "upload" button, which submits the data using POST to a bash
>> script (I use the proccgi for transferring the values).
>>
>> The scripts fetches the URL and launches wget to grab the file, rename it
>> and move it to a specific directory.
>>
>> So far, so good. The script works well.
>>
>> But I have one issue with it: those files are pretty big (1-3 GB) and wget
>> doesn't show anything while it uploads - in the web browser. I tried using
>> some redirect tricks, but it still doesn't show anything on the screen. I
>> can redirect the output to a text file and show it after the upload, but it
>> defeats the purpose of showing some activity.
>>
>> So my question: how can I make WGET (or CURL) show anything on my browser
>> while it downloads the file (uploading it to the server)?
>>
>> Thanks,
>> Hetz
>>
>> ___
>> Linux-il mailing list
>> Linux-il@cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
>
>
>
>
>
>
>
>
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>



-- 
So long, and thanks for all the fish.

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: script to upload from URL

[Amos Shapira]

2011/12/14 David Ronkin 

> Call it asynchronously with ajax or jquery get/post - while waiting for
> response you can print whatever you what.
>

I second that - that's, for instance, how Google's auto-completion (of
addresses in gmail, search terms in search etc) works - it sets a
background JavaScript thread which runs every second in the background,
reads the input field's value and submit it to the server for process if it
changed (and pulls the results, of course). You can skip the "check input
field value" part in your case. There must be some JavaScript libraries
which can already help you there (http://jquery.com/ is the first suspect,
but there must be many more lying around).

Two less related points:

1. Make sure you verify the URL for any hockery-pockery (e.g. that it's a
genuine legitimate "http(s)://" URL and not, for instance
"file:///etc/passwd" or trying to break out of the shell parameter quoting
to inject its own shell commands or encoded javascript that can be used for
cross-site-scripting :)

2. While I'm looking for work, I might be available for such small jobs
(I'm not a JavaScript Guru, more of a server-side guy, but should be able
to do something like this with some research).

--Amos


>
> David
>
>  --
> בברכה,
> דוד רונקין
> נא בקרו בבלוג שלי: http://dronkin.blogspot.com
>
>
> 2011/12/13 Hetz Ben Hamo 
>
>> Hi,
>>
>> I've written a simple bash script to upload a file from a remote server
>> as a CGI script (yes, I know, I should use another language, but it's just
>> a proof-of-concept).
>>
>> It goes like this: A simple HTML page gives the user a text line to enter
>> a URL and "upload" button, which submits the data using POST to a bash
>> script (I use the proccgi for transferring the values).
>>
>> The scripts fetches the URL and launches wget to grab the file, rename it
>> and move it to a specific directory.
>>
>> So far, so good. The script works well.
>>
>> But I have one issue with it: those files are pretty big (1-3 GB) and
>> wget doesn't show anything while it uploads - in the web browser. I tried
>> using some redirect tricks, but it still doesn't show anything on the
>> screen. I can redirect the output to a text file and show it after the
>> upload, but it defeats the purpose of showing some activity.
>>
>> So my question: how can I make WGET (or CURL) show anything on my browser
>> while it downloads the file (uploading it to the server)?
>>
>> Thanks,
>> Hetz
>>
>> ___
>> Linux-il mailing list
>> Linux-il@cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
>>
>
>
>
>
>
>
>
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
>


-- 
 [image: View my profile on LinkedIn]

___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: script to upload from URL

[Moish]

wget2web

On 14/12/2011 03:32, Amos Shapira wrote:

2011/12/14 David Ronkin mailto:dron...@gmail.com>>

Call it asynchronously with ajax or jquery get/post - while waiting
for response you can print whatever you what.


I second that - that's, for instance, how Google's auto-completion (of
addresses in gmail, search terms in search etc) works - it sets a
background JavaScript thread which runs every second in the background,
reads the input field's value and submit it to the server for process if
it changed (and pulls the results, of course). You can skip the "check
input field value" part in your case. There must be some JavaScript
libraries which can already help you there (http://jquery.com/ is the
first suspect, but there must be many more lying around).

Two less related points:

1. Make sure you verify the URL for any hockery-pockery (e.g. that it's
a genuine legitimate "http(s)://" URL and not, for instance
"file:///etc/passwd" or trying to break out of the shell parameter
quoting to inject its own shell commands or encoded javascript that can
be used for cross-site-scripting :)

2. While I'm looking for work, I might be available for such small jobs
(I'm not a JavaScript Guru, more of a server-side guy, but should be
able to do something like this with some research).

--Amos


David

--
בברכה,
דוד רונקין
נא בקרו בבלוג שלי: http://dronkin.blogspot.com



2011/12/13 Hetz Ben Hamo mailto:het...@gmail.com>>

Hi,

I've written a simple bash script to upload a file from a remote
server as a CGI script (yes, I know, I should use another
language, but it's just a proof-of-concept).

It goes like this: A simple HTML page gives the user a text line
to enter a URL and "upload" button, which submits the data using
POST to a bash script (I use the proccgi for transferring the
values).

The scripts fetches the URL and launches wget to grab the file,
rename it and move it to a specific directory.

So far, so good. The script works well.

But I have one issue with it: those files are pretty big (1-3
GB) and wget doesn't show anything while it uploads - in the web
browser. I tried using some redirect tricks, but it still
doesn't show anything on the screen. I can redirect the output
to a text file and show it after the upload, but it defeats the
purpose of showing some activity.

So my question: how can I make WGET (or CURL) show anything on
my browser while it downloads the file (uploading it to the server)?

Thanks,
Hetz

___
Linux-il mailing list
Linux-il@cs.huji.ac.il 
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il









___
Linux-il mailing list
Linux-il@cs.huji.ac.il 
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il




--
View my profile on LinkedIn 



___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il


--
Moish


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: script to upload from URL

[Hetz Ben Hamo]

Hi,

On Wed, Dec 14, 2011 at 9:25 AM, Moish  wrote:

> wget2web
>
>
Link?

Hetz


> On 14/12/2011 03:32, Amos Shapira wrote:
>
>> 2011/12/14 David Ronkin mailto:dron...@gmail.com>>
>>
>>Call it asynchronously with ajax or jquery get/post - while waiting
>>for response you can print whatever you what.
>>
>>
>> I second that - that's, for instance, how Google's auto-completion (of
>> addresses in gmail, search terms in search etc) works - it sets a
>> background JavaScript thread which runs every second in the background,
>> reads the input field's value and submit it to the server for process if
>> it changed (and pulls the results, of course). You can skip the "check
>> input field value" part in your case. There must be some JavaScript
>> libraries which can already help you there (http://jquery.com/ is the
>> first suspect, but there must be many more lying around).
>>
>> Two less related points:
>>
>> 1. Make sure you verify the URL for any hockery-pockery (e.g. that it's
>> a genuine legitimate "http(s)://" URL and not, for instance
>> "file:///etc/passwd" or trying to break out of the shell parameter
>> quoting to inject its own shell commands or encoded javascript that can
>> be used for cross-site-scripting :)
>>
>> 2. While I'm looking for work, I might be available for such small jobs
>> (I'm not a JavaScript Guru, more of a server-side guy, but should be
>> able to do something like this with some research).
>>
>> --Amos
>>
>>
>>David
>>
>>--
>>בברכה,
>>דוד רונקין
>>נא בקרו בבלוג שלי: http://dronkin.blogspot.com
>>
>>
>>
>>2011/12/13 Hetz Ben Hamo mailto:het...@gmail.com>>
>>
>>Hi,
>>
>>I've written a simple bash script to upload a file from a remote
>>server as a CGI script (yes, I know, I should use another
>>language, but it's just a proof-of-concept).
>>
>>It goes like this: A simple HTML page gives the user a text line
>>to enter a URL and "upload" button, which submits the data using
>>POST to a bash script (I use the proccgi for transferring the
>>values).
>>
>>The scripts fetches the URL and launches wget to grab the file,
>>rename it and move it to a specific directory.
>>
>>So far, so good. The script works well.
>>
>>But I have one issue with it: those files are pretty big (1-3
>>GB) and wget doesn't show anything while it uploads - in the web
>>browser. I tried using some redirect tricks, but it still
>>doesn't show anything on the screen. I can redirect the output
>>to a text file and show it after the upload, but it defeats the
>>purpose of showing some activity.
>>
>>So my question: how can I make WGET (or CURL) show anything on
>>my browser while it downloads the file (uploading it to the
>> server)?
>>
>>Thanks,
>>Hetz
>>
>>__**_
>>Linux-il mailing list
>>Linux-il@cs.huji.ac.il 
>>
>> http://mailman.cs.huji.ac.il/**mailman/listinfo/linux-il
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>__**_
>>Linux-il mailing list
>>Linux-il@cs.huji.ac.il 
>>
>> http://mailman.cs.huji.ac.il/**mailman/listinfo/linux-il
>>
>>
>>
>>
>> --
>> View my profile on LinkedIn 
>> 
>> >
>>
>>
>>
>> __**_
>> Linux-il mailing list
>> Linux-il@cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/**mailman/listinfo/linux-il
>>
>
> --
> Moish
>
>
> __**_
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/**mailman/listinfo/linux-il
>



-- 

*חץ בן חמו
חץ-ביז
*השכרה ואירוח של שרתים פיזיים
מעוניין להשתמש בשרותים שחסומים לגולש הישראלי? Hulu? NetFlix? Pandora?
Google Voice? אם כן, היכנס לכאן .
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il