A very efficient way for script-based uploading of new content

[baruchel]

A friend of mine needing to automatically submit his new articles through a script-based process, I had a look at this topic, and found that no efficient solution exists. But I was surprised to notice that attempts seem to focus on the SQL parts of the problem, which should lead to more or less brutal insertions of content in the database unless you know very well the internal structure of the Joomla database. It doesn't look like the other way has been tried, though it leads to an absolutely perfect result : submit your content as you have to do, ie. through HTTP connexion to the site. After having had a look,  it seems you can achieve the whole thing by only two HTTP requests, more precisely two POST requests, the first one being what is usually involved when you log in the administrator web page, the second one being what is usually involved when you click on the "save" button of your content editor. Each request requires a new socket to the port 80 (most often) of your site. According to your exact features, you may have to hack the following lines.

Assuming your site is at http://my.example.org, assuming your log-in form is at /administrator/index.php, assuming your administrator username is "admin", assuming your password is "mickey", assuming your submit button for that page is "Valider" (well, at least in the french version of Joomla; for the english one, it is "Login", and in any case, you will find it by looking at the HTML source code of the /administrator/index.php page), then send something like the 12 lines of code:

POST /administrator/index.php HTTP/1.1
Host: my.example.org
User-Agent: My little HTTP hack 1.0
Accept: */*
Accept-Language: fr-fr,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 40

usrname=admin&pass=mickey&submit=Valider

When fixing the last line, don't forget to count the characters and to fix also the "Content-Length" field.
Don't forget to think about the "end-of-line" coding (look at the "toggle crlf" option of telnet if you use telnet, or think about using "\r\n" or "\n" if you code that in C).

Then read what answers the server before closing the socket: only a few lines, one with a cookie like:
  484f812d4cd1bbf73e898fe5b3004e0c=f409b2f3ea91152e0d6ae0f6e404774a
Extract it, and open a new socket for sending the next request:

POST /administrator/index2.php HTTP/1.1
Host: my.example.org
User-Agent: My little HTTP hack 1.0
Accept: */*
Accept-Language: fr-fr,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://my.example.org/administrator/ind ... 0&task=new
Cookie: 484f812d4cd1bbf73e898fe5b3004e0c=f409b2f3ea91152e0d6ae0f6e404774a
Content-Type: application/x-www-form-urlencoded
Content-Length: 964

title=Un+joli+titre§ionid=1&title_alias=Un+joli+alias+pour+le+titre&catid=15&introtext=Un+joli+texte+d%26%2339%3Bintroduction%3Cbr+%2F%3E&fulltext=Un+joli+texte+principal.%3Cbr+%2F%3E&access=0&created_by_alias=Un+joli+alias+pour+l%27auteur&created_by=62&ordering=0&created=&publish_up=2006-08-25+19%3A07%3A39&publish_down=Jamais&folders=%2F&_source=&_align=&_alt=&_border=&_caption=&_caption_position=bottom&_caption_align=&_width=¶ms%5Bpageclass_sfx%5D=¶ms%5Bback_button%5D=¶ms%5Bitem_title%5D=1¶ms%5Blink_titles%5D=¶ms%5Bintrotext%5D=1¶ms%5Bsection%5D=0¶ms%5Bsection_link%5D=0¶ms%5Bcategory%5D=0¶ms%5Bcategory_link%5D=0¶ms%5Brating%5D=¶ms%5Bauthor%5D=¶ms%5Bcreatedate%5D=¶ms%5Bmodifydate%5D=¶ms%5Bpdf%5D=¶ms%5Bprint%5D=¶ms%5Bemail%5D=¶ms%5Bkeyref%5D=¶ms%5Bdocbook_type%5D=&metadesc=&metakey=&link_name=&id=&version=0&mask=0&option=com_content&redirect=0&task=save&images=&hidemainmenu=0

and it is done ! Just fix the last lines and put your article in it, that's the way your browser send it to Joomla.
If you want to catch the code by yourself, a tool like "tcpflow" will log such things, and then you can hack it.

Regards,
  Thomas Baruchel

[nathandiehl]

thank you. i am moving this thread over to submitted FAQ.
thanks again for the info!

[baruchel]

I noticed that the processus described in my message can be implemented simply by using twice the command 'wget' as follows:

First :
wget --keep-session-cookies
        --save-cookies cookies.txt
        --post-data="usrname=admin&pass=mickey&submit=Valider"
        http://my.example.org/administrator/index.php

(Put "Login" instead of "Valider" for english version of Joomla, or look around)
I put the command on several lines for convenience reasons, but of course there is only one line.
Now, you are logged in as administrator, and the file cookies.txt has been created containing
your session cookie. Then, assuming that article.txt contains your article in HTML form format, type:
wget --load-cookies cookies.txt
        --post-file=article.txt
        --referer="http://my.example.org/administrator/index2.php?option=com_content§ionid=0&task=new"
        http://my.example.org/administrator/index2.php

Your article must have the following format (only one line in your file, with no space):

title=Un+joli+titre§ionid=1&title_alias=Un+joli+alias+pour+le+titre&catid=15
&introtext=Un+joli+texte+d%26%2339%3Bintroduction%3Cbr+%2F%3E&fulltext=Un+joli+t
exte+principal.%3Cbr+%2F%3E&access=0&created_by_alias=Un+joli+alias+pour+l%27aut
eur&created_by=62&ordering=0&created=&publish_up=2006-08-25+19%3A07%3A39&publish
_down=Jamais&folders=%2F&_source=&_align=&_alt=&_border=&_caption=&_caption_posi
tion=bottom&_caption_align=&_width=¶ms%5Bpageclass_sfx%5D=¶ms%5Bback_but
ton%5D=¶ms%5Bitem_title%5D=1¶ms%5Blink_titles%5D=¶ms%5Bintrotext%5D=
1¶ms%5Bsection%5D=0¶ms%5Bsection_link%5D=0¶ms%5Bcategory%5D=0¶ms
%5Bcategory_link%5D=0¶ms%5Brating%5D=¶ms%5Bauthor%5D=¶ms%5Bcreatedat
e%5D=¶ms%5Bmodifydate%5D=¶ms%5Bpdf%5D=¶ms%5Bprint%5D=¶ms%5Bemail
%5D=¶ms%5Bkeyref%5D=¶ms%5Bdocbook_type%5D=&metadesc=&metakey=&link_name=
&id=&version=0&mask=0&option=com_content&redirect=0&task=save&images=&hidemainme
nu=0

Bye.

[baruchel]

I wrote a small shell script using 'wget' for my friend. It should work with your own Joomla
installation. You can find it at the following URL:
  http://baruchel.free.fr/~thomas/code/joomla_upload
Please, read carefully the instructions that are at the beginning of this file (as comments).

You can't expect this URL to be a permanent one; I have not much interest in all that,
and didn't do it for myself. If you think this piece of script has any interest, you should
try to host it by yourself but I won't do it, and will delete this file in some weeks.

Regards,
  Thomas Baruchel