by Elf Qrin (www.ElfQrin.com)
v1.6 r30jul2013 fr18Mar2000
The latest version of this document can be found at www.ElfQrin.com/docs/ngload.php
IMPORTANT: I made this document available on the Web. If you find this document posted in a newsgroup by someone else, it doesn't mean in any way I'm connected with that newsgroup.
NOTE: At the end of this article you'll find a list of all the programs cited herein, with the URLs to download them.
HOW TO UPLOAD
The purpose of this article is to explain how to upload and download binary files to a newsgroup, with a particular attention to large files sent in multiple posts.
Since most newservers limit the size of binary files attached to the messages, you may split a large file into smaller chunks and attack them to several messages. As a general rule, if the file you want to send is larger than 700Kb you'll have to split it into smaller files.
The most popular tool to split large files into chunks in the Windows environment is MasterSplitter, which purpose is "to split large files in order to move them via floppies or for transmission via E-mail." It also can Join chunks or Compare them to verify they are the same of the original file. Anyway, splitting files is not so hard, and you can easily write by yourself a file splitter in C or in QBasic.
All you have to do is to launch MasterSplitter, enter the name of the large file you want to split, and the size of every chunk, and then you are ready to split it.
Another great file splitter/joiner that is also freeware is BLJoin which works in a similar way.
It's better to don't create too many small chunks because some of them could be lost passing from newsserver to newsserver, and you'll have to repost them, in order to satisfy all the audience. A size of about 700Kb should be good, considering that binaries attachment are generally UUencoded, growing thus of 1/4 of their original size, and the additional bytes required for the header and the rest of the body of the message, so that the total size of the message should be less than 1Mb.
Once you have split the file, you can post it to the newsgroup. Make sure the newsserver to which you'll send your messages carry your newsgroup, accepts posting (i.e. it's not a read-only newsserver), and accepts binary attachments. Here you can find a list of free NNTP servers: www.elfqrin.com/hacklab/pages/nntpserv.php
Since attaching every chunk to a message and then send it it's a long and boring duty, you can use an automatic tool such as Autopost.
With Autopost you can put all the files (single files or chunks of a larger file) you want to post in a directory, for example C:\TEMP, then you can launch the program and click on the "Settings" button to enter the option window in which you can enter the NNTP server you want to use (Host), login info (username and password) in case you are not using a free server, the Header information (Name, Reply-To, Organization), and a prefix to the Subject (the Subject of every post will be made by your Prefix, followed by the file name and [1/1] that means the message has 1 binary attachment. Sadly Autopost doesn't add the size of the attachment which is an useful information, especially if who downloads it has a slow connection), the directory containing the files you want to send, and the destination newsgroup.
When you are finished with the options, you can click on OK, and then you can start posting by pressing the "Post" button.
As an extra courtesy, if you are uploading a set of pictures, you can post one or more index pictures that contains the thumbnails with the names of the picture you've posted, and a CSV (Comma Separated Value) index file containing file name, path name, and CRC value (separated by commas). They can be read by utilities (like scansort) to sort and rename files or compare files you have with the ones listed in the CSV to see what's missing.
In case you are uploading a movie, you can post a few pictures with scenes from the movie as a preview, while for mp3s or music, you can upload the lyrics, and a picture of the singer.
If you are doing it for the first time, you can try to post something to alt.test or even better, since they are binary files, to alt.binaries.test : it's not a Good Thing to send binary files to non-binary newsgroups, even if the server allows you to do that. Also when you'll make your actual post, make sure that it's "on topic" with the newsgroup you choose. You'd better lurk in a newsgroup for at least one/two weeks before to make your first post.
A last note about netiquette: make sure you are not using ALL CAPITAL LETTERS not only in the body of your message, but also in your name and subject.
HOW TO DOWNLOAD
To download files from a newsgroup, of course you need to have access to that newsgroup first. So you'll need a program to read newsgroups and a NNTP server that carries the newsgroup you want to open.
If your ISP doesn't have a NNTP server, or it has but it doesn't carry the newsgroup you like, or it only gets a few posts for that newsgroup, you can try one of the free NNTP servers from the list at the URL provided above.
For what concerns the application, a browser like Netscape or IE is good enough. If you want to try a specific news agent, you may try Forte Agent. Forte Agent can also automatically split and rejoin large files. However, Netscape is better under most aspects. It lets you open directly a newsgroup without loading all the groups carried by the server you choose. You can open multiple instances of the messanger window and browse many newsgroups at the same time, and read many posts from the same or different groups (Agent only enqueues all your requests), and you can see pictures while you are downloading them.
To open a newsgroup from a browser, you have to enter in the location field (the one in where you normally enter the URL of a webpage) the "news" protocol followed by the DNS or the IP address of the NNTP server you intend to use, and the name of the newsgroup you want to open, such as in this example:
The content of a newsgroup looks similar to your normal mailbox: you can read the posts by clicking on them. If they have an attachment it usually will appear as a link at the bottom of the message. All you have to do is to right click it and save it to disk. If they are chunks of a large file, name them with a progressive number (for example "File1", "File2", "File3", or just "1", "2", "3"...), to make things easier when you'll have to rejoin them.
If the attachment is a picture it will be shown directly, unless it is encoded, or you have disabled this function, in case your news agent allows it (on Netscape 4.5+ you can disable image loading with Edit|Preferences...|Advanced| and then uncheck "Automatically load images"). Anyway, in this case you also can right click the picture and save it to disk.
It may happen that the message you opened is not what you expected to, for it has nothing to do with the topic of the newsgroup, because it can be a message posted or crossposted to the wrong newsgroup, or more likely spam (commercial message), or a message posted by a "troll" (annoying person who don't believe in free speech online and flood newsgroups he doesn't like with fake messages).
In this case ignore the message and open another one; as a general rule don't even reply to trolls: that's what they want to enhance their otherwise low self-esteem (because they are happy to know someone considered them) and increase flood on the newsgroup. Send your complains to his ISP, instead. If the newsgroup is moderated, moderators will simply delete posts from spammers and trolls (however moderation is not generally considered a Good Thing, because it's anyway a form of censorship, which is against the philosophy of total freedom of the UseNet).
If you are using a browser, you'd better disable JavaScript inside the messages to prevent spammers to redirect your browser to their website. On Netscape 4.5+ you can do that with Edit|Preferences...|Advanced| and then uncheck "Enable JavaScript for Mail and News".
Binary attachments for their same nature (as they are binary files attached to an otherwise pure text file) are encoded in some way, but generally your news agent can decode them automatically, in a transparent way for you, still some encoding can't be decoded "on the fly" (like Base64, which however is more optimized than others) and if you are downloading a picture, you'll see a link to save instead of the picture, even if you enabled the images.
The eldest and still most used encoding system is UUencode (UU stands for "Unix to Unix"). Sometimes the header of the attachment is missing, and you receive it as a pure text file in this format:
as in the following example:
The UUencoded data is a series of lines of ASCII text characters which are normally 60 characters long and begin with the letter "M". Since both Netscape 4.5+ and Agent tell you how many lines there are in a post you can have an idea of how large is a message before to open it. The approximate size in kilobytes of a message can be calculated by multiplying the number of lines by 60 and then dividing the result by 1024. This would be the size of the file you are downloading. To know how many Kb will occupe on your hard disk, you'll have to multiply it by 3 and then divide by 4. If, for example, you find a message which is 1000 lines long, you'll have to download 1000*60/1024=58.5Kb, and it will occupe 58.5*3/4=43.8Kb on disk. When UUencoded files are saved as stand alone files, generally have an ".uu" or ".uue" extension. The permission mode number makes sense only under Unix/Linux.
If you get an undecoded UUencoded attachment, you'll have to decode it "manually" first. You have to proceed this way:
Copy the whole body of the message (usually with CTRL+A to select all, followed by CTRL+C to copy it), then paste it (CTRL+V) on a good text editor. Don't use the standard Windows Notepad because it can't handle large text files and have problems with some special characters.
The best Notepad replacement I ever found (and I've tried them all) is JGsoft EditPad Lite which is also freeware. Download it, unzip it, rename editpad.exe as notepad.exe, copy it in your Windows directory (normally C:\WINDOWS) overwriting the original file, and forget the ugly Windows Notepad has ever existed (note: if you want to keep a copy of the original Notepad, copy it elsewhere but don't move or rename it, because Windows will redirect the File Types to the renamed/moved program).
If you don't need all the extra features of EditPlus but you simply want a NotePad replacement without all the limits of the Windows editor, you can use TheGun, a simple NotePad-looking editor, entirely written in machine code, which size is only 6144 bytes. Another excellent text editor mainly meant for coders is EditPlus (shareware), but it has too many functions to be considered a simple Notepad replacement, so I'd rather suggest you to install it as a second text editor.
However, once you have it in your text editor, delete all the lines which aren't part of the UUencoded file (everything above the "begin" line and below the "end" line) and save it to disk with the name you want. Now you can decode it with Shell Decode Extension (right click on the file name and choose Decode from the context menu). You may also decode it from the MS-DOS command line with an uuencode/uudecode command you may find in some software archive. Remember to delete the UUencoded file you saved after decoding it, for it will be only a waste of space on your hard disk.
If the file you are downloading has been UUencoded and then split in multiple chunks, it may happen that single chunks don't have the "begin" and the "end" lines. In this case, after you have deleted all the useless lines, leaving only the ones that begin with a "M", manually add the begin/end lines, as in the following example:
At this point you can decode it as explained before.
REBUILDING THE BIG FILE
If you have several chunks of a large files on your hard disk, now it's time to rejoin them. You can do that with a tool such as MasterSplitter, or manually, from the MS-DOS command line.
In this case you can use the COPY command like in the following example:
Where DestFile.ext is the name of the destination file name with the appropriate extension (.MPG for MPEG movies, .MP3 for MP3 audio files, .JPG or .GIF for pictures, and so on...)
The option /B is necessary because you are joining Binary files. If you try to join binary files without the /B option, they will be treated as ASCII (pure text) files, and the copy process will end as soon as the computer will meet a byte with a value of 0, because it will consider it as an End-Of-File (EOF) marker.
If you have several chunks and you can't put all of their names in a single MS-DOS command line (there's a limit of 127 characters for a single command line), you'd better join them in larger chunks first, as in the following example:
If there are some missing chunks, you can post a message asking to the original poster or to someone else who got all the posts, to post the missing chunks again. It would be nice to have a second newsgroup dedicated only to reposts (a kind of "subnewsgroup"), to don't bother people that got all the parts of the file.
Before to ask, you can try to open that newsgroup from another NNTP server. Since news posts are passed from a server to another, it may happen that your server didn't get all the posts, but you could find them in another one which has a better "feed" (it gets posts from more newsservers) and a longer history (it keeps old posts for a longer time).
To help recovering files with missing parts, there's also a neat system called PARchive. You may have seen files with extensions like PAR, P00, P01, and so on, posted together with multi part files.
These files can be used by PARchive clients like QuickPAR to rebuild a file. PAR files have to be stored in the same directory containing the parts of the file you want to rebuild. If the number of PAR files is equal or exceeds the number of missing files, the PARchive client will be able to restore the original file.
However, if you can't get all the chunks but the file you were downloading is an MPEG movie, still you can join the chunks you got, if you have at least the first chunk which contains the MPEG header information. You'll notice some weird effects when two non contiguos chunks join.
Another interesting peculiarity of MPEG files is that you can view them during downloading. When you open a message containing a MPEG video, stop it as soon as the link with its file name appear (on Netscape), then right click on the link and save it to disk, so that the file will be saved directly from the Internet to the disk.
Now you can make a copy of the file (click on the file name, then press CTRL+C followed by CTRL+V) and launch it while the rest of the file is still being downloaded.
This is useful to see if the video you are downloading is actually what you expect, rather than a video you already have with a different name, spam, or something that could be inconvenient for you. To see at least a frame of the MPEG video, you should wait until you got at least the first 35Kb (actually it depends from the width/height, color depth and resolution of the video), while if you want to have a more precise idea of the first seconds of animation, you should wait until you got 75/100Kb (since they are encoded, the file you'll save on disk will be only 3/4 of the size of the part of file you've downloaded until that point).
The same peculiarity apply to MP3 audio files and also to JPEG images.
If the file you have downloaded is an executable file, DELETE IT AT ONCE. Never run executable program files downloaded from a newsgroup. They are VERY likely to be trojans or viruses (especially if the program was posted in a newsgroup in which are expressed unpopular ideas). Even if you trust the sender, you can't be sure if instead is someone who forged his identity.
A trojan can format your hard disk, work as a hidden server that keeps a "back door" open to intruders, or send your e-mail address and other personal information to someone else on the Internet, if you are online, or next time you are online.
If for some reason you think you have absolutely to try that file, and you can't download it from somewhere else at least check it with an Antivirus such as Avira Free Antivirus before, which however is a good thing to do for any executable file you downloaded from the Internet, or more specifically from a website you've never been before.
For extra safety, you can get CCleaner to wipe all confidential data stored in your hard disk (you have to configure it to wipe data, that is overwrite and delete, rather than simply delete it), including system/temporary files, and places you've been on the Internet.
DEFERRED RETRIEVING (USING FORTE AGENT)
While a conventional e-mail client with news reader capabilities like Netscape or Outlook Express lets you browse newsgroups in "real time", so that you can open manually one message and see its content, if you have to retrieve a large number of messages, especially multi part files attached to several messages, you'd better use a specific newsgroup client like Forte Agent, that lets you select all the messages you are interested in, and then retrieve them with a single click. This way you can choose several messages and let Agent download them overnight, then the morning after you can save all the downloaded attachments or browse the newsgroups offline.
Forte Agent is a very complete software and lets you do several things, however this is a quick configuration to download messages from the newsgrups. First, go to the Options/User and System Profile menu, click on the System tab, and enter your favorite NNTP server in News Server field. If such NNTP server is not free and requires a password, click on the User tab and set it ("News Server Login"). Under the User tab, you also have to enter your name and e-mail ("Email Address" and "Full Name") but of course you don't need to enter the real ones, if you won't.
Now you are ready to go online: click on the "Go Online/Offline" button or check the "Online" item in the Online menu. At this point you have to retrieve the full list of newsgroups available on the NNTP server you've specified: go to the Online menu and click on "Get New Groups". This task can require up to several minutes but normally you'll have to do it only the first time or when you choose another NNTP server (remember to go offline and then online again when you switch server, also in this case you may want to keep "exinct group" when updating the list so that you don't have to reload the list of groups when you go back to an old server). After updating the list, new groups will be highlighted with a yellow star.
Once you have the list of all the newsgroups available on the chosen NNTP server, you can subscribe one or more of them (select the group name then click on the "Subscribe" button or choose "Subscribe" from the context menu) or simply select one or more by clicking on it (to select more than one keep the CTRL key pressed, also you can select a group of messages keeping the SHIFT key pressed and clicking on the first and the last message of the group). Now you can "Get New Headers in Subscribed Groups" or "Get New Headers in Selected Groups".
When you have all the headers you are ready to download the related messages (and their attachments): select one or more messages, then mark them for retrieval (use the "Mark for Retrieval" button or the same item from the context menu). Finally click on the "Get Marked Message Bodies" button or the same item from the Online menu and let Agent do the work. When it will be finished choose "Save All Attachments" from the File menu. Attachments are generally saved in the same Forte Agent program directory.
Before to mark a group of messages for retrieval, make sure they actually contain what you expect they to, because they could be fake or forged messages by spammers or trolls: select a couple of them (not the first and the last, you'd better choose something in the middle) and click on "Get Selected Message Bodies", then save their attachments, and verify their contents.
YENC ENCODING
Starting from about March 2002, a new encoding -- yEnc, invented by Jrgen Helbing -- appeared on the UseNet and started to become popular especially on binaries newsgroups because it's more efficient than uuencode, typically about 30%, so that the size of the encoded image is less or more the same of the plain unencoded size.
Unfortunately this caused a lot of confusion since most newsgroup clients couldn't handle such new format. This is how a yEnc file looks like:
If you are a Forte Agent registered user, you should have no problems at all, because it offers native support for yEnc since v1.91. Also, Jrgen Helbing joined the Agent developers team.
Otherwise, there are a few applications that handles or process yEnc files. The first one is a command line application called yDec provided directly by the creator of yEnc, but you'd better get yEnc32 which has a graphical interface, can process several files at once (batch) and is also able to encode into yEnc, rather than just decoding it. yEnc32 since v0.0.4 offers a shell extension so that you can save one or more messages on disk, right click on it (them) and choose "yDecode with yEnc32" from the context menu. Now you can choose the destination directory in "Save files to folder" and if you want you can check "Remove encoded files after decoding" and start the unencoding process clicking on "OK".
However, using such programs is kinda tricky because you have save files locally and then convert them. Also, most of the times you'll be unable to decode such files probably because of the way newgroup clients handle text which can result in the loss of some special characters, and you'll get an error messages and a corrupted decoded file. Luckily, there's another great product called yProxy which will stand between your client and the newsgroup server converting automatically and transparently yEnc into UUencode so that virtually any newsgroup client can read yEnc files as it had a native support for it!
Using yProxy is very simple: launch it, and enter your favorite news server into the "NNTP Server Address" field. Then copy the yProxy Address (which is normally localhost or 127.0.0.1 - that means localhost anyway) and use this one as NNTP server in your browser or favorite news reader. Finally make sure the "yEnc Decode" checkbox is checked. Click the Start button to start the local proxy (and obviously Stop to stop it).
Once yProxy is configured a started you can access to a newsgroup like in this example:
This is how yProxy works: it reads data directly from the NNTP server you've chosen, then converts yEnc encoded files and act a local server ("localhost") from which your news reader will actually get the messages.
As a final note about this issue, I'd recommend yEnc32 to create yEncoded files and yProxy together with your favorite news reader to read them.
A newsgroup has been created especially for test-posts in yEncoded format:
HOW TO VIEW A FILE DOWNLOADED FROM A NEWSGROUP
If the file is compressed, you have to decompress it first. There are many compression formats but the most common one is ZIP that might be associated with other kind of compressions.
However the extension of the file will look like .ZIP, .Z, .GZIP, .TAR.ZIP, .RAR . 7-Zip can handle them all. If you enabled the shell extension you can right click on the file and extract it from the context menu.
Once you have uncompressed the file, you can open it.
VLC handles almost all the audio/video formats. Be sure to have the latest version installed.
Pictures in almost any format can be opened with XnView (freeware) or IrfanView (freeware). For more (and uncommon) graphic formats you can try a professional application as Corel PhotoPaint. With PhotoPaint you can also edit pictures to adjust colors, contrast, or sharpness. In this case you should save the file with the same name, but adding an "r" as a suffix to the file name: is a good rule to never change the name of a picture, especially if it's part of a series.
APPENDIX 1 - SOFTWARE PRESENTED IN THIS ARTICLE
MasterSplitter (shareware $10) by TomaSoft Corporation
BLJoin (freeware) by Browny Lads Productions
Mozilla Thunderbird (freeware)
Forte Agent (shareware $29)
Forte Free Agent(freeware, light version of Forte Agent) by Forte
Decode Shell Extension (freeware) by Funduc Software
7-Zip
Avira Antivirus
TheGun.Exe (freeware) by 21st Century Missile Ware
EditPad Classic (freeware) by JGsoft
EditPlus Text Editor (shareware $30) by ES-Computing
yDec (freeware, open source) by yenc.org
yEnc32 (freeware) by eSite Media
yProxy (freeware) by Browny Lads Productions
PARchive
QuickPar
VLC - multimedia player
XnView (freeware)
IrfanView (freeware) by Irfan Skiljan
Acrobat Reader (freeware) by Adobe
CCleaner