README file for KLibido - the KDE Linux Binaries Downloader Version 0.2.5 - 27/Feb/2005 * What it is: KLibido is a KDE program to download encoded articles from the usenet news service, using the nntp protocol. It supports multiple servers, multiple download threads per server, automatic joining and decoding of articles. * What it's not: KLibido is not a NewsReader. It doesn't let you easily display the articles - only their subject, and it discards all non-binary posts. If you want a nice newsreader for KDE, try KNode. * What works: - Getting the group list and subscribe to newsgroups - Getting headers, sorting, filtering by subject, state and "completeness" - downloading and decoding of binary posts (including yEnc). - Queue management: it's possible to drag and drop items around in the download queue to change the order of download, pause the queue, pause and cancel the items in the queue. - Server priorities and balancing - Requeueing: if an article fails on a server, it is automatically tried on the other servers. - Viewing nfo's, covers, et al inside the program using kparts. - Group categories and aliases - .nzb support * What doesn't work: - No documentation :) - There are probably a lot of bugs. Having said that, it quite works, and I've used it to download tens of gigabytes. * How does it do it: - KLibido uses kde and the kde libraries for the interface, Berkeley DB for storing the newsgroups' headers, and UUDeview for decoding the posts. Requirements: - Kde/Qt 3.2 or greater - Berkeley Db 4.x (http://www.sleepycat.com/products/db.shtml) with C++ support. If you compile it from source, remember to pass the "--enable-cxx" option to configure. - UUdeview library version 0.5.20 (http://www.fpx.de/fp/Software/UUDeview/). If you're a Gentoo user, emerge dev-libs/uulib. * How to compile: - You can grab the source from sourceforge (go to http://www.sf.net/projects/klibido). It compiles as usual: ./configure --prefix= make make install You can find out your kde prefix running kde-config --prefix PLEASE NOTE: Lots of problems/crashes have been reported when using optimizations. Please be conservative. If you want to grab the source from CVS, go to http://sourceforge.net/cvs/?group_id=114928 and follow the instructions. PLEASE NOTE: I try to commit only working code, but the CVS can be very unstable at times. Always read the Changelog before compiling/using. * Binary Packages: On the program's homepage there are links/instructions for downloading binary packages for some distributions (Mandrake, Fedora, SuSE, Debian, etc...). See http://klibido.sf.net/ if you don't want to compile the source yourself. * Getting started: Type "klibido" to start the program. Sorry, no fancy wizards at the moment. To get it working: - Configure the directories (a default is provided) - Add your servers (Server->new server) - Get the group list (Server->get list of groups) - Subscribe to some groups (newsgroup->subscribe to newsgroup) - NewsGroup->update subscribed groups - Enjoy :) * A note about error model: KLibido divides errors in two branches, fatal (i.e.: article not found) and not fatal (i.e.: timeout). Fatal errors cause the download of the item to fail, while non-fatal errors trigger a retry. In the server options, there are three values that control the connection timeout/errors: 1. Server timeout: this is the value of timeout for the server (in seconds), i.e.: if no response/data come from the server in "timeout" seconds, a timeout error is triggered (and the item is generally retried) 2. Retries: this value only applies to post downloads (i.e.: not to headers updates) and represents the number of times a download is retried in the event of a *fatal* error. Example: KLibido tries to download an article. The server answers "article not found". The download is retried "Retries" times (default: 3). Why, you may ask? Because some servers are lame/overloaded/whatever, and when you try to download an article you get "article not found" the first time, but if you retry in a few seconds, the download succeeds. If your servers don't show this behavior, put "retries" to 0. 3. Thread timeout: this is the timeout (in minutes) for the connection thread. When a thread fails to successfully connect/login to a server for "thread_timeout" minutes, the thread is paused and must be restarted manually (via the context menu). This is considered a communication error, so it doesn't cause any item in the queue to fail. More in depth: when a thread gets a login error, it is paused for n seconds. "n" depends on the number of *consecutive* fails: first time 1 seconds, second 5 seconds, third 10 seconds, then 30, then one minute. After that, the thread is always paused for one minute. After "thread_timeout" minutes, the thread is put in paused state, and doesn't try to connect anymore until the user restarts it manually. Why? Because a login error can have to much causes: - network is down - wrong login/password - thread limit exceeded - download limit exceeded - etc... Servers often return custom error codes, so in the event of a failed login there is no easy/automated way to know if the login failed because of a wrong password, exceeded daily download limit, too many connected threads, or whatever. So the best thing to do (IMHO) is to pause the thread and retry later. * The story behind it: The main goals I had in mind while writing it were: - Keeping memory usage as low as possible. Update/load/browse large groups (groups with over one million of articles) without using 1 Gb of RAM and swapping for days. - Multiple server support - Multiple download threads support - Fine-grained manipulation of the download queue. I hope to have succeeded. * Who am I: - Does anyone care? :-] * Credits - KNode and Pan developers for letting me steal their icons :) - Sleepycat for the great Berkeley Db - Frank Pilhofer for the UUDeview Library. - The KDevelop team for the great IDE they've made. - The KDE developers for the great GUI. - David Pye (.nzb parsing, yy/uu decoder) * Related software you will find on the 'net: - nget - http://nget.sourceforge.net/ - The *best* command-line nntp grabber. - BNR - http://www.bnr2.org/ - Multi-platform nntp grabber, available for Linux and Windows. - Pan - http://pan.rebelbase.com/ - Great newsreader with binary skills :). - Knzb - http://dmp.org.uk/linux/knzb/ - KDE .nzb (Nezbin) downloader (C) 2004/2005 Alessandro "Bauno" Bonometti,