Two utililties for managing file libs + forums

[Kracken]

Hi guys,

I just got my account yesterday, thanks Rick, these forums are really useful for WG fans.


BTW, I have done two little utilities, no big deal but I'd say they help me appreciate what I always didn't like in majorbbs: filelibs and forums.
Just ask if you want a copy !


1- GALFILIN_HELPER

Compared to Synchronet, importing files into MajorBBS is tedious. Synchro supports recursive scan of a structure, files.bbs+file_id.diz and inherently structure files with 2 level of organization (namely directories/libraries).

In Majorbbs, there is only one level (lib). If you have bunch of files that have file_id.diz, that's fine, but if you're trying to import a cd with dozens of nested subfolders with files.bbs or 00_index.txt everywhere, you'll have a hard time... There's the galfilin offline utility for this task, but it does not support the standard files.bbs. (it works with special "script" files).

I did a tool that scan a folder *recursively*, and document each file as much as possible.

- It tries to extract info from files.bbs, 00_index.txt, *.dir, *.cat whichever it finds in every subfolders
- If none are found, it extract the file_id.diz/desc.sdi of each file, if available
- It also appends longfilename + path of the file inside the structure to the description.
- With all the infos, it creates the SCR needed by galfilin.

It also grabs all the files and put them all in ONE directory in a *8.3* format, so to meet WG requirements for filelibs.

It also does manages duplicates (so if you had two FILE.ZIP anywhere in the structure, the second one will have an ID appended to the basename so it won't overwrite the first.

That way you can add complete CDROMs (or big nested directory structures) in one pass, and all files will be documented as much as possible.

It's not perfect solution, as files are all merged in one folder, but the original dir structure is stored in the description that is imported, and you can search with that, if you want.
It's the only easy way I found to make something useful out of the WG libraries!

I tested it so far and added at least 10 shareware CDroms and I'm happy with the results. I had this idea while browsing files on Penaltybox bbs (a full nightowl cd was imported there in one lib and browsing it wasn't so bad... but it looked tedious to achieve this result without some automation!

Surprisingly, searching in 40000 files descriptions is not that slow, because GALFILKW index keywords in the description in in an effective way.



2- NNTP_IMPORTER

Importing newsfeeds into WG is not simple, because WG sports a passive NNTP server ( it waits to be fed by other nntp through the IHAVE command).
(I'm not in a Synchronet praise, but again, Rob did code this nice piece of software called newslink.js that auto-fetches news from any NNTP and add them to Synchro forums. And it's child easy to setup.)

I tested the DNEWS NTTP server for Windows, and while it can feed WG, I really don't like it. It's a big-bloated convoluted piece of software (sorry little rant here With Dnews I don't feel I have control, as sometimes it does not seem to keep up with the msgids... and trying to figure out the issues in logs is a mess.. So I did a little lightweight script that allow me to get rid of Dnews and enjoy WG forums with fresh feeds as soon as they are submited.

The program will connect to any NNTP server, grabs only the newsfeeds you like and import them the appropriate echo in WG.

Here's the conf file:
------------nntpimporter.conf-------------
[general]
server=192.168.244.150
articles_on_first_pass=5
run_every=600
[newsfeeds]
vert.synchro.net,DOVE-Net.Advertisements
vert.synchro.net,alt.bbs.misc
news.synserver.de,alt.bbs
---------------------------------------------

I set the script to trigger fetching every 600 seconds (run_every parameter). When it launches, it go over each [newsfeed], and downloads *new* messages and import (IHAVE) them into WG.
If the script is doing the first pass, it will download up to 5 articles (as specified by the 'articles_on_first_pass" parameter).
Once done, it retains the ID of the last fetched message for each specific newsfeed, so that next time it runs, it won't re-import same messages.

Notes:
- The script only IMPORT messages so far. I have not handled the other way around (when users (R)eply to messages). But I'm looking at this very soon !
- The WG server doesn't care about the date I send to it through IHAVE, it resets all dates to the current datetime (argg !)
- Make sure the echo exists first. Just create a forum and set the appropriate echo tag : (ie: us:dove-net.advertisements)

[Toyduck]

Hi guys,

I just got my account yesterday, thanks Rick, these forums are really useful for WG fans.


BTW, I have done two little utilities, no big deal but I'd say they help me appreciate what I always didn't like in majorbbs: filelibs and forums.
Just ask if you want a copy !


1- GALFILIN_HELPER

Compared to Synchronet, importing files into MajorBBS is tedious. Synchro supports recursive scan of a structure, files.bbs+file_id.diz and inherently structure files with 2 level of organization (namely directories/libraries).

In Majorbbs, there is only one level (lib). If you have bunch of files that have file_id.diz, that's fine, but if you're trying to import a cd with dozens of nested subfolders with files.bbs or 00_index.txt everywhere, you'll have a hard time... There's the galfilin offline utility for this task, but it does not support the standard files.bbs. (it works with special "script" files).

I did a tool that scan a folder *recursively*, and document each file as much as possible.

- It tries to extract info from files.bbs, 00_index.txt, *.dir, *.cat whichever it finds in every subfolders
- If none are found, it extract the file_id.diz/desc.sdi of each file, if available
- It also appends longfilename + path of the file inside the structure to the description.
- With all the infos, it creates the SCR needed by galfilin.

It also grabs all the files and put them all in ONE directory in a *8.3* format, so to meet WG requirements for filelibs.

It also does manages duplicates (so if you had two FILE.ZIP anywhere in the structure, the second one will have an ID appended to the basename so it won't overwrite the first.

That way you can add complete CDROMs (or big nested directory structures) in one pass, and all files will be documented as much as possible.

It's not perfect solution, as files are all merged in one folder, but the original dir structure is stored in the description that is imported, and you can search with that, if you want.
It's the only easy way I found to make something useful out of the WG libraries!

I tested it so far and added at least 10 shareware CDroms and I'm happy with the results. I had this idea while browsing files on Penaltybox bbs (a full nightowl cd was imported there in one lib and browsing it wasn't so bad... but it looked tedious to achieve this result without some automation!

Surprisingly, searching in 40000 files descriptions is not that slow, because GALFILKW index keywords in the description in in an effective way.



2- NNTP_IMPORTER

Importing newsfeeds into WG is not simple, because WG sports a passive NNTP server ( it waits to be fed by other nntp through the IHAVE command).
(I'm not in a Synchronet praise, but again, Rob did code this nice piece of software called newslink.js that auto-fetches news from any NNTP and add them to Synchro forums. And it's child easy to setup.)

I tested the DNEWS NTTP server for Windows, and while it can feed WG, I really don't like it. It's a big-bloated convoluted piece of software (sorry little rant here With Dnews I don't feel I have control, as sometimes it does not seem to keep up with the msgids... and trying to figure out the issues in logs is a mess.. So I did a little lightweight script that allow me to get rid of Dnews and enjoy WG forums with fresh feeds as soon as they are submited.

The program will connect to any NNTP server, grabs only the newsfeeds you like and import them the appropriate echo in WG.

Here's the conf file:
------------nntpimporter.conf-------------
[general]
server=192.168.244.150
articles_on_first_pass=5
run_every=600
[newsfeeds]
vert.synchro.net,DOVE-Net.Advertisements
vert.synchro.net,alt.bbs.misc
news.synserver.de,alt.bbs
---------------------------------------------

I set the script to trigger fetching every 600 seconds (run_every parameter). When it launches, it go over each [newsfeed], and downloads *new* messages and import (IHAVE) them into WG.
If the script is doing the first pass, it will download up to 5 articles (as specified by the 'articles_on_first_pass" parameter).
Once done, it retains the ID of the last fetched message for each specific newsfeed, so that next time it runs, it won't re-import same messages.

Notes:
- The script only IMPORT messages so far. I have not handled the other way around (when users (R)eply to messages). But I'm looking at this very soon !
- The WG server doesn't care about the date I send to it through IHAVE, it resets all dates to the current datetime (argg !)
- Make sure the echo exists first. Just create a forum and set the appropriate echo tag : (ie: us:dove-net.advertisements)

Welcome! New faces are always welcome here.

Would like to try your utilities; as an old ASP member I have numerous CD's floating around that I have not put back online since I left v6.25. You are right, to tedious, and is it worth it for such old shareware?

With the NNTP; I tried the DNEWS also, but I was hoping for something a little simpler. Having a feed coming in to possibly spark conversations on Worldlink would be nice.

[Kracken]

Hi Toyduck,

I agree, it's not worth it to invest any effort into tinkering with old CD-BBS imports into MajorBBS...
That's why I wanted something that automate the process..

Of course, even the whole point of using file libs today is arguable... I don't think much people still use syncterm/zmodem... there is ftp but well who's interested in 1992 ZOO files or Doom WADs anyway ? possible reasons: nostalgia, boredom, for me I also want to replicate a real bbs experience of the heydays, as files were a big part of it. It's still cool to read all those textfiles directly in WG (using the "Listing" option or CTRL-V).. there are some gems in nightowls... My BBS's will probably be an electro-museum frozen in time some day who knows
I remember burning that midnight oil managing these file libs back then.. it's so much easier todays, with vm/speed/space/python..

I'll package you something and and send you some instructions... hope you like them.

PS: Don't mind the trash code, I just started them both as drafts, with no intention of making that distributable....

stephanegregoire@gmail.com

[Toyduck]

Hi Toyduck,

I agree, it's not worth it to invest any effort into tinkering with old CD-BBS imports into MajorBBS...
That's why I wanted something that automate the process..

Of course, even the whole point of using file libs today is arguable... I don't think much people still use syncterm/zmodem... there is ftp but well who's interested in 1992 ZOO files or Doom WADs anyway ? possible reasons: nostalgia, boredom, for me I also want to replicate a real bbs experience of the heydays, as files were a big part of it. It's still cool to read all those textfiles directly in WG (using the "Listing" option or CTRL-V).. there are some gems in nightowls... My BBS's will probably be an electro-museum frozen in time some day who knows
I remember burning that midnight oil managing these file libs back then.. it's so much easier todays, with vm/speed/space/python..

I'll package you something and and send you some instructions... hope you like them.

PS: Don't mind the trash code, I just started them both as drafts, with no intention of making that distributable....

stephanegregoire@gmail.com

Sometimes trash code can be the best code...particularly if it works

When I ran MBBS 5 and 6 I used to put up the new ASP disk every month. Some of those files could still be popular, particularly with the interest in old DOS games and utilities and the fact that there are ways through VM's and emulators or running these programs like they did in DOS (or DoubleDOS, etc..) and with 'lots' more disk space... I don't miss my 20mg drive 'that' much...

[Kracken]

Hi Toyduck,

Sent you both utilities on your BBS yesterday (gmail refuses me to send you a package containing an EXE, I shipped UNZIP.EXE with galfilin_helper because it needs it to extract fileId.diz and desc.sdi...

BTW while testing w/ different news server, I found one causing bugs with nntpimporter.py... and I fixed the issue (sent you latest update)

(sometimes the server would state that it has articlesID 500 to 520, but there would be missing articles in-between. I've made some error checking there (if program get a fetch error then report and keep going).

--> I checked on the server with telnet and there are definitely "holes" in range of articleids... probably articles that were removed from moderators..

BTW I tested with alt.binaries.*, and as expected, articles are plainly put in the WG forums in full UUENCODED garbage.
Like newslink.js, I'll put on the todo some auto-detection mecanism that does fetch and uudecode() and then put in some file lib.

[Toyduck]

Hi Toyduck,

Sent you both utilities on your BBS yesterday (gmail refuses me to send you a package containing an EXE, I shipped UNZIP.EXE with galfilin_helper because it needs it to extract fileId.diz and desc.sdi...

BTW while testing w/ different news server, I found one causing bugs with nntpimporter.py... and I fixed the issue (sent you latest update)

(sometimes the server would state that it has articlesID 500 to 520, but there would be missing articles in-between. I've made some error checking there (if program get a fetch error then report and keep going).

--> I checked on the server with telnet and there are definitely "holes" in range of articleids... probably articles that were removed from moderators..

BTW I tested with alt.binaries.*, and as expected, articles are plainly put in the WG forums in full UUENCODED garbage.
Like newslink.js, I'll put on the todo some auto-detection mecanism that does fetch and uudecode() and then put in some file lib.

Thanks! I'm in the process of rebuilding my server now (running on a backup one) so I will have the room for these extra files and messages once the new one is up.

Your system up and running?

[Kracken]

actually my board is sort of private for now, it's really plain ugly, just playing with it

I updated the nntpimport.py some more with a very-beta Post functionality.
It seems that when users (R)eply to msgs forums, a new file is dropped in folder c:\wgserv\nntpout and WG try to send it to one external nntp server.. since I want the script to manage this (and the fact that we can have several nntp servers for several feeds), I disable this by setting the param to 'localhost' and handle the nntp out from the script. It works fine, but I had to enable 'authinfo' passing (many nntp allow non-auth read, but auth post). I created a [credentials] section where you can put credentials per server.
Right now, got succesfull post on dovenet/whispernet. YMMV.

PS: received a denied msg from Earthlink trying to reach your toyduck email!

[Toyduck]

actually my board is sort of private for now, it's really plain ugly, just playing with it

I updated the nntpimport.py some more with a very-beta Post functionality.
It seems that when users (R)eply to msgs forums, a new file is dropped in folder c:\wgserv\nntpout and WG try to send it to one external nntp server.. since I want the script to manage this (and the fact that we can have several nntp servers for several feeds), I disable this by setting the param to 'localhost' and handle the nntp out from the script. It works fine, but I had to enable 'authinfo' passing (many nntp allow non-auth read, but auth post). I created a [credentials] section where you can put credentials per server.
Right now, got succesfull post on dovenet/whispernet. YMMV.

PS: received a denied msg from Earthlink trying to reach your toyduck email!

Spam has gotten so bad recently I had to tighten my spam filter up a bit. I put you on my authorize list you shouldn't have any further problems. I did get you message though, it was in my spam folder.

What version you playing with? I'm running the public board on a winserver 2003 using wg32. I run my practice board on a xp vm.

[Kracken]

I have the 3.3 from elwynor running on a Win2008 VM (that's what amazon provides for the free trial) and like you I run a test XP with synchronet + a copy of my main WG3.3
I'm amazed at the stability and speed.. I use to set up 6.25/2.0 on a DOS vm myself, and well, I've settled for good for the nt version, it's solid and quick and most cool stuff are ported or being ported

As a FYI in case you had my issue: I just imported my old backed up QWK message bases into my main WG using a combination of Synchronet and nntpimport... I kept them buried for
a long time, didn't thought that would be possible Synchro permits conferences hard-coded in qwk to be mapped to real conferences, and imports that. Afterward, it's a matter of bridging Synchro and
Wg nntp servers.

[dspain]

Hi guys,

I just got my account yesterday, thanks Rick, these forums are really useful for WG fans.


BTW, I have done two little utilities, no big deal but I'd say they help me appreciate what I always didn't like in majorbbs: filelibs and forums.
Just ask if you want a copy !


1- GALFILIN_HELPER

Compared to Synchronet, importing files into MajorBBS is tedious. Synchro supports recursive scan of a structure, files.bbs+file_id.diz and inherently structure files with 2 level of organization (namely directories/libraries).

In Majorbbs, there is only one level (lib). If you have bunch of files that have file_id.diz, that's fine, but if you're trying to import a cd with dozens of nested subfolders with files.bbs or 00_index.txt everywhere, you'll have a hard time... There's the galfilin offline utility for this task, but it does not support the standard files.bbs. (it works with special "script" files).

I did a tool that scan a folder *recursively*, and document each file as much as possible.

- It tries to extract info from files.bbs, 00_index.txt, *.dir, *.cat whichever it finds in every subfolders
- If none are found, it extract the file_id.diz/desc.sdi of each file, if available
- It also appends longfilename + path of the file inside the structure to the description.
- With all the infos, it creates the SCR needed by galfilin.

It also grabs all the files and put them all in ONE directory in a *8.3* format, so to meet WG requirements for filelibs.

It also does manages duplicates (so if you had two FILE.ZIP anywhere in the structure, the second one will have an ID appended to the basename so it won't overwrite the first.

That way you can add complete CDROMs (or big nested directory structures) in one pass, and all files will be documented as much as possible.

It's not perfect solution, as files are all merged in one folder, but the original dir structure is stored in the description that is imported, and you can search with that, if you want.
It's the only easy way I found to make something useful out of the WG libraries!

I tested it so far and added at least 10 shareware CDroms and I'm happy with the results. I had this idea while browsing files on Penaltybox bbs (a full nightowl cd was imported there in one lib and browsing it wasn't so bad... but it looked tedious to achieve this result without some automation!

Surprisingly, searching in 40000 files descriptions is not that slow, because GALFILKW index keywords in the description in in an effective way.



2- NNTP_IMPORTER

Importing newsfeeds into WG is not simple, because WG sports a passive NNTP server ( it waits to be fed by other nntp through the IHAVE command).
(I'm not in a Synchronet praise, but again, Rob did code this nice piece of software called newslink.js that auto-fetches news from any NNTP and add them to Synchro forums. And it's child easy to setup.)

I tested the DNEWS NTTP server for Windows, and while it can feed WG, I really don't like it. It's a big-bloated convoluted piece of software (sorry little rant here With Dnews I don't feel I have control, as sometimes it does not seem to keep up with the msgids... and trying to figure out the issues in logs is a mess.. So I did a little lightweight script that allow me to get rid of Dnews and enjoy WG forums with fresh feeds as soon as they are submited.

The program will connect to any NNTP server, grabs only the newsfeeds you like and import them the appropriate echo in WG.

Here's the conf file:
------------nntpimporter.conf-------------
[general]
server=192.168.244.150
articles_on_first_pass=5
run_every=600
[newsfeeds]
vert.synchro.net,DOVE-Net.Advertisements
vert.synchro.net,alt.bbs.misc
news.synserver.de,alt.bbs
---------------------------------------------

I set the script to trigger fetching every 600 seconds (run_every parameter). When it launches, it go over each [newsfeed], and downloads *new* messages and import (IHAVE) them into WG.
If the script is doing the first pass, it will download up to 5 articles (as specified by the 'articles_on_first_pass" parameter).
Once done, it retains the ID of the last fetched message for each specific newsfeed, so that next time it runs, it won't re-import same messages.

Notes:
- The script only IMPORT messages so far. I have not handled the other way around (when users (R)eply to messages). But I'm looking at this very soon !
- The WG server doesn't care about the date I send to it through IHAVE, it resets all dates to the current datetime (argg !)
- Make sure the echo exists first. Just create a forum and set the appropriate echo tag : (ie: us:dove-net.advertisements)

nice, i wrote a utility that runs at cleanup and scans all file libraries and auto imports the file as well as am working on a newer NNTP option for wg, i already rewrote the smtp stuff to work using base64 authentication and am working with a sysop now to use a newer rfc compliant nntp client/server like synchronet.

so your utils are EXE files that can be placed in wgscleanup.bat?

[Kracken]

Hi Dan,

That's great your work for the NNTP part (well for Wg in general!) ... it's unfortunate that WG didn't natively had some newslink.js/puller functionality... IMHO nntp is very simple...

for my scripts, I don't think both could not go in wgscleanup.bat b/c:

- For the nntp stuff, it's in python, it's running as a daemon in a while(1) loop and trigger every X secs ... since the goal was to act as a NNTP bridge and get rid of dnews.I also used it as a one-shot tool for my QWK file imports into WG.
- The galfilin helper script is a one-shot dir parser in (ughhh!) vbscript. It essentially converts a folder full of files,subfolders,files.bbs/00_index/*.cat/*.dir/,longfilenames into a WG compatible folder and produces the scr galfilin require before importing into a lib. (out of boredom I started a little helper script and didn't want to open my dev machine, and in the end made it complete for my needs

I could convert them into C# .exe if somebody ever wants a cleaner version.