Page 1 of 1

V2.3 -- Batch Replace no longer working?

Posted: Tue Sep 26, 2006 5:10 pm
by zonk
First off -- Wanted to pass along how much I continue to love this product. Truly among the best $30 I ever spent.

However -- having a problem.

I had been neglectful in updating (mainly because the un-patched version suited me perfectly). However -- ran into a problem this morning trying to do a batch replacement using a tab delimited text file as for the strings, running over an *.sgm file. Kept getting fatal errors in AFR. The file's a fairly large one -- ~90 megs. Sooo... figured maybe it was time to update after all. Downloaded and installed v2.3.

Re-ran -- no crashes -- but no replacements made.

I'm basically replacing some file attributes with some missing values (using my 'normval' attribute as a specific hook to insert the right 'sort-date' attribute where necessary):
Find normval="300,000" sort-date=""
with normval="300,000" sort-date="20060101"

I've got about 200 of these.

I've tried using a *.sgm mask, a *.* mask... I've tried using multiple copies of the files. I've tried importing my list of replacements in different ways (i.e. tried instead saving a csv, then importing, tried a semicolon CSV, tried saving as a configuration).

I've played with all the batch replace options I can think -- checking, then unchecking different coding...

Nothing seems to work. I doublecheck that my search string does exist -- and they do (using my text editor, not an SGML editor-- searched for my "find" value). No line breaks that might be causing this, no extra spacing, no hidden characters....

AFR just no longer seems to be able to find the values. It just spins through the file - -reports finding it, asks me if I wish to process, finishes in a second or 2 without performing any replacements -- empty report, and when I look at the file, it's unchanged.

Any help greatly appreciated.


Posted: Tue Sep 26, 2006 5:14 pm
by Abacre
Please verify that this file does not have "Read-only" attribute.
Could you zip a sample file and send it to us (
together with default.cfg file?

Thanks -

Posted: Tue Sep 26, 2006 5:34 pm
by zonk
I did check the file (and folder) permissions -- I'm runing it off a file on my hard drive. Looks all good (wide open permissions).

Do you want my whole 90 meg file?

FYI -- just to check, I took a small slice of the file I'm running (about 100 lines or so -- ensuring I had at least 1 instance of replacement... just named it test.sgm. File was valid per my DTD, but guessing that wouldn't matter to AFR), and AFR fired properly, making the proper replacement.

Looks like it's gotta be an issue with the size of my file, no?

If necessary, I suppose I could chop the file into smaller chunks.

Posted: Tue Sep 26, 2006 6:13 pm
by Abacre
What is the size of zipped file?

Posted: Tue Sep 26, 2006 6:32 pm
by zonk
File's 18 megs using via winzip....

It's looking increasingly like a size issue...

I divided the file into 6 chunks (8-15 megs each; 200,000 lines each). Some of them were processed, some weren't. Biggest processed was about 13.25 megs -- nothing larger was touched, some smaller files touched - but not all (didn't verify changes existed in each chunk).

Started cutting some of the larger files in half -- ditto... some files start to process (but some ignored).

Does AFR work as a stream editor or strictly as a line editor (w/out using reg exps)? I.e., -- I'm wondering if the fact that the sgm data sometimes streams multiple element sets onto single lines (at least -- that's how my text editor and VIM on the UNIX box see the file) and perhaps some of these lines are choking AFR?

Posted: Tue Sep 26, 2006 6:51 pm
by Abacre
Currently AFR loads whole file into RAM and considers it as a single
string. It's needed for process correctly all regular expressions. Because
some regular expressions may include several lines of text.
We are planning to implement line by line replacement mode.
I think for your task this line-by-line mode will work great because
it seems that Search For text occupies only one line. Right?

OK - seems definitely a file size issue

Posted: Tue Sep 26, 2006 6:52 pm
by zonk
Breaking the file into 12 pieces, each ~100K lines, files about 6-7-8-9 megs each, re-ran and got all but 1 replacement (ran a quick grep to see how many replacements I needed).

I'll check out the one missing -- but could well be an oddball pattern.

Sooo... it would appear just not enough buffer space to crunch this 90 meg file.

Do you think it's worth playing with my virtual memory allocation on my pc at all? I'm guessing that might buy me 6 files instead of 12, but doubtful I'd get enough muscle to crunch a file 10-12 times as large, right?


Posted: Tue Sep 26, 2006 7:20 pm
by zonk
Thanks for your help - and quick responses, Roman.

Breaking the file into chunks isn't the end of the world for me - and I think you're exactly right, a line by line option would probably suit my need perfectly. All my replacements are indeed self-contained in a single line in this specific instance.

Before AFR - I was using a lot of SED shells, but SED was choking on way too many long lines, so I'm still quite pleased with AFR (especially since my limited shell scripting skills suck ;-)

I'll be on the lookout for the next update!