Jaikoz- Discussion Forum

paultaylor wrote:

Sounds like you are jumping to invalid conclusions here have you actually tried increasing memory allocated ?

Memory has nothing to do with locks.

These are my settings in jaikoz.bat:
Code:

 java -Xms300m -Xmx1024m -XX:MaxPermSize=300m -jar lib\jaikoz.jar -l2 -m2 -f

I consistently experience huge delays and inresponsiveness in other applications using the harddisk, when Jaikoz is running, REGARDLESS of the number of files loaded in Jaikoz or the number of files being autocorrected, with ONE exception.

When I use automation software to fix one song at a time.

There is something very wrong with how Jaikoz interacts with the files it is working on.

EDIT: ...it doesn't leave thousands of files open and locked until it has completed auto-correcting all of them.

Right?

RIGHT?

RIGHT?

Ok, this discussion thread was about memory usage which is why I thought you were talking about that. Im not ware of any locking problems using Jaikoz, in fact its only if you have a Save Task as a part of the Autocorrecter, or you have the Retrieve Acoustic Ids tasks configured to save after adding an acoustic id that the files are even accessed.

I will check, but you could try disabling [b]Preferences/Musicbrainz/AmpliFIND/Save Acoustic Ids as Retrieved/b] option and any Save tasks to see if it makes any difference

How do you know its not just problem with cpu contention, most of the autocorrecter tasks are multithreaded and will take advanatge of multiple cpus if your computer has them.

paultaylor wrote:

I will check, but you could try disabling Preferences/Musicbrainz/AmpliFIND/Save Acoustic Ids as Retrieved option and any Save tasks to see if it makes any difference

Ok, I did that and ran autocorrecter with a set of 6500 files.
I clicked cancel after some ~50 files, and tried closing jaikoz when it asked to save files, which I then chose.

When the autocorrecter was running, I observed a negligible queue length (concurrent disk accesses), working set less than assigned for the memory, and very little cpu utilisation.

My setup is:
* win7 64 bit ultimate, updated
* java 64 bit
* i920
* 6 GiB RAM
* 4 TiB WD HDs in a volume

When Jaikoz was saving the files, I observed a sky-high disk queue (100+).
Referring to the resource-monitor, the response-time for the various jaikoz-related threads jumped to between 400 and 500 ms.
Before saving, the response-time was between 5 and 15.
Nothing else utilized the disk.

This indicates to me that Jaikoz is trying to save all files at once.

I can help you out with more information or testing if you like.

EDIT: I just checked with "Save changes" activated as last task in "Manipulators".
These are the problems:

1. Files are not saved as they are changed, but at the end of all other operations
This leads to two problems:
A) Memory bloat, to the point of causing application crash
B) The application is flooding the disk with (concurrent?) write operations (as many as running threads? I saw a very consistent average of 8 with an 8-core CPU) as the final task of correcting many files.

If the writes are in fact concurrent, the whole process of writing out the changes to the files will take much, much longer than if these writes were sequential.
I assume each thread will have the same priority to disk and that this causes contention.

MAzE5h1p69wB wrote:

When Jaikoz was saving the files, I observed a sky-high disk queue (100+).
Referring to the resource-monitor, the response-time for the various jaikoz-related threads jumped to between 400 and 500 ms.
Before saving, the response-time was between 5 and 15.
Nothing else utilized the disk.

This indicates to me that Jaikoz is trying to save all files at once.

Sort of, its actually allocating a thread per cpu for saving files, so if you have 4 cpus there will be 4 threads trying to save files at once. Im not sure why I have done this because there is a comment in the code stating that we should use a maximum of two threads when saving because of file contention. I think it is better to use two threads rather than one in order to allow one thread to do preparation for saving (which doesnt involve file i/o) whilst other is doing I/O

MAzE5h1p69wB wrote:

EDIT: I just checked with "Save changes" activated as last task in "Manipulators".
These are the problems:

1. Files are not saved as they are changed, but at the end of all other operations
This leads to two problems:
A) Memory bloat, to the point of causing application crash
B) The application is flooding the disk with (concurrent?) write operations (as many as running threads? I saw a very consistent average of 8 with an 8-core CPU) as the final task of correcting many files.

If the writes are in fact concurrent, the whole process of writing out the changes to the files will take much, much longer than if these writes were sequential.
I assume each thread will have the same priority to disk and that this causes contention.

Yes, Save Changes as a task in the Autocorrecter uses the same code, so will use one thread per cpu for saving files. I guess because you have an eight cpu machine you are seeing this problem to a greater degree than most, I only have a quad processor machine.

Files are saved at the end of the operation at the moment because Save Changes was envisaged as a separate task and for technical reasons to do with users marking files as deletions. It was an addition because I originally expected users would always want to check changes made by jaikoz before committing.

Memory bloat is due to the fact that the metadata for each file is loaded into memory when the files are opened in Jaikoz, and information can be viewed for any file in the main table. Running the autocorrecter itself only increases memory usage in the way that if it finds more data then there is more metadata to be stored. As discussed before I would like to move to this data being held in a database but this is a big reworking. However you could get round this with your 64bit machine by greatly increasing the maximum memory allocated to jaikoz.

I'm going to limit the cpu's to two now, and upload as Jaikoz 3.7.0 Beta 3 for you to try out. But the Save And Move Autocorrecter task is not concurrent so Id be interested if you try that instead and see what results you get

The memory and Save part of task ideas are bigger problems, but Im already planning on changes to the way autocerrector works for Jaikoz 3.8.0 so the Save as part of task idea should make it that release.

Jaikoz 3.7.0 Beta 3 now available with the threads limited to two when saving files to try out.

Yes, thanks, I found it.

Maybe there should be a beta-forum?

I'll give it a try and see if I can test in a structured manner.

Thanks.

MAzE5h1p69wB wrote:

Maybe there should be a beta-forum?

Maybe, not sure there is enough demand but Ive created one anyway