Jaikoz- Discussion Forum

UnixMover wrote:

The app finally died during a 19.5k run (note it took 4 days before it died). I then implemented the Sun recommendations for the JAVA engine however to no avail, the system died during a 18k data run.

That does sound very slow, what do you have heap set to and what other modifications are you making.

UnixMover wrote:

My assumption is that when you download metadata you store that in an object/structure of some type; do you release those memory areas when the user implements a save?

Yes, but the majority of the memory is held onto by the metadata in the files. i.e if you just load 20,000 songs into Jaikoz (and do nothing else) that will use alot more memory than if you load 5,000 songs.

More memory is used when you autocorrect, but most of that should be freed once the autocorrect has completed. Saving changes means that there is no difference in the metadata in the song on disk and the song in Jaikoz, that will freeup some more memory, however to reiterate the majority of the memory is used up loading songs into Jaikoz

Im well aware of this problem and have a solution to do this, essentially alll the metadata for your loaded songs will be held in database, and only the subset currently displayed will need to be held in memory. but this is
a major change that I am having to implement outside of Jaikoz and then merge back in when I am happy with it.

UnixMover wrote:

I have taken to saving after every operation (see next paragraph for issues related to this trick) I perform on the data. When I work in small bits, things are ok. When I work in 15k and greater chunks the application dies.

You need to increase your max heap, you can see what this is set to in Jaikoz by looking the first line in the Jaikoz console, the default value is approx 800 mb.

UnixMover wrote:

The users have read only permission (774 Un*X style).

It is rather difficult for a Java program to get a comprehensive view of the security setup on Windows, but the situation improves somewhat with Java 7. Id need to see the log files to find out if there are cases where Jaikoz is unable to write when it should be able to write but you may be encountering the following:

Sometimes Jaikoz has to create a new file in a folder, write to that file and delete your original file rather than just write directly to your file. So for that reason you need permissions to create files in the parent folder rather than just be able to edit the files. Whether or not this is required depends upon whether the original file has a metadata tag large enough for the new metadata, and the audio format you are writing to.

UnixMover wrote:

And still other times the console states that it cannot find the file and a ?Null Pointer? error has been thrown.

Would be keen to see the stack trace of this exception, should be in Jaikozdebug0-0.log

Paul, while I truly appreciate all of the effort that has gone into this application-- I just cannot get reliable performance -- so it is with a heavy heart that I thank you for what you have created, and wish you the best of luck in the future. I will check back from time to time to see what progress you have made, however I need to located an application which can perform on "my" dataset of 40k music.

I hope your holiday season is fruitful. I also hope your 2012 finds you prosperous.

Respectfully,

Thomas Klinect
SDVOSB
TPI Inc.
619-270-9533 (V) | 619-342-2684 (F)

UnixMover wrote:

Paul, while I truly appreciate all of the effort that has gone into this application-- I just cannot get reliable performance -- so it is with a heavy heart that I thank you for what you have created, and wish you the best of luck in the future. I will check back from time to time to see what progress you have made, however I need to located an application which can perform on "my" dataset of 40k music.

Hi, when originally developed it was originally envisaged that customers would not want to load their whole library in every time, I assumed they would just load in their incorrect files, fix them, and then load a new batch next time. Additionally the average size of music collections is steadily increasing.

Having said that I have been moving towards making it easier to load even very large sets of files. Stage 1 is that you should be able to load and process very large collections as long as you have a decent machine and allocate enough memory, Stage 2 is to allow large colections to be processed with just the default memory settings and a standard machine.

I'm currently working on both problems, I have one customer who sucessfully loads and processes 250,000 songs in one go, however with this many files there are some problems that can occur during file loading, file processing and file saving. I'm working through these and should have a new release in January, your 40,000 song collection should then work as you expected.

Stage 2 is a longer term, and I'm working on a prototype to solve this problem.

but stage 2 is longer term.