[Logo] Jaikoz and SongKong Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
3.70 beta 4  XML
Forum Index -> Jaikoz Beta
Author Message
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

Couple of observations.

* read settings from previous installation if exists
- for each installation, it shows the choice "Norwegian" as selected, not the choice for my existing installation (this "translatian" seems to have been done by an automatic translator by the way, and worth it be not have more but less makes it so)
- all choices for manipulators are overwritten with jaikoz defaults

* searching in offline help doesn't work

* default choices for jaikoz manipulators seems to be weird and the documentation is less than clarifying
- what is the difference between local correct and correct
- which tasks can run file by file
- what does cluster albums do (please update the documentation at http://www.jthink.net/jaikoz/jsp/help/windows/help.html#ClusterAlbums to reflect what "groups them by artist and album and tries to reduce the number of release ids the tracks are spread over" actually means to the user and why he would want to chose this)
- in what order should manipulators be chosen, so that running task 1-5 and task 3-7 and task 9-10000 actually makes sense or conveys any meaning to the user

* can't run more than one instance of jaikoz at a time
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

MAzE5h1p69wB wrote:
Couple of observations.
* read settings from previous installation if exists
 

Generally it does convert the preferences from the existing preferences, however some preferences are on purposely not updated sometimes because of new changes in the way Jaikoz works. For example the Autocorrecter used to have 'Correct from Musicbrainz' and 'Update from Discogs' as default tasks, but now that Correct from Musicbrainz automatically updates the information from Discogs when it has a link to Discogs there is not normally any point having Update from Discogs as a default task. Also There is now a 'Correct from Discogs' task which is make sense to include. So the preferences for the Autocorrecter have not been preserved for this release.

MAzE5h1p69wB wrote:

- for each installation, it shows the choice "Norwegian" as selected, not the choice for my existing installation (this "translatian" seems to have been done by an automatic translator by the way, and worth it be not have more but less makes it so)
 

Yes, the installer just tries to use the default language of the computer, I don't develop the installer not sure if this can be changed, but within Jaikoz itself your Language and Preferred Country are preserved between versions.

MAzE5h1p69wB wrote:

* searching in offline help doesn't work
 

This is now fixed.

MAzE5h1p69wB wrote:

- what is the difference between local correct and correct
 

Local correct does not requite Internet Access, Correct is really Remote Correct.

MAzE5h1p69wB wrote:

- which tasks can run file by file
 

This is all going to get an overhaul and will probably change in the next release

MAzE5h1p69wB wrote:

- what does cluster albums do
 

You have ten tracks, five linked to release with name 'fred' and release id 1, and another five to another release also called 'fred' but with release id 2. Cluster Albums tries to match all tracks to either release 1 or release 2.

Cluster Albums may well become defunct once I have changed Correct from Muscibrainz to be more release orientated,

MAzE5h1p69wB wrote:

- in what order should manipulators be chosen, so that running task 1-5 and task 3-7 and task 9-10000 actually makes sense or conveys any meaning to the user
 

I don't understand your question but as I say I will be working on changing this, if you have any ideas how you think it should work please let me know.

MAzE5h1p69wB wrote:

* can't run more than one instance of jaikoz at a time
 

That is correct, I cannot see why you would want to do this.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

paultaylor wrote:

MAzE5h1p69wB wrote:

- in what order should manipulators be chosen, so that running task 1-5 and task 3-7 and task 9-10000 actually makes sense or conveys any meaning to the user
 

I don't understand your question but as I say I will be working on changing this, if you have any ideas how you think it should work please let me know.
 


Well, I find the manipulators and what they do, confusing.

Perhaps a tree-view, grouping related tasks, would make things a bit clearer?

Such as grouping together all tasks having to do with correcting metadata.

For example:
* Correct tags
** Correct artist (checked by default)
** Correct album (checked by default)

And also, perhaps grouping:
* Fetch metadata
** Generate acustic
*** Fetch from musicbrainz
*** Fetch from discogs
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Hmm, almost everything is about correcting metadata really, but somethings also have to find a matching track (i.e Correct Metadata from Musicbrainz' whilst others already know the matching track (i.e Update Metadata from existing Musicbrainz Id' , and I think different people would group things differently (i.e Remote versus Local).

What is confusing is the 'Fix Song by Song' option, I added this because a common problem was that a users autocorrecter might consist of Retrieve Acoustic Ids and then Correct Tags from Musicbrainz but they didn't want to wait for Retrieve Acoustic Ids to be run against all songs before Correct tags from Musicbrainz was started so they could see some results quicker.

Both of these tasks can be run Fix Song by Song, so if you just had these two in your Autocorrecter, Jaikoz would Retrivee Acoustic Id for Song 1, then AutoCorrect Metadata from Musicbrainz for Song 1 , then Retrieve Acoustic Id for Song 2 and so on. But the AutoCorrect Metadata from Discogs task does not work song by song, it essentially fixes songs folder by folder/album by album, so if you add this as your third task to the Autocorrecter then Jaikoz would complete Task 1 and Task 2 before starting Task 3. The trouble is that Autocorrect Metadata from Musicbrainz is also going to be more release orientated in the next release so 'Fix song by Song' will not work fot this either, then we are back at the situation we had before 'Fix Song By Song' was added !

My current thinking is to drop 'Fix Song by Song' to 'Fix Folder by Folder'

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

New bug:

Constant disk queue of 50, 8 instances of genpuid.exe with delta I/O reads between 52 and 128 each.

*sigh*

EDIT: I set the affinity for java.exe to use one core only, and the queue dropped to 10. Still seeing 8 instances of genpuid, but I assume that is because jaikoz was running and changing the affinity doesn't kill threads.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Does disabling 'Musicbrainz/AmpliFIND/Save Acoustic Ids as Retrieved' resolve this ?

thanks Paul (Administrator)
Anonymous



Yes.

With that disabled, there is no disk queue.

Nor am I seeing any genpuid.exe subprocesses of java.exe, and running jaikoz.bat with retrieve acustic ids as a manipulator, reports warning createmusicipacusticidworkerthreadN ended.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

warning createmusicipacusticidworkerthreadN should be logLevel config and shouldnt show up, ignore this.

If it is creating acoustic ids it is working properly.

So the solution is just to limit the saving of acoustic id to be single threaded, but still allow 8 threads for an 8 cpu machine, or is there a need to limit threads to a lower number because even with this option disabled there is disc read i/O involved in generating the acoustic ids ?

Edit:I cant have use multiple threads for generating acoustic ids but force any read I/O to go through a single thread because genpuid is a non-Java application provided by AmpliFIND. So the choice is either to have one Genpuid working one file at a time, or have multiple threads each running Genpuid and consequently concurrent disc reads. But the saving of acoustic ids is under my control and that can be made single threaded whilst still having multliple Create Acoustic Id threads.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

paultaylor wrote:

MAzE5h1p69wB wrote:

* can't run more than one instance of jaikoz at a time
 

That is correct, I cannot see why you would want to do this. 


I would want to have multiple instances of Jaikoz running, so that they can all use a separate disk and be limited to using a separate core.

That way I hope to quadruple the throughput, seeing as Jaikoz handles disk-operations in a manner which causes acustic id-generation to fail and alse slows things down enormously.
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

paultaylor wrote:
...or is there a need to limit threads to a lower number because even with this option disabled there is disc read i/O involved in generating the acoustic ids ?

(1)...multiple threads each running Genpuid and consequently concurrent disc reads. But the (2)saving of acoustic ids is under my control and that can be made single threaded whilst still having multliple Create Acoustic Id threads.  


For 1), I'm seeing a queue-length of 50 (maximum allowed, I guess), which causes both huge delays and also fails, as mentioned.

For 2), saving doesn't fail but with a queue-length of 8 I would at a wild guess say saving takes two magnitudes time longer than a sequential save would.

Either way, with non-SSD disks, file-operations should be limited to one read / write at a time.

EDIT: I'll see if I can compare this time required, with the same set of 1000 files, for a non-SSD disk and an SSD raid-0.
Anonymous



Multiple instance of jaikoz running is not the correct solution for a number of reasons but Im going to make a couple of changes for you now which should solve/ease the issue.

1. Save Acoustic Ids will now be sequential write.

2. Max threads for retrieving acoustic ids will be limited to 4 even if you have more physical cpus (of course if you only have one physical cpu you are only allowed one thread)
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Are you using Windows ?

If so Jaikoz 3.7.0 final is now available for download in the beta page For Windows Only

If you run Retrieve Acoustic Ids standalone or as part of the Autocorrecter with Fix Song by Song unchecked it will use a max of four threads for generating acoustic ids. If you run it as part of the Autocorrecter with Fix Song by Song enabled then it will still use 8 threads, however depending on what your autocorrecter tasks are not all threads will be working on Retrieve Acoustic Ids at the same time, (some will be working on the next task) so the resultant file i/o should be similar. In both case file write I/O is constrained to be single threaded.

Give it a go, and let me know

Edit:Now available for Linux and OSX as well

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

MAzE5h1p69wB wrote:

EDIT: I'll see if I can compare this time required, with the same set of 1000 files, for a non-SSD disk and an SSD raid-0. 


I ran a test-case of 1020 files.

Fragmentation was negligible, load from other processes was negligible.

For the RAID-0 with two Intel X25-M G2, the batch took 33m48s.
For the single regular harddisk, it took 34m34s.

Nothing new here, as we have come to the conclusion that what slows things down are concurrent reads and writes.

What I do find strange, though, is that the load was so very, very small, and yet it took so long.

Almost no CPU, net, or IO-load, and yet it took 2 seconds per file.

I think it may be possible to get this down to 200ms per file, if not even less.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Is this is with 3.7 Beta or 3.7 Final ?

Is this running just Retrieve Acoustic Ids or running AutoCorrect , if running AutoCorrect comprises what tasks ?

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

paultaylor wrote:
Is this is with 3.7 Beta or 3.7 Final ?

Is this running just Retrieve Acoustic Ids or running AutoCorrect , if running AutoCorrect comprises what tasks ? 


Beta 4.



Uploaded with ImageShack.us
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Well hangon, consider you are retrieving data from AmpliFind, Musicbrainz and Lyrics Fly, and the last two impose a webservice limit of 1 request per second and you are saving changes to your files I think 2 seconds per song is not too bad.

Generally when you are correcting lots of songs its expected that you will just leave Jaikoz to do its thing.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

paultaylor wrote:
... a webservice limit of 1 request per second and you are saving changes to your files I think 2 seconds per song is not too bad.
 


Well hokay, I did not know that. Mystery solved, I guess.

Then I have really nothing to complain about, except I'd like to just grab ahold of this for a minute:

paultaylor wrote:

Generally when you are correcting lots of songs its expected that you will just leave Jaikoz to do its thing. 


Ideally, I'd like to see Jaikoz running in the background, as a service, monitoring a folder, continuously processing files.

There are applications which do that, very successfully.
Such as for example utorrent and sabnzbd.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

MAzE5h1p69wB wrote:

Ideally, I'd like to see Jaikoz running in the background, as a service, monitoring a folder, continuously processing files.


I have thought about doing this, but it would probably be another program because you get plenty for your money in Jaikoz, and its complex enough as it is without adding a completely new level of functionality.

So it might happen, but I dont want to spread myself too thin.

EDIT:It would be interesting if you could run your test again with jaikoz 3.7.0 to see how much improvment there is with the I/O threading changes.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

Would you like me to compare beta 4 with 3.7 or 3.6 to 3.7?
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Beta 4 to 3.7, so just run 3.7 and see if my changes have resulted in an improvement for you.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

Sure.

What settings do you want me to use?

Given that we understand that trying to read or write from more than one file at a time is like trying to play two songs on a gramophone at the same time, meaning it has some overhead - which is undesirable to the point that it doesn't need to be done.

EDIT: Well, I used a different set of files, but with the same settings as previously mentioned.

With these files, and the error "couldn't connect to amplifind server", I got a throughput of 1.1 seconds per file.
I got the impression that saving files, at the end, was faster and also less "stuttery", if you follow. Previously this saving had created a queue of 5, now there wasn't any queue.

EDIT2: Also it seems that the correcter is working much more "smoothly" now, there aren't any long pauses in between each song, and there are no sudden, inexplicable stops in processing.
And for about the first time, I can use the harddisk that jaikoz is working with, to also do other things, such as watching a movie.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7343
Online

Okay, thanks a good result then.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

Well it seems that the various instances of genpuid.exe can still build up a queue when reading files for the generation of accustic ids.

It seems to be down to a queue of 10, which is better than the disk-max of 50 before, but if these instances can't talk to eachother but floods the disk, I'm just going to have to move music-files to a temporary harddisk that I don' need to access, and let them fight it out between themselves.

EDIT: Also if you are running any other tasks that use the disk even minimally, whilst Jaikoz is whelming it with its queue-buildup, all of the generating accusticid ids FAILS because the queue becomes saturated and Jaikoz times out.

So could we at least have a PAUSE-button or something for Jaikoz?


 
Forum Index -> Jaikoz Beta
Go to:   
Powered by JForum 2.1.6 © JForum Team