[Logo] Jaikoz and SongKong Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
Time it takes to sift through files - an explanation?  XML
Forum Index -> SongKong Questions
Author Message
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

Hi! Now I have the program working - which is awesome. However, I am curious to know why on earth it takes as long as it does. When I specify that each subfolder contains an album - what is the computer doing that means it will take around 45 minutes to write one folder (an album, with maybe 15 tracks) to the harddrive? Im running on an i5 3.4Ghz processor with 16GB memory, and I can see the program is using 80-90% of the processor for hours on end - what in the world is taking place? Is it converting the file to WAV to compare to the original file? Is it working on all the other files in all the other folders as well? What is going on? At the current rate I assume I will be done retagging my collection in march 2013 with the SongKong running 24/7 - which is not going to happen.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

On average I've found it fixes a song a second, so something seems to be seriously wrong here. But if you have asked it to match alot of songs, the first step is to fingerprint songs and this is cpu intensive. SongKong works on folders at a time processing them in a sort of pipeline but it is frontloaded to fingerprints songs more then match them at the start.

Maybe your songs are on a network drive over a unreliable wireless connection and the delay is at save time, but 45 minutes to write one folder, are you sure ?

Without seeing your support files I cant really add anything else at this stage.

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

Hi again. Yes, Im very sure. And I doubt its the fact that its on a networked drive - it copies to it at about 25-30MB a second, so that should be fine, I dont know why that would be an issue. And looking at the timestamp on the folders, it completes about 3-4 folders an hour. Eg. at 1900-2000 hours its written 3-4 complete folders with files in them. So I really dont know whats going on, and Ill start a folder of about 30 subfolders over night and post the results, it might be done by tomorrow, so we can see whats going on. I have a 20Mbit upstream connection so any quearies should be quite swift as well. Either way, will get back to you with a log file tomorrow if you want to take a look at it.

Started a job at 00:21 with 84 subfolders and 1575 files - this should hopefully not take too long.
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

Here is a logfile Id like you to take a look at. Its albums ripped 2 years ago - about 450 albums in individual folders, and the matching time for all of them is about 15 hours. This I cannot - under any circumstance, understand the reason for. Either the software is not properly optimized or its doing something very strange in order to match the INDIVIDUAL FOLDERS that contain an album - meaning, either it does or it doesn't. And for many of the folders NOT being matched to an album; it still writes to them and changes their tags, just it does not rename the folder as it apparently is not a complete album. Go figure - I surely have no clue as to what is going on.

I got a: java.lang.NullPointerException when I tried to add the file as an HTML file (about 6 mb). Trying to zip it and see if that works - nope, cannot upload file at the moment.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Please email support@jthink.net

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

Done.

Ive also noticed that when SongKong has completed writing the new folder (renamed and all), it does so even IF the folder is not complete - meaning it might miss track number 4 on the album - and SongKong will report it as a complete album. This is quite disturbing, as it means that I cannot trust the program to make sure the folder contains all the tracks on the album in that specific folder. I dont know if you find examples of this in the logfile I sent you or not.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Hi, you have sent me your report , thankyou but not the accompanying support files (Advanced/Create Support Files) I need both to investigate.

Regarding incomplete files marked as complete it could be that all your songs in a a folder have been matche dto one album, therefore matched completely but you don't have all songs.

i.e. You have seven songs in folder all matched to one album, but the album contains 8 songs. So all your songs have been matched and the folder renamed, but you don't actually have all songs. This seems to be correct behaviour to me.

Would help if you actually gave me a concrete example.

Paul

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

I have zipped the file - its 158 MB so Ill share it with you. A concrete example is as follows:

Folder names "Das Boot by U96 (1990)" contains 15 files, named 01 - Trackname.mp3 to 15 - Trackname.mp3. Upon using SongKong it correctly creates the folder U96 - Das Boot (1990), and moves the files into it. It does not remove the folder named "Das Boot by U96 (1990)" (even if it was empty), and in this folder the song 04 - Trackname.mp3 is still located - and is not found in the newly created folder U96 - Das Boot (1990). Hence I end up with two folders - one correctly named and tagged - and one file (04 - Trackname.mp3) that is still located in the old folder not in the new one. I dont have a log of this that I collected I think. But this has been the case several times. Again, Ill share the link to the zip file containing the remaining documents you need for the first logfile if that helps you.

Have a good new years eve and lets hope we get the software working properly in 2013

Take care!
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Hi, sorry for the delay in replying (i'm away from office) but you don't seem to have sent the support files I requested using the Help/Create Support Files option, instead you have just send me the image files for the coverimages displayed in the Fix Songs report.

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

God, im sorry. I thought thats what The files were. Either way I give up at the moment and i have a few days left of tagging, i just manually go through everything to get an idea of if the folders are correct or not after running SongKong. When im done i wont have to do such a batch ever again.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Either way, could you just send me the zip file created by Help/Create Support Files (wont be large) so I can try and track the problems youve raised.

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

I can do it tomorrow, when the current batch of 5721 files are done.
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

You should have received the file by now.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Yes, thankyou Im just trying to get the new version of Jaikoz finished then I will spend some time taking a look.

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

Ok great - well as a status update it has taken me about 3 weeks running 24/7 going through my entire collection (+/- 100.000 tracks/10.000 albums) all FLAC files. So it does seem to work, although my major issues are that it matches incomplete albums (missing tracks) as complete albums and the time it takes to complete. Other than that, it seems to do the job fine.
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

On a related note - is it possible to, instead of creating a folder named Various Artists, just simply write VA ? The same goes with Disc 1 and Disc 2 in multi-cd releases. Would like it just to say CD1 instead. I cannot see any info on this in the help files.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

sizetwo wrote:
Ok great - well as a status update it has taken me about 3 weeks running 24/7 going through my entire collection (+/- 100.000 tracks/10.000 albums) all FLAC files. So it does seem to work, although my major issues are that it matches incomplete albums (missing tracks) as complete albums and the time it takes to complete. Other than that, it seems to do the job fine. 


I can see there are issues both saving files and creating acoustids, both pointing to maybe a network problem. Unfortunately there is not much else I can tell from the logs really but I am now starting work on SongKong 1.2 and one of the priority improvements will be better error reporting so I think once this is done we can get to the bottom of your outstanding issues.

thanks Paul (Administrator)
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

sizetwo wrote:
On a related note - is it possible to, instead of creating a folder named Various Artists, just simply write VA ? The same goes with Disc 1 and Disc 2 in multi-cd releases. Would like it just to say CD1 instead. I cannot see any info on this in the help files. 


I would say unrelated, going forward could you please create new forum topics for different questions because it is easier for me to manage and for others to follow topics they might be interested in. But anyway you can do this as follows


Within File Naming:Edit Mask there is one mask for Original Album and one for Compilation albums. These albums will use the Compilation Mask so you need to

Select the compilation mask being used in the Filename Mask dialog and select Edit.

Then Create a New javascript User Defined Function
Code:
 
 function variousartisttova(value1,value2,sep){
      if(value1=="Various Artist")
      {
          return "VA" + sep;
      }
      return value1.length > 0 ? value1 + sep :value2.length > 0 ? value2 + sep:'' ;
  }
 


Now edit the mask (assuming you have had the default)
So that it looks like:
Code:
 variousartisttova(albumartist,artist,'/')
 + ifnotempty(album,'/')
 + variousartisttova(albumartist,artist,' - ')
 + ifnotempty(album,' - ')
 + ifnotempty(pad(track,2),' - ')
 + title
 


As for discnos, SongKong doesn't output discnos by default (although it should I seemed to have missed this) but it won't output 'Disc' just the number.
If it is ouputting 'Disc' then it is either because you havent actually configured SongKong to rename files based on metadata when matched in the Basic tab or 'Disc' is being output as part of the Musicbrainz release title. Before Musicbrainz NGS the Musicbrainz database could only have one cd per release so multiple disc releases often had the disc no appended to the title, this shold no longer happen but some releases in Musicbrainz have not yet been updated to the new standards.

thanks Paul (Administrator)
sizetwo

Pro

Joined: 11/12/2012 13:35:39
Messages: 46
Offline

Ok - I understand. Just to complete this, it does output Disc 1 etc. as long as the "Multi disc releases" dropdown menu is properly selected. It will create folders named as such: Supertramp - Paris (disc 1) (1985) (using my naming format). So what I wanted to know is if its possible to instead of disc 1 have CD1 written instead. I have used a mass renamer to take care of this, so its not a major issue. Other than this, well, Im done using the software for large batches for now, my collection seems properly tagged, and I wish you all the best of luck developing it further, and thanks for the time and effort you put into it - Im sure its quite a bit of work.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Ah I see, you could do something like this in the filename mask
Code:
 album.replace("disc ","CD")
 


but it does now occur to me that 'disc' is not translated, and maybe not the best choice so Ive raised.

http://jthink.net:8081/browse/SONGKONG-186

Glad to hear you won your battle with SongKong.

thanks Paul (Administrator)
 
Forum Index -> SongKong Questions
Go to:   
Powered by JForum 2.1.6 © JForum Team