SP saving via NetMD - some tech info

[Zyan]

Hi all, as I mentioned in the other thread my N707 arrived today. As such, I've also been monitoring the USB traffic and watching what happens when different encoding methods are used.

I'm not sure if this should go in announcements and discoveries (as it kind of is a discovery, but it's related only to NetMD, which is why I've placed it in here).

Well, I did a little test with a short audio sample. First I encoded the sample in to LP2 and LP4. In the four transfer tests I started with identical blank discs.

1st test: I saved the LP4 version to the MD.
2nd test: I saved the LP2 version to the MD.
3rd test: I set OpenMG to transfer the LP4 version in SP mode to the MD.
4th test: I set OpenMG to transfer the LP2 version in SP mode to the MD.

Basically what I did was capture the data to and from the unit for the duration of each transfer, strip off the checking and header packets, and bring it just down to the raw hex data for the audio (sent in blocks of 4096 bytes) being sent to the unit.

Basically the exercise was to figure out just what the unit did when transferring in SP mode (we all know that at this point OpenMG doesn't allow true SP recording using NetMD because it's starting with files already compressed in LP2 or LP4). I had three possibilities in mind...

1. it sent the MDLP2/4 data and left it up to the unit to decode and re-encode it in to SP
2. it decoded the MDLP2/4 data and re-encoded it in SP in software and sent the SP data to the unit
3. it decoded the MDLP2/4 data to some raw format, such as PCM, and sent that to the unit to be encoded in to SP

Now, from the point of view of a person creating software to replace OpenMG, I'd say either 2 or 3 would be the best options, because it means something can be done to make it save it in SP directly (without any hardware changes). 3 is probably the best of all, because if the raw data format can be figured out (I'd assume it'd be close to PCM), then it means an SP encoder doesn't have to be written (because the unit does it).

The result? From what I can see... it's number 3!

How did I get this conclusion? Well...

The amount of data transferred for SP mode when transferring the MDLP2 and MDLP4 tracks was identical (i.e. data sent when doing LP4 -> SP and LP2 -> SP was the same). So this first of all means that it's not just sending the MDLP2 or MDLP4 data and getting the unit to do the work of decoding and re-encoding. The data was different (because the raw audio would have been different due to compression artefacts) however the amount of data was identical, about 1343744 bits. The MDLP2 transfer was about 98304 bits, and the MDLP4 transfer was about 63232 bits.

As you can see, the SP transfer was VERY large in comparison to the LP2 and LP4 transfers. In fact it's so large it couldn't possibly be SP data (which is 292 kbit/s from memory). So we know that it's being sent in a raw, uncompressed format, such as PCM. In fact, it's quite close in file size to PCM. If we look at the number of *bytes*, a raw stereo 44.1 kHz 16-bit PCM version of the sample takes up 161280 bytes, and the data that was sent to the unit is 167968 bytes (1343744 / 8 bits) - so there's 6688 bytes of data padding/header info somewhere in there.

Now, what does this whole rant mean in the scheme of things?

Well, these are just my first packet sniffing attempts and I haven't had time to go over it and verify everything perfectly (I'd like to be more sure of the data format before I go making assumptions). What it means from what I can tell so far, and if my calculations are correct... is that the data being sent to the unit for it to record in SP mode is in fact uncompressed 'PCM-like' raw audio data. This is GREAT news for everyone, because it means that if Sony *want* to, they could allow people to do direct true SP transfers with NetMD (provided they are willing to) because it means the provision is there in the hardware - i.e. you can tell the unit to encode raw audio data in to SP over NetMD. What it also means is that it should be MUCH easier than expected for people to create custom upload programs if they want to, that do SP transfers over NetMD (because no codec has to be written to do an SP save, only the raw audio format has to be figured out).

I hope for those that are interested in the technical side enjoyed (and understand) what I'm rambling on about, LOL

Regards and happy NetMDing

[Upright_Man]

Ok - but can WE engineer a SP recording solution - or do we have to wait for sony?

[Zyan]

Well it'd take a significant amount of work still, but it *seems* a lot easier than I first thought, and I'm pretty sure someone will come up with a way to do it (for ATRAC, probably not ATRAC3 at this point because of encryption as mentioned below).

The main issue to get done is all the transaction parts first... i.e. interfacing with the unit to get it ready to accept a new track.

I've heard some things about certain parts of OpenMG/NetMD being encrypted, but I'd assume that this is more the OpenMG saved files - i.e. it wouldn't make sense to encrypt the raw data just to send it over USB and have the unit decrypt it again before encoding it in to ATRAC, but it would make sense to encrypt the saved files on the HDD (to ensure they can't just be used in whatever way people want, and make copies of copyright music, etc).

The Open/NMD project seems to be making good ground as far as interfacing with the unit is concerned, and as I've mentioned, this is the important part to get done first. The way the handshakes are done are actually quite complex. For example, the unit and OpenMG periodically 'talk' to each other to check the current state of writing the data, the state of the unit, etc. Once this is worked out, and once the raw format is worked out (if my above theory is correct) it should be relatively easy to write a routine to upload raw audio and have the unit encode it in ATRAC 'SP' mode.

It also means that Sony have the possibility to do it with a software upgrade and no changes to the hardware (great news for those of us that have already bought NetMD units).

[rocksteadyMD]

wow! this is great news, if it stands up that is! i wish you luck in a fast and simple solution to true sp and uploading...it'll pay off when people realize what they can buy into! thanks again, i really appreciate all of your hard work!

[Sugarfreegum]

wow! very nice work

[Luke_A_P]

Nice work, opens up all kinds of interesting prospects. I guess the next big question has got to be; is there anyway to read data from the MD across the USB link in the other direction?

This is a facinating project.

Luke

[Savageone79]

I am not sure that is correct. I have talked to technical people at sony and from what I can discern is that the lp2 or lp4 is sent thru usb and the md unit itself recomresses it to SP mode. The Sp codec is nowhere on the computer. Having said this it still would be easy to get true SP tranfers as all they would have to do is allow ucompressed pcm data to be sent thru the usb cable to the unit to compress into sp at 4x speed. I hope they do this. I think they will as the unit becomes more popular and there is more awareness and demand.

[Zyan]

Savageone79, did you read my post and the information on the amount of data sent?

When saving my LP4 sample, 63232 bits of data was sent. When saving the same sample, telling OpenMG to put it on the MD as SP, the amount of data sent was 1343744 bits (the same amount of data sent when the same sample encoded as LP2 was saved as SP).

This means that the data being sent *when OpenMG is told to save the songs as SP*, is the same size regardless of the original encoding rate, which means that the songs are first decoded before being sent (i.e. the LP2/4 data isn't sent to the unit as-is when you tell OpenMG to save as SP, it's decoded before being sent).

So we know that much. Then we need to know if the data that's sent is encoded as SP already, or if it is raw data.

So I compared the bitrate of SP data (292 kbit/s, or 299008 bits/s) to the amount of data sent, 1343744 bits. If the data is sent pre-encoded as SP, then we should be able to divide the amount of data sent, 1343744, by the bitrate of SP, 299008, and get the approximate length of the track.

However 1343744 / 299008 = 4.494 seconds, and the sample that I used was less than one second in length. This means that the data sent, 1343744 bits, was sent in a format that has a much higher bitrate than that of ATRAC SP-mode. In fact the amount of data is nearly identical to that being sent as PCM-encoded audio.

So, after saying that again, it should be clear that when asking OpenMG to save the songs as SP, the software *does not* send the data in its original LP2/4 format, it first decodes it to some other format (which is too large to be SP, in fact it's about as large as uncompress PCM), and then the unit encodes it on-the-fly to SP.

Which as I said is good news, because it means it a) easy for Sony to write a 'true SP' transfer - by going directly from the original audio files you give it without first saving them to the HDD as LP2/4, and, b) allows other people to easily make true SP transfer programs, because there is no codec that will be required, all that has to be worked out is the raw format that's being used (probably a PCM-similar format).

[Luke_A_P]

Savageone;
Yeah, but he's not saying that SP is encoded on the computer at all is he, he's saying raw data is sent. I don't know what your connection with sony is, maybe you got through to someone who has a clue. But if I just called the technical support department I would expect to get trough to someone who is use to telling people to try putting the batteries in the right way round, and I wouldn't trust them to tell me how something actually works.

Luke

[Zyan]

I've been continuing my examination of the format it sends the data in (both for normal transfers of songs, and transfers of songs where you want it to record in SP).

Here's what I've found so far:

When transferring songs in LP2 or LP4, the data is sent exactly as it appears in the OpenMG files (minus the OpenMG file headers at the start and end of the files).

In other words, when you tell it to check out a .OMG file, it doesn't change the data in any way, it just grabs the appropriate data from the file and after setting up the transfer with the unit, sends it as-is.

The OpenMG files also contain other information, including an XML data segment at the end which describes the file. This XML segment could be stripped and parsed if anyone wants to be able to easily get information from the file. It contains:

- package (XML root tag containing a version attribute - version of .OMG file I'd assume, e.g. "2.0")
-- pid (I'd assume this is to identify the track or provide a reference to the encryption key)
-- tracklist (collection)
--- track (with number attribute, e.g. "1")
---- caption (collection)
----- title (title of the track/file, e.g. "My Song")
----- artist (with role attribute, e.g. "principal")
---- renditionlist (collection)
----- rendition (with format attribute, e.g. "audio/atrac3")
------ cid (again, same as the package cid, a 40 hex character value)
----- end rendition
---- end renditionlist
--- end track
-- end tracklist
- end package

Now on to the SP data transfer... I found an annoying problem.

So far I haven't been able to nail down a format that makes sense because it looks like the raw data it's sending is *long pause* encrypted!

It's the right size to be a PCM-like format, but there's no consistent low-high byte order after the header information.

There is one thing that has me wondering about my previous idea that it's sending it in raw data though. It seems that the data it's sending has a similar header to that of the ATRAC3 data it sends in normal sends. Note though that this header is different for every file it sends, it's not consistent in content, just format. I still don't think it's sending SP data though because the amount of data is WAY too much for an SP data rate... I still think that it's sending raw data and the unit is doing the SP encoding.

What is certain is that if it *is* encrypted (certainly looks like it is at this point), it uses the same key each time it transfers the same track, and each track has its own key (because an identical song encoded twice with the same method yields different data and a different cid).

This means the encryption has to be worked out if people want to write their own SP transfer methods.

I'm going to investigate this and see if I'm right and it really is encrypted, however, if it is I might just keep looking at the rest of the format and the way the software interfaces with the unit to get things like MiniDisc TOC information... however I won't probably continue figuring out how the SP transfer is done, because I dunno about you, but the big companies have a nasty habit of coming after people who get around encryption technology (DeCSS anyone?).



The good news is that because normal transfers just grab the data from the .OMG file and send it, it'll mean that people could use OpenMG to create OMG files and then use a custom program to transfer it later (the custom program wouldn't have to know anything about the encryption, it'd just have to know what to send and when - from what I can see a handshake is done first which sends the unit the encryption key, maybe, and then the data is sent after that direct from the file).

At least this means it'll be possible form people to make programs that allow you to transfer normal OMG files as many times as you want, and catelogue them in whatever manner you want. I believe that the Open/NMD has a similar plan in mind once they work out the interfacing codes... that is, using OMG files and transferring those, at least to begin with.

More research has to be done all-round, but it's a start to understanding the format. What I've found seems to be good news in some areas (OMG transfers by custom programs) and bad news in others (SP raw data might be encrypted, dunno why they'd do this though, doesn't seem to make sense to me, I'll keep investigating).